Blog

Situation Overview Sylvera rates projects in the voluntary carbon market with the goal of enabling their customers to invest in the most meaningful initiatives. In order to produce these ratings, Sylvera relies on satellite imagery from providers such as Copernicus, USGS, and NASA. Prior to adopting Arraylake, the engineering team downloaded data across multiple geotiff files stored on individual machines and ran algorithms on the data in a local environment. This process worked for a time but as they began to scale they realized their workflow was not viable. They needed a modern platform to help them manage data and collaborate more effectively. Solutions Assessment As Sylvera looked to improve their data pipeline, they analyzed 3 solutions: One was building a tool in house, which, …

Machine learning has become essential in the utilization of weather, climate, and geospatial data. Sophisticated models such as GraphCast, ClimaX, and Clay are emerging within these domains. The advancement of these models is greatly influenced by the widespread availability of cloud computing resources, particularly GPUs, and the abundance of data stored in cloud repositories. Despite these advancements, there remains a lack of established best practices for efficiently managing machine learning training pipelines due to the diverse range of data formats used when storing scientific data. In this blog post, we discuss an architecture that we have found highly effective in seamlessly integrating multidimensional arrays from cloud storage into machine learning frameworks. The problem At …

December is here, and that means that thousands of Earth System Scientists are getting ready for the annual pilgrimage to AGU! At my first AGU in 2011, I was a fresh-faced grad student, excited to present my latest research on modeling Southern Ocean circulation. Since then, my relationship with this massive conference has evolved a lot. First it was opportunity to reunite with old grad-school friends and catch up on the latest science. Then it became an important opportunity to help my own students and postdocs to build their networks and find their next job opportunity. Now I’m excited to return wearing a new hat: as a vendor in the exhibit hall! 🤠 The Earthmover crew will be on site for the duration of the conference. Come find us at booth 1007! We’ve teamed up with Coiled and Pan…

At Earthmover, we believe that scientific data are key to solving humanity’s greatest challenges. And we know that scientists today are struggling with tools that don’t understand scientific data formats and data models. For the past year, we’ve been hard at work building a platform to transform how scientists interact with data in the cloud. Today, we are thrilled to announce the launch of Arraylake in private beta. Arraylake is a data lake platform built around collections of multidimensional numerical arrays (a.k.a. ND-arrays, tensors)—the native data model of physical, biological, and computational sciences, not to mention deep learning. Given the centrality of tensors to so many disciplines, we were frustrated that they are poorly supported by today’s cloud data infrastructure. Ex…