Articles, announcements, and case studies from the Earthmover team.
The Challenge Eoliann builds proprietary climate risk models that estimate the physical impact of extreme weather events — floods, wildfires, and storms — on critical physical infrastructure: electricity transmission lines, substations, and gas pipelines. Their customers include some of Europe's lar…
COO
When we released Icechunk 1.0 last July, we declared it production-ready and committed to format stability. Since then, adoption has exceeded our expectations. Teams across weather forecasting, climate science, neuroscience, and AI/ML have pushed Icechunk into scenarios we didn't fully anticipate--r…
CEO & Co-founder
Staff Engineer
When we launched the Earthmover Data Marketplace in January, we described it as just the beginning. Today, we're showing that growth. We're excited to announce four new data partners joining the marketplace: Sylvera, Spire, Eagle Rock Analytics, and CarbonPlan. This second cohort adds proprietary da…
COO
Today we’re thrilled to announce IceChunkCoin, the world’s first hyper-dimensional, chunk-native blockchain asset.
CEO & Co-founder
Earthmover built third-wheel, an open-source tool that rewrites Python wheels to install multiple versions of a library in one environment, enabling cross-version compatibility testing for the Icechunk V2 release.
Xarray Community Developer
How we built a Slack bot that alerts our team when it's snowing at their location, using Earthmover's Marketplace and Flux APIs to skip the data pipeline entirely.
Xarray Community Developer
Engineering
Earthmover has been selected by ARIA to provide the Simulation Catalogue for the Forecasting Tipping Points programme, enabling 26 research teams to share and analyze petabyte-scale climate data.
CTO & Co-founder
Earthmover launches the world's first marketplace for AI-ready weather and climate data, offering instant access to analysis-ready cloud-optimized data cubes from leading providers in the open-source Icechunk format.
CEO & Co-founder
A preview of Icechunk 2, featuring node rename, chunk reindexing, rectilinear grids, repository-level metadata, and significant performance improvements with a smooth migration path from Icechunk 1.
Staff Engineer
Zarr Python with Icechunk or Obstore now fully saturates the network between EC2 and S3, achieving the physically maximum possible throughput for reading and writing tensor data in the cloud. Benchmarks compare Zarr, Tensorstore, TileDB, and Parquet stacks across a range of chunk sizes and instance types.
CEO & Co-founder
A roadmap for integrating Xarray and napari to deliver named-dimension-aware, metadata-rich scientific data visualization across biology and geosciences.
Xarray Community Developer
Earthmover co-organizes the Zarr Summit in Rome, bringing together developers and adopters to advance the open-source cloud-native array format as adoption accelerates across major organizations like ESA, NASA, and NVIDIA.
CEO & Co-founder
Woods Hole scientists reduced ocean profile data access from 10 minutes to 10 seconds by converting their OPeNDAP-served NetCDF files to Icechunk repositories on AWS S3.
Postdoctoral Investigator, WHOI
Earthmover is launching a new open-source library xpublish-tiles that powers our new Flux Tiles service, which allows Earthmover Platform users to view their data on a slippy map with dynamically rendered tiles at lower zoom levels than was possible previously.
Forward Deployed Engineer
Engineering
A hands-on walkthrough of calculating historical heatwave frequency over NYC using ERA5 reanalysis data on the Earthmover platform with Arraylake, Icechunk, Xarray, and open-source climate tools.
Software Engineer
Earthmover announces its $7.2M seed round led by Lowercarbon Capital, with participation from Costanoa Ventures and Preston-Werner Ventures, to build the cloud-native data platform for weather, climate, and scientific data.
CEO & Co-founder
GPUs running AI weather forecasts spend over 95% of their time idle, waiting for data. Three optimizations — pre-processing inputs into Icechunk, moving regridding onto the GPU, and writing outputs in parallel — cut inference costs by nearly 90%.
CEO & Co-founder
Zarr lacks built-in support for concurrent readers and writers, leading to inconsistent reads and conflicting writes in team settings. Icechunk solves this by adding atomic updates, consistent snapshots, and Git-like version control on top of Zarr.
Software Engineer
Introducing the Radar DataTree, a new data model that organizes thousands of fragmented weather radar scans into a single time-aware, cloud-native, version-controlled dataset using xarray-datatree, Zarr, and Icechunk.
Data Scientist
Earthmover is sponsoring Ocean Hack Week 2025, providing financial support for participant travel and an Arraylake organization to empower the open ocean science community.
CEO & Co-founder
An introduction to the WMO FM-301 standard for weather radar data and how open-source tools like Xradar are turning fragmented binary radar files into structured, analysis-ready datasets.
Data Scientist
Icechunk 1.0 is now stable and production-ready, bringing transactional safety, efficient versioning, high-performance Rust-based I/O, and virtual references for HDF5 and NetCDF to cloud-native array storage. The release includes manifest splitting, distributed writes, conflict resolution, and a 30 TB ERA5 sample dataset.
CEO & Co-founder
The Earthmover team is attending SciPy 2025 in Tacoma, Washington, with a tutorial on Xarray DataTree and Zarr, multiple talks and posters on Icechunk and Xarray, and a booth showcasing the Earthmover Platform.
CTO & Co-founder
Weather radar captures rich four-dimensional atmospheric data, but legacy binary formats and fragmented archives make large-scale analysis painfully difficult. A modern, cloud-native data model could unlock radar's vast scientific potential.
Data Scientist
Xarray introduces SeasonGrouper and SeasonResampler, two new Grouper objects that enable custom, overlapping, and variable-length seasonal aggregations without workarounds.
Forward Deployed Engineer
Arraylake now supports fine-grained, repository-level permissions with admin, write, and read privilege levels, giving teams precise control over data access and secure sharing with external collaborators.
Software Engineer
Open-data practice in ocean/atmosphere sciences is approximately 170 years old! While it is easy to exclaim, "weather/climate are global, of course we must share data", the actual story is anything but. That story holds valuable inspiration that we can draw from as we face a significant reduction in US climate science research.
Forward Deployed Engineer
Xarray's labeled, multidimensional data structures can solve common pain points in biological data analysis, from tracking microscopy metadata to managing complex genomic datasets. Adoption has been limited by awareness, technical rough edges, and lack of tool integration, but the community is actively working to change that.
Xarray Community Developer
A practical guide to Icechunk's garbage collection and expiration operations, explaining when and how to safely reclaim storage from unused snapshots and dangling objects.
Staff Engineer
Zarr is an open-source, cloud-native protocol for storing chunked, compressed N-dimensional arrays. This guide covers how Zarr works, its ecosystem of tools like Xarray and Icechunk, and when to use it for large-scale scientific and ML data.
Software Engineer
Icechunk stores versioned array data efficiently by never copying or rewriting existing chunks, so each new version only consumes storage for the data that actually changed. Older versions can be expired and garbage-collected when they are no longer needed.
Staff Engineer
Scientific data pipelines are plagued by data swamps, duplicated code, fragile workflows, and siloed teams. TensorOps is a vision for modern practices that bring collaboration, velocity, and reliability to scientific data engineering.
Software Engineer
At the 2025 Cloud-Native Geospatial conference, Zarr adoption was surging across the geospatial domain, with Copernicus Sentinel, USGS Landsat, Google Earth Engine, and ESRI ArcGIS all embracing the format for cloud-optimized array data.
CTO & Co-founder
The Earthmover team is attending the Cloud Native Geospatial Conference 2025 in Snowbird, Utah, leading a hands-on Zarr, Icechunk, and Xarray workshop and presenting talks on cloud-native datacube workflows.
CTO & Co-founder
A practical walkthrough of how Icechunk uses transactions and conflict detection to guarantee data consistency when multiple processes write concurrently. The post demonstrates optimistic concurrency control and the rebase workflow using a bank-account transfer example.
Staff Engineer
Why traditional scientific file formats like NetCDF perform poorly on cloud object storage, and how cloud-optimized formats like Zarr and Icechunk solve the problem by separating metadata and chunking data.
Software Engineer
Earthmover introduces Flux, a managed API layer that serves geospatial data from Arraylake via standard protocols like WMS, EDR, and OPeNDAP, eliminating the need for teams to build and maintain custom data delivery infrastructure.
CEO & Co-founder
Demystifying how S3 prefix sharding actually works and demonstrating that Icechunk can scale to hundreds of thousands of requests per second, far beyond the single-prefix limit.
Staff Engineer
Multidimensional array data about the physical world is fundamentally incompatible with the tabular data model. Benchmarks show that array-native tools like Xarray and Zarr outperform DuckDB and Parquet by up to 10x for common weather data queries.
CEO & Co-founder
Earthmover and Development Seed partnered with NASA to pilot Icechunk, an open-source tensor storage engine that enables 100x faster cloud-native data access for archival Earth science datasets without costly data migration.
CEO & Co-founder
Earthmover customers share how NOAA climate and weather data powers their businesses, from wildfire risk modeling and energy trading to carbon market ratings and precipitation enhancement.
CEO & Co-founder
zarr-python’s performance paradox Last month, we released Zarr-Python 3.0 - a ground-up rewrite of the library (read more about it in this post). Beyond the exciting new features in Zarr V3, we put a lot of work into addressing some long standing performance issues with Zarr-Python 2. With the improvements described in this blog post, we’ve achieved a 14x speedup in loading the ARCO ERA5 dataset! Zarr-Python 2 had a paradoxical performance quirk; although the library could generate massive petabyte-scale datasets, it struggled to perform well when managing large or highly nested hierarchies. For example, listing the contents of a large Zarr group could be painfully slow, particularly if that Zarr group was stored on a high latency storage backend. Zarr users would experience this as long
Software Engineer
Zarr-Python 3.0 is released with full support for the Zarr V3 specification, chunk-sharding for more flexible storage, major performance improvements from a fully asynchronous core, and a modernized extensible codebase.
CTO & Co-founder
Earthmover announces Icechunk, an open-source transactional storage engine for Zarr that brings ACID transactions, time travel, data versioning, and high-performance Rust-based I/O to multidimensional array data in cloud object storage.
CEO & Co-founder
Vector data cubes extend the familiar raster data cube concept to geospatial vector data, using arrays indexed by geometries instead of gridded coordinates. The Xvec package brings this capability to Xarray, enabling powerful multidimensional analysis of point, line, and polygon data.
Software Engineer
The ALIVE research team at UW-Madison uses Arraylake to manage GOES-R satellite data for near real-time carbon and water flux estimation, benefiting from version control, ACID transactions, and seamless remote collaboration.
CEO & Co-founder
A practical guide to building planetary-scale Earth observation datacubes in Zarr using serverless computing, comparing frameworks like Coiled, Modal, and Lithops for massively parallel satellite image processing.
CEO & Co-founder
The Zarr-Python project is undergoing a major refactor toward version 3.0, bringing full support for the Zarr V3 specification, new asynchronous APIs for better performance, and a modernized plugin system for codecs and storage backends.
CTO & Co-founder
Carbon market ratings company Sylvera adopted Arraylake to centralize millions of scattered geotiff files into cloud-optimized arrays, enabling incremental data ingestion and version-tracked auditing across their geospatial pipelines.
CEO & Co-founder
A practical guide to building a high-performance PyTorch dataloader that streams Zarr data directly from cloud storage using Xarray, Xbatcher, and Dask, achieving a 15x speedup over naive approaches.
CTO & Co-founder
Earthmover will be at AGU 2023 in booth 1007 alongside Coiled and Pangeo, demoing Arraylake and presenting three talks on cloud-native scientific data workflows.
CEO & Co-founder
Earthmover launches Arraylake in private beta, a cloud-native data lake platform purpose-built for multidimensional arrays with a built-in data catalog, ACID transactions, version control, and virtual file support.
CEO & Co-founder
Earthmover is hiring two founding engineers to help build a modern data stack for science, tackling climate and planetary challenges with cloud-native software.
CTO & Co-founder
Earthmover was founded to build a modern cloud data stack for scientific data, inspired by the success of the Pangeo open-source community and the urgent need for better tooling around multidimensional array datasets in climate tech and beyond.
CEO & Co-founder