Your data layer matters more than ever
Agents can build the app. Turning raw files into a dataset still takes a human, so share the dataset, and do it without copying the data.
Ian Hunt-Isaak
Xarray Community Developer
blog · tag
#Zarr
30 posts
Fundamentals: What is zero-copy ingest?
You want to use Zarr, but your data is in HDF5, TIFF, or ND2. Zero-copy ingest lets you read it as Zarr without converting it.
Ian Hunt-Isaak
Xarray Community Developer
Convert Zarr to Icechunk with just a few clicks!
Arraylake makes it trivial to ingest and view existing Zarr stores in the cloud.
Tom Nicholas
Software Engineer
A Data Layer for Scientific AI: Icechunk for Agentic Single-Cell Biology Workflows
Agents work best when they can branch, explore in parallel, and converge. Using examples from single-cell biology, we show how Icechunk turns a shared cloud Zarr store into a versioned, transactional system of record for agentic science.
Ryan Abernathey
CEO & Co-founder
Luiz Irber
Software Engineer
Ian Hunt-Isaak
Xarray Community Developer
Weather Forecast Data: Icechunk vs. Iceberg Head-to-Head
TLDR: Operational weather forecast data workloads built on Icechunk are 10-100x faster and 5-8x cheaper than Iceberg. Warning: This article is long. It's written for a system architect planning to expend significant budget on weather data infrastructure and looking to deeply understand the options a
Ryan Abernathey
CEO & Co-founder
Our compute roadmap: introducing Zax SQL
We're introducing Zax, a new query and compute engine for multidimensional scientific data, and Zax-SQL, the first interface to it. Zax plans your computation for you, so you are no longer responsible for tuning chunk sizes, worker counts, and memory limits.
Ryan Abernathey
CEO & Co-founder
Brian Davis
Head of Product
Joe Hamman
CTO & Co-founder
Deepak Cherian
Principal Research Engineer
Sebastian Galkin
Head of Engineering
Making open-source AI weather forecasting models easy to run
A joint post from Earthmover and Hugging Face on running open-source AI weather forecasting models like Aurora using free ERA5 data from the Earthmover Marketplace — with a live demo space and a full local tutorial.
Emma Scharfmann
ML for Science, Hugging Face
Aaron Spring
ML, AI and geospatial data engineering
Ryan Abernathey
CEO & Co-founder
Kashif Rasul
AI Research Science, Hugging Face
Having it all: virtual and materialized data products
A guest post from dynamical.org on how virtual Icechunk Zarrs complement materialized products to deliver low-latency, complete, and fast-access weather data — plus the low-latency NOAA HRRR dataset it describes, now on the Earthmover Data Marketplace.
Alden Keefe Sampson
dynamical.org
Marshall Moutenot
dynamical.org
Virtually Gribberish - Bringing Icechunk clarity to GRIB archives
GRIB data is notoriously hard to work with, so we made it easy to create virtual Icechunk stores from GRIB archives such as the NOAA National Blend of Models, allowing cloud-optimized access to GRIB data archives without copying the data.
Tom Nicholas
Software Engineer
Matt Iannucci
Staff Engineer
Virtual chunks that just work — but securely
Delivering enterprise-grade access control for virtual chunks - surprisingly subtle!
Tom Nicholas
Software Engineer
Sebastian Galkin
Head of Engineering
Old format, no problem!: Cloud-optimizing the GOES-16 archive as Virtual Zarr
Access billions of chunks of satellite imagery as a single Zarr store, without copying any data!
Tom Nicholas
Software Engineer
Low Latency Icechunk ERA5 Now Available on the Earthmover Data Marketplace
We released Icechunk-ERA5, a performance-optimized, daily updating ERA5 data cube available now in the Earthmover data marketplace.
Ryan Abernathey
CEO & Co-founder
Variable length chunks in Zarr
A new extension to Zarr just landed: the rectilinear chunk grid lets you specify arbitrarily sized chunks along each axis, aligning chunk boundaries with the natural structure of your data instead of forcing a regular grid.
Joe Hamman
CTO & Co-founder
Max Jones
Cloud Engineer @ DevSeed
Davis Bennet
Software Engineer (Freelance)
Ship fast, break nothing: Engineering rigor in Icechunk with property and stateful testing
How we engineer rigor into Icechunk and its upstream brethren using property and stateful testing techniques. Also, a cautionary tale.
Deepak Cherian
Principal Research Engineer
Sebastian Galkin
Head of Engineering
How Kettle Uses Earthmover to Power Wildfire Risk Modeling at Scale
The Company Kettle is not a typical insurance company. Using AI to build smarter insurance products, Kettle provides insurance for property owners in areas affected by catastrophic climate events, with a particular focus on wildfire. Their AI models consume over 130 terabytes of satellite, weather,
Margaret Francis
COO
From GeoTIFF Chaos to Cloud-Native Climate Risk: How Eoliann Built Airis on Earthmover
Eoliann builds proprietary climate risk models that estimate the physical impact of extreme weather events — floods, wildfires, and storms — on critical physical infrastructure: electricity transmission lines, substations, and gas pipelines…
Margaret Francis
COO
Announcing Icechunk 2: Better Consistency, Performance, and Reliability for Tensor Storage
When we released Icechunk 1.0 last July, we declared it production-ready and committed to format stability. Since then, adoption has exceeded our expectations. Teams across weather forecasting, climate science, neuroscience, and AI/ML have pushed Icechunk into scenarios we didn't fully anticipate--r
Ryan Abernathey
CEO & Co-founder
Sebastian Galkin
Head of Engineering
I/O-Maxing Tensors in the Cloud
Zarr Python with Icechunk or Obstore now fully saturates the network between EC2 and S3, achieving the physically maximum possible throughput for reading and writing tensor data in the cloud. Benchmarks compare Zarr, Tensorstore, TileDB, and Parquet stacks across a range of chunk sizes and instance types.
Ryan Abernathey
CEO & Co-founder
Building the Future of Scientific Data at the Zarr Summit
Earthmover co-organizes the Zarr Summit in Rome, bringing together developers and adopters to advance the open-source cloud-native array format as adoption accelerates across major organizations like ESA, NASA, and NVIDIA.
Ryan Abernathey
CEO & Co-founder
Multi-Player Mode: Why Teams That Use Zarr Need Icechunk
Zarr lacks built-in support for concurrent readers and writers, leading to inconsistent reads and conflicting writes in team settings. Icechunk solves this by adding atomic updates, consistent snapshots, and Git-like version control on top of Zarr.
Lindsey Nield
Senior Engineer
Radar DataTree: Transforming thousands of scans into a single cohesive model
Introducing the Radar DataTree, a new data model that organizes thousands of fragmented weather radar scans into a single time-aware, cloud-native, version-controlled dataset using xarray-datatree, Zarr, and Icechunk.
Alfonso Ladino-Rincon
Data Scientist
Icechunk 1.0: Production-Grade Cloud-Native Array Storage Is Here
Icechunk 1.0 is now stable and production-ready, bringing transactional safety, efficient versioning, high-performance Rust-based I/O, and virtual references for HDF5 and NetCDF to cloud-native array storage. The release includes manifest splitting, distributed writes, conflict resolution, and a 30 TB ERA5 sample dataset.
Ryan Abernathey
CEO & Co-founder
Fundamentals: What Is Zarr? A Cloud-Native Format for Tensor Data
Zarr is an open-source, cloud-native protocol for storing chunked, compressed N-dimensional arrays. This guide covers how Zarr works, its ecosystem of tools like Xarray and Icechunk, and when to use it for large-scale scientific and ML data.
Lindsey Nield
Senior Engineer
Zarr takes Cloud-Native Geospatial by storm
At the 2025 Cloud-Native Geospatial conference, Zarr adoption was surging across the geospatial domain, with Copernicus Sentinel, USGS Landsat, Google Earth Engine, and ESRI ArcGIS all embracing the format for cloud-optimized array data.
Joe Hamman
CTO & Co-founder
Learning about Icechunk consistency with a clichéd but instructive example
A practical walkthrough of how Icechunk uses transactions and conflict detection to guarantee data consistency when multiple processes write concurrently. The post demonstrates optimistic concurrency control and the rebase workflow using a bank-account transfer example.
Sebastian Galkin
Head of Engineering
Fundamentals: What is Cloud-Optimized Scientific Data?
Why traditional scientific file formats like NetCDF perform poorly on cloud object storage, and how cloud-optimized formats like Zarr and Icechunk solve the problem by separating metadata and chunking data.
Tom Nicholas
Software Engineer
Accelerating Xarray with Zarr-Python 3
zarr-python’s performance paradox Last month, we released Zarr-Python 3.0 - a ground-up rewrite of the library (read more about it in this post). Beyond the exciting new features in Zarr V3, we put a lot of work into addressing some long standing performance issues with Zarr-Python 2. With the improvements described in this blog post, we’ve achieved a 14x speedup in loading the ARCO ERA5 dataset! Zarr-Python 2 had a paradoxical performance quirk; although the library could generate massive petabyte-scale datasets, it struggled to perform well when managing large or highly nested hierarchies. For example, listing the contents of a large Zarr group could be painfully slow, particularly if that Zarr group was stored on a high latency storage backend. Zarr users would experience this as long
Davis Bennet
Software Engineer (Freelance)
Zarr-Python 3 is here!
Zarr-Python 3.0 is released with full support for the Zarr V3 specification, chunk-sharding for more flexible storage, major performance improvements from a fully asynchronous core, and a modernized extensible codebase.
Joe Hamman
CTO & Co-founder
Announcing Icechunk!
Earthmover announces Icechunk, an open-source transactional storage engine for Zarr that brings ACID transactions, time travel, data versioning, and high-performance Rust-based I/O to multidimensional array data in cloud object storage.
Ryan Abernathey
CEO & Co-founder
Toward Zarr-Python 3.0
The Zarr-Python project is undergoing a major refactor toward version 3.0, bringing full support for the Zarr V3 specification, new asynchronous APIs for better performance, and a modernized plugin system for codecs and storage backends.
Joe Hamman
CTO & Co-founder
Related Posts
Blog Tags
access controls access-control agents ai AI weather AIFS announcement architecture arraylake Arraylake aurora Big Data biology blog post blog-post case study case-study chat cirrus climate risk cloud cloud native compliance compute conference data archives data cube data infrastructure data workflows dev note dynamical earth observation earth-2 earth-observation energy engineering enterprise era5 event FAIR feature release financial-services flux forecast fundamentals garbage collection geospatial GOES-R governance grib guest-post hrrr hugging-face hypothesis iceberg icechunk Icechunk IFS LangChain LangGraph marketplace mcp NASA natural language processing NCEI netCDF NetCDF noaa NOAA nowcasting nvidia nws oceanography open data open science open source performance product property-testing public-sector python Python Radar rust Rust scverse security serverless shuttle single-cell soc2 sql tables tabular data teams technical tensorops tensors testing third-wheel toxiproxy trust vector data virtual-zarr virtualizarr virtualization weather wildfire WOD xarray Xarray zarr Zarr zax zero-copy-ingest