blog · tag
35 posts
Apache Iceberg comes to Arraylake. Tables now live alongside Icechunk arrays in the same catalog, under the same permissions, in your own buckets — queryable from DuckDB, Spark, Trino, Snowflake, and Zax-SQL.
CTO & Co-founder
Staff Engineer
We're introducing Zax, a new query and compute engine for multidimensional scientific data, and Zax-SQL, the first interface to it. Zax plans your computation for you, so you are no longer responsible for tuning chunk sizes, worker counts, and memory limits.
CEO & Co-founder
Head of Product
CTO & Co-founder
Principal Research Engineer
Head of Engineering
A joint post from Earthmover and Hugging Face on running open-source AI weather forecasting models like Aurora using free ERA5 data from the Earthmover Marketplace — with a live demo space and a full local tutorial.
ML for Science, Hugging Face
ML, AI and geospatial data engineering
CEO & Co-founder
AI Research Science, Hugging Face
A guest post from dynamical.org on how virtual Icechunk Zarrs complement materialized products to deliver low-latency, complete, and fast-access weather data — plus the low-latency NOAA HRRR dataset it describes, now on the Earthmover Data Marketplace.
dynamical.org
dynamical.org
GRIB data is notoriously hard to work with, so we made it easy to create virtual Icechunk stores from GRIB archives such as the NOAA National Blend of Models, allowing cloud-optimized access to GRIB data archives without copying the data.
Software Engineer
Staff Engineer
How AQUAVIEW reorganized NOAA's World Ocean Database around instrument types, powered it with Arraylake, and built a natural language interface that turns hours-long data workflows into seconds.
Director of Research Development and Scientific Entrepreneurship, University of Southern Mississippi
Director, Institute for Advanced Analytics and Society, University of Southern Mississippi
Delivering enterprise-grade access control for virtual chunks - surprisingly subtle!
Software Engineer
Head of Engineering
Earthmover has teamed up with Booz Allen Hamilton on NWS CIRRUS — NOAA's new cloud-based operational data platform for the National Weather Service. Icechunk will serve as a core data format for the operational data lake.
CTO & Co-founder
Access billions of chunks of satellite imagery as a single Zarr store, without copying any data!
Software Engineer
We released Icechunk-ERA5, a performance-optimized, daily updating ERA5 data cube available now in the Earthmover data marketplace.
CEO & Co-founder
How we used third-wheel and Hypothesis to interleave Icechunk 1.x and 2.x in a single stateful test — and the bugs that fell out.
Xarray Community Developer
The Company: Long Range AI Weather Prediction Beyond Weather is an Amsterdam-based AI weather forecasting startup with its roots in academic research. Founded in 2023 by a master's student, a PhD researcher, and a professor from the Vrije Universiteit Amsterdam's climate extremes and societal risk r
COO
We describe our approach to rigor in Icechunk 2 including concurrency permutation testing and more!
Principal Research Engineer
How we engineer rigor into Icechunk and its upstream brethren using property and stateful testing techniques. Also, a cautionary tale.
Principal Research Engineer
Head of Engineering
The Company Kettle is not a typical insurance company. Using AI to build smarter insurance products, Kettle provides insurance for property owners in areas affected by catastrophic climate events, with a particular focus on wildfire. Their AI models consume over 130 terabytes of satellite, weather,
COO
Earthmover's filtered subscriptions allow data providers to create secure, read-only views into multidimensional data cubes, enabling more granular cloud-native data exchange between provider and consumer.
CEO & Co-founder
Head of Engineering
Senior Engineer
Eoliann builds proprietary climate risk models that estimate the physical impact of extreme weather events — floods, wildfires, and storms — on critical physical infrastructure: electricity transmission lines, substations, and gas pipelines…
COO
When we released Icechunk 1.0 last July, we declared it production-ready and committed to format stability. Since then, adoption has exceeded our expectations. Teams across weather forecasting, climate science, neuroscience, and AI/ML have pushed Icechunk into scenarios we didn't fully anticipate--r
CEO & Co-founder
Head of Engineering
Today we’re thrilled to announce IceChunkCoin, the world’s first hyper-dimensional, chunk-native blockchain asset.
CEO & Co-founder
Earthmover built third-wheel, an open-source tool that rewrites Python wheels to install multiple versions of a library in one environment, enabling cross-version compatibility testing for the Icechunk V2 release.
Xarray Community Developer
A preview of Icechunk 2, featuring node rename, chunk reindexing, rectilinear grids, repository-level metadata, and significant performance improvements with a smooth migration path from Icechunk 1.
Head of Engineering
Zarr Python with Icechunk or Obstore now fully saturates the network between EC2 and S3, achieving the physically maximum possible throughput for reading and writing tensor data in the cloud. Benchmarks compare Zarr, Tensorstore, TileDB, and Parquet stacks across a range of chunk sizes and instance types.
CEO & Co-founder
Woods Hole scientists reduced ocean profile data access from 10 minutes to 10 seconds by converting their OPeNDAP-served NetCDF files to Icechunk repositories on AWS S3.
Postdoctoral Investigator, WHOI
Zarr lacks built-in support for concurrent readers and writers, leading to inconsistent reads and conflicting writes in team settings. Icechunk solves this by adding atomic updates, consistent snapshots, and Git-like version control on top of Zarr.
Senior Engineer
Introducing the Radar DataTree, a new data model that organizes thousands of fragmented weather radar scans into a single time-aware, cloud-native, version-controlled dataset using xarray-datatree, Zarr, and Icechunk.
Data Scientist
Icechunk 1.0 is now stable and production-ready, bringing transactional safety, efficient versioning, high-performance Rust-based I/O, and virtual references for HDF5 and NetCDF to cloud-native array storage. The release includes manifest splitting, distributed writes, conflict resolution, and a 30 TB ERA5 sample dataset.
CEO & Co-founder
A practical guide to Icechunk's garbage collection and expiration operations, explaining when and how to safely reclaim storage from unused snapshots and dangling objects.
Head of Engineering
Zarr is an open-source, cloud-native protocol for storing chunked, compressed N-dimensional arrays. This guide covers how Zarr works, its ecosystem of tools like Xarray and Icechunk, and when to use it for large-scale scientific and ML data.
Senior Engineer
Icechunk stores versioned array data efficiently by never copying or rewriting existing chunks, so each new version only consumes storage for the data that actually changed. Older versions can be expired and garbage-collected when they are no longer needed.
Head of Engineering
At the 2025 Cloud-Native Geospatial conference, Zarr adoption was surging across the geospatial domain, with Copernicus Sentinel, USGS Landsat, Google Earth Engine, and ESRI ArcGIS all embracing the format for cloud-optimized array data.
CTO & Co-founder
A practical walkthrough of how Icechunk uses transactions and conflict detection to guarantee data consistency when multiple processes write concurrently. The post demonstrates optimistic concurrency control and the rebase workflow using a bank-account transfer example.
Head of Engineering
Why traditional scientific file formats like NetCDF perform poorly on cloud object storage, and how cloud-optimized formats like Zarr and Icechunk solve the problem by separating metadata and chunking data.
Software Engineer
Demystifying how S3 prefix sharding actually works and demonstrating that Icechunk can scale to hundreds of thousands of requests per second, far beyond the single-prefix limit.
Head of Engineering
Earthmover and Development Seed partnered with NASA to pilot Icechunk, an open-source tensor storage engine that enables 100x faster cloud-native data access for archival Earth science datasets without costly data migration.
CEO & Co-founder
Earthmover announces Icechunk, an open-source transactional storage engine for Zarr that brings ACID transactions, time travel, data versioning, and high-performance Rust-based I/O to multidimensional array data in cloud object storage.
CEO & Co-founder
