blog

field notes from the science data cloud

Articles, announcements, and case studies from the Earthmover team.

NVIDIA: Building a Low-Latency Earth-2 Nowcasting Pipeline with Earthmover
Blog Post

NVIDIA: Building a Low-Latency Earth-2 Nowcasting Pipeline with Earthmover

NVIDIA and Earthmover built a low-latency nowcasting workflow that turns Meteosat satellite observations into a two-hour nowcast using Earth-2 StormScope-EU, and serves it directly from versioned data through map tiles and SQL.

Negin Sobhani
Negin Sobhani

Senior Machine Learning Performance Engineer, NVIDIA

Aaron Spring
Aaron Spring

Data and AI Freelancer

Jussi Leinonen
Jussi Leinonen

Solutions Architect, NVIDIA

Marius Koch
Marius Koch

Senior Solution Architect for Climate and Energy Transition, NVIDIA

Weather Forecast Data: Icechunk vs. Iceberg Head-to-Head
Blog Post

Weather Forecast Data: Icechunk vs. Iceberg Head-to-Head

TLDR: Operational weather forecast data workloads built on Icechunk are 10-100x faster and 5-8x cheaper than Iceberg. Warning: This article is long. It's written for a system architect planning to expend significant budget on weather data infrastructure and looking to deeply understand the options a…

Ryan Abernathey
Ryan Abernathey

CEO & Co-founder

Science data comes in tables and tensors
Announcement

Science data comes in tables and tensors

Apache Iceberg comes to Arraylake. Tables now live alongside Icechunk arrays in the same catalog, under the same permissions, in your own buckets — queryable from DuckDB, Spark, Trino, Snowflake, and Zax-SQL.

Joe Hamman
Joe Hamman

CTO & Co-founder

Matt Iannucci
Matt Iannucci

Staff Engineer

Our compute roadmap: introducing Zax SQL
Blog Post

Our compute roadmap: introducing Zax SQL

We're introducing Zax, a new query and compute engine for multidimensional scientific data, and Zax-SQL, the first interface to it. Zax plans your computation for you, so you are no longer responsible for tuning chunk sizes, worker counts, and memory limits.

Ryan Abernathey
Ryan Abernathey

CEO & Co-founder

Brian Davis
Brian Davis

Head of Product

Joe Hamman
Joe Hamman

CTO & Co-founder

Deepak Cherian
Deepak Cherian

Principal Research Engineer

Sebastian Galkin
Sebastian Galkin

Head of Engineering

Making open-source AI weather forecasting models easy to run
Blog Post

Making open-source AI weather forecasting models easy to run

A joint post from Earthmover and Hugging Face on running open-source AI weather forecasting models like Aurora using free ERA5 data from the Earthmover Marketplace — with a live demo space and a full local tutorial.

Emma Scharfmann
Emma Scharfmann

ML for Science, Hugging Face

Aaron Spring
Aaron Spring

ML, AI and geospatial data engineering

Ryan Abernathey
Ryan Abernathey

CEO & Co-founder

Kashif Rasul
Kashif Rasul

AI Research Science, Hugging Face

Announcing the Arraylake Community Tier
Announcement

Announcing the Arraylake Community Tier

Arraylake's refreshed Community Tier unlocks powerful developer capabilities and open-science workflows for free: create your own repos, bring your own object store or use 10 GB of managed storage, and serve data through Earthmover Compute with 50 compute credits per month.

Lindsey Nield
Lindsey Nield

Senior Engineer

Brian Davis
Brian Davis

Head of Product

Product Announcement: Teams in Arraylake
Announcement

Product Announcement: Teams in Arraylake

Teams is here for Arraylake paid accounts: group users and API keys, grant roles once, and every member inherits them. Group-level access management for growing data teams.

Brian Davis
Brian Davis

Head of Product

Matt Iannucci
Matt Iannucci

Staff Engineer

Having it all: virtual and materialized data products
Guest Post

Having it all: virtual and materialized data products

A guest post from dynamical.org on how virtual Icechunk Zarrs complement materialized products to deliver low-latency, complete, and fast-access weather data — plus the low-latency NOAA HRRR dataset it describes, now on the Earthmover Data Marketplace.

Alden Keefe Sampson
Alden Keefe Sampson

dynamical.org

Marshall Moutenot
Marshall Moutenot

dynamical.org

Multiscale overviews in Arraylake: GeoZarr-Powered Pyramids in Flux - and How to Generate Them in Icechunk
Blog Post

Multiscale overviews in Arraylake: GeoZarr-Powered Pyramids in Flux - and How to Generate Them in Icechunk

Overviews, Multiscales, and Pyramids—oh my! We’re introducing multiscale support across Arraylake and Flux Tiles, powered by GeoZarr conventions. This post bridges the product and the data pipeline: how Flux uses precomputed overviews for efficient multiscale visualization, and how to generate those overviews so your datasets can take advantage of this capability.

Lindsey Nield
Lindsey Nield

Senior Engineer

The University of Southern Mississippi makes 90 million ocean profiles conversational: AQUAVIEW's approach to the World Ocean Database
Case Study

The University of Southern Mississippi makes 90 million ocean profiles conversational: AQUAVIEW's approach to the World Ocean Database

How AQUAVIEW reorganized NOAA's World Ocean Database around instrument types, powered it with Arraylake, and built a natural language interface that turns hours-long data workflows into seconds.

Henry Jones PhD
Henry Jones PhD

Director of Research Development and Scientific Entrepreneurship, University of Southern Mississippi

Joshua Hill PhD
Joshua Hill PhD

Director, Institute for Advanced Analytics and Society, University of Southern Mississippi

Announcing the Arraylake MCP Server
Announcement

Announcing the Arraylake MCP Server

The Arraylake MCP server is now publicly available. Connect Claude, ChatGPT, Cursor, or any MCP-compatible client to your Arraylake data and let an AI assistant discover repositories, inspect schemas, query Flux, and render interactive maps.

Matt Iannucci
Matt Iannucci

Staff Engineer

Variable length chunks in Zarr
Blog Post

Variable length chunks in Zarr

A new extension to Zarr just landed: the rectilinear chunk grid lets you specify arbitrarily sized chunks along each axis, aligning chunk boundaries with the natural structure of your data instead of forcing a regular grid.

Joe Hamman
Joe Hamman

CTO & Co-founder

Max Jones
Max Jones

Cloud Engineer @ DevSeed

Davis Bennet
Davis Bennet

Software Engineer (Freelance)

How Kettle Uses Earthmover to Power Wildfire Risk Modeling at Scale
Case Study

How Kettle Uses Earthmover to Power Wildfire Risk Modeling at Scale

The Company Kettle is not a typical insurance company. Using AI to build smarter insurance products, Kettle provides insurance for property owners in areas affected by catastrophic climate events, with a particular focus on wildfire. Their AI models consume over 130 terabytes of satellite, weather,…

Margaret Francis
Margaret Francis

COO

Filtered Subscriptions: Fine-Grained Cloud-Native Data Sharing
announcement

Filtered Subscriptions: Fine-Grained Cloud-Native Data Sharing

Earthmover's filtered subscriptions allow data providers to create secure, read-only views into multidimensional data cubes, enabling more granular cloud-native data exchange between provider and consumer.

Ryan Abernathey
Ryan Abernathey

CEO & Co-founder

Sebastian Galkin
Sebastian Galkin

Head of Engineering

Lindsey Nield
Lindsey Nield

Senior Engineer

Announcing the Earthmover Data Marketplace
Announcement

Announcing the Earthmover Data Marketplace

Earthmover launches the world's first marketplace for AI-ready weather and climate data, offering instant access to analysis-ready cloud-optimized data cubes from leading providers in the open-source Icechunk format.

Ryan Abernathey
Ryan Abernathey

CEO & Co-founder

I/O-Maxing Tensors in the Cloud
Blog Post

I/O-Maxing Tensors in the Cloud

Zarr Python with Icechunk or Obstore now fully saturates the network between EC2 and S3, achieving the physically maximum possible throughput for reading and writing tensor data in the cloud. Benchmarks compare Zarr, Tensorstore, TileDB, and Parquet stacks across a range of chunk sizes and instance types.

Ryan Abernathey
Ryan Abernathey

CEO & Co-founder

Plotting NYC heatwaves during NYC Climate Week
Blog Post

Plotting NYC heatwaves during NYC Climate Week

A hands-on walkthrough of calculating historical heatwave frequency over NYC using ERA5 reanalysis data on the Earthmover platform with Arraylake, Icechunk, Xarray, and open-source climate tools.

Tom Nicholas
Tom Nicholas

Software Engineer

Multi-Player Mode: Why Teams That Use Zarr Need Icechunk
Blog Post

Multi-Player Mode: Why Teams That Use Zarr Need Icechunk

Zarr lacks built-in support for concurrent readers and writers, leading to inconsistent reads and conflicting writes in team settings. Icechunk solves this by adding atomic updates, consistent snapshots, and Git-like version control on top of Zarr.

Lindsey Nield
Lindsey Nield

Senior Engineer

Icechunk 1.0: Production-Grade Cloud-Native Array Storage Is Here
Announcement

Icechunk 1.0: Production-Grade Cloud-Native Array Storage Is Here

Icechunk 1.0 is now stable and production-ready, bringing transactional safety, efficient versioning, high-performance Rust-based I/O, and virtual references for HDF5 and NetCDF to cloud-native array storage. The release includes manifest splitting, distributed writes, conflict resolution, and a 30 TB ERA5 sample dataset.

Ryan Abernathey
Ryan Abernathey

CEO & Co-founder

Meet the Earthmover Team at SciPy 2025 in Tacoma!
Event

Meet the Earthmover Team at SciPy 2025 in Tacoma!

The Earthmover team is attending SciPy 2025 in Tacoma, Washington, with a tutorial on Xarray DataTree and Zarr, multiple talks and posters on Icechunk and Xarray, and a booth showcasing the Earthmover Platform.

Joe Hamman
Joe Hamman

CTO & Co-founder

The Untapped Promise of Weather Radar Data
Blog Post

The Untapped Promise of Weather Radar Data

Weather radar captures rich four-dimensional atmospheric data, but legacy binary formats and fragmented archives make large-scale analysis painfully difficult. A modern, cloud-native data model could unlock radar's vast scientific potential.

Alfonso Ladino-Rincon
Alfonso Ladino-Rincon

Data Scientist

Announcing Fine-Grained Access Controls
Announcement

Announcing Fine-Grained Access Controls

Arraylake now supports fine-grained, repository-level permissions with admin, write, and read privilege levels, giving teams precise control over data access and secure sharing with external collaborators.

Brian Davis
Brian Davis

Head of Product

Xarray for Biology
Blog Post

Xarray for Biology

Xarray's labeled, multidimensional data structures can solve common pain points in biological data analysis, from tracking microscopy metadata to managing complex genomic datasets. Adoption has been limited by awareness, technical rough edges, and lack of tool integration, but the community is actively working to change that.

Ian Hunt-Isaak
Ian Hunt-Isaak

Xarray Community Developer

Icechunk: Efficient storage of versioned array data
Blog Post

Icechunk: Efficient storage of versioned array data

Icechunk stores versioned array data efficiently by never copying or rewriting existing chunks, so each new version only consumes storage for the data that actually changed. Older versions can be expired and garbage-collected when they are no longer needed.

Sebastian Galkin
Sebastian Galkin

Head of Engineering

TensorOps: Scientific Data Doesn't Have to Hurt
Blog Post

TensorOps: Scientific Data Doesn't Have to Hurt

Scientific data pipelines are plagued by data swamps, duplicated code, fragile workflows, and siloed teams. TensorOps is a vision for modern practices that bring collaboration, velocity, and reliability to scientific data engineering.

Brian Davis
Brian Davis

Head of Product

Zarr takes Cloud-Native Geospatial by storm
Blog Post

Zarr takes Cloud-Native Geospatial by storm

At the 2025 Cloud-Native Geospatial conference, Zarr adoption was surging across the geospatial domain, with Copernicus Sentinel, USGS Landsat, Google Earth Engine, and ESRI ArcGIS all embracing the format for cloud-optimized array data.

Joe Hamman
Joe Hamman

CTO & Co-founder

Fundamentals: Tensors vs. Tables
Blog Post

Fundamentals: Tensors vs. Tables

Multidimensional array data about the physical world is fundamentally incompatible with the tabular data model. Benchmarks show that array-native tools like Xarray and Zarr outperform DuckDB and Parquet by up to 10x for common weather data queries.

Ryan Abernathey
Ryan Abernathey

CEO & Co-founder

How Our Customers Use NOAA Data
Blog Post

How Our Customers Use NOAA Data

Earthmover customers share how NOAA climate and weather data powers their businesses, from wildfire risk modeling and energy trading to carbon market ratings and precipitation enhancement.

Ryan Abernathey
Ryan Abernathey

CEO & Co-founder

Accelerating Xarray with Zarr-Python 3
Blog Post

Accelerating Xarray with Zarr-Python 3

zarr-python’s performance paradox Last month, we released Zarr-Python 3.0 - a ground-up rewrite of the library (read more about it in this post). Beyond the exciting new features in Zarr V3, we put a lot of work into addressing some long standing performance issues with Zarr-Python 2. With the improvements described in this blog post, we’ve achieved a 14x speedup in loading the ARCO ERA5 dataset! Zarr-Python 2 had a paradoxical performance quirk; although the library could generate massive petabyte-scale datasets, it struggled to perform well when managing large or highly nested hierarchies. For example, listing the contents of a large Zarr group could be painfully slow, particularly if that Zarr group was stored on a high latency storage backend. Zarr users would experience this as long

Davis Bennet
Davis Bennet

Software Engineer (Freelance)

Zarr-Python 3 is here!
Announcement

Zarr-Python 3 is here!

Zarr-Python 3.0 is released with full support for the Zarr V3 specification, chunk-sharding for more flexible storage, major performance improvements from a fully asynchronous core, and a modernized extensible codebase.

Joe Hamman
Joe Hamman

CTO & Co-founder

Announcing Icechunk!
Announcement

Announcing Icechunk!

Earthmover announces Icechunk, an open-source transactional storage engine for Zarr that brings ACID transactions, time travel, data versioning, and high-performance Rust-based I/O to multidimensional array data in cloud object storage.

Ryan Abernathey
Ryan Abernathey

CEO & Co-founder

Vector data cubes in Xarray
Blog Post

Vector data cubes in Xarray

Vector data cubes extend the familiar raster data cube concept to geospatial vector data, using arrays indexed by geometries instead of gridded coordinates. The Xvec package brings this capability to Xarray, enabling powerful multidimensional analysis of point, line, and polygon data.

Emma Marshall
Emma Marshall

Software Engineer

Toward Zarr-Python 3.0
Blog Post

Toward Zarr-Python 3.0

The Zarr-Python project is undergoing a major refactor toward version 3.0, bringing full support for the Zarr V3 specification, new asynchronous APIs for better performance, and a modernized plugin system for codecs and storage backends.

Joe Hamman
Joe Hamman

CTO & Co-founder

Case Study: Sylvera
Case Study

Case Study: Sylvera

Carbon market ratings company Sylvera adopted Arraylake to centralize millions of scattered geotiff files into cloud-optimized arrays, enabling incremental data ingestion and version-tracked auditing across their geospatial pipelines.

Ryan Abernathey
Ryan Abernathey

CEO & Co-founder

Earthmover and Pangeo at AGU 2023
Event

Earthmover and Pangeo at AGU 2023

Earthmover will be at AGU 2023 in booth 1007 alongside Coiled and Pangeo, demoing Arraylake and presenting three talks on cloud-native scientific data workflows.

Ryan Abernathey
Ryan Abernathey

CEO & Co-founder

Arraylake Now Available in Private Beta
Announcement

Arraylake Now Available in Private Beta

Earthmover launches Arraylake in private beta, a cloud-native data lake platform purpose-built for multidimensional arrays with a built-in data catalog, ACID transactions, version control, and virtual file support.

Ryan Abernathey
Ryan Abernathey

CEO & Co-founder

Earthmover is hiring
Announcement

Earthmover is hiring

Earthmover is hiring two founding engineers to help build a modern data stack for science, tackling climate and planetary challenges with cloud-native software.

Joe Hamman
Joe Hamman

CTO & Co-founder

Why we started Earthmover
Blog Post

Why we started Earthmover

Earthmover was founded to build a modern cloud data stack for scientific data, inspired by the success of the Pangeo open-source community and the urgent need for better tooling around multidimensional array datasets in climate tech and beyond.

Ryan Abernathey
Ryan Abernathey

CEO & Co-founder