Our compute roadmap: introducing Zax SQL

Multi-dimensional data has a computing problem.

For years, the scientific Python ecosystem has done an extraordinary job building tools for working with multidimensional data. Xarray gave scientists an intuitive model for arrays. Dask made those computations distributable. Zarr brought arrays to the cloud.

But despite more than 10-years of effort, using these tools in production still requires an enormous amount of expertise or trial-and-error. You have to decide how to chunk your data, how much memory to provision, how many workers to run, and how to recover when something inevitably fails.

We think it should work differently. The system should make those decisions, not the user.

Today we’re introducing Zax, a new query and compute engine for tensor data. Zax plans the computation for you: it chooses a read strategy, schedules I/O against the data layout on-disk, and sizes the work to the resources available. Users are no longer responsible for tuning the low-level details.

Zax is the next step in the journey we’ve been on since we started Earthmover: making scientific data in the cloud as easy, fast, and economical to work with as modern tabular data.

Zax-SQL is the first interface to the engine, and it’s available today on every Arraylake plan, including the free Community Tier.

We’re in the early days of Zax development. Zax-SQL showcases some of what the engine can already do, but we’re far from done with Zax. This post chronicles our journey from storage to compute, and offers a primer on Zax and where we’re headed with this part of the Earthmover Platform.

From data to compute

We started Earthmover because working with scientific data was too slow and too painful.

Scientific array datasets were still living in opaque binary files and collections of scripts, while the tabular data ecosystem was rapidly moving toward a streamlined, interoperable, cloud-native future. Working with arrays meant slow I/O, duplicated data, fragile pipelines, expensive infrastructure, and enormous amounts of engineering work just to get to the point where the science could begin.

The obvious place to begin was the bottom of the stack: data storage.

Object storage gave us a scalable data layer, and Zarr gave us a much better foundation on top of it: chunked, compressed multidimensional arrays that can be read efficiently from object storage. But Zarr alone wasn’t enough for production, so we built Icechunk.

Icechunk brings database-like capabilities — transactions, branching, rollbacks, distributed writer support, and version history — to the Zarr format. It also supports zero-copy ingestion of existing NetCDF, HDF5, GRIB, H5AD, and TIFF archives through virtualization, which lets organizations utilize petabyte-scale archives in the cloud without rewriting petabytes of data. And it is the fastest way we know of to read and write tensors in the cloud.

Icechunk is free and open source, and it’s rapidly becoming the standard for teams building serious scientific data systems.

Today, Icechunk is used in production across industry, government, non-profits, and academia, including RWE, Brightband, NOAA’s National Weather Service, NASA, The Nature Conservancy, CarbonPlan, Columbia University, and the University of Washington.

To make the most of this powerful new cloud storage model, we created Arraylake: a managed catalog and governance layer for Icechunk, with discovery, lineage, authentication, authorization, teams, and integration with cloud-provider IAM. Next came the first version of Earthmover Compute, which we launched as Flux in early 2025. It solved the “last mile” problem we kept watching teams struggle with: instead of every customer hand-rolling a JSON API over their own data cubes, Flux APIs served Icechunk datasets over established standards — OGC EDR, WMS, and OPeNDAP — with no service to build, scale, or operate.

As our customers used Arraylake, we kept hearing the same thing: this is great for managing the data we already have, but how do we find the high-quality data we need?

So we built the Data Marketplace. Today the Marketplace contains more than 60 PB of data from 22 providers, including datasets like ERA5, giving organizations a governed way to discover and access large cloud-optimized scientific datasets.

The problems we started Earthmover to solve — finding, sharing, collaborating on, iterating on, and distributing data — are increasingly becoming solved problems.

Along the way, though, our users kept telling us that the other half of their work was still painful. Flux answered the delivery question, but its APIs are read-only and generally only support simple queries. Everything upstream of that, the pipelines that actually compute the data, was still theirs to build, tune, and operate. The Xarray and Dask stack wasn’t meeting their needs — and where it did, it was too expensive to tune and operate. This has been one of the most consistent pieces of feedback we’ve received, and it long predates the current wave of AI workloads, though training and evaluating models against petabyte-scale archives has made it much harder to ignore.

Converting data into insight is now the bottleneck.

Array computing today

If you’ve worked with Xarray and Dask, you know the workflow. You open a dataset, construct an Xarray computation, let Dask build a task graph, submit that graph to workers, and then start tuning.

Is the chunking right? Do I have enough memory? Too many workers? Too few? Am I reading efficiently? Should I rechunk, and pay for a second copy on disk? Why did the cluster run for three hours? Why are my workers dying?

Xarray and Dask have enabled an enormous amount of computation that would otherwise have been impossible. We use them ourselves, and several members of the Zax team are core maintainers of Xarray and Dask.

But the fundamental problem remains: the user is still responsible for making all the layers of the stack cooperate.

We’ve diagnosed this problem as this: today’s software stack separates data layout, I/O, computation, task partitioning, compute resources, and execution environment. Different tools make decisions about different pieces, and no tool optimizes across all of them. The scientist or data engineer has to stitch it together and absorb the consequences when the pieces disagree.

Modern analytical databases take a different approach. Their query optimizers reason about a query as a whole: what data needs to be read, how it should be filtered and partitioned, and how the computation should be executed.

We think scientific arrays deserve the same treatment.

Zax

Zax is a new query engine for multi-dimensional chunked arrays, with a sophisticated query planner at its core.

The planner works much like a database planner, but in multi-dimensional space. It looks at a workload, reasons about the available resources, optimizes the query across all layers of the stack, and produces a streaming execution plan. Where necessary, it inserts additional operations to make sure the query executes reliably and efficiently.

Critically, the planner isolates the user from chunk shape concerns. This is important as we’ve learned that chunk shape is the single largest source of pain in array workloads today. We are trying to remove the choice of chunking from the user’s purview. Doing that well requires distilling a decade of experience building Xarray, Dask, and Zarr into planner heuristics, which is most of what we’ve been working on.

Like Icechunk, Zax is written in Rust, which matters for more than performance. Most scientific computing today is done in Python, and Python will remain a first-class way to interact with Zax. But the scientific data community is much larger than Python. Scientists use R, Julia, MATLAB, and other languages; data engineers and application developers may not use a scientific programming language at all.

The compute engine shouldn’t force all of those users through one programming language. Rust gives us a native, high-performance engine that can sit underneath many different interfaces — including a native Rust API for embedding Zax directly, alongside the Python API.

Why SQL first

Ultimately, Zax will have a dedicated Python API that feels natural to users of Xarray, and a standalone open-source execution engine.

We’re deliberately not starting there. Zax is still evolving rapidly, and we expect to learn a great deal about both the engine and its eventual Python API by putting it in front of real workloads. Shipping a Python API first would mean committing to it now, and we aren’t ready to do that.

Instead, we’re exposing Zax through established client-server protocols. Zax-SQL speaks the PostgreSQL wire protocol and Arrow Flight SQL. These are open standards with many implementations, so the queries, notebooks, dashboards, and pipelines you build against Zax aren’t tied to an Earthmover-specific client.

There’s a second reason to start with SQL, and it has become more pressing over the past year. Large language models, and the agents built on them, write competent SQL. They are much worse at writing memory-aware Python that traverses chunked, multi-dimensional arrays without falling over — which is precisely the expertise Xarray and Dask still demand of a human. Putting a SQL-compatible interface in front of petabyte-scale Icechunk data means an agent can query scientific tensor data through the tabular interface it already knows, instead of generating array code that an expert then has to review and tune.

How Zax-SQL works

SQL is the lingua franca of mainstream data engineering, but it’s rarely used with scientific data. The reason is simple: SQL is designed for relational (i.e. tabular) data, while most scientific data — imaging and sensor systems, simulations, model output — is distributed using the tensor data model (see our explainer on tensors vs. tables). But many scientific workflows do end up producing tabular results, or need to hand results to a relational system.

Zax-SQL bridges that divide. It maps the NetCDF data model onto the relational model and lets you query tensors as if they were rows and columns. The mapping is one-way and equivalent to flattening the dataset: coordinates and data variables all appear as columns of the same length, variables of differing dimensionality are lazily broadcast against each other, and the row count is the outer product of the dimensions.

We’ve done two things to make this practical rather than merely possible:

• efficient pushdown of SQL filter operations into selection operators in multidimensional space, so a WHERE clause becomes a targeted read rather than a full scan
• efficient streaming I/O for turning tensor data into tabular record batches

From there, Zax-SQL relies on Apache DataFusion for the relational parts of the processing chain.

A monthly climatology at a single grid point looks like this:

SELECT DATE_PART('month', valid_time) as month,
       AVG(t2m) as t2m_mean,
       STDDEV(t2m) as t2m_std
FROM era5.'single/temporal'
WHERE longitude = 45.0 AND latitude = 60.0
GROUP BY month
ORDER BY month

That query runs against a 125 TB Icechunk dataset, but nothing close to 125 TB is read. The planner turns the WHERE clause into a coordinate selection in multidimensional space, works out which chunks actually intersect that point, and streams only those blocks out of object storage as Arrow record batches for DataFusion to aggregate. You write a filter (e.g. a WHERE clause) and the planner decides what that means in terms of I/O.

What works today (and what doesn’t)

Zax-SQL is the first thing we’ve built on Zax, and it exposes a small fraction of the engine.

Works today	In progress
Any PostgreSQL-compatible client, or any Arrow Flight SQL client Filters, projections, aggregations, and joins over flattened tensor data Icechunk datasets in your own Arraylake organization or from the Data Marketplace	Writing results back to Icechunk Spatial extensions: PostGIS geometry types and operators Rolling windows, regridding, and other general array operations

If these gaps are important to you, tell us in the Earthmover community Slack and we’ll work to prioritize them for you.

We also want to shout out some of the excellent other projects in this space:

• The Beacon project has a nifty client-server architecture for querying NetCDF-type data with SQL.
• The xarray-sql project provides a similar capability in a standalone serverless architecture.
• The growing raster support in SedonaDB and Apache Sedona offers some similar functionality via GIS-style spatial SQL operators.

Zax-SQL was developed independently of any of these, but we’re excited to see the whole space of array-based computing mature!

Managed compute

The engine is only part of the story. We’re also taking responsibility for the infrastructure required to run it.

Scientific computing still often means running and maintaining clusters: Kubernetes, schedulers, workers, autoscaling, networking, credentials, monitoring, dependencies, and all the operational machinery required to keep a distributed system healthy.

With Zax running as a managed service in the same cloud environment as the data, compute becomes another part of the workflow rather than another infrastructure project. Concretely: you stop paying for an idle cluster while you stare at a Dask dashboard trying to work out which worker died and why.

It also changes the cost profile. A planner that sizes work to the resources available is a planner that doesn’t over-provision, and a managed service with defined resource limits is one that can’t quietly run up a cloud bill overnight. Zax means less manual work, less data movement, and more predictable infrastructure costs.

What’s next

Zax is young, and we’re still experimenting with what the perfect array computing interface should look like. We’re being deliberate about what we build next: each new capability should stand on its own, but also make the rest of the system more useful.

The most important item is a native Python interface for Zax that feels natural to Xarray users, and that lets you run arbitrary array workloads rather than the narrow set the current services expose. Shorter term, we’re working on spatial extensions for Zax-SQL, fuller support for geospatial workflows through OpenEO and the OGC APIs, and continued work on the planner so that more of the optimization work disappears from your workflow entirely.

But the roadmap extends beyond the engine itself.

We want queries to become reusable objects, rather than something that disappears when a notebook finishes running. A query should be something you can save, version, share, inspect, and run again against new data. Views should work the same way, giving you a way to define derived datasets without immediately materializing another copy of the underlying data.

From there, the line between data and compute starts to blur in useful ways. A derived dataset doesn’t necessarily need to be written out to a new collection and handed off to another system. Computed results can be published directly through the Data Marketplace, or served through the same interfaces as any other dataset — SQL, OGC tiles, and the rest.

We’re also thinking about how these capabilities fit into the broader scientific data ecosystem. The goal isn’t to replace the tools people already use. Xarray, Dask, Zarr, Icechunk, OpenEO, SQL, and the projects around them are valuable precisely because they’re interoperable. We’re designing Zax to compose with that ecosystem rather than becoming an isolated, standalone tool.

We’ve spent the first chapter of Earthmover making scientific data cloud-native. Now we’re going after compute.