A Data Layer for Scientific AI: Icechunk for Agentic Single-Cell Biology Workflows

Our mission at Earthmover is to bring down the data bottlenecks that stand in the way of scientific progress. We do this by identifying challenges common to many different scientific domains and building foundational technology to address them.

Luiz and I just returned from the scverse conference in Copenhagen, an inspiring gathering of single-cell and spatial ‘omics researchers developing open-source tools for data analysis and AI. There, we observed specific instances of two trends that are playing out across science:

• Agents are getting embedded deeper in the scientific workflow. A short time ago, they were seen as relatively limited coding assistants. Today they are testing hypotheses independently with researcher supervision. At some frontier labs—and likely soon at a university near you—they are making new discoveries autonomously. The workshop on Claude Science from Pablo Garcia-Nieto of Anthropic really drove this home.
• The need for “AI-ready data” is getting more urgent. As teams develop agentic workflows and build foundation models, this once over-hyped term is now crystallizing into much more precise requirements. Cell atlases, in particular, represent a perfect target for open-ended agentic discovery.

With this context, it’s easy to understand why Luiz’s talk on Icechunk received such an enthusiastic response: Icechunk is a natural data layer and system of record for scientific AI.

In the blog post How Agents Use Systems Differently, Davis Treybig argues that, in order to support the patterns preferred by agents, data systems must provide snapshotting, time travel, branching, and copy-on-write:

Agents benefit from high degrees of parallelism, exploring many different ideas or options simultaneously and then converging on the optimal ones.

Inspired by this framing, this blog post lays out some of the specific use cases for a transactional, versioned, cloud-based Zarr store in agentic science workflows, using examples from single cell biology. We’re actively seeking feedback on these ideas, so please reach out with your reaction!

The Status Quo: Downloading and Copying Files Endlessly

Agents were trained on us; by default, they do things the way we already do things. So when we tell an agent to perform a single-cell analysis, it’s going to reach for the most common, well-documented workflows.

Today this workflow is based on files—lots and lots of files. In the scverse community, the traditional file format has been a specific profile of HDF5 called h5ad.

Want to read a dataset? Download some files.

Got a new result to share? Upload some more files.

Want to try a different analysis method? Copy your files. (That’s how you get feature_bc_matrix.v4.1-ryans-annotations.final-v2.h5. Sound familiar?)

This workflow is so ingrained that it may seem pointless to even question it. Is there really any alternative to file-centric workflows in bioinformatics?

Yes: the alternative is cloud-native data storage that functions more like a database than a jumble of files. With formats like Zarr, the user can instantly open the dataset directly over the network without downloading everything up front; the data streams to the client on demand, as needed by the workflow. There does not need to be a local copy of the data sitting on a hard drive.

There are two scenarios where this is advantageous over the file-download approach:

• When datasets are large—the pain starts around 10 GB and by 10 TB it becomes functionally impossible to download the full dataset. And of course, very large data corpuses are necessary for building the most ambitious AI models.
• When many users or agents need to interact with the data concurrently. Even if the dataset is small, storing many duplicate copies of it adds up. In the old world, concurrency was limited by the number of scientists; but now, agents can generate orders of magnitude more concurrent data usage!

When these two patterns combine—many agents working concurrently over a large dataset—the cloud-native approach becomes not just convenient but necessary.

	Few users / agents	Many users / agents
Big data	Cloud-native is more performant	Cloud-native is necessary!
Small data	Just download the files	Cloud-native is more cost effective

The cloud-native approach to data distribution is also starting to take hold in the scverse community, with Zarr being adopted as the core data container across AnnData, SpatialData, and MuData.

From Read-Only to Concurrent Collaboration

Today, many scverse users are probably familiar with the read-only experience of using Zarr from cloud storage. For example, the Wellcome Sanger Institute publishes some example OME-Zarr data from the 10x Genomics Visium platform. But if they want to actually do anything useful besides just look at the data (annotation, segmentation, etc.), they probably have to download it anyway.

This is because most single-cell workflows involve mutating or augmenting the raw dataset. Starting from raw data output by a transcriptomics platform, a typical workflow involves:

• Quality control, such as removing doublets, empty droplets, and low-quality cells.
• Embedding - each cell is reduced to a low-dimensional representation (PCA, or a learned latent space such as scVI) that the downstream steps work on; UMAP or t-SNE then projects that to 2D for visualization.
• Clustering - cells with similar expression profiles are grouped into clusters, using methods such as Leiden clustering.
• Segmentation - for spatial data, segmentation techniques applied to histology imagery can be used to identify individual cells, and these geometries can be merged with the transcripts.
• Annotation - human or machine generated labels and details are assigned to the cells and / or clusters. For example: “this is a cancer cell”.

Each of these augmentations creates data and metadata that need to be kept together with the original raw data. (Indeed that’s exactly what the AnnData format solves; a single container for the data and its annotations.) But when multiple different scientists—or AI agents—are working on the same original datasets via the download method, this can result in an explosion of dataset copies, each with its own incompatible annotations. Merging these results back together becomes impossible.

Experienced teams have workarounds for this. You can keep every variant in one AnnData object under different obsm and obs keys; you can write small sidecar files per method next to an immutable base; or you can let Nextflow or Snakemake give each run its own output directory. While these approaches avoid the explosion of data volume from redundant copies, none of them gives you isolation between concurrent writers (two processes adding to one object will clobber each other), atomicity (a crashed job leaves a half-written file that looks like a result), or any guarantee that a result was computed against the version of the base data you now have. Unfortunately, these are some of the most likely pitfalls encountered in multi-agent workflows.

Instead, what if everyone (and their agents) could work together on the same, cloud-based dataset, collaboratively iterating from raw data to publication-grade discovery with a single source of truth shared by all, plus full history and provenance tracking? And what if the only infrastructure needed to enable this was standard cloud object storage?

This is the problem that Icechunk solves.

What is Icechunk? A Serverless Database for Zarr

Before explaining Icechunk however, we have to review Zarr. (For a deeper dive, see our explainer Fundamentals: What is Zarr?)

Zarr is a hierarchical container for storing ND-array data (sometimes called tensors) plus metadata, inspired by the HDF data model. HDF and Zarr are critical in scientific and engineering fields for storing complex datasets that don’t fit into tabular data systems.

An important thing to remember is that, unlike HDF, Zarr is not actually a file format. It’s more like a “directory format”; a specification for a group of interrelated files that are stored together. The files are either metadata—stored as simple JSON—or numerical data chunks, stored as compressed binary. Because Zarr is not constrained to fit into a single file, the arrays can be arbitrarily large (such as the petabyte-scale mouse-brain images) and the hierarchies can be arbitrarily deep. The separation of metadata from numerical data together with the flexible chunking of arrays is what makes the format cloud native. A single Zarr store can accommodate all of the data and metadata associated with an AnnData dataset, and the user or agent doesn’t have to think in terms of the individual underlying files.

However, Zarr by itself can’t support the kind of collaborative workflow described above because it isn’t safe for uncoordinated concurrent writes. There’s no protection against one user (or agent) overwriting another’s annotations, segmentations, etc. This makes a shared Zarr store a totally unreliable source of truth for collaborative, multi-agent workflows because of the lack of isolation between them. This pushes us back towards file downloads—duplicating the datasets to local storage is an expensive but effective way to achieve isolation—or sidecar workarounds.

The way to resolve this tension is to make Zarr work more like a database, while keeping the same data model. The gold standard for databases is ACID: updates should be atomic, consistent, isolated, durable. Icechunk is an additional layer that sits underneath Zarr which provides ACID guarantees. It batches updates to multiple entities in the Zarr store into atomic transactions. It enforces isolation between concurrent readers and writers. And it serializes updates so that all clients agree on the history of how a dataset evolved, which also enables data versioning, branching, and time travel. Seen from a different perspective, Icechunk is like Git for Zarr.

Icechunk does all of these things without introducing a server we have to manage into our architecture. All of its ACID capabilities ride on top of the object store’s core primitives, such as the atomic compare-and-swap operation. And it does this without a read-performance penalty: for bulk reads, Icechunk is generally faster than vanilla Zarr (see our comprehensive benchmarks).

So, to summarize, with Icechunk we get:

• A cloud-based Zarr store that can accommodate hierarchical array (tensor) data of any scale
• A mechanism for humans or agents to safely mutate that store, adding annotations, segmentations, etc. potentially in a parallel / distributed fashion, with no partial writes ever visible to readers and no lost updates between concurrent writers
• A full record of how the store evolves over time, and the ability to rewind to a previous state without unnecessary data duplication
• Zero-copy branches for agentic sandbox explorations
• No additional infrastructure or performance penalty compared to vanilla Zarr

We’ll now demonstrate what that enables via two specific example notebooks, both available in the earth-mover/icechunk-bio-notebooks repo on GitHub. Each one can be opened and run directly in your browser on molab.

Example 1: Single user, Icechunk snapshots as provenance

This is a very simple example to introduce Icechunk’s core mechanism, the snapshot, as the atomic unit of dataset evolution.

We reprise the basic Getting Started with AnnData tutorial, but substituting the h5ad file for an Icechunk repo. We start from the reference PBMC3k dataset, containing the transcriptome from 3000 peripheral blood mononuclear cells, provided by 10x Genomics and packaged with AnnData.

We walk through six stages of analysis: ingest, QC, normalization, embedding, clustering, and annotation. Each stage creates a snapshot in the Icechunk repo which stores only the chunks that changed since the previous step. The commit metadata record parameters and details of the step. At the end, we have a full record of exactly what was modified in each operation, AND we can revert the dataset to its earlier state. In our example, we use this to rewind the history and correct a mislabeled cluster.

To achieve a similar level of auditability and version history without Icechunk, we would need to output a separate HDF file for each stage of the pipeline, leading to roughly 6x more data being stored. (The count matrix dominates the file size and would be duplicated in every copy.) That may be tolerable for a tiny demo dataset, but not for a petabyte-scale multi-modal cell atlas!

Example 2: Multiple agents testing batch integration methods

We now take on a more ambitious and realistic task: batch integration over many experiments.

Batch integration in single-cell RNA sequencing is the process of combining datasets from different experiments, samples, laboratories, or sequencing runs while correcting for unwanted technical differences. The goal is to make cells group together based on their biology, rather than which experiment produced them. Batch integration enables cross-study analysis and large-scale cell atlases, making it a foundational operation for multi-dataset single-cell platforms.

Batch integration is a good workflow to delegate to agents because a) there are many different methods to experiment with and b) there is a clear metric to use to evaluate which one is best.

The data uses 32,484 cells in 9 batches from the human immune dataset of the scIB benchmark (Luecken et al. 2022), with 16 expert-annotated cell types. The notebook compares 7 batch-integration methods: unintegrated, harmony, combat, scanorama, bbknn, scvi, and scanvi.

Our workflow delegates one LLM agent to tune each method. Each agent writes to its own Icechunk branch, and all 7 agents run at the same time. (In this notebook, they’re all running on the same machine, but in principle, we could farm the work out over a distributed cluster while still sharing a single copy of the data.) An evaluator agent then reads all branches and promotes the best result to main.

flowchart TD
    data["scIB human immune data: 32,484 cells, 9 batches"] --> base["Base snapshot, tag immune-base-v2: HVGs and a 50-PC PCA"]
    base --> main["Branch run-main"]
    base --> byhand["Branch run-by-hand: one Harmony iteration with no LLM, then deleted"]
    base --> agent
    subgraph agent["7 method agents in parallel, each on branch run-method"]
        llm["LLM picks the params"] --> fit["run_method"]
        fit --> scib["score with scib-metrics"]
        scib --> commit["write_method_output and commit"]
        commit --> llm
    end
    agent --> evaluator["Evaluator LLM: branch_scores, then picks one winner"]
    evaluator --> lineage["check_lineage"]
    lineage --> promote["promote: fast-forward to the best iteration, move the labels"]
    promote --> main
    promote --> tag["tag_winner"]
    tag --> discard["discard the other branches"]
    discard --> gc["collect_garbage"]
    main --> result["Result on main: UMAP, ARI and NMI"]
    gc --> scenarios["4 failure scenarios, each in its own run: Icechunk against a sidecar baseline"]

Before the evaluator runs, Icechunk can draw every branch of the repo as one tree. Each of the 7 method branches forks from the base snapshot, and each commit is one iteration of its agent.

At the end of the run, the history of main shows the full lineage of the result. Icechunk draws this graph directly in the notebook:

Each agent scores its own iterations with scib-metrics, the standard benchmark for batch integration: a bio-conservation score (do cells of the same type still group together?) and a batch-correction score (are the batches mixed?), combined as Total = 0.6 × bio conservation + 0.4 × batch correction. scanvi is the only supervised method, so the notebook hides the cell-type labels of three bone-marrow batches (9,581 cells) from it, and computes bio conservation only on those held-out cells. The evaluator reads the best iteration of every branch, picks the winner on Total, and checks that the branch still descends from the current base before promoting it. In our run, scanvi won with a Total of 0.649 (bio conservation 0.710, batch correction 0.557), ahead of harmony at 0.646. On the held-out cells, the promoted labels scored an ARI of 0.455 and NMI of 0.721 against the dataset’s expert cell-type annotations. Note that the scanvi agent ran 3 iterations, but the notebook promoted iteration 1, not the last one. The evaluator chose only the method. The promotion step then moved main to the commit with the highest Total on that branch. Icechunk keeps every iteration as a commit, so a worse later attempt does not replace the best result.

The UMAP of the winning scanvi embedding shows the expert cell types next to the promoted Leiden clusters. Each cluster takes the color of its most common cell type, in a lighter or darker tone when several clusters share a type. Thus a cell type that the clusters split, such as the CD4+ T cells, shows as several tones of one color.

Branches make this workflow safe and cheap. Each agent writes only the paths of its method, and all branches share the base data. Promotion moves a branch ref and makes two small commits; zero data is copied. Once the agents are done and the winner is promoted, we delete the other branches and run Icechunk’s garbage collection. It reclaims the storage of every discarded attempt, so only the base data and the winning result data remain. We killed an agent mid-write; its branch stayed at its last commit, without interfering with the others, or with the main branch. We re-ran QC on main while agents were running; the evaluator refused to promote the branches forked from the old base. The guardrails worked.

This workflow is still somewhat artificial; the data are not huge, and the amount of parallel agents (7) is modest. However, it clearly demonstrates the value that Icechunk brings. Rather than duplicating our data files over and over for each agent, all the agents can work against branches of the same cloud-based data source while maintaining strict isolation. A single source of truth for the entire agent swarm.

This approach scales in both directions. Creating a new branch is cheap whether there are 7 or 700. And while the data here were small, we know that Zarr stores can hold petabytes. (In the geosciences, where Icechunk adoption is farther along, Icechunk stores of this size exist—for example, the NOAA GEFS 35-day weather forecast, curated by Dynamical.org, currently sits at 6.5 PB.)

Conclusions and Next Steps

We’re excited by the enthusiastic feedback we’ve received from the single cell community about Icechunk. As demonstrated by our notebooks, Icechunk offers two key capabilities that are essential for reproducible, agent-driven bioinformatics workflows:

• A system of record which permits tracking updates, corrections, and annotations over time without needless data duplication.
• A robust, safe way for many users or agents to concurrently branch, experiment, and iterate on a shared dataset stored in cloud object storage.

We’re dedicated to working with this community to make Icechunk even more useful and powerful. Some of the future directions include:

• Virtualization (i.e. zero-copy ingest) - Virtualization is a technique that allows legacy file formats (h5ad, TIFF, etc) to be wrapped in a virtual Icechunk / Zarr layer, instantly transforming them into a cloud-native format without copying all the bytes. We have some compelling examples of applying virtualization at scale to existing archives that we’re planning to share soon.
• Spatial data - The Icechunk approach currently requires that all the data fit the Zarr data model. However, the spatial data community frequently embeds other file formats, such as parquet, into the Zarr hierarchy—an out-of-spec use of Zarr that is motivated by real requirements. We’re exploring how we can either:
  • Implement Zarr extensions that make these workarounds unnecessary (e.g. better vector data support), or…
  • Relax Icechunk’s schemas to accommodate out-of-band files within the hierarchy.
• Visualization - The single-cell community has a rich ecosystem of data visualizers (napari, Vitessce, mersfisheyes, etc) which work with h5ad and Zarr but generally not Icechunk. Adding broader Icechunk support is an easy lift that will make the data more accessible and interoperable.

Above all, we’re interested in learning more from you about how AI is transforming spatial ‘omics, and where data bottlenecks are standing in the way. Our mission as a company is to bring down these bottlenecks.

If you want to follow what we’re building at Earthmover, please join our mailing list.