Multiscale overviews in Arraylake: GeoZarr-Powered Pyramids in Flux - and How to Generate Them in Icechunk

Announcing multiscale support in Flux

We’re excited to announce support for multiscale (overview) datasets in Flux Tiles, powered by the GeoZarr conventions for proj, spatial, and multiscales.

This unlocks a major improvement in how high-resolution geospatial data can be visualized across the Earthmover platform. Instead of attempting to render full-resolution data at every zoom level, Flux can now intelligently select from precomputed overviews to serve appropriately downsampled data as users zoom out on the map. The result is faster rendering, lower memory usage, and the ability to smoothly visualize datasets that were previously too large to explore at global scale.

From xpublish-tiles to production-grade visualization

This capability is built on top of xpublish-tiles, the open-source backend that powers Flux Tiles. When we released xpublish-tiles last year, it enabled dynamic tile generation directly from Icechunk-backed Zarr datasets, bringing slippy map visualization to cloud-native scientific data.

But there was still a fundamental limitation: at low zoom levels, rendering still required scanning and aggregating large volumes of native-resolution data on the fly.

That limitation became especially painful for high-resolution datasets (think: 10m, 30m, 90m data). Even when tiling worked well at moderate zoom levels, zooming out too far overwhelmed the system with the sheer volume of data required to render a single global view.

Overviews change that.

By incorporating precomputed multiscale representations that follow the GeoZarr conventions, xpublish-tiles can now select the overview that best matches the requested zoom level, choosing the coarsest available representation that still provides sufficient detail. This eliminates the need to scan and aggregate large volumes of high-resolution data when zoomed out.

Flux Tiles builds on top of this OSS foundation as a hosted, production-grade visualization layer. It uses xpublish-tiles under the hood, but adds the operational and ergonomic layer needed for real-world use: seamless dataset discovery, consistent rendering behavior across datasets, and an experience tuned for interactive exploration rather than backend configuration. The core capability is open, but the product layer makes it usable at scale.

Sentinel-2 NDVI over CONUS, rendered with precomputed overviews. As you zoom in and out, Flux Tiles automatically selects the best-matching resolution—read on to learn how we built these overviews at scale.

Powered by the GeoZarr Standard

Supporting multiscales in Flux is only half of the solution. To take advantage of this capability, datasets first need to contain overviews and the metadata needed to describe them. Overviews are a key part of the popular Cloud Optimized GeoTIFF format, but until recently, there was no standard way to store overviews in Zarr.

That’s where GeoZarr comes in.

GeoZarr provides a standard way to describe multiscale datasets, allowing tools like xpublish-tiles to discover available overviews and select the appropriate resolution for a given zoom level. But before Flux can use those overviews, they need to be generated and stored alongside the native-resolution data.

The remainder of this post covers both sides of that workflow: what GeoZarr is, what overviews are and why they matter, how to decide which overviews to generate, and how to build them efficiently for large Icechunk-backed datasets. We’ll also walk through a real Sentinel-2 example where overviews made it possible to seamlessly visualize billions of data points across space, time, and derived variables in Flux.

Fundamentals: What is GeoZarr?

GeoZarr is a set of Zarr metadata conventions designed as a modular, composable way to represent multidimensional georeferenced grid data. It was created to standardize how geospatial metadata is stored in Zarr datasets by building on a small set of interoperable conventions that describe projection, spatial structure, and multiresolution data. ¹

If you need a refresher on metadata in Zarr, check out our What is Zarr? post!

In practice, GeoZarr gives us a reliable way to discover and interpret the key pieces of information needed for visualization and analysis, including identifying and selecting overviews for rendering geospatial data in xpublish-tiles.

The standard is composed of three core conventions:

• Proj convention: https://github.com/zarr-conventions/proj
• Spatial convention: https://github.com/zarr-conventions/spatial
• Multiscale convention: https://github.com/zarr-conventions/multiscales

By adding proj, spatial, and multiscale metadata to your Icechunk dataset, tools like Arraylake and xpublish-tiles can automatically discover coordinate systems, spatial extents, and available overviews. This makes it possible to intelligently select the appropriate resolution level when serving data for a given map zoom level.

Fundamentals: What are overviews?

“Overviews,” “multiscales,” and “pyramids” all refer to the same concept: precomputed, downsampled versions of a dataset that represent the same data at progressively coarser resolutions.

Multiscale pyramids are especially useful for visualizing high-resolution Earth science data. They allow you to render a low-resolution global view quickly and then smoothly transition into higher-resolution data as you zoom in.

At a low zoom level (i.e. zoomed out), loading the full-resolution dataset is both unnecessary and expensive; you would need to read and process an enormous amount of data just to draw a single global image! Instead, a coarser representation of the data is visually sufficient. As you zoom in, you progressively need more detail, but still not the full native resolution until you reach very high zoom levels (i.e. when the zoom matches the native resolution).

Overviews are precomputed downsampled versions of the dataset stored alongside the native resolution. Each level is created using an aggregation method such as mean, median, nearest-neighbor/interpolation, or mode, depending on the data type and desired visual effect, which creates coarser and coarser “views” of the data.

These precomputed levels are then used during visualization, allowing the system to select the smallest dataset that still provides sufficient detail for a given zoom level. In most cases, you will still want to access the native-resolution data directly when performing analysis or retrieving actual values from the dataset.

Choosing what overviews to generate

Most tiling systems generate overviews by repeatedly downsampling by a factor of 2: 1x (native), 2x, 4x, 8x, and so on. This mirrors Web Mercator zoom levels, which also change resolution by a factor of two at each zoom step.

A common assumption is that generating a complete pyramid of overviews is expensive, but in practice the storage overhead is modest. In a 2x pyramid, each overview contains one quarter as many pixels as the previous level, so the total additional storage is approximately (1/4 + 1/16 + 1/64 + \cdots = 1/3), or about 33% on top of the original dataset. A 4x pyramid is even cheaper, adding only (1/16 + 1/256 + \cdots = 1/15), or roughly 6.7%. An 8x pyramid drops to ~1.6%, and beyond that the cost becomes negligible even for multi-terabyte datasets.

Crucially, this cost is highly skewed toward the finest levels. In a 2x pyramid, the 2x overview alone accounts for roughly 75% of total overview storage, and the 2x and 4x levels together account for over 90%. Coarser levels (8x and beyond) contribute very little storage while still providing substantial performance benefits for zoomed-out visualization. For that reason, there is usually little reason to omit the coarse end of the pyramid: it is cheap, and it significantly reduces the amount of data required for low-resolution views.

The more important question, then, is not whether to generate the full pyramid, but how densely to space them. In other words: do we actually need every intermediate 2x level?

Although a full 2x pyramid is relatively inexpensive in aggregate, the cost is heavily concentrated in the finest levels. The 2x overview alone can represent a substantial amount of data for multi-terabyte datasets, even though coarser levels quickly become negligible in terms of storage. This creates a key trade-off: the finest overview levels primarily reduce data access cost near native resolution, but they are also where most of the storage overhead is concentrated.

At the same time, neighboring 2x overview levels often provide surprisingly little additional value in practice. Map renderers do not require an exact match between the data resolution and the display resolution to produce good-looking results. Standard resampling methods (e.g., mean, nearest-neighbor, bilinear interpolation, etc.) can tolerate modest mismatches with little or no visible degradation. This is already how dynamic tiling works in xpublish-tiles for datasets without precomputed overviews: a single source resolution routinely serves multiple adjacent zoom levels.

Because zoom levels form a logarithmic scale, a single overview level can often support several zoom levels while maintaining acceptable visual quality and performance. This reframes the core decision: not as a choice between “2x vs 4x strategies,” but as a choice about whether to include the expensive finest step at all. If users frequently navigate near native resolution and responsiveness is critical, including a 2x level may still be worthwhile, since it reduces data access cost at those zooms while preserving fidelity. If interactions are concentrated at coarser scales (for example, continental to state-level exploration), starting at 4x often provides a better balance between storage cost and interactive performance, without meaningfully degrading visual output.

In practice, we recommend selecting a spacing factor that reflects the zoom ranges you want to optimize for rather than assuming that every intermediate 2x level is necessary. The real decision is where to start the hierarchy and how quickly to step down: whether to include the (potentially) expensive 2x level, or begin at 4x and rely on the fact that coarser levels are extremely cheap and still highly effective for visualization. This framing makes the design space simpler: most of the pyramid costs very little, and most of the visual benefit comes from having some coarsening rather than every intermediate step.

Generating overviews at scale in Icechunk

There are already some great open-source tools for generating multiscale pyramids from Zarr datasets. CarbonPlan’s ndpyramid and topozarr are both solid options; topozarr even writes GeoZarr-compliant metadata out of the box!

For many use cases, these tools are enough. But once you start working with very large datasets, or you want more control over how your pyramid is structured, some limitations show up pretty quickly.

For example, topozarr always coarsens by a factor of 2 between levels, which makes it impossible to build something like a sparse 4x pyramid. ndpyramid is more flexible in terms of choosing factors, but both tools ultimately give you in-memory (or Dask-backed) outputs, leaving you to handle persistence yourself. More importantly, neither is really designed around distributed write patterns for cloud-native stores like Icechunk.

That starts to matter when you’re dealing with Earth observation-scale data with terabytes of inputs spread across thousands of spatial tiles. At that point, you need a workflow that can scale out computation, write safely in parallel, recover from failures, and avoid doing unnecessary work on empty or sparse regions.

A reference workflow for large datasets

We’ve put together a reference implementation in https://github.com/earth-mover/icechunk-multiscales-demo that shows what generating overviews at scale can look like in practice.

The core computation uses xarray’s coarsen() method to downsample tiles, but the workflow around it is designed for distributed, incremental processing against Icechunk.

Instead of treating overview generation as a single batch job, this workflow breaks it into a set of scalable patterns:

• Flexible pyramid structure: You’re not locked into 2x steps; you can generate 4x, 16x, 64x levels depending on what actually makes sense for your data and your visualization needs.
• Two-stage build process: First, the full spatial structure is initialized. Then only tiles that contain data are filled in. This is especially useful for sparse datasets.
• Parallel writes with fork/merge: Each worker writes independently to its own forked Icechunk session, and those forks are merged in batches. This avoids write conflicts while still scaling horizontally.
• Resumable execution: If something fails halfway through, you don’t start over, you resume from where you left off.
• Pyramids built from pyramids: Higher-level overviews can be derived from lower-resolution ones (for example, 16x built from 4x), which reduces how much data you need to read.
  • Caveat: This only applies for composable resampling methods (mean, min, max, etc.) Non-composable resampling methods (median, mode, nearest neighbor) should always use the native data for resampling.

Along the way, the workflow writes GeoZarr metadata at every level, enabling tools like xpublish-tiles to automatically discover what overviews exist and choose the right one for any given zoom level.

The Icechunk storage engine is also an important part of our approach. Icechunk enables transactional updates and strict serializable version history for Zarr stores. This is important when generating overviews, to ensure consistency between all of the zoom levels. With icechunk, when the native resolution data change, you can also update the overviews, and bundle all these changes into a single transaction—users only ever see consistent data.

Adapt it to your needs

Because every dataset and workflow is different, this is not meant to be a one-size-fits-all tool. It is a reference implementation for a set of patterns you can build from.

The demo repository is intentionally structured to make those patterns explicit: two-stage initialization, distributed tile-based processing, safe parallel writes via fork/merge, batched commits, and GeoZarr-aware outputs. It uses Coiled as a compute backend, but the same approach carries over to Dask, Modal, or any othernd distributed execution framework.

Read through the code to understand the workflow, then adapt it to your own dataset and infrastructure. You can use it as-is, or treat it as a starting template: swap in your preferred compute backend, adjust the coarsening strategy, or reshape it around your data model and scaling requirements. And if it helps, throw it at your favorite AI agent and iterate from there.

Once it’s running, the important part is already handled: GeoZarr-compliant metadata is written alongside each overview, and xpublish-tiles can use this metadata and overviews to select the appropriate resolution for each zoom level when visualizing your data with Flux.

Putting it all together: A Sentinel-2 example

To validate the approach, we generated overviews for a large Sentinel-2-derived dataset stored in Icechunk.

The dataset spans the continental United States (CONUS) at ~90m resolution — roughly 70,800 × 31,200 pixels, or over 2 billion pixels per time step — across 51 monthly time steps (January 2020–March 2024). From Sentinel-2’s 11 spectral bands, we computed 229 spectral indices along with an RGB median composite, resulting in 230 data variables per pixel. Altogether, that’s roughly 19 trillion data points.

Using the workflow described above, we generated three overview levels (4x, 16x, and 64x) alongside the native-resolution data. Overview generation was performed in parallel across tiles, variables, and time steps using Icechunk fork/merge workflows and batched commits. Each overview level was written with GeoZarr-compliant metadata.

The figure below compares the native-resolution Normalized Difference Vegetation Index (NDVI) with the 4x, 16x, and 64x overview levels generated for our Sentinel-2 dataset at time step 01-01-2020:

At full-scene scale, the 4x and 16x overviews preserve nearly all of the visual structure of the native data while containing dramatically fewer pixels. Even the 64x overview retains the large-scale patterns needed for low-zoom visualization. The zoomed panels highlight the tradeoff: as resolution decreases, fine-scale detail is smoothed away, but the information needed for visualization remains

From overviews to interactive visualization

Once these overviews are present, Flux can use them automatically.

For each tile request, xpublish-tiles examines the available GeoZarr multiscale levels and selects the coarsest overview whose resolution is still sufficient for the requested zoom level. As users zoom in, Flux progressively transitions from the 64x overview to 16x, then 4x, and eventually to the native-resolution data.

The result is a dramatically better visualization experience: lower memory usage, fewer chunks read per request, faster tile generation, and smooth exploration of datasets that would otherwise be impractical to render at global scale.

Conclusion

What emerges is a simple pattern for making large geospatial datasets explorable at any scale: generate multiscale overviews, describe them using GeoZarr conventions, and let visualization systems automatically choose the right representation for the task at hand.

Flux brings that final piece to life. Instead of forcing users to think about resolutions, zoom thresholds, or pyramid levels, it continuously selects the most appropriate data as they move from a global view down to native detail. The result is a visualization experience that feels effortless, even when backed by datasets containing billions or trillions of values.

The full reference implementation is available in the Icechunk multiscales demo repo: https://github.com/earth-mover/icechunk-multiscales-demo.

Acknowledgements

This work builds on the contributions of many in the open-source community. We’re grateful to Max Jones and Emmanuel Mathot for their work advancing the GeoZarr standard, and to the authors of ndpyramid and topozarr: Raphael Hagen, Anderson Banihirwe, Max Jones, and Joe Hamman.

Footnotes

1. The multiscales convention originates from bioimaging workflows (see: OME-NGFF image spec), but it is domain-agnostic and can be used for multi-resolution representations of gridded geospatial data too. ↩