Cursed venvs, Confident Releases: Testing Icechunk Across Major Versions

tl;dr: Technical details of how we do surgery on python wheels in order to do cross version compatibility testing.

We’re getting ready to release Icechunk V2 — the next evolution of our tensor storage engine. People run real workloads on Icechunk V1. They’re not all going to upgrade on the same day. So before we ship V2, we need to be absolutely certain that both versions can coexist reading and writing to V1 repos.

For this kind of testing we love Hypothesis stateful tests. They’re great at pushing libraries past the happy path and catching weird bugs like this one.

But unfortunately there’s not an easy way to coordinate a stateful test with two different Icechunk versions. The Hypothesis stateful test orchestrator runs in-process — it doesn’t spawn and control subprocesses. So we needed both versions of Icechunk importable in the same Python environment, and that’s not something any package manager will let you do.

Try putting this in your pyproject.toml:

icechunk < 2
icechunk >= 2

Dependency solvers like uv will (correctly!) tell you no. Your venv only gets one icechunk.

So there’s no way to do this… right?

Enter third-wheel: an intentional man-in-the-middle attack on your packages

That’s why we made third-wheel — a tool that performs an intentional man-in-the-middle attack on your wheel installation to let you install two (or more!) versions of any library into a single environment. You’d never ship the resulting venv to production, but for testing it’s exactly what we needed. It can either run as a proxy PyPI server that renames packages on the fly, or download and patch wheel files directly.

Here’s what it looks like in practice. First, download icechunk v1 and rename it to icechunk_v1:

third-wheel download icechunk --version "<2" --rename icechunk_v1 -o ./wheels/

🛞 Downloaded: wheels/icechunk-1.1.19-cp312-cp312-macosx_11_0_arm64.whl
🛞 Renamed: wheels/icechunk_v1-1.1.19-cp312-cp312-macosx_11_0_arm64.whl

Then download icechunk v2 normally:

third-wheel download icechunk --version "==2.0.0a0" -o ./wheels/

🛞 Downloaded: wheels/icechunk-2.0.0a0-cp312-cp312-macosx_11_0_arm64.whl

Now install both into the same venv:

uv pip install wheels/icechunk_v1-1.1.19-*.whl wheels/icechunk-2.0.0a0-*.whl

Installed 10 packages in 172ms
 + icechunk==2.0.0a0
 + icechunk-v1==1.1.19
 ...

Two versions of icechunk. One venv. Not something you’d see every day.

What did we actually do to that wheel?

You might be wondering: what about the compiled extensions? The internal imports?

Wheel files are basically just zip files, so you can crack them open and change whatever you want. third-wheel doesn’t just rename the file — it opens the wheel up and rewrites it from the inside out. Here’s what that looks like in practice.

Every file path inside the wheel gets rewritten:

- icechunk/__init__.py
- icechunk/_icechunk_python.cpython-312-darwin.so
- icechunk/repository.py
- icechunk/session.py
- icechunk/store.py
- icechunk-1.1.19.dist-info/METADATA
+ icechunk_v1/__init__.py
+ icechunk_v1/_icechunk_python.cpython-312-darwin.so
+ icechunk_v1/repository.py
+ icechunk_v1/session.py
+ icechunk_v1/store.py
+ icechunk_v1-1.1.19.dist-info/METADATA

All internal imports are rewritten to match:

 # icechunk_v1/store.py
-from icechunk._icechunk_python import PyStore, VirtualChunkSpec
+from icechunk_v1._icechunk_python import PyStore, VirtualChunkSpec
 # icechunk_v1/session.py
-from icechunk import (
+from icechunk_v1 import (
     ChunkType,
     ConflictSolver,
     Diff,
     RepositoryConfig,
 )
-from icechunk._icechunk_python import PySession
-from icechunk.store import IcechunkStore
+from icechunk_v1._icechunk_python import PySession
+from icechunk_v1.store import IcechunkStore

Even the package metadata and checksums are regenerated:

 # METADATA
 Metadata-Version: 2.4
-Name: icechunk
+Name: icechunk_v1
 Version: 1.1.19

 # RECORD
-icechunk/__init__.py,sha256=P4esqOVN-jLnCKY6LEIYgbR5fOW7M2h6wnAViUNt3jg,5482
-icechunk/repository.py,sha256=Ijx_z3AWMwaNjwh52CpndSX8ojExV0Vt2nriBgCjLYw,53189
+icechunk_v1/__init__.py,sha256=pwn2nLVxhCeSE0AGKIXSsGvRiK2-yYgQO6zWaKo1_FM,5500
+icechunk_v1/repository.py,sha256=a6xkwvyRy0tCpdLdrU-DHeHsUFW2m2j1NjoRniJZF1w,53204

And what about conflicting dependencies? third-wheel can rename those too. Both versions of icechunk depend on zarr, but we need each version pinned to its own zarr — so we can have zarr_v1 and zarr living side by side in the same environment.

The payoff: two icechunks, one interpreter

With both versions installed, the code we actually wanted to write just works:

import icechunk_v1 as ic1
import icechunk as ic2
print(ic1.__version__, ic2.__version__)

icechunk_v1 version: 1.1.19
icechunk    version: 2.0.0a0

This is what unlocks proper Hypothesis stateful testing across major versions. We can create a repo with v1, read it with v2, write with v2, read with v1, and throw randomized sequences of operations with our two versions until we’re confident nothing breaks.

Why bother?

Icechunk has an on-disk format, so version compatibility isn’t optional. In fact it is one of our stated guarantees that all future versions of Icechunk will be able to read any previously written data.

third-wheel let us build the test suite we actually needed for Icechunk, and now we’re confident that the move to Icechunk V2 won’t break our promise and ruin anyone’s day. Icechunk 2 is available now in alpha — give it a try.

Check out third-wheel on GitHub if you’ve ever needed to test across library versions — or if you just want to see how the madness works.