Here’s my take. One of our models at Dira GeoSystems takes an aerial image and returns polygons. Testing that the service still returns the right polygons after a code change or a library update was more challenging than I thought.

The first idea was a snapshot test: I thought I could just run a few fixed tiles, commit the output, and let tests fail if anything changes. I found out quickly that that doesn’t even work on CPU (yes, CPU, not GPU: our CI runners don’t have GPUs). Floating point details differ between machines, so the same code gives slightly different coordinates on my laptop and on the GitHub runner.

So I realized that the simplest possible test probably would just be to compare polygon by polygon. The number of polygons has to match exactly, and every expected polygon needs its own counterpart with an IoU of at least 0.999. Strict enough to catch a real change, loose enough to survive a different CPU.

On GPU, even that fails. We run inference in mixed precision: on the GPU, most of the math runs in 16-bit floats instead of 32-bit, which is faster but only keeps about three significant digits. Each GPU generation uses its own optimised kernels, which add numbers up in a different order, so the results differ slightly. Same model, same tile: an RTX 5090 and an RTX 6000 Ada gave a merged-area IoU between 0.988 and 0.9995, and one polygon more or less. So depending on the card, you get a different number of polygons.

That’s why the GPU test compares the total area instead: IoU of at least 0.98, and the count may differ by two.

A few more things I ran into:

• One fixture is a JPEG-compressed tile. JPEG decoding depends on the GDAL version, so a library update can change the input pixels before the model even sees them.
• The GPU test is expensive (build the image, start a pod, run the tiles), so it doesn’t run on every change (more on that in a future post).
• When the strict CPU test fails after an update, it’s usually a tiny shift. If the benchmark moves by around 0.1 percentage points, I accept it and recapture the fixtures. No big analysis.

The fixtures have one more simple but very important job. A lot of the service code is written by AI assistants, and the fixtures are the one thing the assistant is not allowed to change. Only a script regenerates them, and only when I decide the output should change.

If you do it differently, I’d like to hear how.

My argument is simple: precisely because this is all very early and nobody runs these at scale yet, the temptation to cut corners is strongest.

Why do I say that?

Because in the last couple of years I built roughly 50 prototypes and the main lesson is that it takes much more time to redesign a rocket ship when you originally blueprinted a pair of rollerblades. Usually the rollerblades were already sold as a rocket ship, gold paint and two bottle rockets duct taped on.

I am still a huge fan of YAGNI. But avoiding premature optimization is not the same as ignoring architectural integrity. You do not need to build every future feature now. But you do want to avoid today’s prototype assumptions that make tomorrow incredibly expensive to change.

Three decisions that keep our inference pipeline clean:

• Clients ask for a target, like “buildings”, never a model version. Internally that maps to an adapter, a checkpoint, and a queue. Swap the model, and the client contract does not move.
• The orchestrator cannot see the models. It knows tiles, batches, and progress, nothing about neural networks. CI checks read the syntax tree and block model imports from leaking past the adapter layer.
• Loading fails loud on a version mismatch. Every checkpoint carries its provenance. The adapter checks it against the installed package and refuses incompatible weights, so a silent mismatch never gets blamed on the model.

The honest constraint: today the API hands the worker a local file path, so both need the same filesystem. Moving to shared object storage is the main thing standing between us and workers we can distribute freely. But that change sits at the infrastructure boundary. It does not touch the client contract or the core pipeline.

My opinion: good architectural boundaries make scale far less painful. But scale is not a free byproduct of good architecture.

Here’s a situation I hit a lot. I pull in some repo I want to try, maybe from a paper. The authors published and moved on, so the repo is stale and the pinned versions (if they even pinned) are ancient. So I bump torch, numpy, python and see what breaks. You’re stuck guessing at a version combo that works, and the trap is that a wrong guess still builds green.

Because if you’re on a newer card, say a Blackwell 50-series, you can hit something nastier.

You run torch.cuda.is_available(), it returns True, everything looks fine. Then you actually run something and the process dies with:

no kernel image is available for execution on the device

What’s going on: PyTorch wheels ship precompiled kernels for a fixed set of GPU architectures. If you pin an old torch and run it on a newer architecture (sm_120 for Blackwell), the wheel just doesn’t contain kernels for your card. It still builds. is_available() still says True. There’s nothing to actually launch, so every CUDA op fails at runtime. PyTorch only warns about this with a UserWarning, which is easy to miss in logs. Worst case you don’t find out until the container hits the GPU server and the job errors out.

I contributed a PR to cog that catches this early. It’s unglamorous infra work, but the kind that saves time and money: cog doctor now reads your GPU’s compute capability, figures out the exact torch wheel cog will install, and compares that against a compatibility table. If they don’t line up, you get a clear warning before you build instead of after you’ve shipped.

It doesn’t fix the incompatibility. It just points it out early so you can react to it. For me, working somewhere without a big infra team where a lot of this is still manual, that’s worth a lot.

Hope this helps some fellow ML engineers in the same boat. Thanks to the cog reviewer for a thorough pass.

Reproducibility.

If you work with Python, you know that Python makes reproducibility tough in a lot of cases. When you use Python in Bazel, however, you might know that Bazel is one of the few ecosystems that treats the CPython interpreter itself as a pinned build input rather than whatever is on the host machine. But even there, the Python version typically lives in three places: pyproject.toml, the toolchain registration, and pip.parse. Nothing keeps them in sync and you have to handle these three versions manually, which is annoying and leads to errors.

But your days of endless crying are over.

With the next rules_python release, my recently merged PR ships. Point the build at your pyproject.toml and the requires-python field becomes the single source of truth. The interpreter Bazel fetches, the registered toolchain, and the pip resolution all follow that one file. No need for python_version in BUILD files or .python-version files in your project root. A wrong Python version in the build is no longer a state the system can reach.

The details: the toolchain is auto-registered, pyproject.toml is parsed with a pure-Starlark TOML decoder (reading the version needs no host Python), and the file is watched. Which makes automated upgrades trivial: e.g. Renovate just bumps requires-python, and that single-line PR upgrades interpreter, toolchain and resolved deps together.

By design it expects exact version pinning (==X.Y.Z). That fits application repos where you want exact, reproducible and hermetic builds (as opposed to published packages where you might be more lenient with version ranges) and is what makes the automation clean.

I know this all sounds incredibly boring. But in my opinion boring and reliable wins over shiny and unreliable.