Two small computers beat one big one: a fair local-LLM A/B test

Two small computers run the language models in my house: two ASUS GX10 units (the retail version of NVIDIA’s DGX Spark platform) wired together into one cluster. Bigger machines keep coming out — the new Mac Studio Ultra, for instance — and the tempting assumption is that bigger is simply better. So when a new model lands, I don’t argue about it. I test it.

Over the past months I’ve run several of these bake-offs, and the method matters more than any single result. Three rules turned out to be the difference between a useful test and a wasted week.

Rule 1: same reasoning effort, no token ceiling

A fair comparison gives both models the same thinking budget. I run both sides at maximum reasoning and let each finish naturally — no token caps. The only hard limit is an outer time limit with a clear definition of what “failed” means before the test starts.

This matters because models differ wildly in how much they think per task. A model that reasons twice as long isn’t worse; it’s different. Cutting tokens to make the race fair actually makes it unfair.

Rule 2: full names, explained per task

“Model A beat Model B” is a useless conclusion. The useful form is: which models, on which task, with what explanation — Danish prose transcription, tool calling, long-context retrieval — and where each fell over. On my Danish-language tests, the differences between models that score equally on English benchmarks become enormous. One model that leads on English benchmarks loses a Danish transcription test by a 3× error-rate margin to a model three years older. Benchmarks don’t capture that; a local test does.

Rule 3: the control experiment is the real test

This is the one that changed how I work. During one evening test, a configuration change measured +10% on code generation — a solid-looking win. Before accepting it, I re-ran the unchanged baseline configuration as a control. It came back at half its normal performance.

Nothing had changed in the model. What had changed: another process on the network was quietly issuing large requests to the same server the whole evening. My “win” was measuring someone else’s traffic, not the configuration.

So now every test runs a guard first: check what else is using the model, run a control experiment on a known configuration, and only if the control reproduces the known numbers do the actual test arms count. Half my tests would have shipped wrong conclusions without this. If you benchmark anything shared — a model server, a database, a network link — the control experiment isn’t optional rigor. It’s the only part that tells you whether you measured your change or measured the world.

What the tests have decided

The small cluster has held its ground every time: it’s stayed the quality baseline while newer, flashier models came and went, and the workflow decisions (which model for prose, which for code, which for vision) now come from measurements instead of release notes. When something new arrives, the nightly harness runs it while the house sleeps — guarded, with a control experiment, and with a rule I learned the hard way: if the baseline doesn’t reproduce, the problem is the environment, not the model.

That’s the whole trick, really. Don’t test harder. Test so you can tell the difference between your change and everything else.