Close to the metal, close to the wire

At XT26 last month, we swapped our editors/IDEs AI coding agent TUIs for a day of talks - mostly AI experience reports, as our industry attempts to find some stability in a sea of change.

As a database engineer, though, my favourite talk of the day was naturally the one that had nothing to do with AI.

Tom Dellmann, Head of Engineering at Chronicle Software, gave a talk called 'Close to the metal', on the journey from idiomatic Java all the way down to the kind of low-latency Java that trades in double-digit microseconds - sometimes single-digit, and all of it in software.

We’re an HTAP database, not a matching engine - safe to say microseconds aren’t quite the eternity for us that they are for Tom. But he’s a generous guide to a world most of us never get to see, and one layer of his talk landed very close to home indeed.

The lie of the average

Tom started where he says he always starts with a client: measurement.

He made a very salient point straight away: it’s easy to look at your averages, pronounce your performance "pretty good", and completely miss that the tail is falling apart at the higher percentiles. The whole latency histogram is the thing to watch, not the middle of it.

From there he took an order-book matching engine and worked it down, layer by layer: serialisation, garbage collection, data-structure choice, and finally the operating system and kernel themselves. The layer I want to dwell on - because it’s where XTDB has quietly made exactly the same bet - is serialisation.

The cost of turning objects into bytes

Any time you move data between two processes you have to turn your in-memory objects into bytes, push them over the wire, and turn them back again at the far end. That serialise/deserialise round-trip is pure overhead, and when your whole latency budget is a microsecond, it can quietly eat a big chunk of it.

Tom walked through a spectrum of approaches, each trading usability for speed:

a 'self-describing' format, which writes type and field information onto the wire - lovely and readable, you don’t even need the schema to read it back, but you pay for carting all that metadata around.
a 'user-controlled layout', where you decide exactly which bytes land at which offset - faster, but read the message back even slightly differently from how you wrote it and you’re in trouble.
and, at the far end, 'trivially copyable' - where, instead of walking the fields one by one, you just memcpy the object’s bytes straight out and straight back in again.

That last one is the quick one. There’s barely any translation to do, because the bytes on the wire already look almost exactly like the bytes in memory.^[1]

The same bet, a different shape

We make the same bet at the very heart of XTDB.

XTDB’s storage and query engine is built on Apache Arrow, and one of the reasons we reached for Arrow in the first place is precisely Tom’s point: Arrow’s in-memory format and its on-the-wire format are, by design, very nearly the same thing. So when XTDB pulls a block of data off object storage, or hands a batch of rows from one query operator to the next, there’s very little 'deserialisation' left to do - the bytes arrive already in the shape we want to compute over.

The one real difference is the shape. Tom’s matching engine is dealing in rows - a whole order as one contiguous struct, copied out in a single block. XTDB is columnar - we’re laying out and copying whole columns at a time rather than whole rows. But apart from that, we’ve got an awful lot in common: in both worlds the win comes from making the serialised form and the in-memory form look as alike as you possibly can, so there’s nothing left to translate.^[2]

A few other things we nodded along to

Tom spends a lot of the talk on the garbage collector, because on the hot path allocation is the enemy - every object you create is something the collector has to come and clean up, and those pauses show up right where it hurts: in the tail.

We don’t chase zero-GC across a whole query the way Tom does across a trade, but we’re just as allergic to allocating per row: wherever we can, we allocate once for the query as a whole, and where we really can’t, once per batch of ~1,000 rows - never once for every row flowing through. It falls out naturally from the columnar, batch-at-a-time way Arrow wants you to work anyway.

Arrow’s Java library helps us out here too: all of its allocations are off heap and manually-managed, so we typically see very little GC impact in XTDB.

He also made a point I completely agree with - that, where you can, you should try not to write multi-threaded programs at all. A lot of XTDB is deliberately single-threaded for exactly his reasons: we ingest predominantly on a single thread, so there are no locks and vanishingly few atomic references near the hot path, and where we genuinely do share a structure between threads we reach for optimised 'persistent' (immutable) data structures, so readers and writers never contend.

Not the sort of thing that makes for a demo

None of this comes for free, of course. Tom’s refrain all the way through was "it depends", and he’s right - my team will probably tell you this is the most common answer I ever give to their questions. Keeping data off-heap and copy-friendly is memory you have to manage yourself, and, as he put it, "there are lots of ways to get into trouble with that". We know - we’ve spent a fair proportion of our lives with our heads buried in the profiler chasing exactly those sorts of troubles.

But it’s a trade we’d make again every time. For a database whose day job is shuttling large volumes of data between memory, storage and the wire, making those three representations look as alike as possible is a genuinely big win - not the sort of thing that makes for an exciting demo, but the sort you’re grateful for on every single query.

Towards the end, Tom showed a flame graph of an application with the garbage collector finally gone quiet, and admitted it made him "unreasonably happy, which is a bit sad, really".

I laughed - because, well, guilty as charged. 🤓

Please do go and watch the talk when the XT26 recordings drop - it’s a lovely tour of a corner of our craft most of us never visit. And thank you, Tom, for coming to talk to us - it clearly struck a chord, and it certainly did with me.

Until next time!

James

1. Tom told a lovely war story about the sharp edge here - a 'trivially copyable' object serialised on one machine came back garbled on another, because the JVM quietly changed its object-layout algorithm between JDK 11 and 17. The fast path has teeth!

2. Close enough to home, in fact, that I’ve spent a good chunk of the last week trying to shave the overhead of writing Arrow’s schema out ahead of the data - it turns out even the little envelope around the buffers is worth sweating when you’re doing it often enough. 😅