Development Diary #7

Much of our effort in late 2021 was spent hardening XTDB. As more and more companies are deploying XTDB into production, we’ve been given an increasing number of opportunities to squash edge-case bugs and solidify XTDB’s behaviours.

1.18.1 predominantly brought Lucene improvements. We added a :refresh-frequency parameter, similar to Elastic’s refresh_interval setting, and improved Lucene’s performance. We also improved custom indexes by including the original transaction operations in the callback.

1.19.0 was, in some ways, our biggest release yet (in terms of raw lines of code!) — we renamed the database to XTDB. We’ll speak about this more in its own section.

1.20.0 was an hardening release purely for Lucene’s benefit, ensuring that it can cope with absent documents and be restored from a checkpoint. We also changed the pull behaviour to return nil on empty joins.

1.21.0 was a much bigger hardening release with a few long-awaited new features. TPC-H performance improved, caches were stabilized with better hash distribution, Clojure was bumped to 1.11, many small edge case errors were corrected, and a number of behaviours were made more consistent: pull-many, failed match, Lucene :timeout, auto-incrementing identifiers in Postgres and MySQL, and more.

In 1.21.0, three big new features landed. First, :find clauses now support simple Clojure expressions in projections and aggregations. Second, you can now import Transaction Time from an upstream source — a huge win for temporal power users. Last, but certainly not least, users on Apple M1 chips now have fast, native ARM build support for RocksDB.

XTDB hasn’t always been XTDB. Most of our users remember a time when it was a little database lovingly known as Crux. Crux grew out of a very real bitemporal data processing problem which surfaced in a JUXT project for a Tier-1 bank. The vision and technical scope of XTDB has continued to evolve ever since, and in the meantime performance has improved and APIs have stabilized.

By the time XTDB found its final name, it was already a very solid product used in many organizations — most of which we don’t know! Unless someone has a commercial contract with us, or reaches out to us via Discuss / Twitter / etc, we’re none the wiser. The beauty of open source.

XTDB isn’t just a new name, though. With it comes quite a grand vision for the future. We feel — now more than ever — that immutability and time play a fundamental role in every system we have ever built, repaired, or maintained. XTDB is a fun play on the JUXT logo, but we’re also quite serious when we describe it as the "Cross Time Database".

XTDB originated from the domain of End-of-Day Risk calculations, so it’s no surprise that we hear about users applying it to financial systems quite frequently. Trading firms are using XTDB as a "tactical bitemporal database" across a variety of grey box trading, instrument, and analytical services. Quant firms are using XTDB as an HTAP database for algo research. Big banks use XTDB for risk, auditing, and compliance.

But we’re surprised every week by the industries putting XTDB into production. In the education sector, it’s used to track teachers and class loads. In workforce management, it’s used as a flexible graph database that allows users to model data themselves. The list goes on.

Many of these companies are household names. Unfortunately, just as we hand-wave over "Tier-1 banks", we are often forced to hand-wave over these logos. Over the coming year, as more companies sign up for our Partner Program and Commercial Support, we look forward to sharing their stories with you.

HYTRADBOI was an information-packed virtual conference at the end of April. It consisted of 10-minute pre-recorded talks woven together using only open source technologies. The conference was held on a self-hosted Matrix server and the afterparty was hosted on Jitsi. I was fortunate enough to give a talk: Baking in time at the bottom of the database, a technical background and justification for our creation of XTDB.

It was a great opportunity to hear about Martin Kleppmann’s latest work on CRDTs and I thought the Ultorg demo was particularly impressive. Huge thanks to Jamie Brandon for putting on the event!

If you enjoyed that talk, I encourage you to come to my workshop in September at Strange Loop 2022! I’ll be taking participants through real-world bitemporal data challenges with and without XTDB.

From the moment we started pursuing XTDB as a commercial product, the team has spent hundreds of hours interviewing users, partners, customers, and advisors. We’ve asked everyone to look at XTDB, retrospectively, and tell us their honest opinion about data problems their company faces. The results are consistent.

First, no one wants to babysit a database in 2022. A large part of XTDB’s current appeal is how fast and easy it is to get started: add a dependency on com.xtdb/xtdb-core and start adding data. Bam, done. Everyone we spoke to wants this level of operational simplicity — but across all languages and platforms.

The second problem everyone wants solved is equally unsurprising: easy, immutable data. Whether your business thinks of it as "lineage", "provenance", or "perfect audits", absolutely everyone is tired of working around mutable databases (AKA "the queryable cache") with event sourcing. Event sourcing should be for, well, events — not a bandaid on the fact that almost every major database throws out data with each UPDATE operation (or makes the old record unqueryable).

From there, everyone has an itch to scratch. CTOs are very interested in solving the "two database problem" with Hybrid Transactional-Analytical Processing (HTAP), and they don’t want to maintain an ETL (or ELT) pipeline anymore. Developers want real time-traveling queries and versioning for free in their OLTP systems, they don’t want to shoehorn their realtime data into an OLAP system anymore. The DDD folks want nested records, they don’t want to cram strings into Postgres JSON columns anymore. Data scientists want first-class SQL, they don’t want to learn a new query language.

We’ve been listening.

…​but we haven’t only been listening. We’ve also been researching and building. It’s our goal to take what everyone currently loves about XTDB (in the words of Jacob O’Bryant, "the database that sparks joy") and grow it. Much of that growth is interlinked. We think of these interlinked qualities as the pillars of XTDB.

Håkan will accompany me to Strange Loop in September. He’ll be giving a talk on "Light and adaptive indexing for immutable databases" where he’ll explain the past two years of research and how it affects XTDB’s new design.

Here are some cool things users have been sharing recently:

XTDB wouldn’t exist without its high-energy community. Thank you to everyone who shares in XTDB-related excitement with us across the globe. We can’t wait to share the updates of the coming months!

As always, do get in touch with the team if you have any questions, issues, or feedback.