Monday, 26 February, 2024
Development Diary #10 - v2 primary index
James Henderson Welcome to our second dev diary of 2024!
In our previous dev diary we announced a preview of the Java and Kotlin API for our upcoming major release of XTDB v2 ('XT2'). This month, we’ve been busy working behind the scenes on XT2’s primary index and benchmarking, as well as responding to initial user feedback.
Thanks to everyone who’s gotten in touch so far - and please do keep your thoughts and ideas coming!
Indexing in XT2
In XT 1.x, every node kept a local copy of the indices it needed to serve queries, using an on-disk K/V store (e.g. RocksDB, LMDB).
In 2.x, we’re relaxing this constraint: it’s no longer a requirement that every node has a full copy of all of your data. Each node keeps a cache of the data it needs to serve the queries asked of it, and data that isn’t queried as frequently (often historical data) can be relegated to significantly cheaper object storage. (Naturally, it helps if each node can cache as much as possible locally - they key difference is that it’s no longer a hard requirement.) If you’ve seen previous XT2 blogs, we’ve been referring to this as 'separating storage from compute'.
We do this using log-structured merge trees ('LSM' trees) - when transactions arrive on the transaction log, these are written as quickly as possible to the lowest 'level' of the tree ('level 0', or 'L0'). Nodes in the cluster then collaborate to compact these L0 log files into more efficient sorted indices, in deeper levels of the tree (L1, L2 etc). Every file contains serialized Apache Arrow columnar data.
XT differs slightly from typical LSM tree implementations in that, during compaction, standard implementations will remove superseded versions of your data to save space. In XT, we don’t ever throw data away [1] - these old versions become part of the history of your database, to be queried whenever you need to go back in time.
Safe to say, there’s plenty more we could talk about on this subject (usually, to anyone who’ll listen!) - keep your eyes peeled on our usual channels for future blog posts, demos and talks.
Indexing version bump (breaking change)
To implement this LSM tree, we’ve introduced a new index version, which will require an explicit maintenance task for anyone with an existing XT2 cluster that they wish to preserve.
-
If you have an infinite (or effectively infinite) retention tx-log - e.g. standalone, or a Kafka log with suitably long compaction time that no events have yet been compacted - your XT nodes will automatically and safely migrate your object store when they’re upgraded.
-
We are not currently aware of any live XT2 instances where this approach is not sufficient. While XT2 is still in early access/alpha, there may be other such changes before a stable release - during this period, we’ll assume that the majority of users won’t yet be putting live, business-critical data into XT2.
More importantly, we’ll assume that anyone who does have a live XT2 instance containing business-critical data is already in regular contact with the XTDB team.
-
This change will create a
/v1directory in the root of your object-store - once the new nodes have stabilised, any other files within the object-store, outside of this directory, can be safely removed. -
As always, if you have any issues, problems or questions, please let us know through the usual channels!
Elsewhere in XT2
We’ve been busy fixing bugs, simplifying functionality and documentation, and adding to our standard library.
For example, we’ve already had quite a few people asking us for date_trunc, to truncate a timestamp to the start of the day/month/year etc - not a SQL-standard function, but apparently a popular addition!
Some keen early users of v2 have already been experimenting with building their client libraries, e.g. for Python and Common Lisp - a good opportunity to ask: which language(s) would you like to see official support for?
Get in touch!
As always, we’d love to hear from you - we’ve already had many insightful conversations with people who’ve given XT2 a spin which have influenced our roadmap, and it’d be great to get your thoughts too:
-
Discuss XTDB (public forums)
Similarly, if you’d like a demo of XT2, or to be kept informed about its progress, come say hello at hello@xtdb.com.
Cheers,
James