Development Diary #13 - Beta Release

We are very pleased to announce that we consider XTDB v2 ('XT2') ready for end-to-end validation and deployment with our Design Partners, alongside our first 2.0.0-beta1 release!

Nearly 18 months ago we announced Early Access as a means of guiding development on the new 2.x architecture and API surface area. Since then we have been working with key Design Partners to define the MVP feature set and validate the essential performance characteristics. Huge thanks to everyone who has been involved and given us invaluable feedback to get to this point, including our many 1.x users.

With the new beta1 release available we can now focus on the operational aspects of XT2 with our Design Partners, who we are working with closely to deploy across AWS, Azure, and Google Cloud. General Availability for XT2 will be announced in due course once we are satisfied with the stability of these deployments. If you’re interested to get involved with helping us on the journey to General Availibility, it’s not too late!

We have also taken the opportunity to refresh the SQL Quickstart documentation with interactive SQL widgets and instructions for use with Postgres' psql CLI - feedback is welcome.

Big things ahead…

We are planning to 'launch' XT2 more publicly over the coming weeks, alongside a deluge of upcoming written and visual content. This launch timeline also factors in our upcoming presence at two important conferences:

Speaking at Heart of Clojure 2024, Belgium

This is the second instalment of the conference, and it will be a wonderful opportunity for the Clojure community to connect again in Europe since the last time in 2019.

Jeremy Taylor will be giving a talk on "Richer SQL". JUXT was originally founded over 11 years ago purely because of Clojure, the modern JVM-based Lisp that Rich Hickey gave to the world, and as a collective of Clojure-loving engineers we have been applying Rich’s ideas and perspectives to the world of software ever since. Not only is XTDB largely written in Clojure, but the entire design and feature set essentially carries forward many of Rich’s ideals to a mainstream SQL audience. This talk will explore the 'epochal time model', consistency guarantees, and the nature of evolving data in business systems.

Speaking at Open Source in Finance Forum (OSFF), NYC, by FINOS

Some of the team attended the The Linux Foundation’s OSFF event in London earlier this summer and it became clear that FINOS was a very relevant community of engineers and business leaders for us to get more acquainted with.

At the NYC OSFF Jeremy will present on "Reconciliation, Risk, ML and Other Use Cases for Bitemporal Systems" - an overview of how IT systems in financial services traditionally cope with the problems that the bitemporal model solves.

We look forward to meeting many new faces (though hopefully some familiar ones also) soon!

Release Notes

For good measure, we have included the release notes from the Github release below…

Our main aims of XTDB 2.x have been:

Hybrid transactional/analytical processing (HTAP).
1. XT1 is a great transactional database. In XT2, we’ve improved the support for OLAP queries, to reduce the need for users to write and maintain ETL processes to a separate OLAP system. As part of this, data in XT is now all stored in the open Apache Arrow format, and we now have a shiny new dynamic, columnar query engine.
2. We’ve also decoupled data storage from compute nodes, so that the two can be scaled independently - more details in the architecture discussion below.
Across-time bitemporal queries
1. XT1 is great at point-in-time queries, and can also return the history for a single entity; XT2 additionally supports fully general bitemporal queries - including across-time aggregations and temporal joins.
Addition of first-class SQL:2011 support
1. While we still love EDN Datalog, it’s hard to deny the position of SQL:2011 in our industry - the ecosystem, knowledge and tooling is unrivalled.
2. The SQL:2011 standard introduced a specification for bitemporality in tables and queries, which we’ve implemented.
3. In bringing in SQL, one of the major changes in XT2 is that your documents all now live in specific, but dynamic 'tables'. In keeping with XT’s dynamic nature, these tables are best thought of as bags of related documents - same as you would in a SQL database, but there’s no requirement for up-front schema, and no requirement for all of the documents in a table to have the same column types, or even the same columns.
4. We’ve also stayed true to our data-oriented query roots, introducing 'XTQL' - a composable, data-oriented query language heavily inspired by the theoretical bases of both Datalog and the relational algebra.

In terms of talking to XT2:

XT2 comes with two primary APIs: a Postgres wire-compatible server, and a Clojure in-process/HTTP API.
You can run a Docker image, a Github Codespace, or visit https://play.xtdb.com
We have specific deployment Docker images for AWS and Azure (GCP to come).

For example, to start a local node:

docker run -ti --name xtdb -p 5432:5432 ghcr.io/xtdb/xtdb:2.0.0b1
psql -h localhost

Then:

-- no `CREATE TABLE` required, just start inserting
-- `INSERT ... VALUES` also supported
INSERT INTO users RECORDS {_id: 'jms', name: 'James'}, {_id: 'jdt', name: 'Jeremy'};

INSERT INTO users RECORDS {_id: 'mal', name: 'Malcolm'};

SELECT *, _valid_from, _valid_to FROM users;

For more examples, see the SQL Quickstart.

Architecturally, XT2 differs slightly from XT1:

In XT1, the 'golden stores' were an infinite-retention transaction log (e.g. Kafka) and a document store (e.g. AWS S3). In XT2, this changes slightly: we now require a transient transaction log (i.e. dropping the requirement for infinite retention), and an 'object store'. The object-store, while fulfilled by similar services and implementations to the document store, is now intended to store significantly larger blobs.
XTDB nodes no longer need to store a full copy of the data - they pull blobs from the object store as required.
The primary index is a hash-oriented, distributed LSM trie of Arrow files for each table, on which the individual nodes share LSM compaction responsibilities.

Until next time

For support and questions, please do get in touch with us - either at https://discuss.xtdb.com or hello@xtdb.com - and please feel free to raise issues on the XTDB Github repo in the usual way.

Cheers!

Jeremy & the XT Team