XTDB v2 Hub

Quick links: Getting started | Documentation | GitHub | XTDB discussion forum

Over the last couple of years, we’ve been working hard on the next version of XTDB, and we’re excited to share it with you now!

To summarise, XTDB 2:

• is based on a new, lightning-fast columnar/relational query engine
• is queried through either SQL or ‘XTQL’, our new data-oriented, composable query language
• targets both OLTP and OLAP use cases (i.e. ‘hybrid’, or ‘HTAP’), reducing the need for separate ETL processes, and with infrequently accessed data stored in shared, cheaper object storage
• supports ‘fully bitemporal’ queries (rather than just point-in-time)

We’re now at a point where we can gather feedback on what we have so far, particularly from a usability point-of-view. Throughout the alpha process, as we work towards general availability, we’ll be spending more time improving the performance, stability and operational aspects, as well as extending its reach via client libraries for other languages.

With this in mind, we’re very much in listening mode right now - as an existing XT user and/or keen early adopter, we’d love to hear your first impressions, thoughts and opinions on where we’re headed. Please do get in touch!

Similarly, if you’d like to be kept informed about the progress of v2, drop us an email at hello@xtdb.com.

Coming up:

• What’s new in V2?
• Main differences to 1.x
• What’s happening to 1.x?
• Getting in touch

What’s new in V2?

The primary aims of V2 are as follows:

1. Hybrid transactional/analytical processing (‘HTAP’):

   XTDB v1 is mostly targeted towards transactional queries - in v2, we’ve improved the support for OLAP queries, to reduce the need for users to write and maintain ETL processes to a separate OLAP system.

   As part of this, we’ve ‘separated storage from compute’ - data that is not accessed as often (historical data, say) can be kept in significantly cheaper storage (e.g. Amazon’s S3).

   Additionally, data in XT is now all stored in the open Apache Arrow format, and we have a shiny, new, fast, columnar query engine.

2. First-class SQL support:

   While we obviously love Datalog, it’s hard to deny the position of SQL in our industry - there’s such a wealth of knowledge, experience and tooling for SQL that you can now use with XTDB.

   XTDB 2 therefore has first-class support for two query languages - one is SQL, the other is ‘XTQL’ (#4). They are fully interoperable - data entered with either SQL or XTQL can be queried with the other.

   In order for the two to be interoperable, documents in XTDB now live in specific, but dynamic tables - same as you would in an SQL database, but there’s no requirement for up-front schema, and no requirement for all of the documents in a table to have the same column types, or even the same columns.

3. Full bitemporality:

   Where XTDB has previously been optimized for point-in-time historical queries, v2 now supports ‘cross-time’ queries, e.g. temporal joins and temporal range scans. This unlocks the complete history of data for rich analysis.

   We support the SQL:2011 standard for bitemporal functionality, and it’s built into the heart of XTQL too.

4. XTQL:

   … because it just wouldn’t be XTDB without also having a more data-oriented, composable query language!

   XTQL is inspired by the strong theoretical bases of both Datalog and relational algebra. These two combine to create a joyful, productive, interactive development experience, with the ability to build queries iteratively, testing and debugging smaller parts in isolation. XTQL is also highly amenable to being dynamically generated - because, as an industry, haven’t we all spent enough time constructing SQL strings (not to mention avoiding the concomitant ie injection attacks)?

   XTQL also has DML operations - so, a lot of use cases that previously required transaction functions for strong consistency can now use XTQL’s declarative ‘insert’, ‘update’ and ‘delete’ transaction operations. These have full access to the current state of the database, the full query engine, can be submitted to the transaction log without pre-registration, and are evaluated on the single writer thread.

   It’s available in both JSON and EDN, with other client libraries to come in the near future.

   For more information about XTQL, see our ‘What is XTQL?’ article.

Main differences to 1.x

XTDB 2.x is very much in keeping with the core ideas and concepts behind XTDB 1.x - you’ll still find a schema-less, documented-oriented database with bitemporal immutability as standard. That said, with such a major release, there’s obviously plenty that’s new and/or improved:

• Tables: every document in XTDB now lives within a table - transaction operations specify the tables to insert into/update in addition to the documents themselves; queries specify the tables to pull from.

  In practice, we found that the majority of production XT1 use cases had a ‘type’ attribute of some kind - this standardises the pattern, but also means that our query planner can now use finer-grained, per-table information and statistics to better optimise queries rather than trying to infer this from the rest of the query.

• Architecturally, the XTDB golden stores have evolved:

  • The document store is now a shared object store - rather than individual documents, this now stores large, compacted, columnar Apache Arrow index files.
  • It still has a transaction log to impose a total ordering on transactions, but this is no longer required to have infinite retention (reducing the write amplification) - the full history is now stored in the object store.
  • The index files that were previously required on every node are now stored in the object-store - each node now maintains a (large) local cache of its working set, but can fetch missing files from the shared store on demand.

• Transaction functions are still supported (even though we’d strongly recommend using XTQL DML where possible) - they are now evaluated in a Small Clojure Interpreter (SCI) sandbox, for security.

• match has been replaced with assert-exists/assert-not-exists - rather than being limited to checking the whole of a single document, these can run any arbitrary query to determine whether the transaction should proceed or not.

• Rather than supporting arbitrary Clojure functions, queries in XTDB 2 now use an XT standard library of functions, drawn from the SQL standard.

What’s happening to 1.x?

Rest assured, version 1 of XTDB will remain stable and supported with security and bug fix releases for the foreseeable future - we have a vibrant community and existing install base that we’re very proud of.

• If you have a production installation of XTDB 1.x and would like us to keep you informed, you can sign up to our mailing list - send us an email at hello@xtdb.com.
• For teams that would like to migrate from v1 to v2, we will be publishing migration tools and documentation on migration strategies soon - but please also get in touch if you’d like our help doing so.

Getting in touch

We’d love to hear from you - please do get in touch:

• via email: hello@xtdb.com
• on the XTDB forum: discuss.xtdb.com

Cheers!

James, Jeremy and the XTDB Team