Tuesday, 13 May, 2025
XTDB: Data Compliance Assurance with Bitemporality
Jason Bloomberg By Jason Bloomberg, Managing Director, Intellyx (guest author)
In the first three installments of this series of articles, my colleague Eric Newcomer and I made the case for bitemporal databases.
In the first installment, Eric introduced the central challenge of data compliance: how traditional relational databases struggle to keep track of how values change over time.
I followed up with an explanation of how the everyday UPDATE and DELETE operations so familiar to database professionals are inherently flawed, as they erase any information about the previous state of the data they overwrite.
In the third article, Eric explored the bitemporality challenge in depth, differentiating between ‘current’ databases that track values in the present vs. bitemporal databases that can track various values at points in the past as well as when those values changed.
This article, in turn, wraps up the series. Sure, bitemporality is a nice-to-have feature, but when is it a must-have capability? How do you decide whether there is a place for a database like XTDB in your organization?
Addressing the Primary Pushback against Bitemporality
The primary pushback we received against bitemporal databases from the previous articles in this series was the fact that it’s possible to achieve the benefits of bitemporality without adding a separate bitemporal database like XTDB to an already crowded IT infrastructure portfolio.
Detractors pointed out that legacy data infrastructure products like J.D. Edwards and Teradata have built-in bitemporality features. Other skeptics referred to the fact that the latest version of the SQL standard includes long-needed temporal operators.
Both concerns are valid – but miss the central value proposition of a bitemporal database.
Alternative approaches are either proprietary or require extensive hand-coding – or both. In contrast, bitemporality is built into XTDB.
As a result, XTDB makes it much simpler and easier to keep track of both the ‘valid time’ (supporting queries like ‘when was the price equal to $1.00?’) as well as the ‘system time’ (answering questions like ‘when did somebody change the price?’).
The argument for XTDB thus becomes the value of deploying simpler and easier bitemporality despite the additional complexity and overhead of adding a new database system to the mix – a tradeoff that arguably could go either way.
This argument, however, is missing a critical consideration: data compliance assurance is only getting more complicated – and the more complicated it gets, the greater the XTDB value proposition.
Real-World Example: Exploding Data Compliance Assurance Complexity
Real-time equities trading is perhaps the most technically demanding of regulated industries. Not only must trades execute in ten seconds or less to avoid being considered late, but the trading systems must also handle after-hours trading.
Bitemporality has obvious applications for trading during normal market hours. Auditors must be able to review equities pricing at any point in the past. Any attempt to retroactively fudge with the numbers must be impossible to hide. Trading firms must also implement sophisticated error correction procedures that can account for mis-pricing and many other error conditions.
From the auditor perspective, therefore, tracking prices at points in the past as well as any corrective actions that techs may have performed are both important considerations that require bitemporality.
Adding after-hours trading to the mix, however, complicates matters even further. According to FINRA, trades executed after hours must be reported on an ‘as/of’ basis by the next morning – or the next business day in the case of weekends or holidays.
In other words, each such trade has both a reporting time when the trade is officially executed as well as an ‘as/of’ time that dictates the price for the trade – and these two times will always be different.
Auditors must now keep track of all these factors, as well as the error conditions that might arise. For example, FINRA rules state that if the trade reporting is late, then it must be reported on an ‘as/of’ basis on a subsequent date – with rules dictating what that date must be (and firms must also then be prepared to ‘re-report’ as rules change).
Without the power of a bitemporal database like XTDB, coding the necessary business logic directly into the supporting data infrastructure becomes an increasingly complex, convoluted tangle – a mess that risks slowing down the proper execution of trades.
And the last thing anyone wants in the equities trading business is something that slows down trade execution.
The Intellyx Take
Equities trading is but one example of a regulated industry that must keep track of a complex web of events in real-time – as well as supporting requirements both for error correction as well as auditing.
For any regulated organization weighing the benefits of implementing a new bitemporal database like XTDB vs. continuing to code bitemporal capabilities within existing data infrastructure, be warned: complexity only increases – and with it, cost and performance considerations.
There’s no question that implementing a new database, either alongside or replacing existing databases, is a decision that should not be taken lightly. Be sure to factor in future cost and performance issues for the status quo vs. the flexibility of an open platform like XTDB to make the right decision for your organization.
Copyright © Intellyx B.V. Intellyx is editorially responsible for this document. No AI bots were used to write this content. At the time of writing, JUXT is an Intellyx client.
