Tuesday, 30 January, 2024
Development Diary #9
James Henderson Welcome to our first dev diary of 2024!
In December we released the first early access preview of XTDB v2, and gave a webinar on what’s new.
Introducing Java/Kotlin support:
We’re introducing the first public preview of our first-class Java/Kotlin API for JVM users - including a fluent, strongly-typed XTQL query and transaction DSL.
For example, you can now connect to a node, submit documents, and query them using Kotlin:
// open up an in-memory XTDB node
Xtdb.openNode().use { xtdb ->
// put three order docs into the `orders` table
xtdb.submitTx(
putDocs(
"orders",
mapOf("xt\$id" to 1, "customerId" to 1, "orderValue" to 142.32),
mapOf("xt\$id" to 2, "customerId" to 1, "orderValue" to 389.12),
mapOf("xt\$id" to 3, "customerId" to 2, "orderValue" to 85.99),
)
)
// run an SQL query
val sqlRes = xtdb.openQuery(
"""
SELECT customer_id, COUNT(*) AS order_count, SUM(order_value) AS total_value
FROM orders
GROUP BY customer_id
ORDER BY total_value DESC
"""
).use { it.toList() }
// that same query, but in XTQL
val xtqlRes = xtdb.openQuery(
pipeline(
from("orders") { bindAll("customerId", "orderValue") },
aggregate {
bind("customerId")
bindAll(
"orderCount" to expr { "rowCount"() },
"totalValue" to expr { "sum"("orderValue".sym) },
)
},
orderBy(orderSpec("totalValue").desc())
)
).use { it.toList() }
}For more examples, have a look at the XTQL transaction and query documentation, and click on the language icons.
Please do give this a try and let us know what you think - especially while it’s in preview, we’re more than happy to take suggestions on ways we can improve it!
For Clojure users: the equivalent in EDN:
;; transaction
[:put-docs :orders
{:xt/id 1, :customer-id 1, :order-value 142.32}
{:xt/id 2, :customer-id 1, :order-value 389.12}
{:xt/id 3, :customer-id 2, :order-value 85.99}]
;; query
(-> (from :orders [:customer-id :order-value])
(aggregate :customer-id
{:order-count (row-count)}
{:total-value (sum order-value)})
(order-by {:expr total-value, :dir :desc}))… and also JSON
As promised in the webinar, we’ve been working on a JSON API for XT2 - this now has support for both XTQL and SQL transactions and queries.
While it can be written directly, we’d imagine that this will more likely be used as a compile target for language-specific client libraries/DSLs (like the Java/Kotlin API above, even).
// transaction - submit some order documents
{
"putDocs": [
{ "xt$id": 1, "customerId": 1, "orderValue": 142.32 }
{ "xt$id": 2, "customerId": 1, "orderValue": 389.12 }
{ "xt$id": 3, "customerId": 2, "orderValue": 85.99 }
],
"into": "orders"
}
// then query:
// SELECT customer_id, COUNT(*) AS order_count, SUM(order_value) AS total_value
// FROM orders
// GROUP BY customer_id
// ORDER BY total_value DESC
[
{ "from": "orders", "bind": [ "customerId", "orderValue" ] },
{
"aggregate": [
"customerId",
{ "orderCount": { "xt:call": "rowCount" }},
{ "totalValue": { "xt:call": "sum", "args": [{ "xt:lvar": "orderValue" }] } }
]
},
{
"orderBy": [ { "expr": { "xt:lvar": "totalValue" }, "dir": "desc" } ]
}
]For more examples, have a look at the XTQL transaction and query documentation.
Node configuration: YAML
We’ve added support for configuring an XTDB node through a YAML file - for example, to configure a node backed by Kafka and AWS S3, you can start the node with something like this:
txLog: !Kafka
bootstrapServers: !Env XTDB_KAFKA_BOOTSTRAP_SERVERS
topicName: xtdb-tx-log
storage: !Remote
localDiskCache: /var/cache/xtdb/storage
objectStore: !S3
bucket: !Env XTDB_S3_BUCKET
snsTopicArn: !Env XTDB_SNS_TOPIC_ARNFor more information, see the configuration guide.
Breaking changes
If you’re an existing (Clojure) early access user, here are the breaking changes:
-
The http-client module has been renamed from
com.xtdb/xtdb-http-client-cljtocom.xtdb/xtdb-http-client-jvm -
The transaction operation builders in
xtdb.apihave been removed - these are now supplied as vectors.See the Clojure docs for more examples. Particularly, for those of you coming from XT1:
-
:putis now:put-docs,:delete→:delete-docs,:erase→:erase-docs -
All of the above are now variadic - i.e.
:put-docscan accept multiple documents (for the same table and valid-time range) - and should give a slight performance advantage over individual operations.If you’re unsure how to migrate your operations, give us a shout.
-
-
:key-fnhas been made consistent. It defaults to:kebab-case-keywordthrough the Clojure API, but can also accept:snake-case-keyword,:camel-case-keyword, or any of these with-string. -
Node configuration has been significantly simplified, particularly when configuring the transaction log and object-store. Object store configuration is now nested under 'storage' - there are examples in the Clojure docs.
Again, if you’re unsure how to migrate your config, give us a shout.
Next up
Next up on our roadmap, as it stands:
-
We’ve had a number of people asking for specific functions which aren’t part of the SQL spec but are nevertheless used on a daily basis in practice (good ol' Postgres et al creating de-facto standards here!) - these are on their way.
-
We’re looking to add support for a first handful of common SQL-industry tools, especially in the BI area.
-
We’ll also be spending a significant proportion of the next few months with our heads buried in a profiler ⏱
Get in touch!
As always, we’d love to hear from you - we’ve already had many insightful conversations with people who’ve given XT2 a spin which have influenced our roadmap, and it’d be great to get your thoughts too:
-
Discuss XTDB (public forums)
Similarly, if you’d like a demo of XT2, or to be kept informed about its progress, come say hello at hello@xtdb.com.
Cheers,
James
