Development Diary #9

Welcome to our first dev diary of 2024!

In December we released the first early access preview of XTDB v2, and gave a webinar on what’s new.

Introducing Java/Kotlin support:

We’re introducing the first public preview of our first-class Java/Kotlin API for JVM users - including a fluent, strongly-typed XTQL query and transaction DSL.

For example, you can now connect to a node, submit documents, and query them using Kotlin:

// open up an in-memory XTDB node
Xtdb.openNode().use { xtdb ->
    // put three order docs into the `orders` table
    xtdb.submitTx(
        putDocs(
            "orders",
            mapOf("xt\$id" to 1, "customerId" to 1, "orderValue" to 142.32),
            mapOf("xt\$id" to 2, "customerId" to 1, "orderValue" to 389.12),
            mapOf("xt\$id" to 3, "customerId" to 2, "orderValue" to 85.99),
        )
    )

    // run an SQL query
    val sqlRes = xtdb.openQuery(
        """
        SELECT customer_id, COUNT(*) AS order_count, SUM(order_value) AS total_value
        FROM orders
        GROUP BY customer_id
        ORDER BY total_value DESC
        """
    ).use { it.toList() }

    // that same query, but in XTQL
    val xtqlRes = xtdb.openQuery(
        pipeline(
            from("orders") { bindAll("customerId", "orderValue") },

            aggregate {
                bind("customerId")
                bindAll(
                  "orderCount" to expr { "rowCount"() },
                  "totalValue" to expr { "sum"("orderValue".sym) },
                )
            },

            orderBy(orderSpec("totalValue").desc())
        )
    ).use { it.toList() }
}

For more examples, have a look at the XTQL transaction and query documentation, and click on the language icons.

Please do give this a try and let us know what you think - especially while it’s in preview, we’re more than happy to take suggestions on ways we can improve it!

For Clojure users: the equivalent in EDN:

;; transaction
[:put-docs :orders
 {:xt/id 1, :customer-id 1, :order-value 142.32}
 {:xt/id 2, :customer-id 1, :order-value 389.12}
 {:xt/id 3, :customer-id 2, :order-value 85.99}]

;; query
(-> (from :orders [:customer-id :order-value])
    (aggregate :customer-id
               {:order-count (row-count)}
               {:total-value (sum order-value)})
    (order-by {:expr total-value, :dir :desc}))

… and also JSON

As promised in the webinar, we’ve been working on a JSON API for XT2 - this now has support for both XTQL and SQL transactions and queries.

While it can be written directly, we’d imagine that this will more likely be used as a compile target for language-specific client libraries/DSLs (like the Java/Kotlin API above, even).

// transaction - submit some order documents
{
  "putDocs": [
    { "xt$id": 1, "customerId": 1, "orderValue": 142.32 }
    { "xt$id": 2, "customerId": 1, "orderValue": 389.12 }
    { "xt$id": 3, "customerId": 2, "orderValue": 85.99 }
  ],

  "into": "orders"
}

// then query:
//   SELECT customer_id, COUNT(*) AS order_count, SUM(order_value) AS total_value
//   FROM orders
//   GROUP BY customer_id
//   ORDER BY total_value DESC

[
  { "from": "orders", "bind": [ "customerId", "orderValue" ] },

  {
    "aggregate": [
      "customerId",
      { "orderCount": { "xt:call": "rowCount" }},
      { "totalValue": { "xt:call": "sum", "args": [{ "xt:lvar": "orderValue" }] } }
    ]
  },

  {
    "orderBy": [ { "expr": { "xt:lvar": "totalValue" }, "dir": "desc" } ]
  }
]

For more examples, have a look at the XTQL transaction and query documentation.

Node configuration: YAML

We’ve added support for configuring an XTDB node through a YAML file - for example, to configure a node backed by Kafka and AWS S3, you can start the node with something like this:

txLog: !Kafka
  bootstrapServers: !Env XTDB_KAFKA_BOOTSTRAP_SERVERS
  topicName: xtdb-tx-log

storage: !Remote
  localDiskCache: /var/cache/xtdb/storage

  objectStore: !S3
    bucket: !Env XTDB_S3_BUCKET
    snsTopicArn: !Env XTDB_SNS_TOPIC_ARN

For more information, see the configuration guide.

Breaking changes

If you’re an existing (Clojure) early access user, here are the breaking changes:

The http-client module has been renamed from com.xtdb/xtdb-http-client-clj to com.xtdb/xtdb-http-client-jvm
The transaction operation builders in xtdb.api have been removed - these are now supplied as vectors.

See the Clojure docs for more examples. Particularly, for those of you coming from XT1:
- :put is now :put-docs, :delete → :delete-docs, :erase → :erase-docs
- All of the above are now variadic - i.e. :put-docs can accept multiple documents (for the same table and valid-time range) - and should give a slight performance advantage over individual operations.
  
  If you’re unsure how to migrate your operations, give us a shout.
:key-fn has been made consistent. It defaults to :kebab-case-keyword through the Clojure API, but can also accept :snake-case-keyword, :camel-case-keyword, or any of these with -string.
Node configuration has been significantly simplified, particularly when configuring the transaction log and object-store. Object store configuration is now nested under 'storage' - there are examples in the Clojure docs.

Again, if you’re unsure how to migrate your config, give us a shout.

Next up

Next up on our roadmap, as it stands:

We’ve had a number of people asking for specific functions which aren’t part of the SQL spec but are nevertheless used on a daily basis in practice (good ol' Postgres et al creating de-facto standards here!) - these are on their way.
We’re looking to add support for a first handful of common SQL-industry tools, especially in the BI area.
We’ll also be spending a significant proportion of the next few months with our heads buried in a profiler ⏱

Get in touch!

As always, we’d love to hear from you - we’ve already had many insightful conversations with people who’ve given XT2 a spin which have influenced our roadmap, and it’d be great to get your thoughts too:

Discuss XTDB (public forums)
hello@xtdb.com

Similarly, if you’d like a demo of XT2, or to be kept informed about its progress, come say hello at hello@xtdb.com.

Cheers,

James