Getting started with XTDB and Luminus

Want to get started on a web project using Clojure and XTDB? Luminus is a great way to do that. Luminus is a customizable project template that will generate a skeleton app for you to work with, using your desired components.

In this tutorial we’ll look at how you can spin up a simple REST API to manage users using XTDB as the database.

Requirements

All you’ll need installed is either Leiningen or Boot, though Leiningen is more common and what we’ll use in the example below.

Starting a Luminus app

To generate the project, run

lein new luminus my-app +service +xtdb

+service will give us a JSON based API rather than returning HTML. +xtdb gives us everything we need to interact with XTDB: a Mount state for the XTDB node, config for different environments and some example queries.

To start the server, start a REPL in your editor and run

(start)
; ==>
{:started ["#'my-app.config/env"
           "#'my-app.handler/init-app"
           "#'my-app.handler/app-routes"
           "#'my-app.core/http-server"]}

If you browse to http://localhost:3000 you should see a Swagger page that documents some example routes.

Wiring up the API with XTDB

Let’s add a couple of new API routes that create and query users in XTDB.

The XTDB parts of the template are in these files:

src/clj/my_app/db/core.clj - this starts our XTDB node and contains some example transactions and queries.
test/clj/my_app/db/core_test.clj - some tests which spin up a node and check that the queries and transactions work as expected.
dev-config.edn - contains our XTDB config for the dev environment. By default this will store data in <project directory>/data locally via RocksDB.
test-config.edn - contains our XTDB config for the test environment. By default this will use an in memory node.

Note

The Luminus template includes dev-config.edn and test-config.edn in the gitignore file, for the reason that devs may want to use different dev setups and for safety against accidentally committing sensitive info like passwords. If you’d prefer to commit the config, feel free to remove these files from .gitignore.

The API routes are defined in src/clj/my_app/routes/services.clj. Let’s add a couple of routes which use our queries:

;; src/clj/my_app/routes/services.clj

(ns my-app.routes.services
  (:require
    ; ...
    ; add a require for the db namespace
    [my-app.db.core :as db])
  ; add an import for UUID
  (:import (java.util UUID)))

; ...
["/ping"
  {:get (constantly (ok {:message "pong"}))}]

; our shiny new xtdb routes
["/users"
 ["" {:post {:summary "create a new user"
      :parameters {:body any?}
      :handler (fn [{{user :body} :parameters}]
                 (let [user-with-id (db/create-user! db/node user)]
                   {:status 201 :body user-with-id}))}}]

 ["/:id"
  {:get {:summary "get a user"
         :parameters {:path {:id string?}}
         :handler (fn [{{:keys [id]} :path-params}]
                    (if-let [user (db/find-user-by-id db/node (UUID/fromString id))]
                      {:status 200 :body user}
                      {:status 404 :body {:message "User not found"}}))}}]]

["/math"
 {:swagger {:tags ["math"]}}
; ...

For now, we’ll leave out validating the request, but we have the option to add that in later, for example by replacing the any? above with a Clojure spec for a user.

If we reload this namespace and run a (restart) in the repl and refresh Swagger in the browser, we should see our new routes. We can try out the new POST /users route in Swagger, posting in some JSON:

{
   "user/email": "one@example.com",
   "user/name": "User One"
}

The API should respond with our user, with "user/id" added, telling us that our user has been successfully saved to XTDB! 🎉 Trying out GET /users/<some-uuid> should give us back the user.

Warning

If the get user route is called with an ID that’s not a valid UUID, (UUID/fromString id) will throw an IllegalArgumentException, and a 500 error will be returned rather than the 400 or 404 it should be. We could add some validation for this, or could potentially use string UUIDs rather than UUID objects as our IDs in XTDB to bypass the issue.

To namespace or not to namespace?

Currently, if we want our XTDB documents to include namespaced keywords, the posted JSON needs to include the namespace e.g. "user/email". Arguably this is fine, and you might be happy to leave it, but it can look a bit strange and could make our API harder to interact with if called from languages without namespaced keys. To deal with this we can add some conversions at the API layer that add and remove the namespace as appropriate:

;; src/clj/my_app/routes/services.clj

; ...
(defn- nsmap->map
  "Strips the namespace from the keys of a map"
  [m]
  (reduce-kv (fn [m k v]
               (assoc m (keyword (name k)) v))
             {} m))

(defn- map->nsmap
  "Adds the given namespace to the keys of a map"
  [m ns]
  (reduce-kv (fn [m k v]
               (assoc m (keyword ns (name k)) v))
             {} m))

(def json->user #(map->nsmap % "user"))
(def user->json nsmap->map)

; ...
["/users"
 [""
  {:post {:summary "create a new user"
          :parameters {:body any?}
          :handler (fn [{{user :body} :parameters}]
                     (let [user-with-id (db/create-user! db/node (json->user user))]
                       {:status 201 :body (user->json user-with-id)}))}}]

 ["/:id"
  {:get {:summary "get a user"
         :parameters {:path {:id string?}}
         :handler (fn [{{:keys [id]} :path-params}]
                    (if-let [user (db/find-user-by-id db/node (UUID/fromString id))]
                      {:status 200 :body (user->json user)}
                      {:status 404 :body {:message "User not found"}}))}}]]
; ...

Now our create user endpoint can be given more familiar JSON:

{
   "email": "two@example.com",
   "name": "User Two"
}

As the app grows, it might be nice to move these transformations into coercers or middleware.

Note

The queries in the template use the convention of namespacing keywords in XTDB documents e.g. :user/id rather than :id. This isn’t enforced by XTDB and there’s nothing to stop you removing the namespaces. There are tradeoffs with both approaches and this topic could have its own blog post, but as a quick summary: namespacing keywords in XTDB documents generally helps discoverability of keywords and helps protect against keyword collisions. The downside is that it will take some extra code to wrangle the data to and from formats like JSON. In this post I’ve chosen to use namespaced keywords to show how you might handle them.

In summary

We hope this gives a flavour of how to get building with Luminus and XTDB. To take this further you could implement validation, routes to update and delete users, or a route that returns a history of a user, harnessing some of XTDB’s temporal powers!

As always, feel free to reach out to us on the 'Discuss XTDB' forum, the #xtdb channel on the Clojurians' Slack or via hello@xtdb.com.