dtm CLI - User Guide

A high-level walkthrough of working with dtm (Delta Table Manager): from bootstrapping a new project to running it day-to-day.

Mental model

dtm keeps three pictures of your database in sync:

Picture	What it is	Where it lives
Solution	The schema you want - CREATE TABLE files	sql/ folder
Statefile	The schema dtm thinks the database currently has	state/<target>/*.json
Database	The schema actually deployed in Unity Catalog	The live workspace

Every command is some flavour of "diff two of these, then act on the result".

User-authored SQL that doesn't fit a typed diff (DDL tweaks, data fixes, seed inserts) lives in pre- and post-deployment migrations - plain .sql files run by filename order, with optional depends_on metadata.

Invocation basics

uv run dtm --target <name> <command>

• --target is required and must match a key under targets: in project.yaml (e.g. dev, prod).
• --config defaults to ./project.yaml.
• --render-output <dir> keeps the rendered SQL on disk for inspection - handy when debugging unexpected diffs.

---

Flow 1 - Bootstrapping a new project

1. Lay down project.yaml

Declare:

• the catalogs the project will use under targets.<env>.databases (they must already exist in the workspace - dtm never creates catalogs, see scope),
• the SQL/migration folders,
• target_platform and any platform_options (e.g. delta.columnMapping.mode: name - set this now, not later, if you ever want to drop columns).

Minimal working template:

state_backend: local
log_level: INFO

migrations:
  database_sql_folder: ./sql              # relative to project.yaml
  migration_root_path: ./migrations       # relative to project.yaml
  pre_deployment_glob: "pre_deployment/**/*.sql"   # relative to migration_root_path
  post_deployment_glob: "post_deployment/**/*.sql" # relative to migration_root_path
  conflict_strategy: warn

target_platform: azure_databricks

# Emitted as TBLPROPERTIES on every CREATE TABLE.
# Set column mapping now if you ever want to drop columns later.
platform_options:
  tables:
    properties:
      delta.columnMapping.mode: name

targets:
  dev:
    workspace_url: https://adb-1234567890123456.7.azuredatabricks.net
    state_folder: ./state/dev
    databases:
      db1: dtm_demo1_dev
      db2: dtm_demo2_dev
    variables:
      var1: value1_dev
  prod:
    workspace_url: https://adb-1234567890123456.7.azuredatabricks.net
    state_folder: ./state/prod
    databases:
      db1: dtm_demo1_prod
      db2: dtm_demo2_prod
    variables:
      var1: value1_prod

2. Connect to the workspace

Export Databricks credentials before running commands.

Required:

• CLOE_DBX_WORKSPACE_URL - Databricks workspace URL
• CLOE_DBX_SQL_WAREHOUSE_ID - SQL warehouse (serverless compute) used for execution

Authentication: set exactly one method

Method	Required environment variables
Service principal	CLOE_DBX_CLIENT_ID, CLOE_DBX_CLIENT_SECRET
Azure Entra ID service principal	CLOE_AZURE_TENANT_ID, CLOE_AZURE_CLIENT_ID, CLOE_AZURE_CLIENT_SECRET
Personal access token (PAT)	CLOE_DBX_PAT*

If Databricks CLI auth is already configured, you can fetch and export CLOE_DBX_PAT with:

export CLOE_DBX_PAT="$(databricks auth token --host "$CLOE_DBX_WORKSPACE_URL" 2>/dev/null \
| python3 -c "import json,sys; print(json.load(sys.stdin)['access_token'])" 2>/dev/null)"

Optional SQL defaults

• CLOE_DBX_DEFAULT_CATALOG
• CLOE_DBX_DEFAULT_SCHEMA
• CLOE_DBX_STATEMENT_TIMEOUT (for example 50s; Databricks synchronous max is 50s)

3. Initialise state

uv run dtm --target dev init

Creates empty state/dev/*.json files. Safe to re-run.

4. Author the desired schema

Drop CREATE SCHEMA / CREATE TABLE files under sql/. Use {db1}-style placeholders that resolve from targets.<env>.databases. Only the operations in supported-operations are allowed.

5. Validate before you touch anything

uv run dtm --target dev validate

Checks project.yaml, SQL syntax, and the migration dependency graph. Fix errors here - it's free.

6. Plan

uv run dtm --target dev plan              # outputs: filenames only
uv run dtm --target dev plan --detailed   # outputs:  filenames + rendered SQL

Shows the pre-deployment migrations, the auto-generated schema diff, and the post-deployment migrations that deploy would execute. Always run plan before deploy.

7. Deploy

uv run dtm --target dev deploy        # prompts for confirmation
uv run dtm --target dev deploy -y     # CI / non-interactive

Runs everything plan showed, in order: pre → auto-diff → post. On success, statefiles are updated to record what ran.

---

Flow 2 - Everyday work

The loop is the same as bootstrapping, minus init.

Routine: change a table

1. Edit the CREATE TABLE file in sql/ (add a column, change a type, add a comment, …).
2. uv run dtm --target dev validate - quick sanity check.
3. uv run dtm --target dev plan - eyeball the generated ALTERs.
4. uv run dtm --target dev deploy - apply.
5. Commit the SQL change and the updated statefile together.

Routine: ad-hoc SQL (data fix, backfill, one-off DDL)

Add a file under migrations/pre_deployment/ or migrations/post_deployment/. Use the first comment in the file as YAML metadata if order matters:

/* depends_on:
  - 001_create_table.sql
  - 002_alter_table.sql
*/

depends_on accepts either a single path or a list of paths. Paths are relative to the migration file's directory. The metadata must be the first comment in the file and must be valid YAML. Then plan → deploy as above. Each migration runs exactly once - its hash is tracked in the statefile.

Routine: inspect drift

uv run dtm --target dev compare solution statefile
uv run dtm --target dev compare solution database

compare prints the typed diff between any two of solution, statefile, database. Use it to answer "what would the next deploy do?" without generating SQL.

Routine: someone changed the DB by hand

Pull the live schema back into the statefile:

uv run dtm --target dev state refresh

Then compare solution statefile to see what drifted, and either update your sql/ files to match reality or deploy to push reality back to the solution.

Routine: promoting to prod

Same commands, different --target:

uv run dtm --target prod validate
uv run dtm --target prod plan
uv run dtm --target prod deploy -y

Each target has its own state and its own catalog mapping, so dev and prod statefiles evolve independently.

---

Command cheat-sheet

Command	Purpose	When
init	Create empty statefiles for a target	Once per target
validate	Static checks on config, SQL, migration graph	Before every plan/deploy
plan [--detailed]	Show pending pre-/auto-/post-migrations	Before every deploy
deploy [-y]	Run pre → auto-diff → post, then update state	To apply changes
compare A B	Diff between solution / statefile / database	Drift checks, dry runs
state refresh	Rebuild statefile from the live database	After out-of-band changes
state generate	Write solution_state.json from SQL files (debug)	Debugging drift

---

Gotchas

• --target is required on every call - even for validate and init.
• Catalogs are pre-conditions, never deliverables - create them out of band; dtm refuses to.
• Set delta.columnMapping.mode = name on day one. Retrofitting it later needs a manual ALTER TABLE migration (see platform-options).
• Renames look like drop + add in v1. Plan accordingly if the table holds data you can't lose.