Index
Delta Table Manager package for managing delta table migrations and state.
AutoMigrationRunner
¶
Bases: MigrationRunner
Generate migrations from a state diff and execute them.
The runner compares a current state (read from the database statefile)
against a desired state (parsed from SQL files) and generates the SQL
migrations needed to reconcile the two. After execution the applied
migrations are persisted to auto_migration_state_path for observability.
Source code in src/cloe_delta_table_manager/migrations/runner/auto.py
__init__(*, executor=None, state_client=None, platform_options=None)
¶
Initialize the AutoMigrationRunner.
Source code in src/cloe_delta_table_manager/migrations/runner/auto.py
execute_migrations(*, current_state, desired_state, statefile_path)
¶
Generate and execute auto migrations, then persist state.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
current_state
|
DatabaseState
|
The database state as recorded in the statefile. |
required |
desired_state
|
DatabaseState
|
The desired state derived from SQL source files. |
required |
statefile_path
|
str | Path
|
Path to persist the auto migration state after execution. |
required |
Source code in src/cloe_delta_table_manager/migrations/runner/auto.py
list_pending_migrations(*, current_state, desired_state)
¶
Return auto-generated migrations without executing them.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
current_state
|
DatabaseState
|
The database state as recorded in the statefile. |
required |
desired_state
|
DatabaseState
|
The desired state derived from SQL source files. |
required |
Returns:
| Type | Description |
|---|---|
list[Migration]
|
List of generated Migration objects. |
Source code in src/cloe_delta_table_manager/migrations/runner/auto.py
FileMigration
¶
Bases: Migration
Represents a user-defined migration script with additional metadata.
Source code in src/cloe_delta_table_manager/migrations/migration.py
resolve_depends_on_paths()
¶
Resolve depends_on_paths relative to source_path directory.
Source code in src/cloe_delta_table_manager/migrations/migration.py
validate_depends_on_path(v)
classmethod
¶
Validate that depends_on_paths contains only relative paths.
Source code in src/cloe_delta_table_manager/migrations/migration.py
FileMigrationRunner
¶
Bases: MigrationRunner
Manages reading, ordering, and executing user-defined database migrations.
Source code in src/cloe_delta_table_manager/migrations/runner/file.py
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 | |
__init__(executor=None, state_client=None)
¶
Initialize the FileMigrationRunner.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
executor
|
SQLExecutor | None
|
SQL executor for running statements against Databricks. |
None
|
state_client
|
StateClient | None
|
Client for reading/writing state. Defaults to LocalStateClient. |
None
|
Source code in src/cloe_delta_table_manager/migrations/runner/file.py
execute_migrations(statefile_path, root_path, glob_pattern, conflict_strategy=None)
¶
Execute user-defined migrations.
Source code in src/cloe_delta_table_manager/migrations/runner/file.py
list_pending_migrations(statefile_path, root_path, glob_pattern, conflict_strategy=None)
¶
Return pending migrations in execution order without running them.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
statefile_path
|
Path | str
|
Path to the migration state file. |
required |
root_path
|
Path | str
|
Path to directory containing migration SQL files. |
required |
glob_pattern
|
str
|
Glob pattern to match migration files. |
required |
conflict_strategy
|
MigrationConflictStrategy | None
|
Strategy for handling modified/removed migrations. |
None
|
Returns:
| Type | Description |
|---|---|
list[Migration]
|
List of NEW migrations in dependency-resolved execution order. |
Source code in src/cloe_delta_table_manager/migrations/runner/file.py
IgnoreConflict
¶
Bases: MigrationConflictStrategy
Silently ignores modified or removed migrations and continues execution.
Source code in src/cloe_delta_table_manager/migrations/conflict_strategy.py
on_modified(migrations)
¶
Migration
¶
Bases: BaseModel
Represents a database migration with operations and dependencies.
Source code in src/cloe_delta_table_manager/migrations/migration.py
operation_hash
property
¶
Compute a deterministic migration id as a SHA-256 hash of all operations.
Uses hashlib instead of Python's built-in hash() to guarantee identical values across processes regardless of PYTHONHASHSEED.
__repr__()
¶
Detailed string representation showing id, status, number of operations, and dependencies.
Source code in src/cloe_delta_table_manager/migrations/migration.py
__str__()
¶
from_diff(diff, on_unsupported='fail', platform_options=None)
classmethod
¶
Create a Migration object from a typed Diff object.
Source code in src/cloe_delta_table_manager/migrations/migration.py
MigrationConflictStrategy
¶
Bases: ABC
Abstract base class defining how to react to modified or removed migrations.
Source code in src/cloe_delta_table_manager/migrations/conflict_strategy.py
on_modified(migrations)
abstractmethod
¶
Handle migrations that have been modified since the last execution.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
migrations
|
list[Migration]
|
Modified migrations detected by the runner. |
required |
Source code in src/cloe_delta_table_manager/migrations/conflict_strategy.py
on_removed(migrations)
abstractmethod
¶
Handle migrations that have been removed since the last execution.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
migrations
|
list[Migration]
|
Removed migrations detected by the runner. |
required |
Source code in src/cloe_delta_table_manager/migrations/conflict_strategy.py
MigrationOrchestrator
¶
Orchestrates the migration process by coordinating loading and running of migrations.
Source code in src/cloe_delta_table_manager/migrations/migration_orchestrator.py
79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 | |
__init__(config, target_platform='azure_databricks', connector=None, state_client=None, platform_options=None)
¶
Initialize the MigrationOrchestrator.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
MigrationOrchestratorConfig
|
Migration paths and state file configuration. |
required |
target_platform
|
Literal['azure_databricks']
|
Target platform for migrations (for now, only "azure_databricks" is supported). |
'azure_databricks'
|
state_client
|
StateClient | None
|
Client for reading/writing state. Defaults to LocalStateClient. |
None
|
connector
|
DatabricksConnector | None
|
Databricks connector. Required to call |
None
|
platform_options
|
AzureDatabricksOptions | None
|
Platform-specific render-time options forwarded to the auto runner. |
None
|
Source code in src/cloe_delta_table_manager/migrations/migration_orchestrator.py
collect_pending_migrations(current_database_state)
¶
Return pending migrations across all phases in deployment order without executing them.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
current_database_state
|
DatabaseState
|
Live database state fetched from the target. |
required |
Returns:
| Type | Description |
|---|---|
list[tuple[str, list[Migration]]]
|
List of |
list[tuple[str, list[Migration]]]
|
and post-deployment phases. Migrations within each phase are in dependency order. |
Source code in src/cloe_delta_table_manager/migrations/migration_orchestrator.py
compare_sql_file_state_with_database_statefile()
¶
Compare the SQL file state with the database state and determine which migrations need to be run.
Source code in src/cloe_delta_table_manager/migrations/migration_orchestrator.py
deploy(current_database_state)
¶
Orchestrate the migration process: pre-deployment → auto → post-deployment.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
current_database_state
|
DatabaseState
|
Live database state fetched from the target. |
required |
Source code in src/cloe_delta_table_manager/migrations/migration_orchestrator.py
MigrationOrchestratorConfig
¶
Bases: BaseModel
Configure migration paths and state file locations for the orchestrator.
Source code in src/cloe_delta_table_manager/migrations/migration_orchestrator.py
MigrationRunner
¶
Bases: ABC
Manages reading, ordering, and executing database migrations.
Source code in src/cloe_delta_table_manager/migrations/runner/base.py
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 | |
applied_migrations
property
¶
Get list of applied migrations.
Returns:
| Type | Description |
|---|---|
list[Migration]
|
List of migrations that are in the APPLIED state. |
failed_migrations
property
¶
Get list of failed migrations.
Returns:
| Type | Description |
|---|---|
list[Migration]
|
List of migrations that are in the FAILED state. |
file_migrations
property
¶
Get list of file-based migrations only.
Returns:
| Type | Description |
|---|---|
list[FileMigration]
|
List of FileMigration instances. |
new_migrations
property
¶
Get list of new migrations.
Returns:
| Type | Description |
|---|---|
list[Migration]
|
List of migrations that are in the NEW state. |
pending_migrations
property
¶
Get list of pending migrations.
Returns:
| Type | Description |
|---|---|
list[Migration]
|
List of migrations that are in the PENDING state. |
__init__(executor=None, state_client=None)
¶
Initialize the MigrationRunner.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
executor
|
SQLExecutor | None
|
SQL executor for running statements against Databricks. |
None
|
state_client
|
StateClient | None
|
Client for reading/writing state. Defaults to LocalStateClient. |
None
|
Source code in src/cloe_delta_table_manager/migrations/runner/base.py
MigrationStatus
¶
Bases: Enum
Enum representing the status of a migration.
Source code in src/cloe_delta_table_manager/migrations/migration.py
RaiseOnConflict
¶
Bases: MigrationConflictStrategy
Raises ValueError when modified or removed migrations are detected.
This is the default behaviour, preserving strict integrity guarantees.
Source code in src/cloe_delta_table_manager/migrations/conflict_strategy.py
on_modified(migrations)
¶
Raise ValueError listing all modified migrations.
Source code in src/cloe_delta_table_manager/migrations/conflict_strategy.py
on_removed(migrations)
¶
Raise ValueError listing all removed migrations.
Source code in src/cloe_delta_table_manager/migrations/conflict_strategy.py
WarnOnConflict
¶
Bases: MigrationConflictStrategy
Emits a :mod:warnings warning and continues when conflicts are detected.
Source code in src/cloe_delta_table_manager/migrations/conflict_strategy.py
on_modified(migrations)
¶
Emit a warning listing all modified migrations.
Source code in src/cloe_delta_table_manager/migrations/conflict_strategy.py
on_removed(migrations)
¶
Emit a warning listing all removed migrations.