Building Blocks¶
Below is a list of building blocks currently available to projects and developers within the CLOE toolbox. Each block is designed to streamline development, enhance functionality, and provide robust solutions to common challenges.
Architecture Overview¶
Our toolbox follows a three-tier architecture designed for maximum reusability and maintainability:
graph TD
subgraph "Type 3: Project-Specific"
A[Data Masking Tools]
B[Access Control]
C[Custom Validators]
end
subgraph "Type 2: Intermediary"
D[Database Crawlers]
E[API Clients]
F[Data Processors]
end
subgraph "Type 1: Foundational"
G[Database Connectors]
H[Git Clients]
I[Authentication]
end
A --> D
B --> D
C --> E
D --> G
D --> H
E --> G
E --> I
F --> GBuilding Block Types¶
The following blocks are available (More info here):
- Type 1: Foundational Building Blocks - Core infrastructure components used by other packages
- Type 2: Intermediary Building Blocks - Versatile components for both projects and other packages
- Type 3: Project-Specific Building Blocks - Specialized solutions for specific project challenges
Type 3¶
-
DBT Project Template
Kickstarts and sets up a new dbt environment with devcontainer and VSCode extensions.
Tags: dbt, build tools, sql
-
Azure Data Factory Auto Config
Automatically configures Azure Data Factory using metadata.
Tags: adf, azure data factory, etl, elt, metadata-driven
-
Snowflake Policy Creator
Creates Snowflake policies like data masking using metadata.
Tags: policies, snowflake
-
Nessy
Builds a lakehouse in Databricks.
Tags: spark, Databricks, Fabric
-
Python Package Template
Quickly sets up a Python environment with devcontainers and pre-commit hooks.
Tags: python, template, package
-
Snowflake Clone Creator
Easily and automatically creates customizable Snowflake database clones for feature-driven development.
Tags: clone, cicd, ci/cd, snowflake
-
Snowflake RBAC Tool
Automates the lifecycle of role-based access control in Snowflake.
Tags: snowflake, security, rbac, metadata-driven
-
SQL Composer
Simplifies the creation of SQL for Spark or Snowflake using a simplified metadata format.
Tags: transform, metadata-driven, simplified
-
SQL Transformations
Provides various transformations using the standard metadata format, including automatic lookups and complex table mappings.
Tags: transform, metadata-driven, sql
-
CLOE GUI
A GUI with many features to simplify and automate writing and reading of metadata.
Tags: GUI, metadata, api, utility
-
Snowflake Firewall Updater
A tool that automatically updates the Snowflake firewall with the IPs from the Azure Service Tags.
Tags: snowflake, network, terraform, utility, data-platform
-
Synthetic Data Generator
A tool that automatically generates synthetic data.
Tags: data generation, synthetic, test, utility, databricks
Type 2¶
-
ADF Job Template
Provides Azure Data Factory pipeline templates for data extraction and transfer, deployable with Terraform.
Tags: adf, template, templates
-
Logging
Provides a standardized logging interface for Python projects.
Tags: python, logging, utility
-
DDL Generator
Generates DDLs for creating tables and procedures in Snowflake from metadata.
Tags: ddl, metadata to code, code, sql
-
Snowflake Crawler
Crawls a Snowflake instance and saves the information in the common metadata format.
Tags: snowflake, crawler, utility
-
SAP Crawler
Scans a SAP system and saves the information in the common metadata format.
Tags: crawler, sap, utility
Type 1¶
-
API
Adds functionality to the GUI, like auto mapping between columns for use in the GUI.
Tags: GUI, UI
-
Metadata Definition
Holds definitions for the metadata format in Pydantic classes for secure metadata handling.
Tags: metadata, objects
-
Git Client Tool
Offers basic Git features like retrieving files from a specific commit or tag.
Tags: git, utility
-
Snowflake Connector
Provides a standardized interface to connect with Snowflake, including CI/CD scenarios and various authentication methods.
Tags: snowflake, connector, utility
Legacy Tools¶
-
Airflow Components
Adds additional functionality to Airflow. Like new tasks
Tags: Tag1, Tag2
-
Airflow Metadata Configurator
Tool to create DAG from metadata in Snowflake. Needs the airflow components.
Tags: Tag1, Tag2
-
MSSQL Crawler
Tool to crawl MSSQL instances and save information into metadata.
Tags: Tag1, Tag2
-
MSSQL Connector
Provides a standardized interface to connect with MSSQL, including CI/CD scenarios and various authentication methods.
Tags: Tag1, Tag2
-
SQL Orchestrator
Metadata-driven standalone SQL Orchestrator for executing stored procedures on a SQL server.
Tags: Tag1, Tag2
-
Information Schema Converter
Converts a information Schema CSV export to metadata.
Tags: Tag1, Tag2