CLOE Util Snowflake Connector Documentation¶
Overview¶
The CLOE Util Snowflake Connector is a Python framework that provides a robust and flexible interface for connecting to and interacting with Snowflake databases. Built on top of the official Snowflake Python connector, this framework simplifies authentication, connection management, and query execution for applications that need to integrate with Snowflake.
Key Features¶
- Multiple Authentication Methods: Supports password-based and key pair authentication
- Flexible Private Key Handling: Accommodates various private key formats (encrypted/unencrypted, with/without headers, different line break formats)
- Azure DevOps Integration: Seamless integration with Azure DevOps pipelines using environment variables and secure files
- Connection Management: Simplified connection initialization and management
- Query Execution: Support for single and batch query execution with optional result retrieval
- Type Safety: Built with Pydantic for robust validation and type checking
Table of Contents¶
- Installation
- Authentication Methods
- Configuration
- Key Pair Authentication Setup
- Usage Examples
- Azure DevOps Integration
- API Reference
- Troubleshooting
Installation¶
Add the package to your project dependencies:
Requirements¶
- Python 3.11 or higher
- Dependencies (automatically installed):
snowflake-connector-python~=3.16.0pydantic~=2.11.7pydantic-settings~=2.10.1snowflake-core~=1.7.0
Authentication Methods¶
The connector supports two primary authentication methods:
1. Password Authentication¶
Username and password-based authentication is the simplest method, but is about to be depricated by Snowflake:
from cloe_util_snowflake_connector import connection_parameters, snowflake_interface
conn_params = connection_parameters.ConnectionParameters(
user="your_username",
password="your_password",
account="your_account.region.cloud",
warehouse="your_warehouse",
database="your_database",
schema="your_schema",
role="your_role"
)
snowflake_conn = snowflake_interface.SnowflakeInterface(conn_params)
2. Key Pair Authentication (Recommended for Production)¶
Key pair authentication provides enhanced security and is recommended for production environments, especially when integrating with CI/CD pipelines.
Advantages: - No password storage or rotation required - More secure for automated processes - Compliant with enterprise security policies - Ideal for Azure DevOps and other CI/CD platforms
See Key Pair Authentication Setup for detailed configuration instructions.
Configuration¶
The connector uses environment variables with the CLOE_SNOWFLAKE_ prefix for configuration. This approach allows for flexible deployment across different environments.
Required Environment Variables¶
| Variable | Description | Example |
|---|---|---|
CLOE_SNOWFLAKE_USER |
Snowflake username | service_account |
CLOE_SNOWFLAKE_ACCOUNT |
Snowflake account identifier | myorg.west-europe.azure |
Optional Environment Variables¶
| Variable | Description | Default |
|---|---|---|
CLOE_SNOWFLAKE_PASSWORD |
Password for authentication | - |
CLOE_SNOWFLAKE_WAREHOUSE |
Default warehouse | - |
CLOE_SNOWFLAKE_DATABASE |
Default database | - |
CLOE_SNOWFLAKE_SCHEMA |
Default schema | - |
CLOE_SNOWFLAKE_ROLE |
Role to assume | - |
CLOE_SNOWFLAKE_AUTOCOMMIT |
Enable/disable autocommit | True |
Key Pair Authentication Variables¶
| Variable | Description | Use Case |
|---|---|---|
CLOE_SNOWFLAKE_PRIVATE_KEY |
Private key as string | Azure DevOps secrets |
CLOE_SNOWFLAKE_PRIVATE_KEY_PASSPHRASE |
Passphrase for encrypted private key | When using encrypted key string |
CLOE_SNOWFLAKE_PRIVATE_KEY_FILE |
Path to private key file | Azure DevOps secure files |
CLOE_SNOWFLAKE_PRIVATE_KEY_FILE_PWD |
Password for encrypted private key file | When using encrypted key file |
Important: You must provide exactly one authentication method:
- CLOE_SNOWFLAKE_PASSWORD, OR
- CLOE_SNOWFLAKE_PRIVATE_KEY, OR
- CLOE_SNOWFLAKE_PRIVATE_KEY_FILE
Providing multiple authentication methods will raise a validation error.
Using .env Files¶
You can store configuration in a .env file:
CLOE_SNOWFLAKE_USER=your_username
CLOE_SNOWFLAKE_PASSWORD=your_password
CLOE_SNOWFLAKE_ACCOUNT=myorg.west-europe.azure
CLOE_SNOWFLAKE_WAREHOUSE=COMPUTE_WH
CLOE_SNOWFLAKE_DATABASE=ANALYTICS_DB
CLOE_SNOWFLAKE_SCHEMA=PUBLIC
CLOE_SNOWFLAKE_ROLE=ANALYST_ROLE
CLOE_SNOWFLAKE_AUTOCOMMIT=True
Load configuration from the file:
from pathlib import Path
from cloe_util_snowflake_connector import connection_parameters, snowflake_interface
conn_params = connection_parameters.ConnectionParameters.init_from_env_variables(env_file=Path(".env"))
snowflake_conn = snowflake_interface.SnowflakeInterface(conn_params)
Key Pair Authentication Setup¶
Key pair authentication is the recommended approach for production environments. This section provides a step-by-step guide to setting it up.
Step 1: Generate Key Pair¶
Generate an RSA key pair using OpenSSL:
# Generate private key (unencrypted)
openssl genrsa -out rsa_key.pem 2048
# Generate private key (encrypted - recommended)
openssl genrsa -out rsa_key.p8 2048
openssl pkcs8 -topk8 -inform PEM -in rsa_key.pem -out rsa_key.p8 -v2 des3
# Generate public key
openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub
Step 2: Register Public Key in Snowflake¶
- Get the public key content (excluding headers/footers):
# View public key
cat rsa_key.pub
# Extract just the key content (remove headers)
grep -v "BEGIN PUBLIC KEY" rsa_key.pub | grep -v "END PUBLIC KEY" | tr -d '\n'
- Assign the public key to your Snowflake user:
-- As a user with appropriate privileges (SECURITYADMIN or ACCOUNTADMIN)
ALTER USER your_username SET RSA_PUBLIC_KEY='MIIBIjANBgkq...'; -- Your public key here
-- Verify the key is set
DESC USER your_username;
Step 3: Configure the Connector¶
The connector supports multiple private key formats to accommodate different deployment scenarios:
Format 1: Multiline String with Headers (Standard PEM)¶
Best for: Secure files in Azure DevOps
CLOE_SNOWFLAKE_PRIVATE_KEY="-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDGqwy4wwnBWFlH
hQ9xNF4Q/xk9nggb28M7Tqh8TzkRb5DOhMYvhuq/ut6vWr4Wzx6108FA5pVQpmRp
...
0OoaOEQzasKe5Ltt1DEWA+D8BYmWDtGDeo8Kd4ncf1ZjmUpi9G8b8G05NiF0cztH
4ZmdBEp3WUxmCDvyn2I2wJY=
-----END PRIVATE KEY-----"
Format 2: Single Line with Escaped Line Breaks¶
Best for: Azure DevOps pipeline secrets
CLOE_SNOWFLAKE_PRIVATE_KEY="-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDGqwy4wwnBWFlH\nhQ9xNF4Q/xk9nggb28M7Tqh8TzkRb5DOhMYvhuq/ut6vWr4Wzx6108FA5pVQpmRp\n...\n0OoaOEQzasKe5Ltt1DEWA+D8BYmWDtGDeo8Kd4ncf1ZjmUpi9G8b8G05NiF0cztH\n4ZmdBEp3WUxmCDvyn2I2wJY=\n-----END PRIVATE KEY-----"
Format 3: Base64 Content Only (No Headers)¶
Best for: Simplified secret storage
CLOE_SNOWFLAKE_PRIVATE_KEY="MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDGqwy4wwnBWFlHhQ9xNF4Q/xk9nggb28M7Tqh8TzkRb5DOhMYvhuq/ut6vWr4Wzx6108FA5pVQpmRp...0OoaOEQzasKe5Ltt1DEWA+D8BYmWDtGDeo8Kd4ncf1ZjmUpi9G8b8G05NiF0cztH4ZmdBEp3WUxmCDvyn2I2wJY="
Format 4: Private Key File¶
Best for: Local development or when using Azure DevOps secure files
CLOE_SNOWFLAKE_PRIVATE_KEY_FILE="/path/to/rsa_key.p8"
# If encrypted:
CLOE_SNOWFLAKE_PRIVATE_KEY_FILE_PWD="your_key_password"
Step 4: Handle Encrypted Keys¶
If your private key is encrypted (recommended for production), provide the passphrase:
For key string:
CLOE_SNOWFLAKE_PRIVATE_KEY="-----BEGIN ENCRYPTED PRIVATE KEY-----\n..."
CLOE_SNOWFLAKE_PRIVATE_KEY_PASSPHRASE="your_passphrase"
For key file:
CLOE_SNOWFLAKE_PRIVATE_KEY_FILE="/path/to/encrypted_key.p8"
CLOE_SNOWFLAKE_PRIVATE_KEY_FILE_PWD="your_passphrase"
Testing Key Pair Authentication¶
from pathlib import Path
from cloe_util_snowflake_connector import connection_parameters, snowflake_interface
# Initialize from environment variables
conn_params = connection_parameters.ConnectionParameters.init_from_env_variables()
# Create connection
snowflake_conn = snowflake_interface.SnowflakeInterface(conn_params)
# Test the connection
snowflake_conn.test_connection()
print("Connection successful!")
# Close connection
snowflake_conn.close()
Usage Examples¶
Basic Connection and Query Execution¶
from cloe_util_snowflake_connector import connection_parameters, snowflake_interface
# Initialize connection
conn_params = connection_parameters.ConnectionParameters.init_from_env_variables()
snowflake_conn = snowflake_interface.SnowflakeInterface(conn_params)
# Test connection
snowflake_conn.test_connection()
# Execute a single query with results
result = snowflake_conn.run_one_with_return("SELECT * FROM my_table LIMIT 10")
print(result) # Returns list of dictionaries
# Close connection
snowflake_conn.close()
Batch Query Execution¶
# Execute multiple queries without returning results
queries = [
"CREATE TABLE IF NOT EXISTS test_table (id INT, name STRING)",
"INSERT INTO test_table VALUES (1, 'Alice')",
"INSERT INTO test_table VALUES (2, 'Bob')"
]
snowflake_conn.run_many(queries)
Batch Queries with Results¶
# Execute multiple queries and get all results
queries_dict = {
"users": "SELECT * FROM users",
"orders": "SELECT * FROM orders WHERE date = CURRENT_DATE()",
"products": "SELECT * FROM products WHERE active = TRUE"
}
results = snowflake_conn.run_many_with_return(queries_dict)
# Access results by query key
user_data = results["users"]
order_data = results["orders"]
product_data = results["products"]
Using Session Parameters¶
from cloe_util_snowflake_connector import connection_parameters, snowflake_interface
# Set query tag for tracking
session_params = connection_parameters.SessionParameters(query_tag="DataPipeline_ETL")
conn_params = connection_parameters.ConnectionParameters.init_from_env_variables()
conn_params.session_parameters = session_params
snowflake_conn = snowflake_interface.SnowflakeInterface(conn_params)
Advanced: Using Snowflake Core API¶
The connector provides access to the underlying Snowflake connection for advanced use cases:
# Get root object for Snowflake Core API
root = snowflake_conn.get_root_object()
# Get raw connection object
connection = snowflake_conn.get_connection_object()
# Use for advanced Snowflake operations
databases = root.databases.iter()
for db in databases:
print(db.name)
Azure DevOps Integration¶
The connector is designed to work seamlessly with Azure DevOps pipelines, supporting both secrets and secure files.
Method 1: Using Pipeline Secrets¶
This method stores the private key as an Azure DevOps secret variable.
Step 1: Prepare Your Private Key
Convert your private key to a single-line format:
# For unencrypted key
cat rsa_key.pem | tr '\n' '\\n'
# Result example:
# -----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkq...\n-----END PRIVATE KEY-----
Step 2: Add Secret to Azure DevOps
- Go to your Azure DevOps project
- Navigate to Pipelines → Library → Variable groups (or Pipeline → Edit → Variables)
- Add a new secret variable:
- Name:
CLOE_SNOWFLAKE_PRIVATE_KEY - Value: Your single-line private key (click the lock icon to mark as secret)
- If encrypted, also add:
CLOE_SNOWFLAKE_PRIVATE_KEY_PASSPHRASE
Step 3: Configure Pipeline
# azure-pipelines.yml
trigger:
- main
pool:
vmImage: 'ubuntu-latest'
variables:
- group: snowflake-credentials # Reference your variable group
- name: CLOE_SNOWFLAKE_USER
value: 'service_account'
- name: CLOE_SNOWFLAKE_ACCOUNT
value: 'myorg.west-europe.azure'
- name: CLOE_SNOWFLAKE_WAREHOUSE
value: 'COMPUTE_WH'
- name: CLOE_SNOWFLAKE_DATABASE
value: 'ANALYTICS_DB'
- name: CLOE_SNOWFLAKE_SCHEMA
value: 'PUBLIC'
- name: CLOE_SNOWFLAKE_ROLE
value: 'ANALYST_ROLE'
steps:
- task: UsePythonVersion@0
inputs:
versionSpec: '3.11'
displayName: 'Use Python 3.11'
- script: |
pip install cloe-util-snowflake-connector
displayName: 'Install dependencies'
- script: |
python your_script.py
displayName: 'Run Snowflake script'
env:
CLOE_SNOWFLAKE_PRIVATE_KEY: $(CLOE_SNOWFLAKE_PRIVATE_KEY)
CLOE_SNOWFLAKE_PRIVATE_KEY_PASSPHRASE: $(CLOE_SNOWFLAKE_PRIVATE_KEY_PASSPHRASE)
Method 2: Using Secure Files¶
This method stores the private key as a secure file in Azure DevOps.
Step 1: Upload Secure File
- Go to Pipelines → Library → Secure files
- Click "+ Secure file"
- Upload your private key file (e.g.,
rsa_key.p8)
Step 2: Configure Pipeline
# azure-pipelines.yml
trigger:
- main
pool:
vmImage: 'ubuntu-latest'
variables:
- name: CLOE_SNOWFLAKE_USER
value: 'service_account'
- name: CLOE_SNOWFLAKE_ACCOUNT
value: 'myorg.west-europe.azure'
- name: CLOE_SNOWFLAKE_WAREHOUSE
value: 'COMPUTE_WH'
steps:
- task: DownloadSecureFile@1
name: privateKey
displayName: 'Download private key'
inputs:
secureFile: 'rsa_key.p8'
- task: UsePythonVersion@0
inputs:
versionSpec: '3.11'
displayName: 'Use Python 3.11'
- script: |
pip install cloe-util-snowflake-connector
displayName: 'Install dependencies'
- script: |
python your_script.py
displayName: 'Run Snowflake script'
env:
CLOE_SNOWFLAKE_PRIVATE_KEY_FILE: $(privateKey.secureFilePath)
CLOE_SNOWFLAKE_PRIVATE_KEY_FILE_PWD: $(SNOWFLAKE_KEY_PASSWORD) # From variable group
Best Practices for Azure DevOps¶
- Use Variable Groups: Store common configuration in variable groups for reusability
- Mark Secrets: Always mark sensitive variables (keys, passwords) as secrets
- Separate Environments: Use different variable groups for dev, staging, and production
- Rotate Keys: Implement a key rotation policy and update secure files/secrets regularly
- Limit Permissions: Restrict who can view/edit secure files and secret variables
- Use Service Principals: Create dedicated Snowflake users for automated processes
API Reference¶
ConnectionParameters¶
Main class for configuring Snowflake connections.
Attributes:
- user (str): Snowflake username
- password (str | None): Password for authentication
- private_key (bytes | None): Private key in bytes format
- private_key_file (str | None): Path to private key file
- private_key_file_pwd (str | None): Password for encrypted private key file
- account (str): Snowflake account identifier
- warehouse (str | None): Default warehouse
- database (str | None): Default database
- schema (str | None): Default schema
- role (str | None): Role to assume
- autocommit (bool | None): Enable autocommit (default: True)
- session_parameters (SessionParameters | None): Additional session parameters
Methods:
@classmethod
def init_from_env_variables(cls, env_file: Path = Path(".env")) -> ConnectionParameters
Parameters:
- env_file (Path): Path to .env file (default: ".env")
Returns: ConnectionParameters instance
Example:
from pathlib import Path
from cloe_util_snowflake_connector import connection_parameters
conn_params = connection_parameters.ConnectionParameters.init_from_env_variables(env_file=Path("config/.env"))
SnowflakeInterface¶
Main interface for interacting with Snowflake.
Initialization:
Parameters:
- connection_params (ConnectionParameters): Connection configuration
Methods:
test_connection()¶
Tests the Snowflake connection by executing a simple query.Raises: Exception if connection fails
Example:
run_one_with_return()¶
Execute a single query and return results.Parameters:
- query (str): SQL query to execute
Returns: List of dictionaries, where each dictionary represents a row
Example:
results = snowflake_conn.run_one_with_return("SELECT * FROM users LIMIT 5")
for row in results:
print(row['user_id'], row['username'])
run_many()¶
Execute multiple queries asynchronously without returning results.Parameters:
- queries (list[str]): List of SQL queries to execute
- continue_on_error (bool): Continue executing remaining queries if one fails (default: True)
Example:
queries = [
"UPDATE users SET last_login = CURRENT_TIMESTAMP() WHERE user_id = 1",
"INSERT INTO audit_log VALUES (1, 'login', CURRENT_TIMESTAMP())"
]
snowflake_conn.run_many(queries)
run_many_with_return()¶
def run_many_with_return(
self,
queries: dict[Any, str],
continue_on_error: bool = True
) -> dict[Any, list[dict[str, str]] | None]
Parameters:
- queries (dict[Any, str]): Dictionary mapping query IDs to SQL queries
- continue_on_error (bool): Continue if a query fails (default: True)
Returns: Dictionary mapping query IDs to results (or None if query failed)
Example:
queries = {
"active_users": "SELECT * FROM users WHERE active = TRUE",
"recent_orders": "SELECT * FROM orders WHERE order_date >= DATEADD(day, -7, CURRENT_DATE())"
}
results = snowflake_conn.run_many_with_return(queries)
active_users = results["active_users"]
recent_orders = results["recent_orders"]
get_root_object()¶
Get the Snowflake Core API root object for advanced operations.Returns: snowflake.core.Root instance
Example:
get_connection_object()¶
Get the underlying Snowflake connection object.Returns: snowflake.connector.SnowflakeConnection instance
Example:
close()¶
Close the Snowflake connection.Example:
SessionParameters¶
Optional session-level parameters.
Attributes:
- query_tag (str): Tag for query tracking in Snowflake
Example:
from cloe_util_snowflake_connector import connection_parameters
session_params = connection_parameters.SessionParameters(query_tag="ETL_Pipeline_v2")
conn_params.session_parameters = session_params
Troubleshooting¶
Common Issues and Solutions¶
1. Authentication Error: "Multiple authentication methods provided"¶
Error:
Solution: Ensure only one authentication method is configured. Check your environment variables and remove any unused authentication variables:
# For password auth, remove these:
del os.environ['CLOE_SNOWFLAKE_PRIVATE_KEY']
del os.environ['CLOE_SNOWFLAKE_PRIVATE_KEY_FILE']
# For key pair auth, remove this:
del os.environ['CLOE_SNOWFLAKE_PASSWORD']
2. Private Key Format Error¶
Error:
Solution:
- Verify your private key includes proper headers (-----BEGIN PRIVATE KEY-----)
- Check for escaped newlines (\n) vs actual newlines
- If using encrypted key, ensure passphrase is provided
- Try converting your key to standard PEM format:
3. Connection Timeout¶
Error:
Solution:
- Verify your account identifier is correct (e.g., myorg.west-europe.azure)
- Check network connectivity and firewall rules
- Ensure Snowflake account is active
- Try removing .snowflakecomputing.com from account identifier
4. Invalid Account Identifier¶
Error:
Solution:
The connector automatically cleans account identifiers. Ensure you provide it in one of these formats:
- myorg.west-europe.azure
- https://myorg.west-europe.azure.snowflakecomputing.com
Both will be normalized to myorg.west-europe.azure.
5. Autocommit Value Error¶
Error:
Solution: Use string values "True" or "False" (case-sensitive) in environment variables:
6. Private Key Passphrase Issues¶
Error:
Solution: - If your private key is unencrypted, do NOT provide a passphrase - If you need encryption, regenerate an encrypted key:
7. Azure DevOps Secure File Not Found¶
Error:
Solution: Ensure you're using the correct variable reference in your pipeline:
- task: DownloadSecureFile@1
name: snowflakeKey # This name matters
inputs:
secureFile: 'rsa_key.p8'
- script: |
python script.py
env:
CLOE_SNOWFLAKE_PRIVATE_KEY_FILE: $(snowflakeKey.secureFilePath) # Use the task name
Debug Mode¶
Enable detailed logging to troubleshoot connection issues:
import logging
from cloe_util_snowflake_connector import connection_parameters, snowflake_interface
# Enable debug logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger("cloe_util_snowflake_connector")
logger.setLevel(logging.DEBUG)
# Now run your connection code
conn_params = connection_parameters.ConnectionParameters.init_from_env_variables()
snowflake_conn = snowflake_interface.SnowflakeInterface(conn_params)
snowflake_conn.test_connection()
Getting Help¶
If you encounter issues not covered here:
- Check the Snowflake Python Connector documentation
- Review your Snowflake account settings and user permissions
- Verify your environment variables are set correctly
- Contact your Snowflake administrator for account-specific issues
Security Best Practices¶
Key Management¶
- Never commit private keys to version control
- Add
*.pem,*.p8to.gitignore -
Use secure storage (Azure Key Vault, AWS Secrets Manager)
-
Use encrypted private keys in production
-
Rotate keys regularly
- Establish a key rotation policy (e.g., every 90 days)
-
Update Snowflake user with new public key before deploying
-
Limit key access
- Use Azure DevOps permissions to restrict who can view secrets
- Implement separate keys for different environments
Migration Guide¶
Migrating from Password to Key Pair Authentication¶
Step 1: Generate and Register Key Pair Follow the Key Pair Authentication Setup guide.
Step 2: Update Configuration
Before:
CLOE_SNOWFLAKE_USER=myuser
CLOE_SNOWFLAKE_PASSWORD=mypassword
CLOE_SNOWFLAKE_ACCOUNT=myorg.west-europe.azure
After:
CLOE_SNOWFLAKE_USER=myuser
CLOE_SNOWFLAKE_PRIVATE_KEY_FILE=/path/to/rsa_key.p8
CLOE_SNOWFLAKE_PRIVATE_KEY_FILE_PWD=key_password
CLOE_SNOWFLAKE_ACCOUNT=myorg.west-europe.azure
Step 3: Test in Non-Production Test the new authentication method in development/staging before deploying to production.
Step 4: Update CI/CD Pipelines Update Azure DevOps pipelines to use secure files or secret variables instead of password.
Step 5: Remove Password Once confirmed working, remove the password variable and optionally disable password authentication in Snowflake:
Additional Resources¶
- Snowflake Key Pair Authentication Documentation
- Azure DevOps Secure Files
- Azure DevOps Variable Groups
- Snowflake Python Connector
Version Information¶
- Current Version: 1.0.6
- Python Compatibility: 3.11+
- Last Updated: January 2026
For changelog and version history, see the project repository.