Skip to main content

Snowflake

For context on getting started with ingestion, check out our metadata ingestion guide.

Setup#

To install this plugin, run pip install 'acryl-datahub[snowflake]'.

Capabilities#

This plugin extracts the following:

  • Metadata for databases, schemas, views and tables
  • Column types associated with each table
  • Table, row, and column statistics via optional SQL profiling
tip

You can also get fine-grained usage statistics for Snowflake using the snowflake-usage source described below.

Quickstart recipe#

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:  type: snowflake  config:    # Coordinates    host_port: account_name    warehouse: "COMPUTE_WH"
    # Credentials    username: user    password: pass    role: "sysadmin"
sink:  # sink configs

Config details#

Note that a . is used to denote nested fields in the YAML recipe.

As a SQL-based service, the Athena integration is also supported by our SQL profiler. See here for more details on configuration.

FieldRequiredDefaultDescription
usernameSnowflake username.
passwordSnowflake password.
host_portโœ…Snowflake host URL.
warehouseSnowflake warehouse.
roleSnowflake role.
env"PROD"Environment to use in namespace when constructing URNs.
options.<option>Any options specified here will be passed to SQLAlchemy's create_engine as kwargs.
See https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine for details.
database_pattern.allowList of regex patterns for databases to include in ingestion.
database_pattern.deny"^UTIL_DB$"
"^SNOWFLAKE$"
"^SNOWFLAKE_SAMPLE_DATA$"
List of regex patterns for databases to exclude from ingestion.
database_pattern.ignoreCaseTrueWhether to ignore case sensitivity during pattern matching.
table_pattern.allowList of regex patterns for tables to include in ingestion.
table_pattern.denyList of regex patterns for tables to exclude from ingestion.
table_pattern.ignoreCaseTrueWhether to ignore case sensitivity during pattern matching.
schema_pattern.allowList of regex patterns for schemas to include in ingestion.
schema_pattern.denyList of regex patterns for schemas to exclude from ingestion.
schema_pattern.ignoreCaseTrueWhether to ignore case sensitivity during pattern matching.
view_pattern.allowList of regex patterns for views to include in ingestion.
view_pattern.denyList of regex patterns for views to exclude from ingestion.
view_pattern.ignoreCaseTrueWhether to ignore case sensitivity during pattern matching.
include_tablesTrueWhether tables should be ingested.
include_viewsTrueWhether views should be ingested.

Compatibility#

Coming soon!

Snowflake Usage Stats#

For context on getting started with ingestion, check out our metadata ingestion guide.

Setup#

To install this plugin, run pip install 'acryl-datahub[snowflake-usage]'.

Capabilities#

This plugin extracts the following:

  • Statistics on queries issued and tables and columns accessed (excludes views)
  • Aggregation of these statistics into buckets, by day or hour granularity

Note: the user/role must have access to the account usage table. The "accountadmin" role has this by default, and other roles can be granted this permission.

Note: the underlying access history views that we use are only available in Snowflake's enterprise edition or higher.

note

This source only does usage statistics. To get the tables, views, and schemas in your Snowflake warehouse, ingest using the snowflake source described above.

Quickstart recipe#

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:  type: snowflake-usage  config:    # Coordinates    host_port: account_name    warehouse: "COMPUTE_WH"
    # Credentials    username: user    password: pass    role: "sysadmin"
    # Options    top_n_queries: 10
sink:  # sink configs

Config details#

Note that a . is used to denote nested fields in the YAML recipe.

FieldRequiredDefaultDescription
usernameSnowflake username.
passwordSnowflake password.
host_portโœ…Snowflake host URL.
warehouseSnowflake warehouse.
roleSnowflake role.
env"PROD"Environment to use in namespace when constructing URNs.
bucket_duration"DAY"Duration to bucket usage events by. Can be "DAY" or "HOUR".
start_timeLast full day in UTC (or hour, depending on bucket_duration)Earliest date of usage logs to consider.
end_timeLast full day in UTC (or hour, depending on bucket_duration)Latest date of usage logs to consider.
top_n_queries10Number of top queries to save to each table.

Compatibility#

Coming soon!

Questions#

If you've got any questions on configuring this source, feel free to ping us on our Slack!