Skip to main content

Athena

For context on getting started with ingestion, check out our metadata ingestion guide.

Setup#

To install this plugin, run pip install 'acryl-datahub[athena]'.

Capabilities#

This plugin extracts the following:

  • Metadata for databases, schemas, and tables
  • Column types associated with each table
  • Table, row, and column statistics via optional SQL profiling

Quickstart recipe#

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:  type: athena  config:    # Coordinates    aws_region: my_aws_region_name    work_group: my_work_group
    # Credentials    username: my_aws_access_key_id    password: my_aws_secret_access_key    database: my_database
    # Options    s3_staging_dir: "s3://<bucket-name>/<folder>/"
sink:  # sink configs

Config details#

Note that a . is used to denote nested fields in the YAML recipe.

As a SQL-based service, the Athena integration is also supported by our SQL profiler. See here for more details on configuration.

FieldRequiredDefaultDescription
usernameAutodetectedUsername credential. If not specified, detected with boto3 rules. See https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
passwordAutodetectedSame detection scheme as username
databaseAutodetected
aws_regionโœ…AWS region code.
s3_staging_dirโœ…Of format "s3://<bucket-name>/prefix/". The s3_staging_dir parameter is needed because Athena always writes query results to S3.
See https://docs.aws.amazon.com/athena/latest/ug/querying.html.
work_groupโœ…Name of Athena workgroup.
See https://docs.aws.amazon.com/athena/latest/ug/manage-queries-control-costs-with-workgroups.html.
env"PROD"Environment to use in namespace when constructing URNs.
options.<option>Any options specified here will be passed to SQLAlchemy's create_engine as kwargs.
See https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine for details.
table_pattern.allowList of regex patterns for tables to include in ingestion.
table_pattern.denyList of regex patterns for tables to exclude from ingestion.
table_pattern.ignoreCaseTrueWhether regex matching should ignore case or not
schema_pattern.allowList of regex patterns for schemas to include in ingestion.
schema_pattern.denyList of regex patterns for schemas to exclude from ingestion.
schema_pattern.ignoreCaseTrueWhether regex matching should ignore case or not
view_pattern.allowList of regex patterns for views to include in ingestion.
view_pattern.denyList of regex patterns for views to exclude from ingestion.
view_pattern.ignoreCaseTrueWhether regex matching should ignore case or not
include_tablesTrueWhether tables should be ingested.

Compatibility#

Coming soon!

Questions#

If you've got any questions on configuring this source, feel free to ping us on our Slack!