Skip to main content

Glue

For context on getting started with ingestion, check out our metadata ingestion guide.

Setup#

To install this plugin, run pip install 'acryl-datahub[glue]'.

Note: if you also have files in S3 that you'd like to ingest, we recommend you use Glue's built-in data catalog. See here for a quick guide on how to set up a crawler on Glue and ingest the outputs with DataHub.

Capabilities#

This plugin extracts the following:

  • Tables in the Glue catalog
  • Column types associated with each table
  • Table metadata, such as owner, description and parameters
  • Jobs and their component transformations, data sources, and data sinks

Quickstart recipe#

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:  type: glue  config:    # Coordinates    aws_region: "my-aws-region"
sink:  # sink configs

Config details#

Note that a . is used to denote nested fields in the YAML recipe.

FieldRequiredDefaultDescription
aws_regionโœ…AWS region code.
env"PROD"Environment to use in namespace when constructing URNs.
aws_access_key_idAutodetectedSee https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
aws_secret_access_keyAutodetectedSee https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
aws_session_tokenAutodetectedSee https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
aws_roleAutodetectedSee https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
extract_transformsTrueWhether to extract Glue transform jobs.
database_pattern.allowList of regex patterns for databases to include in ingestion.
database_pattern.denyList of regex patterns for databases to exclude from ingestion.
database_pattern.ignoreCaseTrueWhether to ignore case sensitivity during pattern matching.
table_pattern.allowList of regex patterns for tables to include in ingestion.
table_pattern.denyList of regex patterns for tables to exclude from ingestion.
table_pattern.ignoreCaseTrueWhether to ignore case sensitivity during pattern matching.
underlying_platformglueOverride for platform name. Allowed values - glue, athena

Compatibility#

Coming soon!

Questions#

If you've got any questions on configuring this source, feel free to ping us on our Slack!