Skip to main content
Version: Next

DynamoDB

Testing

Important Capabilities​

CapabilityStatusNotes
Detect Deleted Entities✅Optionally enabled via stateful_ingestion.remove_stale_metadata
Platform Instance✅By default, platform_instance will use the AWS account id

This plugin extracts the following:

AWS DynamoDB table names with their region, and infer schema of attribute names and types by scanning the table

Prerequisities​

In order to execute this source, you need to attach the AmazonDynamoDBReadOnlyAccess policy to a user in your AWS account. Then create an API access key and secret for the user.

For a user to be able to create API access key, it needs the following access key permissions. Your AWS account admin can create a policy with these permissions and attach to the user, you can find more details in Managing access keys for IAM users

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "iam:ListAccessKeys",
        "iam:CreateAccessKey",
        "iam:UpdateAccessKey",
        "iam:DeleteAccessKey"
      ],
      "Resource": "arn:aws:iam::${aws_account_id}:user/${aws:username}"
    }
  ]
}

CLI based Ingestion​

Install the Plugin​

pip install 'acryl-datahub[dynamodb]'

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: dynamodb
  config:
    platform_instance: "AWS_ACCOUNT_ID"
    aws_access_key_id: "${AWS_ACCESS_KEY_ID}"
    aws_secret_access_key: "${AWS_SECRET_ACCESS_KEY}"
    #
    # If there are items that have most representative fields of the table, users could use the
    # `include_table_item` option to provide a list of primary keys of the table in dynamodb format.
    # For each `region.table`, the list of primary keys can be at most 100.
    # We include these items in addition to the first 100 items in the table when we scan it.
    #
    # include_table_item:
    #   region.table_name:
    #     [
    #       {
    #         "partition_key_name": { "attribute_type": "attribute_value" },
    #         "sort_key_name": { "attribute_type": "attribute_value" },
    #       },
    #     ]

sink:
  # sink configs

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
aws_access_key_id ✅
string
AWS Access Key ID.
aws_secret_access_key ✅
string(password)
AWS Secret Key.
platform_instance
string
The instance of the platform that all assets produced by this recipe belong to
env
string
The environment that all assets produced by this connector belong to
Default: PROD
classification
ClassificationConfig
For details, refer Classification.
Default: {'enabled': False, 'sample_size': 100, 'max_worker...
classification.enabled
boolean
Whether classification should be used to auto-detect glossary terms
Default: False
classification.info_type_to_term
map(str,string)
classification.max_workers
integer
Number of worker threads to use for classification. Set to 1 to disable.
Default: 4
classification.sample_size
integer
Number of sample values used for classification.
Default: 100
classification.classifiers
array
Classifiers to use to auto-detect glossary terms. If more than one classifier, infotype predictions from the classifier defined later in sequence take precedance.
Default: [{'type': 'datahub', 'config': None}]
classification.classifiers.DynamicTypedClassifierConfig
DynamicTypedClassifierConfig
classification.classifiers.DynamicTypedClassifierConfig.type ❓
string
The type of the classifier to use. For DataHub, use datahub
classification.classifiers.DynamicTypedClassifierConfig.config
object
The configuration required for initializing the classifier. If not specified, uses defaults for classifer type.
classification.column_pattern
AllowDenyPattern
Regex patterns to filter columns for classification. This is used in combination with other patterns in parent config. Specify regex to match the column name in database.schema.table.column format.
Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True}
classification.column_pattern.ignoreCase
boolean
Whether to ignore case sensitivity during pattern matching.
Default: True
classification.column_pattern.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
classification.column_pattern.allow.string
string
classification.column_pattern.deny
array
List of regex patterns to exclude from ingestion.
Default: []
classification.column_pattern.deny.string
string
classification.table_pattern
AllowDenyPattern
Regex patterns to filter tables for classification. This is used in combination with other patterns in parent config. Specify regex to match the entire table name in database.schema.table format. e.g. to match all tables starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'
Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True}
classification.table_pattern.ignoreCase
boolean
Whether to ignore case sensitivity during pattern matching.
Default: True
classification.table_pattern.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
classification.table_pattern.allow.string
string
classification.table_pattern.deny
array
List of regex patterns to exclude from ingestion.
Default: []
classification.table_pattern.deny.string
string
domain
map(str,AllowDenyPattern)
A class to store allow deny regexes
domain.key.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
domain.key.allow.string
string
domain.key.ignoreCase
boolean
Whether to ignore case sensitivity during pattern matching.
Default: True
domain.key.deny
array
List of regex patterns to exclude from ingestion.
Default: []
domain.key.deny.string
string
include_table_item
map(str,array)
include_table_item.key.object
object
table_pattern
AllowDenyPattern
Regex patterns for tables to filter in ingestion. The table name format is 'region.table'
Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True}
table_pattern.ignoreCase
boolean
Whether to ignore case sensitivity during pattern matching.
Default: True
table_pattern.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
table_pattern.allow.string
string
table_pattern.deny
array
List of regex patterns to exclude from ingestion.
Default: []
table_pattern.deny.string
string
stateful_ingestion
StatefulStaleMetadataRemovalConfig
Base specialized config for Stateful Ingestion with stale metadata removal capability.
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Advanced Configurations​

Using include_table_item config​

If there are items that have most representative fields of the table, users could use the include_table_item option to provide a list of primary keys of the table in dynamodb format. We include these items in addition to the first 100 items in the table when we scan it.

Take AWS DynamoDB Developer Guide Example tables and data as an example, if a account has a table Reply in the us-west-2 region with composite primary key Id and ReplyDateTime, users can use include_table_item to include 2 items as following:

Example:

# The table name should be in the format of region.table_name
# The primary keys should be in the DynamoDB format
include_table_item:
  us-west-2.Reply:
    [
      {
        "ReplyDateTime": { "S": "2015-09-22T19:58:22.947Z" },
        "Id": { "S": "Amazon DynamoDB#DynamoDB Thread 1" },
      },
      {
        "ReplyDateTime": { "S": "2015-10-05T19:58:22.947Z" },
        "Id": { "S": "Amazon DynamoDB#DynamoDB Thread 2" },
      },
    ]

Code Coordinates​

  • Class Name: datahub.ingestion.source.dynamodb.dynamodb.DynamoDBSource
  • Browse on GitHub

Questions

If you've got any questions on configuring ingestion for DynamoDB, feel free to ping us on our Slack.