Skip to main content
Version: 0.14.0

DataHubGc

Testing

CLI based Ingestion

Install the Plugin

The datahub-gc source works out of the box with acryl-datahub.

Config Details

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
cleanup_expired_tokens
boolean
Whether to clean up expired tokens or not
Default: True
truncate_index_older_than_days
integer
Indices older than this number of days will be truncated
Default: 30
truncate_indices
boolean
Whether to truncate elasticsearch indices or not which can be safely truncated
Default: True
truncation_sleep_between_seconds
integer
Sleep between truncation monitoring.
Default: 30
truncation_watch_until
integer
Wait for truncation of indices until this number of documents are left
Default: 10000

Code Coordinates

  • Class Name: datahub.ingestion.source.gc.datahub_gc.DataHubGcSource
  • Browse on GitHub

Questions

If you've got any questions on configuring ingestion for DataHubGc, feel free to ping us on our Slack.