For context on getting started with ingestion, check out our metadata ingestion guide.
acryl-datahub out of the box.
This plugin pulls metadata from dbt's artifact files:
- dbt manifest file
- This file contains model, source and lineage data.
- dbt catalog file
- This file contains schema data.
- dbt does not record schema data for Ephemeral models, as such datahub will show Ephemeral models in the lineage, however there will be no associated schema for Ephemeral models
- dbt sources file
- This file contains metadata for sources with freshness checks.
- We transfer dbt's freshness checks to DataHub's last-modified fields.
- Note that this file is optional – if not specified, we'll use time of ingestion instead as a proxy for time last-modified.
- The data platform you are enriching with dbt metadata.
- data platforms
- Load schemas from dbt catalog file, not necessary when the underlying data platform already has this data.
- Use model identifier instead of model name if defined (if not, default to model name).
- Use this filter to exclude and include node types using allow or deny method
Check out the following recipe to get started with ingestion! See below for full configuration options.
For general pointers on writing and running a recipe, see our main recipe guide.
source: type: "dbt" config: # Coordinates manifest_path: "./path/dbt/manifest_file.json" catalog_path: "./path/dbt/catalog_file.json" sources_path: "./path/dbt/sources_file.json" # Options target_platform: "my_target_platform_id" load_schemas: True # note: if this is disabled sink: # sink configs
Note that a
. is used to denote nested fields in the YAML recipe.
|✅||Path to dbt manifest JSON. See https://docs.getdbt.com/reference/artifacts/manifest-json|
|✅||Path to dbt catalog JSON. See https://docs.getdbt.com/reference/artifacts/catalog-json|
|Path to dbt sources JSON. See https://docs.getdbt.com/reference/artifacts/sources-json. If not specified, last-modified fields will not be populated.|
|Environment to use in namespace when constructing URNs.|
|✅||The platform that dbt is loading onto.|
|✅||Whether to load database schemas. If set to |
|Whether to use model identifiers instead of names, if defined (if not, default to names)|
|List of regex patterns for dbt nodes to include in ingestion.|
|List of regex patterns for dbt nodes to exclude from ingestion.|
|Whether to ignore case sensitivity during pattern matching.|
If you've got any questions on configuring this source, feel free to ping us on our Slack!