For context on getting started with ingestion, check out our metadata ingestion guide.
To install this plugin, run
pip install 'acryl-datahub[nifi]'.
This plugin extracts the following:
- Nifi flow as
- Ingress, egress processors, remote input and output ports as
- Input and output ports receiving remote connections as
- Lineage information between external datasets and ingress/egress processors by analyzing provenance events
- Limited ingress/egress processors are supported
Check out the following recipe to get started with ingestion! See below for full configuration options.
For general pointers on writing and running a recipe, see our main recipe guide.
# sink configs
Note that a
. is used to denote nested fields in the YAML recipe.
|✅||URI to connect|
|Site name to identify this site with, useful when using input and output ports receiving remote connections|
|Nifi authentication. must be one of : NO_AUTH, SINGLE_USER, CLIENT_CERT|
|Nifi username, must be set for |
|Nifi password, must be set for |
|Path to PEM file containing the public certificates for the user/client identity, must be set for |
|Path to PEM file containing the client’s secret key|
|The password to decrypt the client_key_file|
|Path to PEM file containing certs for the root CA(s) for the NiFi|
|time window to analyze provenance events for external datasets|
|Lookup to find site_name for site_url, required if using remote process groups in nifi flow|
|List of regex patterns for process groups to include in ingestion.|
|List of regex patterns for process groups to exclude from ingestion.|
|Whether to ignore case sensitivity during pattern matching.|
|Environment to use in namespace when constructing URNs.|
If you've got any questions on configuring this source, feel free to ping us on our Slack!