For context on getting started with ingestion, check out our metadata ingestion guide.
To install this plugin, run
pip install 'acryl-datahub[kafka-connect]'.
This plugin extracts the following:
- Kafka Connect connector as individual
- Creating individual
- Lineage information between source database to Kafka topic
- Currently works for JDBC and Debezium source connectors only.
Check out the following recipe to get started with ingestion! See below for full configuration options.
For general pointers on writing and running a recipe, see our main recipe guide.
source: type: "kafka-connect" config: # Coordinates connect_uri: "http://localhost:8083" cluster_name: "connect-cluster" # Credentials username: admin password: password sink: # sink configs
Note that a
. is used to denote nested fields in the YAML recipe.
|URI to connect to.|
|Kafka Connect username.|
|Kafka Connect password.|
|Cluster to ingest from.|
|Whether to create the input and output Dataset entities|
|List of regex patterns for connectors to include in ingestion.|
|List of regex patterns for connectors to exclude from ingestion.|
|Whether to ignore case sensitivity during pattern matching.|
|Environment to use in namespace when constructing URNs.|
If you've got any questions on configuring this source, feel free to ping us on our Slack!