Skip to main content

Kafka Metadata

For context on getting started with ingestion, check out our metadata ingestion guide.


To install this plugin, run pip install 'acryl-datahub[kafka]'.


This plugin extracts the following:

  • Topics from the Kafka broker
  • Schemas associated with each topic from the schema registry

Quickstart recipe#

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:  type: "kafka"  config:    # Coordinates    connection:      bootstrap: "broker:9092"
      schema_registry_url: http://localhost:8081
sink:  # sink configs

Config details#

Note that a . is used to denote nested fields in the YAML recipe.

conection.bootstrap"localhost:9092"Bootstrap servers.
connection.schema_registry_urlhttp://localhost:8081"Schema registry location.
connection.schema_registry_config.<option>Extra schema registry config. These options will be passed into Kafka's SchemaRegistryClient. See
connection.consumer_config.<option>Extra consumer config. These options will be passed into Kafka's DeserializingConsumer. See and
connection.producer_config.<option>Extra producer config. These options will be passed into Kafka's SerializingProducer. See and
topic_patterns.allowList of regex patterns for topics to include in ingestion.
topic_patterns.denyList of regex patterns for topics to exclude from ingestion.
topic_pattern.ignoreCaseTrueWhether to ignore case sensitivity during pattern matching.

The options in the consumer config and schema registry config are passed to the Kafka DeserializingConsumer and SchemaRegistryClient respectively.

For a full example with a number of security options, see this example recipe.


Coming soon!


If you've got any questions on configuring this source, feel free to ping us on our Slack!