With the exception of
KAFKA_SCHEMAREGISTRY_URL, Kafka is configured via spring-boot, specifically with KafkaProperties. See Integration Properties prefixed with
Below is an example of how SASL/GSSAPI properties can be configured via environment variables:
These properties can be specified via
application.yml files, or as command line switches, or as environment variables. See Spring's Externalized Configuration to see how this works.
See Kafka Connect Security for more ways to connect.
DataHub components that connect to Kafka are currently:
- Various ingestion example apps
By default, ingestion relies upon the
FailedMetadataChangeEvent kafka topics by default for
We've included environment variables to customize the name each of these topics, if your company or organization has naming rules for your topics.
METADATA_CHANGE_EVENT_NAME: The name of the metadata change event topic.
METADATA_AUDIT_EVENT_NAME: The name of the metadata audit event topic.
FAILED_METADATA_CHANGE_EVENT_NAME: The name of the failed metadata change event topic.
KAFKA_MCE_TOPIC_NAME: The name of the metadata change event topic.
KAFKA_FMCE_TOPIC_NAME: The name of the failed metadata change event topic.
KAFKA_TOPIC_NAME: The name of the metadata audit event topic.
Please ensure that these environment variables are set consistently throughout your ecosystem. DataHub has a few different applications running which communicate with Kafka (see above).
Kafka Consumers in Spring are configured using Kafka listeners. By default, consumer group id is same as listener id.
We've included an environment variable to customize the consumer group id, if your company or organization has specific naming rules.
KAFKA_CONSUMER_GROUP_ID: The name of the kafka consumer's group id.
- For quickstart, add these environment variables to the corresponding application's docker.env
- For helm charts, add these environment variables as extraEnvs to the corresponding application's chart. For example,
We are using the Spring Boot framework to start our apps, including setting up Kafka. You can use environment variables to set system properties, including Kafka properties. From there you can set your SSL configuration for Kafka.
If Schema Registry is configured to use security (SSL), then you also need to set this config.
Note In the logs you might see something like
The configuration 'kafkastore.ssl.truststore.password' was supplied but isn't a known config.The configuration is not a configuration required for the producer. These WARN message can be safely ignored. Each of Datahub services are passed a full set of configuration but may not require all the configurations that are passed to them. These warn messages indicate that the service was passed a configuration that is not relevant to it and can be safely ignored.