Metadata Audit Event Consumer Job
The Metadata Audit Event Consumer is a Spring job which can be deployed by itself, or as part of the Metadata Service.
Its main function is to listen to change log events emitted as a result of changes made to the Metadata Graph, converting changes in the metadata model into updates against secondary search & graph indexes (among other things)
Today the job consumes from two important Kafka topics:
Where does the name Metadata Audit Event come from? Well, history. Previously, this job consumed a single
MetadataAuditEventtopic which has been deprecated and removed from the critical path. Hence, the name!
- You need to have JDK8
installed on your machine to be able to build
DataHub Metadata Service.
Metadata Audit Event Consumer Job is already built as part of top level build:
However, if you only want to build
MAE Consumer Job specifically:
Metadata Audit Event Consumer Job, you need to make sure that Kafka, Schema Registry & Zookeeper,
Elasticsearch, and Neo4j Docker containers are up and running.
Start via Docker image
The quickest way to try out
Metadata Audit Event Consumer Job is running the Docker image.
Start via command line
If you do modify things and want to try it out quickly without building the Docker image, you can also run the application directly from command line after a successful build:
MCL_CONSUMER_ENABLED=true ./gradlew :metadata-jobs:mae-consumer-job:bootRun
Spring boot actuator has been enabled for MAE Application.
info web endpoints are enabled by default.
healthcheck - http://localhost:9091/actuator/health
metrics - http://localhost:9091/actuator/metrics
To retrieve a specific metric - http://localhost:9091/actuator/metrics/process.uptime