The models defined in snapshot and delta are used to build the schema for several metadata Kafka events. As these events have the prefix
Metadata and suffix
Event, they’re collectively referred to as MXE.
We also model MXEs using PDL and rely on the pegasus gradle plugin to convert them into AVSC. However, we also need to rename all the namespaces of the generated AVSC to avoid namespace clashes for projects that depend on both the PDL models and MXEs.
As the AVSC and PDL model schemas are 100% compatible, it’d be very easy to convert the in-memory representation from one to another using Pegasus’ DataTranslator.
MCE is a "proposal" for a metadata change, as opposed to MAE, which is conveying a committed change. Consequently, only successfully accepted and processed MCEs will lead to the emission of a corresponding MAE. A single MCE can contain both snapshot-oriented and delta-oriented metadata change proposal. The use case of this event is explained in Metadata Ingestion.
We’ll also generate a dead letter queue event, Failed Metadata Change Event (FMCE), for any rejected MCE. The event simply wraps the original MCE and an error message, which contains the reason for rejection. This event can be used for debugging any potential ingestion issues, as well as for re-playing any previous rejected proposal if ever needed.
A Metadata Audit Event captures the change made to one or multiple metadata aspects associated with a particular entity, in the form of a metadata snapshot before the change, and a metadata snapshot after the change.
Every source-of-truth for a particular metadata aspect is expected to emit a MAE whenever a change is committed to that aspect. By ensuring that, any listener of MAE will be able to construct a complete view of the latest state for all aspects. Furthermore, because each MAE contains the "after image", any mistake made in emitting the MAE can be easily mitigated by emitting a follow-up MAE with the correction. By the same token, the initial bootstrap problem for any newly added entity can also be solved by emitting a MAE containing all the latest metadata aspects associated with that entity.