This container is used to automatically apply upgrades from one version of DataHub to another.
As of today, there are 2 supported upgrades:
- NoCodeDataMigration: Performs a series of pre-flight qualification checks and then migrates metadata_aspect table data
to metadata_aspect_v2 table. Arguments:- *batchSize* (Optional): The number of rows to migrate at a time. Defaults to 1000.- *batchDelayMs* (Optional): The number of milliseconds of delay between migrated batches. Used for rate limiting. Defaults to 250.
- NoCodeDataMigrationCleanup: Cleanses graph index, search index, and key-value store of legacy DataHub data (metadata_aspect table) once the No Code Data Migration has completed successfully. No arguments.
To run the
datahub-upgrade container, some environment variables must be provided in order to tell the upgrade CLI
where the running DataHub containers reside.
Below details the required configurations. By default, these configs are provided for local docker-compose deployments of
docker/datahub-upgrade/env/docker.env. They assume that there is a Docker network called datahub_network
where the DataHub containers can be found.
These are also the variables used when the provided
datahub-upgrade.sh script is executed. To run the upgrade CLI for non-local deployments,
follow these steps:
- Define new ".env" variable to hold your environment variables.
The following variables may be provided:
- Pull (or build) & execute the
The primary argument required by the datahub-upgrade container is the name of the upgrade to perform. This argument
can be specified using the
-u flag when running the
For example, to run the migration named "NoCodeDataMigration", you would do execute the following:
In addition to the required
-u argument, each upgrade may require specific arguments. You can provide arguments to individual
upgrades using multiple
For example, the NoCodeDataMigration upgrade provides 2 optional arguments detailed above: batchSize and batchDelayMs.
To specify these, you can use a combination of
-a arguments and of the form argumentName=argumentValue as follows: