Using Docker Images During Development
We've created a special
docker-compose.dev.yml override file that should configure docker images to be easier to use
Normally, you'd rebuild your images from scratch with
docker-compose build (or
docker-compose up --build). However,
this takes way too long for development. It has to copy the entire repo to each image and rebuild it there.
docker-compose.dev.yml file bypasses this problem by mounting binaries, startup scripts, and other data to
special, slimmed down images (of which the Dockerfile is usually defined in
These dev images will use your locally built code, so you'll need to build locally with gradle first (and every time you want to update the instance). Building locally should be much faster than building on Docker.
We highly recommend you just invoke the
docker/dev.sh script we've included. It is pretty small if you want to read it
to see what it does, but it ends up using our
The default dev images, while set up to use your local code, do not enable debugging by default. To enable debugging, you need to make two small edits (don't check these changes in!).
- Add the JVM debug flags to the environment file for the service.
- Assign the port in the docker-compose file.
For example, to debug
# Add this line to docker/datahub-gms/env/docker.env. You can change the port and/or change suspend=n to y.
# Change the definition in docker/docker-compose.dev.yml to this
ports: # <--- Add this line
- "5005:5005" # <--- And this line. Must match port from environment file.
Tips for People New To Docker
It is highly recommended you use Docker Desktop's dashboard to access service logs. If you double click an image it will pull up the logs for you.
If you ran
docker/quickstart.sh before, your machine may already have a container for DataHub. If you want to run
docker/dev.sh instead, ensure that the old container is removed by running
docker container prune. The opposite also
Note this only removes containers, not images. Should still be fast to switch between these once you've launched both at least once.
If you are using Windows WSL (with Ubuntu) and receive an error of 'unexpected character "." in variable name...' while executing
docker/dev.sh try these steps:
- Open up Docker Desktop, click gear icon at top to open the settings and uncheck the "Use Docker Compose V2" option. Close your terminal, open a new one and try to rerun the command
docker/dev.sh. In some cases, unchecking the box may not be sufficient, if the problem persists try executing
docker-compose disable-v2from your terminal.
- Next, try
sudo docker/dev.shand finally, try moving the file
~/.docker/config.json.bakand retry the command with
Running a specific service
docker-compose up will launch all services in the configuration, including dependencies, unless they're already
running. If you, for some reason, wish to change this behavior, check out these example commands.
docker-compose -p datahub -f docker-compose.yml -f docker-compose.override.yml -f docker-compose.dev.yml up datahub-gms
Will only start
datahub-gms and its dependencies.
docker-compose -p datahub -f docker-compose.yml -f docker-compose.override.yml -f docker-compose.dev.yml up --no-deps datahub-gms
Will only start
datahub-gms, without dependencies.