Skip to main content

Using Docker Images During Development

We've created a special override file that should configure docker images to be easier to use during development.

Normally, you'd rebuild your images from scratch with docker-compose build (or docker-compose up --build). However, this takes way too long for development. It has to copy the entire repo to each image and rebuild it there.

The file bypasses this problem by mounting binaries, startup scripts, and other data to special, slimmed down images (of which the Dockerfile is usually defined in <service>/debug/Dockerfile).

These dev images will use your locally built code, so you'll need to build locally with gradle first (and every time you want to update the instance). Building locally should be much faster than building on Docker.

We highly recommend you just invoke the docker/ script we've included. It is pretty small if you want to read it to see what it does, but it ends up using our file.


The default dev images, while set up to use your local code, do not enable debugging by default. To enable debugging, you need to make two small edits (don't check these changes in!).

  • Add the JVM debug flags to the environment file for the service.
  • Assign the port in the docker-compose file.

For example, to debug datahub-gms:

# Add this line to docker/datahub-gms/env/docker.env. You can change the port and/or change suspend=n to y.
# Change the definition in docker/ to this
image: linkedin/datahub-gms:debug
context: datahub-gms/debug
dockerfile: Dockerfile
ports: # <--- Add this line
- "5005:5005" # <--- And this line. Must match port from environment file.
- ./datahub-gms/
- ../gms/war/build/libs/:/datahub/datahub-gms/bin

Tips for People New To Docker

Accessing Logs

It is highly recommended you use Docker Desktop's dashboard to access service logs. If you double click an image it will pull up the logs for you.

Conflicting containers

If you ran docker/ before, your machine may already have a container for DataHub. If you want to run docker/ instead, ensure that the old container is removed by running docker container prune. The opposite also applies.

Note this only removes containers, not images. Should still be fast to switch between these once you've launched both at least once.

Unexpected character

If you are using Windows WSL (with Ubuntu) and receive an error of 'unexpected character "." in variable name...' while executing docker/ try these steps:

  • Open up Docker Desktop, click gear icon at top to open the settings and uncheck the "Use Docker Compose V2" option. Close your terminal, open a new one and try to rerun the command docker/ In some cases, unchecking the box may not be sufficient, if the problem persists try executing docker-compose disable-v2 from your terminal.
  • Next, try sudo docker/ and finally, try moving the file ~/.docker/config.json to ~/.docker/config.json.bak and retry the command with sudo.

Running a specific service

docker-compose up will launch all services in the configuration, including dependencies, unless they're already running. If you, for some reason, wish to change this behavior, check out these example commands.

docker-compose -p datahub -f docker-compose.yml -f docker-compose.override.yml -f up datahub-gms

Will only start datahub-gms and its dependencies.

docker-compose -p datahub -f docker-compose.yml -f docker-compose.override.yml -f up --no-deps datahub-gms

Will only start datahub-gms, without dependencies.