Docker for Developers
上QQ阅读APP看书，第一时间看更新
上一章目录下一章

Introduction to Docker Compose

A composing system for containers is a tool that allows us to describe the whole microservices architecture program in a configuration file and then perform operations on the system described. Docker Compose is one such tool. Before we get into what Docker Compose is and does, let's look at the reason why we need a tool like this.

The problem with .sh scripts

So far, we've been using .sh scripts to make working with our microservices application easy. We have used the following scripts:

  • start-mongodb.sh
  • start-redis.sh
  • start-mosca.sh
  • subscriber/start-subscriber.sh
  • publisher/start-publisher.sh
  • subscriber/build.sh
  • publisher/build.sh
  • subscriber/push.sh
  • publisher/push.sh

Instead of having to invoke each of these as separate commands, we can make a single start-all.sh script that invokes them all:

#!/bin/sh

./start-mosca.sh

./start-mongodb.sh

./start-redis.sh

cd subscriber && ./start-subscriber.sh & cd ..

cd publisher && ./start-publisher.sh & cd ..

Note

The start-all.sh script is presented for informational purposes. We will not be using it going forward!

This approach works, but the information about what ports are open and other container-specific access information is hidden within those .sh scripts. For example, the mongodb.sh script starts MongoDB and binds port 27017 of the container to port 27017 of the host.  

Making changes to the configuration may require editing each of those .sh scripts, and maybe even the start-all.sh script itself, as well as its counterpart, stop-sll.sh. We have several additional scripts as well for building and publishing the containers and to perform other housekeeping tasks. This approach is both inconvenient and error-prone.

The Docker Compose tool solves most of the issues with .sh scripts, although we might still want to use .sh scripts to invoke the docker-compose command with its various command-line arguments.

Docker Compose configuration files

Configuration for Docker Compose is done via .yml files, the contents of which are YAML.  YAML is a markup language that allows data serialization. It is similar to JSON format but is much more human-friendly in its syntax.

A file named docker-compose.yml is Docker Compose's default configuration file. You may have multiple configuration files, and you can tell Docker Compose which configuration files to use via a command-line switch.

Let's look at the docker-compose-example.yml file in the chapter4/ directory in the repository. The Docker Compose tool can replace the shell script methodology we've used so far:

# Example Docker Compose file for our chapter 4 application

version: '3'

services:

Docker Compose supports different versions of the docker-compose.yml format. Newer versions have higher version numbers and add additional docker-compose features. In the services section, we describe each of the containers that are to be built and run.

We have our redis container under the services section. The image field specifies that we will be using the redis image from Docker Hub. We persist the database in /tmp/redis so that the data is not lost when the container is stopped and restarted:

  redis:

    image: redis

    volumes:

      - /tmp/redis:/data

    ports:

      - 6379:6379

We expose port 6379, the default Redis port, on the host. Exposing this port allows the host and other containers to access the Redis server.

After Redis, we have our MongoDB container. We are going to use the mongo image from Docker Hub. We persist the data in the host's /tmp/mongo directory so that the database's contents are retained between stopping and restarting the container:

  mongodb:

    image: mongo

    volumes:

      - /tmp/mongo:/data/db

    ports:

      - 27017:27017

The default TCP port for MongoDB is 27017, and we expose it to map port 27017 in the container to port 27017 on the host. Tools on the host and within our containers can access MongoDB via localhost, and we don't need to specify a port on the command lines since the default is configured.

Next is the Mosca container. We are using the matteocollina/mosca image from Docker Hub. We set the /db volume in the container to /tmp/mosca on the host to persist Mosca's state:

  mosca:

    image: matteocollina/mosca

    volumes:

      - /tmp/mosca:/db

    ports:

      - 1883:1883

      - 80:80

We expose ports 1883 and 80 as the same ports on the host. Port 1883 is the default MQTT port. Port 80 is provided to support MQTT over WebSocket, so you can use MQTT in JavaScript programs in the browser.

In our publisher container, the build: line tells docker-compose that we need to build the container specified in the publisher/ directory. The Dockerfile in the publisher directory is used to define how the container is to be built:

  publisher:

    build: publisher

    environment:

    - MQTT_HOST=${HOSTIP}

    - REDIS_HOST=${HOSTIP}

    - MONGO_HOST=${HOSTIP}

    ports:

      - 3000:3000

We expose port 3000 so that we can access the web server that is running in the container using a web browser on the host.

In our subscriber container, the build: line tells docker-compose that we need to build the container specified in the subscriber/ directory. The Dockerfile in the subscriber directory is used to define how the container is to be built:

  subscriber:

    build: subscriber

    environment:

    - MQTT_HOST=${HOSTIP}

    - REDIS_HOST=${HOSTIP}

    - MONGO_HOST=${HOSTIP}

We don't expose anything—the subscriber performs all of its I/O operations via direct API calls for MongoDB and Redis, as well as accepting commands and reporting status via MQTT.

Some things to note are as follows:

  • All the containers are described neatly within the single configuration file.
  • The containers still expose the same ports on the host as with the .sh scripts.
  • The containers must still find the database and MQTT broker containers via the HOSTIP environment variable. This variable must still be set as explained in the previous chapter.

To use our docker-compose-example.yml script to bring up all five microservices, we use the docker-compose up command. The -f switch tells docker-compose which Docker Compose .yml file to use:

% docker-compose -f docker-compose-example.yml up

By default, docker-compose runs all the containers in the configuration file in debug mode. They will print their output to the Terminal/console in the order that the lines are printed. You may see lines printed by the subscriber, then lines printed by the publisher, then lines printed by subscriber again. If you hit Ctrl + C, it will terminate all of the containers and return you to Command Prompt.

If you want the containers to run in detached or daemon mode, use the -d switch:

% docker-compose -f docker-compose-example.yml up -d

In detached or daemon mode, the containers will not print output to the Terminal/console and you will be returned to the prompt right away.

To stop all five microservices, we use a similar docker-compose command:

% docker-compose -f docker-compose-example.yml down

If we do not specify the Docker Compose configuration file to use (-f docker-compose-example.yml), then the docker-compose command will look for and use a file named docker-compose.yml instead.

The docker-compose up/down commands allow us to start and stop one or more of our services as well. For example, we can start only the mongodb and redis containers:

% docker-compose -f docker-compose-example.yml up mongodb redis

The existing mongodb and/or redis containers will be stopped and new ones started. It is up to your programs to detect whether the connections to these services were stopped and to handle the error accordingly.  

We can build any or all of our services using docker-compose:

% docker-compose -f docker-compose-example.yml build publisher

This command builds our publisher container but does not start any containers.

The key takeaway from the ability to specify none (none means all) or one or more of our containers (by name) replaces several of our old .sh scripts. We don't need start scripts anymore because we can use docker-compose up; we don't need stop scripts because we can use docker-compose down; we don't need build scripts because we can use docker-compose build; and more! See https://docs.docker.com/compose/reference/ for details on other docker-compose command functionality.

We are likely to have different setups for development and production, if not additional scenarios. With .sh scripts, we have a debug.sh and run.sh script for development and production. The problem with this .sh file scheme is that we have almost identical docker run commands in each, with only minor differences.  

Docker Compose has an inheritance feature where multiple configuration files can be specified on the docker-compose command line.  

Inheritance using multiple configuration files

We can implement a base docker-compose.yml file and then override the settings in that file with our own override configuration files. This feature is called inheritance—we will inherit the base settings from the docker-compose file and override the settings for our purposes.

Docker Compose starts with the first configuration file on the command line, then merges the second one into it, then merges the third (if there is one), and so on. To merge means to apply settings in the second (or third) configuration file to the current state of the configuration, which will ultimately be used. Any settings in the second configuration file will replace the ones in the first configuration file, if they exist, or will add new services or settings if they don't already exist.

Let's look at the docker-compose.yml base file, which we'll use from now on:

version: '3'

services:

  redis:

    image: redis

  mongodb:

    image: mongo

    volumes:

      - /tmp/mongo:/data/db

  mosca:

    image: matteocollina/mosca

    volumes:

      - /tmp/mosca:/db

  publisher:

    build: publisher

    

    depends_on:

      - "mosca"

      - "subscriber"

  subscriber:

    build: subscriber

    depends_on:

      - "redis"

      - "mongodb"

      - "mosca"

This looks like the docker-compose-example.yml file from the previous section, but you may notice a couple of differences:

  • There are two depends_on options—one for the publisher and one for the subscriber.
  • We are no longer exposing or binding the container's ports to the host's ports.

Let's take a look at them in detail in the following sections.

The depends_on option

The depends_on option allows us to control the start-up order of the containers (refer to https://docs.docker.com/compose/startup-order/). Additionally, depends_on expresses an interdependency between containers. Refer to https://docs.docker.com/compose/compose-file/#depends-on#depends_on for more information about the depends_on option.

Service dependencies cause the following behaviors:

  • docker-compose up starts services in dependency order. In our example, redis, mongo, and the mosca services are started before the subscriber container, and both mosca and subscriber are started before publisher.
  • docker-compose up SERVICE automatically includes dependencies under SERVICE.

docker-compose stop stops services in dependency order (mosca, then mongodb, then redis in our docker-compose.yml file).

The order in which the services are started is important because if we start publisher before mosca is running, the logic to connect to the MQTT broker in the publisher program will fail. Similarly, starting subscriber before the database and MQTT broker services would likely cause the logic in subscriber to connect to the databases and the MQTT broker to fail. It doesn't make sense to start publisher before subscriber is running because anything publisher sends via MQTT will fall on deaf ears, so to speak.

Even though a container has started, there is no guarantee that the container's program will have completed its initialization by the time the microservices that use them try to connect.  In our publisher and subscriber code, we created a wait_for_services() method that ensures that we can connect to the services only when they are up and ready.  

We call wait_for_services() first thing in our publisher and subscriber programs to ensure we have waited just long enough for the dependent services to be up and ready.

The wait_for_services() method in publisher/index.js is as follows:

/**

* wait_for_services

*

* This method is called at startup to wait for any dependent containers to be running.

*/

const waitOn = require("wait-on"),

  wait_for_services = async () => {

  try {

    await waitOn({ resources: [`tcp:${mqtt_host}:${mqtt_port}`] });

  } catch (e) {

    debug("waitOn exception", e.stack);

  }

};

Our publisher microservice only connects to the MQTT broker, so the wait_for_services() method only waits for our MQTT broker's TCP port to be accessible.

The wait_for_services() method in subscriber/index.js is a bit more complicated:

/**

* wait_for_services

*

* This method is called at startup to wait for any dependent containers to be running.

*/

const waitOn = require("wait-on"),

  wait_for_services = async () => {

  try {

    debug(`waiting for mqtt (${mqtt_host}:${mqtt_port})`);

    await waitOn({ resources: [`tcp:${mqtt_host}:${mqtt_port}`] });

    debug(`waiting for redis (${redis_host}:${redis_port})`);

    await waitOn({ resources: [`tcp:${redis_host}:${redis_port}`] });

    debug(`waiting for mongo (${mongo_host}:${mongo_port})`);

    await waitOn({ resources: [`tcp:${mongo_host}:${mongo_port}`] });

  } catch (e) {

    debug("***** exception ", e.stack);

  }

};

The subscriber microservice needs to connect to the MQTT broker, the redis server, and the mongo server. We wait for the TCP ports of those servers to be accessible.

There are other ways to wait for services to be available that involve installing command-line programs/scripts in the container and running them before starting our publisher or subscriber service. For example, you might use this handy wait-for-it.sh script, which can be found at https://github.com/vishnubob/wait-for-it.

The lack of options in the docker-compose.yml file to expose container ports is not an oversight. We are fully able to specify those options in an override file that can provide options to existing containers.

Adding port bindings using overrides

In the chapter4/ directory in the code repository, we have a docker-compose-simple.yml file that is an example of an override file:

version: '3'

services:

  redis:

    ports:

      - 6379:6379

  mongodb:

    ports:

      - 27017:27017

  mosca:

    ports:

      - 1883:1883

      - 80:80

  publisher:

    environment:

    - MQTT_HOST=${HOSTIP}

    - REDIS_HOST=${HOSTIP}

    - MONGO_HOST=${HOSTIP}

    ports:

      - 3000:3000

  subscriber:

    environment:

    - MQTT_HOST=${HOSTIP}

    - REDIS_HOST=${HOSTIP}

    - MONGO_HOST=${HOSTIP}

Here, we specify the ports for each container. We are inheriting the options from our docker-compose.yml file and adding options to expose the ports for each of our containers.  

We don't expose any ports for the subscriber microservice because it never exposes any ports to the host's ports.

We also define three environment variables to be used by the publisher and subscriber containers to access the MQTT_HOST (mosca), REDIS_HOST (redis), and MONGO_HOST (mongodb) services.

The docker-compose command to bring up our services using the two configuration files (inheritance) is as follows:

% HOSTIP=192.168.0.21 docker-compose -f docker-compose.yml -f docker-compose-simple.yml up

Since we are not using the -d switch, our containers are not detached but print their console/debug output to the Terminal. You cannot enter more commands until you hit Ctrl+ C. Doing this will stop all the containers in reverse depends_on order and return you to Command Prompt:

% HOSTIP=192.168.0.21 docker-compose -f docker-compose.yml -f docker-compose-simply.yml up -d

Adding the -d switch causes all the containers to be started in daemon mode. They run in the background and you immediately get a command-line prompt. No further output is sent to the Terminal.

If containers are running in daemon mode, you can stop them using the docker-compose down command:

% HOSTIP=192.168.0.21 docker-compose -f docker-compose.yml -f docker-compose-simple.yml down

We can use three or more configuration files as well. Each additional file specified on the command line further extends the containers and options specified within.

What we have so far is effectively a production that is set up using inheritance. Debugging using this is particularly painful because your only means of diagnosing errors is to add debug() calls to the publisher and/or subscriber, then rebuilding the container(s), and then rerunning the whole application.

To improve our development and debugging cycles, we can bind/mount our publisher/ and subscriber/ directories to the /home/app directory in the containers. The Dockerfiles for both containers use the nodemon (https://nodemon.io/) utility to start the application within the container.

The nodemon utility does a bit more than just starting our program:

  • It also monitors the state of the program, and if it stops, nodemon will restart it.  This is useful because our Node.js programs might detect an error from which they cannot easily be recovered, so they just exit and allow nodemon to restart them.
  • For development, nodemon also monitors the timestamps of the files in the code directory and will restart the program if any of the files change.  

Since we can bind/mount our source code directly in the container, any changes we make to the files using our editor or IDE on the host will immediately affect the changes in the container.

We can create a docker-compose-simple-dev.yml file, which adds our bind/mounts to publisher and subscriber:

version: '3'

services:

  publisher:

    volumes:

      - ./publisher:/home/app

  subscriber:

    volumes:

      - ./subscriber:/home/app

We run this using the docker-compose up command:

% HOSTIP=192.168.0.21 docker-compose -f docker-compose.yml -f docker-compose-simple.yml -f dockercompose-simple-dev.yml up -d

If we edit, say, the publisher/index.js file on the host, we can see that nodemon sees the change and restarts the publisher program:

publisher_1   | [nodemon] restarting due to changes...

publisher_1   | [nodemon] starting `node ./index.js`

publisher_1   | 2020-03-30T18:03:39.537Z publisher publisher microservice, about to wait for MQTT host(192.168.0.21, 1883

publisher_1   | 2020-03-30T18:03:39.546Z publisher ---> wait succeeded

publisher_1   | 2020-03-30T18:03:39.587Z publisher publisher connecting to MQTT mqtt://192.168.0.21

publisher_1   | 2020-03-30T18:03:39.591Z publisher connected to  192.168.0.21 port 1883

publisher_1   | 2020-03-30T18:03:39.638Z publisher listening on port  3000

We now have a good handle on docker-compose, but we are binding ports from our containers to the host's ports. This is problematic if you have a container that wants to bind to port 80 on the host but the host is running a web server or another container for another project that also wants to bind to port 80.

Fortunately, Docker provides a facility to only expose our ports to our containers!

上一章目录下一章