Implementing a sample microservices application
We can use the Mosca, MongoDB, and Redis containers, along with a couple of custom containers, to implement a simple but complete application:
Figure 3.3 – Diagram of our sample microservices application
The publisher and subscriber will communicate with each other using MQTT. The subscriber will listen for a handful of MQTT topics that direct it to operate on or retrieve information from the MongoDB and Redis databases. The publisher will send these MQTT topics and print the responses.
The publisher will be based on Node.js version 11 and the subscriber will be based on Node.js version 12. Without Docker or a virtual machine, running two Node.js versions on the same machine concurrently requires the use of Node Version Manager (nvm) and having multiple versions of Node.js installed on your workstation. Docker containers make it simple to use as many versions as you need and to package the version, along with the app that uses it, in a nice package (a container).
The publisher and subscriber apps are in their own publisher/ and subscriber/ subdirectories of chapter3/ in the companion repository. These programs each need their own Dockerfile so that we can build the two separate containers. They also have their own helper .sh scripts (debug.sh, run.sh, build.sh, and so on). The publisher app only needs to have an MQTT library. The subscriber app needs the MQTT library and a MongoDB library and a Redis library. These libraries will be installed using npm (the Node.js package manager) within the containers.
The publisher and subscriber apps demonstrate how a microservices architecture works, using multiple Docker containers.
The subscriber connects to the MongoDB and Redis containers using Node.js packages/libraries, which are installed in the container with npm. The subscriber provides basic Create, Read, Update, and Delete (CRUD) functions for adding, listing, removing, and retrieving count of records in each of the MongoDB and Redis databases. The publisher sends MQTT messages to the subscriber to invoke this functionality.
Our topics are strings that are derived from a pattern: container/command. If we want to communicate with the subscriber, the pattern is subscriber/command. If we want to communicate with the publisher, the pattern is publisher/command. This convention makes it obvious which topics each microservice would want to subscribe or publish to.
The MQTT topics and messages are as follows:
- subscriber/mongo-count: Responds with the count of records in the MongoDB database.
- subscriber/mongo-add: Adds the message content to the MongoDB database.
- subscriber/mongo-list: Returns a JSON object that contains a list of records in the MongoDB database. If the message is a non-zero length string, it is used to filter the list of records returned.
- subscriber/mongo-remove: Removes a record from the MongoDB database. The message may contain a string or an object (JSON) suitable for passing to MongoDB's collection.deleteOne() method.
- subscriber/mongo-removeall: Deletes all records from the MongoDB database.
- subscriber/redis-count: Responds with the count of records in the Redis database.
- subscriber/redis-flushall: Removes all the records from the Redis database.
- subscriber/redis-set: Adds a record to the Redis database; the message is of the key=value form.
- subscriber/redis-list: Lists all the records in the Redis database and returns a JSON array of records.
- subscriber/redis-del: Deletes a record from the Redis database.
- subscriber/commands: Returns a list of available commands (MQTT topics).
There are shell scripts in the root of the chapter3/ directory that individually start Redis (start-redis.sh), MongoDB (start-mongodb.sh), and the Mosca MQTT broker (start-mosca.sh), as well as a script, start-all.sh that starts all three.
We've already detailed the workings of the start-mongodb.sh script earlier. The start-redis.sh and start-mosca.sh scripts are roughly the same; just the names of the programs that are started (Redis and Mosca) are changed.
It is important to note that the start-mongodb.sh script connects the host's port 27017 to the container's port 27017. This is so that other containers can reach MongoDB via the default port. The start-mosca.sh script connects ports 1883 and 80 to the host so that MQTT and MQTT, over WebSocket, can be used from any of the containers. The start-redis.sh script connects port 6379 to the host so that Redis can be accessed from the containers via the default Redis port. Of course, the host can access any of the containers as well.
The subscriber/start-subscriber.sh and publisher-start-publisher.sh scripts both run the applications locally on the host, not in containers. This allows host native debugging functionality, using WebStorm or another IDE or Node.js debugger. Developing and debugging our publisher and subscriber entirely within Docker containers is covered in the next chapter.
Note
To use the start-subscriber.sh and start-publisher.sh scripts, you will need to install Node.js and yarn on your development workstation. Ensure that you run yarn install in both subscriber/ and publisher/ directories.
This is what start-subscriber.sh looks like:
#!/bin/sh
yarn start
The start-publisher.sh script is identical to the start-subscriber.sh script. The package.json file in the publisher directory signals yarn start to launch the publisher program.
The HOSTIP variable must be set to your host machine's IP, available to our publisher and subscriber, and is used by our Node.js programs to address the MQTT broker, MongoDB server, and Redis server when connecting.
To find your IP on macOS (assuming you use 192.168.*.* as your home network IP address range):
# ifconfig | grep 192
inet 192.168.0.19 netmask 0xffff0000 broadcast 192.168.255.255
The IP of the host is 192.168.0.19.
To find your IP on Linux, use the following command:
$ ip address | grep 192
inet 192.168.0.21/16 brd 192.168.255.255 scope global dynamic enp0s31f6
The IP of this host is 192.168.0.21.
You will run the start-publisher.sh script using the following command:
HOSTIP=192.168.0.19 ./start-publisher.sh
To run the start-subscriber.sh script use the following command:
HOSTIP=192.168.0.19 ./start-subscriber.sh
The publisher program is relatively simple. It connects to the MQTT broker and listens for topics starting with publisher/. The topics and messages received are then converted into the subscriber/ format topics and published to MQTT. The subscriber responds with the publisher topic and the response message.
With both the publisher and subscriber running, we use the MQTT command-line tool to send messages to the publisher. In the following screenshot, you can see how we exercise a few of the subscriber commands.
These two scripts assume that we have Mosca installed on our host. We don't need to install it for the MQTT broker, but for the command-line tools. Being able to send MQTT topics/commands from the command line on the host, in .sh scripts on the host, and in crontabs on the host is very useful. You can also use Mosca as a library to implement a broker in your own Node.js code.
Note
For curious readers, the screenshot is of a Terminal window running tmux with three panes. tmux is a terminal multiplexer: it enables several terminals to be created, accessed, and controlled from a single screen. The tmux GitHub repository can be found at https://github.com/tmux/tmux.
In the following screenshot, you can see how we exercise a few of the subscriber commands:
Figure 3.4 – Three shells demonstrating the publisher and subscriber working together
As we can see, the publisher and subscriber work as expected, as do the database queries between containers and the host. We can edit and debug the publisher and subscriber programs to get them working to our satisfaction.
Now that we have these working publisher and subscriber containers, we want to share them with the rest of the development team.