Use Docker, Registrator, Consul, Consul Template, and Nginx to achieve a highly scalable Web framework

Consul is a service software that supports multi-data center distributed high availability service discovery and configuration sharing, developed by HashiCorp Corporation in Go language, open protocol based on Mozilla Public License 2.0. This article describes how to use Consul to combine multiple Docker containers to provide a highly scalable Web service.
When you start assembling the container to build your system, you will find Docker very interesting. Recently, I was playing Consul and tried to use it to build a highly scalable Web application framework. Consul is a service discovery and configuration project developed by HashiCorp (the creator of Vagrant ).

I used to try to use Consv to create an extensible framework with SRV records (see here : here ), but I found this way a bit complicated. And I pursue the simple, so I found the Consul Template , when adding or removing services, Consul Template can be connected through the Consul update configuration and restart the application.

In this article, I will talk about how to use Docker to consolidate Consul, Consul Template, Registrator, and Nginx into a trustworthy and extensible framework – DR CoN. Once the assembly is complete, DR CoN allows you to add and remove services in this framework, the key is that you do not need to rewrite any configuration, do not need to restart any service, everything can run!


Docker wraps some APIs on LXC (Linux Containers), so it can only run on Linux (Note [1]). Since I am using the OS X operating system (most of the readers may be), I wrote this article " Run Docker on OSX via Boot2Docker ". The following is a brief introduction:

  Brew install boot2docker 
Boot2docker init
Boot2docker up

After execution, a virtual machine will be run on Ubuntu to run the Docker daemon. To connect to the daemon, execute the following command:

  Export DOCKER_IP = `boot2docker ip` 
Export DOCKER_HOST = `boot2docker socket`

You can verify that the Docker was successfully installed by executing the following command:

  Docker ps 

Build a simple Web service with Docker

We need a service to test Dr Con this framework. To do this, let's start building a simple service that I know (see this article for more information). Create a Dockerfile file and write the following:

  FROM python: 3 
CMD ["python", "-m", "http.server"]

Execute the following command in the directory where the Dockerfile file is located:

  Docker build -t python / server. 

After execution, a mirror named python/server will be created and a container will be run by executing the following command:

  Docker run -it \ 
-p 8000: 80 python / server

We can call this simple service through the curl command to test whether it can run properly:

  Curl $ DOCKER_IP: 8000 


Consul is a service with DNS and HTTP APIs. It also has many other features, such as service health detection, cross-host cluster build and "key-value" on the repository. Run the following command to run Consul in the Docker container:

  Docker run -it -h node \ 
-p 8500: 8500 \
-p 8600: 53 / udp \
Progrium / consul \
-server \
-bootstrap \
-advertise $ DOCKER_IP \
-log-level debug

If you have access to $DOCKER_IP:8500 through the browser, you will see all the services already registered in the Consul on the control panel.

We are able to register a service with Consl's Web API via curl :

  Curl -XPUT \ 
$ DOCKER_IP: 8500 / v1 / agent / service / register \
-d '{
"ID": "simple_instance_1",
"Name": "simple",
"Port": 8000,
"Tags": ["tag"]
} '

Then, we can use the dig command to query Consul for this provided DNS API:

  Dig @ $ DOCKER_IP -p 8600 simple.service.consul 

After execution, the results are as follows:

  ; << >> DiG 9.8.3-P1 << >> simple.service.consul 
;; global options: + cmd
;; Got answer:
;; - >> HEADER << - opcode: QUERY, status: NOERROR, id: 39614
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

; Simple.service.consul. IN A

Simple.service.consul. 0 IN A

;; Query time: 1 msec
;; SERVER: # 53 (
; WHEN: Mon Jan 12 15:35:01 2015
;; MSG SIZE rcvd: 76

And so on, which has a problem, where is the service port Unfortunately, the DNS record does not return the service port number. In order to get the port number, we must check the SRV record:

  Dig @ $ DOCKER_IP -p 8600 SRV simple.service.consul 

After execution, the results are as follows:

  ; << >> DiG 9.8.3-P1 << >> SRV simple.service.consul 
;; global options: + cmd
;; Got answer:
;; - >> HEADER << - opcode: QUERY, status: NOERROR, id: 3613
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

; Simple.service.consul. IN SRV

Simple.service.consul. 0 IN SRV 1 1 8000 node.node.dc1.consul.

Node.node.dc1.consul. 0 IN A

;; Query time: 1 msec
;; SERVER: # 53 (
WHEN: Mon Jan 12 15:36:54 2015
;; MSG SIZE rcvd: 136

SRV records are difficult to use because many technologies do not support them.

Srv-router This container is used by Consul and Nginx to route requests to the corresponding service (see here for details). However, here we will introduce a simpler way for Nginx to route these requests to the corresponding service.


After the Docker container is started, the Registrator configures the corresponding environment variable and registers the container on the Consul, as follows:

  Docker run -it \ 
-v /var/run/docker.sock:/tmp/docker.sock \
-h $ DOCKER_IP progrium / registrator \
Consul: // $ DOCKER_IP: 8500

After the Registrator starts, we run a service:

  Docker run -it \ 
-e "SERVICE_NAME = simple" \
-p 8000: 80 python / server

After the command is executed, the service is automatically added to Consul, and if we close the service, it will be automatically removed on the Consul. Since Registrator does not require us to manually register the service to Consul, it is the first part of DR CoN to be assembled.

Consul Template

The role of the Consul Template is that when it finds that the service on the Consul has changed, it uses the Consul to update the file and execute the appropriate command.

For example, it can rewrite the nginx.conf file, all the routing information will be included in it, and then reload Nginx configuration, so that a number of similar services to achieve load balancing, or to a variety of services to provide a single terminal.

I made the following modifications to this Docker container (see here for details):

  FROM nginx: 1.7 

#Install Curl
RUN apt-get update -qq && apt-get -y install curl

#Download and Install Consul Template
RUN curl -L $ CT_URL | \
Tar -C / usr / local / bin --strip-components 1 -zxf -

#Setup Consul Template Files
RUN mkdir / etc / consul-templates
ENV CT_FILE /etc/consul-templates/nginx.conf

#Setup Nginx File
ENV NX_FILE /etc/nginx/conf.d/app.conf

#Default Variables
ENV CONSUL consul: 8500
ENV SERVICE consul-8500

# Command will
# 1. Write Consul Template File
# 2. Start Nginx
# 3. Start Consul Template

CMD echo "upstream app {\ n \
Least_conn; \ n \
{{Range service \ "$ SERVICE \"}} \ n \
Server {{.Address}}: {{. Port}}; \ n \
{{Else}} server;{{end}} \ n \
} \ N \
Server {\ n \
Listen 80 default_server; \ n \
Location / {\ n \
Proxy_pass http: // app; \ n \
} \ N \
} "> $ CT_FILE; \
/ Usr / sbin / nginx -c /etc/nginx/nginx.conf \
& CONSUL_TEMPLATE_LOG = debug consul-template \
-consul = $ CONSUL \
-template "$ CT_FILE: $ NX_FILE: / usr / sbin / nginx -s reload";

To download this Dockerfile, please click here .

Note: The above \n\ stands for another line, and removes the line feed for the Docker's multi-line command.

The Docker container will run Consul Template and Nginx, and when the service changes, it will overwrite Nginx's app.conf file and reload Nginx.

Build the container with the following command:

  Docker build -t drcon 

Then, start the container:

  Docker run -it \ 
-e "CONSUL = $ DOCKER_IP: 8500" \
-e "SERVICE = simple" \
-p 80:80 drcon

SERVICE option is used to specify the services on the Consul. Therefore, the above DR CoN container will be through all the services named simple to achieve load balancing.

Assembled together

Now let's assemble everything.

Run Consul:

  Docker run -it -h node \ 
-p 8500: 8500 \
-p 53: 53 / udp \
Progrium / consul \
-server \
-bootstrap \
-advertise $ DOCKER_IP

Run the Registrator:

  Docker run -it \ 
-v /var/run/docker.sock:/tmp/docker.sock \
-h $ DOCKER_IP progrium / registrator \
Consul: // $ DOCKER_IP: 8500

Run DR CoN:

  Docker run -it \ 
-e "CONSUL = $ DOCKER_IP: 8500" \
-e "SERVICE = simple" \
-p 80:80 drcon

Execute curl $DOCKER_IP:80 command, the results are as follows:

  Curl: (52) Empty reply from server 

Now open simple services:

  Docker run -it \ 
-e "SERVICE_NAME = simple" \
-p 8000: 80 python / server

After execution, it will happen:

  1. Registrator registers this service with Consul;
  2. Consul Template rewrite nginx.conf, and then reload the configuration.

Now execute curl $DOCKER_IP:80 will be successfully routed to this service.

If we start another simple service on another port:

  Docker run -it \ 
-e "SERVICE_NAME = simple" \
-p 8001: 80 python / server

The request will load balance through these two simple services.

If you close or start a simple service, the script is still running fast and all requests can be done gracefully.

to sum up

With Docker, it is easier to describe, distribute, and implement DR CoN frameworks, and of course the Consul is so easy to use. Using Docker's more powerful tools to assemble the service is very interesting and useful, and now I can create a horizontally scalable framework, and everything can work properly.

Note [1]: Docker in the 1.2 version after the default use of libcontainer, rather than LXC, although the Docker support LXC. Since both libcontainer and LXC are based on the namespace and Cgroups in the Linux kernel, they can only run on Linux.

Source: Scalable Architecture DR CoN: Docker, Registrator, Consul, Consul Template and Nginx (Translated by: Xiao Yuanhao Proof: Li Yingjie)


Translator introduction

Xiao Yuanhao, graduate student, studying at the Beijing University of Aeronautics and Astronautics Computer College ACT laboratory, currently engaged in virtualization, cloud computing direction of the study. We hope to communicate and study Docker with DockerOne.

    Heads up! This alert needs your attention, but it's not super important.