Talk about the new network features of Docker 1.9

Docker introduced a set of custom network commands and cross-host network support in version 1.9. This is a major change since the libnetwork project was pulled from Docker's main repository. Whether or not you have noticed, Docker's new network is about to make a noticeable change to the user's habits.
The same time as the above-

Libnetwork and Docker networks

The separation of the libnetwork project from the lincontainer and the Docker code was completed in the Docker version 1.7 (extracted from the Docker version 1.6 network code). After that, the container's network interface becomes a replaceable plug-in module. As the change is very smooth, as Docker users almost do not feel the difference, but this change for the next series of extensions planted a good foreshadowing.

In summary, libnetwork's core thing is to define a set of standard container network model (Container Network Model, referred to as CNM), as long as the network interface to meet the model can be used for communication between containers, and communication The process and details can be achieved entirely by the network interface.

Docker's container network model was originally proposed by Cisco employees Erik, and it was interesting to note that Erik himself was not a direct contributor to Docker and libnetwork code. Initially Erik was just to expand the Docker network capabilities, designed a Docker bridge expansion prototype, and this idea back to the Docker community. However, his bold vision was recognized by the Docker team and, after extensive discussions with other Docker partners, gradually formed the embryonic form of libnetwork.

In this network model, three terms are defined: Sandbox, Endpoint, and Network.


As shown in the figure above, they are the abstracts of the container network environment, container virtual network card, and host virtual network card / bridge, respectively, in container communication.

  1. Sandbox: corresponds to a container in the network environment, including the corresponding network card configuration, routing table, DNS configuration. CNM image it is expressed as a "sandbox" of the network because such a network environment is created with the creation of the container and does not exist with the destruction of the container;
  2. Endpoint: In fact, is a container in the virtual network card, the container will be displayed as eth0, eth1 and so on;
  3. Network: refers to a communication network can communicate with each other, joined the same network of containers directly through the name of each other can be connected to each other. Its entity is essentially a virtual network card or bridge on the host.

This abstraction brings a very smooth transition to Docker's version 1.7, except that the three classic "network modes" in the document are replaced by "network plug-ins", and the user feels almost no difference.

Until the arrival of the 1.9 version, Docker finally the network control capabilities completely open to the end user, and thus changed the operation of the two containers connected to the operation (of course, Docker compatibility for the effort, so even if you do not know the following All of these differences, still do not affect the continued use of Docker in the past).

The network-related changes in Docker 1.9 are embodied in the new "docker network" command.

“ “
$ Docker network –help

Usage: docker network [OPTIONS] COMMAND [OPTIONS]

Ls List all networks
Rm Remove a network
Create Create a network
Connect connect container to a network
Disconnect Disconnect container from a network
Inspect Display detailed network information

“ “

Briefly explain the role of these commands.

1, docker network ls

This command lists all the networks on the current host or Swarm cluster:

“ “
$ Docker network ls
6e6edc3eee42 bridge bridge
1caa9a605df7 none null
D34a6dec29d3 host host

“ “

By default, you will see three networks, which are created by the Docker Deamon process. They actually correspond to the Docker past three "network model":

  1. Bridge: the container uses a separate network Namespace and connects to the docker0 virtual network card (default mode)
  2. None: The container does not have any network card, suitable for containers that do not need to communicate with the outside through the network
  3. Host: The container and the host share the network Namespace, with the same network equipment as the host

After the introduction of libnetwork, they are no longer fixed "network mode", but only three different "network plug-in" entity. Say they are entities because users now use Docker's network commands to create more networks that are similar to the default network, each of which is an entity of a particular type of network plug-in.

02, docker network create / docker network rm

These two commands are used to create or delete a container network, and create a network plug-in that can be used with the "-driver" parameter, for example:

$ docker network create –driver=bridge br0

Now this host has a new bridge type Docker network:
$ docker network ls
b6942f95d04a br0 bridge

The Docker container can specify the network used by the "-net" parameter when it is created, and the containers connected to the same network can communicate directly with each other.
When a container network is no longer needed, you can delete it:

$ docker network rm br0

03, docker network connect / docker network disconnect

These two commands are used to dynamically add a container to an existing network or remove the container from the network. For a clearer explanation of this, let's look at an example.

Create two networks by referring to the scenario in the previous libnetwork container network model:

$ docker network create –driver=bridge frontend
$ docker network create –driver=bridge backend

And then run three containers, so that the first container access frontend network, the second container at the same time access to two networks, three containers only access backend network. First, you can easily create the first and third containers with the "-net" parameter:

$ docker run -td –name ins01 –net
$ docker run -td –name ins03 –net

How do I create a container that joins both networks at the same time? Since the "-net" parameter can only be assigned a network name when creating a container, you need to add another network with the docker network connect command after it is created:

$ docker run -td –name ins02 –net
$ docker network connect backend ins02

Now use the ping command to test the connectivity between these containers:

$ docker exec -it ins01 ping ins02
$ docker exec -it ins01 ping ins03
$ docker exec -it ins02 ping ins01
$ docker exec -it ins02 ping ins03
$ docker exec -it ins03 ping ins01
$ docker exec -it ins03 ping ins02

The result also confirms that the two containers in the same network can use each other to find each other directly, and that the containers in different networks are not directly communicable. The docker networkdisconnect can also be used to dynamically remove the specified container from the specified network:

$ docker network disconnect backend ins02
$ docker exec -it ins02 ping ins03

Visible, the ins02 container instance removed from the backend network, it can not directly connect ins03 container instance.

04, docker network inspect

Finally, this command can be used to display the specified container network information, and all connected to the list of containers in this network:

$ docker network inspect bridge
“Id”: “6e6edc3eee42722df8f1811cfd76d7521141915b34303aa735a66a6dc2c853a3”,
“Scope”: “local”,
“IPAM”: {
“Config”: [{“Subnet”: “”}]
“Containers”: {
“IPv4Address”: “”,

It is worth noting that each of the different networks on the same host has a different network address segment, so a container that belongs to multiple networks will have multiple virtual NICs and multiple IP addresses.

It can be seen, libnetwork brings the most intuitive change is actually: docker0 is no longer the only container network, and users can create any number of docker0 similar to the network to isolate the communication between the containers. However, to be careful, the user-defined network and the default network is still not the same place.

The default three networks can not be deleted, and the user-defined network can be deleted with the "docker networkrm" command;

The containers that connect to the default bridge network connection need to be explicitly specified at startup using the "-link" parameter to connect to each other in the container using the container name. And connect to the custom network of the container, do not need any configuration can be directly used to connect to any container name belongs to the same network in the container. This design is to facilitate the communication between the containers, but also can effectively limit the scope of communication, increase network security;

In the Docker 1.9 document has made it clear that the container is no longer recommended to use the default bridge card, its existence is only compatible with the early design. The "-link" communication between containers has also been marked as "outdated" and may be completely removed in a future version.

Docker's built-in Overlay network

Built-in cross-host network communication has been Docker's much-anticipated feature, and before the 1.9 version, there are many third-party tools or methods in the community trying to solve this problem, such as Macvlan, Pipework, Flannel, Weave and so on. Although there are many differences in the implementation of these programs, but its ideas are divided into two types: two-tier VLAN network and Overlay network.

In simple terms, the idea of ​​a two-tier VLAN network to solve cross-host communication is to transform the original network architecture into a large Layer 2 network, which is directly routed through a specific network device to achieve communication between the point-to-point of the container. This scheme is superior to the Overlay network in terms of transmission efficiency, but it also has some inherent problems.

  • This approach requires Layer 2 network equipment support, versatility and flexibility as the latter;
  • Because the number of VLANs that are usually available on the switch is about 4000, this will limit the size of the container cluster and can not meet the deployment requirements of the public cloud or large private cloud.
  • Large-scale data center deployment VLAN, will lead to any one of the VLAN's broadcast data will be flooded throughout the data center, a lot of consumption of network bandwidth, bring maintenance difficulties.

In contrast, the Overlay network refers to a new data format that encapsulates a Layer 2 message over an IP message through a convention communication protocol without changing the existing network infrastructure. This will not only take full advantage of the mature IP routing protocol process data distribution, but also in the Overlay technology using extended isolation number of bits, can break the VLAN 4000 limit, support up to 16M users, and if necessary, can be broadcast traffic For the multicast traffic, to avoid the proliferation of broadcast data. Therefore, Overlay network is actually the most mainstream container cross-node data transmission and routing solutions.

In the 1.9 version of Docker officially joined the official support for cross-node communication solutions, and this built-in cross-node communication technology is the use of Overlay network approach.

Speaking of Overlay network, many people's first reaction is: inefficient, this understanding is in fact biased. There are many ways to implement the Overlay network. The IETF (Internet Engineering Task Force) has developed three implementations of Overlay: Virtual Scalable LAN (VXLAN), Network Virtualization (NVGRE) with universal routing encapsulation, And stateless transport protocol (SST), which VXLAN support the most abundant manufacturers, can be said that the delay network of the de facto standard.

In addition to these three standards there are many non-standard Overlay communication protocols, such as Weave, Flannel, Calico and other tools include a set of custom Overlay network protocol (Flannel also supports VXLAN mode), these custom network protocol Of the communication efficiency is far lower than the IETF standard protocol [5], but because they are very convenient to use, has been widely adopted and it is generally believed that Overlay network inefficient impression. However, according to some of the online test data, the use of VXLAN network transmission rate and two-tier VLAN network is basically the same.
With the help of these concerns, the good news is that Docker's built-in Overlay network is the VETLAN standard using the IETF standard and is the SDN Controller model in VXLAN that is generally considered the most suitable for large-scale cloud computing virtualization environments.

So far everything is so beautiful, we are not trying to try it?

And slow, until I first sprinkle some cold water. The key to this feature is that Docker's new "overlay" type of network card requires only the "-driver = overlay" parameter when the user creates a network card with the "docker networkcreate" command. Looks like this:

$ docker network create –driver=overlay ovr0

But the reality is that until now, Docker's Overlay network is tightly integrated with its Swarm cluster, so the easiest way to use Docker's built-in cross-node communication is to adopt Swarm as a clustered solution. This is why Docker 1.9 will be released with Swarm1.0, and marks Swarm has Product-Ready. In addition, there are some additional conditions:

  • All Swarm nodes have a Linux system kernel version of no less than 3.16
  • Requires an additional configuration storage service, such as Consul, Etcd, or ZooKeeper
  • All nodes are able to connect to the IP and port of the storage service
  • The Docker daemon running by all nodes needs to use the "-cluster-store" and "–cluster-advertise" parameters to specify the configuration store service address used

We do not explain why we must use Swarm, we will soon find out the reasons. Assuming that both of the above conditions 1 and 3 are met, then you need to create an external configuration storage service, for the sake of simplicity to consider the high availability, you can use a single point of service.

To Consul, for example, use Docker to start it, taking into account the domestic visit Docker Hub is relatively slow, it is recommended to use "Ling bird" Docker mirror warehouse:

$ docker run -d \
–restart=”always” \
–publish=”8500:8500″ \
–hostname=”consul” \
–name=”consul” \ -server -bootstrap

If you use Etcd, you can use the following command to start the container, the same can be used "Ling bird" Docker mirror warehouse:

$ docker run -d \
–restart=”always” \
–publish=”2379:2379″ \
–name=”etcd” \ etcd \
-name etcd0-advertise-client-urls http://<Etcd所在主机IP>:2379 \
-listen-client-urlshttp:// -initial-cluster-state new

And then modify each host Docker background process startup script in the "DOCKER_OPTS" variable content, if Consul plus the following two items:

–cluster-store=consul://<Consul所在主机IP>:8500 -–cluster-advertise=eth1:2376

If it is Etcd plus:


And then restart the Docker daemon for each host, everything is ready. Of course, due to modify and restart the Docker background process itself is more troublesome thing, if the user business may use cross-node network communication, it is recommended to set up Docker cluster when the preparation of storage services in advance, and then directly to the host node You can add the appropriate parameters to the Docker startup configuration.

As for the location where the storage service is configured, it is generally recommended to separate the nodes that run the business container and use a separate service node to ensure that all nodes running the business container are stateless and can be scheduled and assigned evenly.

Then the time to create the Overlay network, the problem came, we want to build this network is across all nodes, that is to say in each node should have a name, ID and attribute exactly the same network, between them But also mutual recognition of each other for their own copy of the different nodes. How to achieve this effect? The current Docker network command can not be done, so only by means of Swarm.

Build Swarm cluster method here is not going to elaborate, only to demonstrate the operation of the order. For the sake of simplicity, we use Swarm's official public token service as the storage location for node networking information. First, get a token on any node with the following command:

$ docker run –rm swarm create 6856663cdefdec325839a4b7e1de38e8

Optionally select one of the nodes as the master node of the cluster and run the Swarm Master service on the host:

$ docker run -d -p 3375:2375 swarm manage token://<前面获得的token字符串>

Run the Swarm Agent service on other nodes that are running as Docker service containers:

$ docker run -d swarm join –addr=<当前主机IP>:2375token://<前面获得的token字符串>

This will get a Swarm cluster. Of course, we can also use the previously established Consul or Etcd service to replace the official token service, just a little modify the startup parameters can be, the specific details can refer to Swarm documents.

Swarm provides an API that is fully compatible with the Docker service, so you can use the docker command directly. Note that the external port number 3375 specified when creating the master service in the above command is the address used to connect the Swarm service. Now we can create a network of type Overlay:

$ docker -H tcp://<Master节点地址>:3375network create –driver=overlay ovr0

This command is sent to the Swarm service, and Swarm adds an Overlay network of exactly the same attributes to all Agent nodes. In other words, any of the agent nodes on the implementation of "docker networkls" command can see it, and use the "docker network inspect" command to view its information, will be in each node to get exactly the same content. Through the Docker connected to the Swarm cluster implementation network ls command you can see the whole cluster network of the whole picture:

$ docker -H tcp://<Master节点地址>:3375network ls
$ docker network ls
445ede8764da swarm-agent-1/bridge bridge
2b9c1c73cc5f swarm-agent-2/bridge bridge

90f6666a9c5f ovr0 overlay

In the Swarm network, each network name will be preceded by the name of the node, but the Overlay type of network is not the prefix, which also shows that this type of network is shared by all nodes.
Here we create two containers in the Swarm connected to the Overlay network, and use the Swarm filter to restrict the two containers to run on different nodes.

$ docker -H tcp://<Master节点地址>:3375 run-td –name ins01 –net ovr0 –env=”constraint:node==swarm-agent-1″
$ docker -H tcp://<Master节点地址>:3375 run-td –name ins02 –net ovr0 –env=”constraint:node==swarm-agent-2″

And then try to connect the ins02 container from the ins01 container:

$ docker -H tcp://<Master节点地址>:3375 exec-it ins01 ping ins02

At this point, we have been in the Docker Overlay network successfully carried out cross-node data communication.

Want a simple point? With the spirit of the cloud it

I do not know what we found, whether it is the past Pipework, Flannel, Weave way, or Docker 1.9 built Overlay network, although all of these programs are to promote their own easy to use, to build cross-node communication container is still a The last resort.

The reason why this is said, because in the real application scenarios often due to the direct connection of the service, the number of containers that need to communicate with each other is far beyond the capacity of a single host node, which makes us have to be in the existing foundation Implementation of a self-maintenance service mechanism, the communication between the containers extended to multiple nodes. But maintaining these cross-node communication infrastructure itself is not for the enterprise to bring real business value!

In the moment, the cloud environment for the rapid development of the environment has created for many enterprises the opportunity to overtake the corner: through the use of cloud platform, companies save their own acquisition and the formation of large-scale network and computing resources and management of the huge operation and maintenance team investment , Just focus on the business itself, the design and design will be able to harvest huge profits. At the same time, the container as a new generation of service deployment and scheduling tools are increasingly widely used, and solve the problem of container communication should not be the focus of business needs, and now has become the scale of service can not be ignored in front of the moment Hinder.

Lingque cloud as a container cloud service platform leader, for the container users to shield the container running node details, making the user completely feel the difference caused by cross-node communication. So how do you run two containers in the spirits and communicate with each other?

First install the alauda command line tool, use the "alauda login" command login.

$ alauda login
Username: fanlin
Password: *灵雀密码*
[alauda] Successfully logged in as fanlin.
[alauda] OK

And then run two containers, the former running Nginx service, which is used to test the connection with the former.

$ alauda service run –publish=80/http web
[alauda] Creating and starting service “web”
[alauda] OK
$ alauda service run client
[alauda] Creating and starting service “client”
[alauda] OK

Try to access the HTTP service provided by the web container instance from the client container instance, noting the address format automatically generated by the spy tree.

$ alauda service exec client wget -O-
Password for

<title>Welcome to nginx!</title>

You can see the contents of the Nginx default home page to prove that the two containers can be connected between. In addition to creating a single task, you can use the "alauda compose" command to quickly create a collection of multiple containers.

So the two containers are running on which the mainframe? Control him, the spirit of the bird has all these basic details of the hidden hiding, so just continue to create more services to the cloud it, no longer have to worry about the host resources exhausted, how cross-node communication problems face friends.

About the author: Lin Fan, ThoughtWorks Software Engineer and DevOps Consultant, has extensive experience in continuous delivery and server operation and maintenance automation, focusing on DevOps and container technology. In the InfoQ, CSDN website and "programmer" magazine published a number of related areas of the article, author of "CoreOS practice road" a book.

    Heads up! This alert needs your attention, but it's not super important.