Docker Open Source Components: Introduction to HyperKit, VPNKit, and DataKit

[Editor's note] Anil Madhavapeddy is Docker's technology, and he teaches at Cambridge University's computer lab. Anil is the original developer of Xen Hypervisor and is now working on unikernel. Anil worked at NetApp, XenSource, Citrix, Intel and NASA. He is the most active contributor to the open source OpenBSD operating system.

@Container container technology conference will be held on June 4 at the Shanghai Everbright Convention and Exhibition Center International Hotel, from Rancher, Ctrip, PPTV, ants gold clothes, Jingdong, Zhejiang Mobile, Haier Electric, only goods, eBay, China's Minsheng Bank, Changan Automobile and other companies in charge of technical personnel will bring practical experience to share, welcome to the sense of the United States, the United States, the United States, the United States, the United States, the United States, Interested classmates attend.

We've done a lot of built-in Docker support on Mac and Windows systems , so that Docker usage habits on each operating system can be seamlessly switched. Mac and Docker on Windows have been able to connect storage and network efficient bridges to Docker containers like Linux Docker containers without the help of any third party software.

Docker is based entirely on the Open Source Foundation (Solomon Hykes gave a keynote address at OSCON 2016, which introduces the history of Docker and open source.) Today, we support Docker's core technology for Mac and Windows desktop applications to the open source community.

Mac and Windows versions of Docker technology include hardware virtualization, embedded operating systems and unikernel technology, the user did not have this experience before. Let's take a closer look at how the source works, so that the user can better use it for his own project or contribute his own code.

Running Docker on a Mac will run a lightweight hipervisor (running an embedded linux instance) that contains the latest stable version of the Docker engine. Unlike other hypervisors, because of the built-in hypervisor framework (introduced from OSX10.10), it does not require admin priority. The Docker application also has built-in libraries that support Docker VM networks and storage, which provide interaction only between Linux and OSX / Windows.
1.png
Today, we are pleased to release these open source components, we apply them to Mac and Windows under the Docker, these new components include:

  • HyperKit : Lightweight Virtualization Toolkit running on OSX
  • DataKit : Modern Distributed Component Framework
  • VPNKit : Embedded Virtual Network Library

In order to develop Mac and Windows Docker, each component package can be used alone or in combination. This is just the beginning: then we will develop other mature components (such as: file system framework), some of which include: HyperKit, DataKit and VPNKit.
2.png

HyperKit

HyperKit is a lightweight virtualization approach based on the introduction of the Hypervisor framework after MacOSX10.10. HyperKit applications can use hardware virtualization to run VMs, but do not require special permissions or complex management tool stack.

HyperKit is based on the xHyve and vHyve projects and works with other related functional modules (such as VPNKit and DataKit) to make it more efficient. Because HyperKit architecture and the library above, so it will be connected with the unikernel library is very direct. For example, we can join the persistence block device by using the MirageOS QCow library (written in OCaml).

What can users contribute?

There are several important aspects:

  • Support for more guest operating systems. Linux is currently the only supported operating system. FreeBSD can be started, but the need to run the installer is therefore not seamless. In order to start Windows, OpenBSD or NetBSD must patch to add more BIOS support, and need more detailed testing.
  • Support for more high-level language bindings. Because HyperKit is expressed as a library, you can use standard external functional interfaces to integrate with high-level languages.
  • Hypervisor function. Some traditional hypervisor functions, such as sleep, recovery, real-time hardware performance, are not supported. To ensure that HyperKit is lightweight and easy to embed, these features can be added in a library.

We will ensure that contributors are well organized and that their tuning code will be well integrated into the upstream project.

What can users do?

Applications that require special or short-time virtual machines can benefit from HyperKit, such as Linux's traditional operating system, or some unikernel projects .

DataKit

DataKit is a toolkit that is compatible with the Git-compatible file system. It reuses the UNIX pipeline concept and the Plan9 9P protocol, while using tree structure stream data instead of bare text. DataKit allows users to define complex workflows between loosely coupled processes by using simple scripts and complex file systems.

DataKit is around the data stream reconstruction of the application architecture, using Plan9 "everything is a file" concept, in the git era put forward the "everything is a versioned file" concept. Because the Mac and Windows Docker in the extensive use of DataKit and 9P, we also open source go-p9p project, it is a modern, for the Go environment permanent 9P library.

What can users do?

The following is an example of creating a CI (continuous integration) system with DataKit, with only 50 lines of script: github.com/docker/datakit/tree/master/ci .

README also describes how to integrate DataKit in Github. DataKit can be used for any scenario that needs to be coordinated between data (version data).

What can users contribute?

DataKit support for Github PR is still very original, very much need more contributions. DataKit can be used in many scenes, welcome to share the user's own project.

VPNKit

VPNKit is in the bare Ethernet network traffic and MacOSX / Windows socket between the transmission of data between the network library. Based on the MirageOS TCP / IP unikernel stack, it is a library written in OCaml. VPNKit will be useful if you need granular control over the network within user-space, and you can easily extend it in high-level languages.

What can users contribute?

VPNKit provides a network interaction point between Docker containers under Mac and Windows. It can be extended by using the method of capturing packets and monitoring. The specific packet mode is forwarded to the filter through the proxy protocol or visualized by HTTP protocol for web application Troubleshooting.

What can users do?

If VPNKit supports more endpoint types, it can also be used to test network traffic without actually generating or sending data to cause stress. You can also create a lightweight overlay network between applications.

The next step

VPNKit and DataKit for Mac and Windows Docker provides a special module, open source they may be able to bring more surprises. This argument is not mutually exclusive, we hope to hear everyone's own projects. In their respective projects, you can write down their own bug.

If you can meet with the relevant project maintainer at our conference at OSCON , you can find a presentation on Docker on Mac and Windows by following links: slides for the talk I about yesterday at OSCON .

If you are not present, you can register with Docker for Mac and Windows beta and use feedback to send back to us. Finally, our team is all interested in sharing this project in open source.

English translation: Improving Docker with Unikernels: Introducing HyperKit, VPNKit and DataKit (translation: Yang Feng)

    Heads up! This alert needs your attention, but it's not super important.