Expand the cloud in MySQL, via Vitess and Kubernetes

Editor's Note Anthony Yeh is a YouTube software engineer. The article focuses on how YouTube uses Vitess and Kubernetes to scale the deployment of MySQL.

A new site may grow at exponential speed. Soon after, you will start planning expansion to meet this unexpected demand. You can of course continue to add more front-end server to solve the problem, but in the end, the database will become your bottleneck, which will lead you:

  • Add multiple copies for better read throughput and data durability
  • Introduce partitions to extend your write throughput and allow your datasets to grow more than a single machine
  • Create a separate copy pool for batch jobs and backups to isolate real-time traffic and replicas
  • Cloning is deployed throughout multiple world-wide data centers to meet disaster recovery and lower latency needs

On YouTube, we completed the expansion of MySQL deployment, which now handles hundreds of millions of daily video views and 300 hours of new video per minute. In order to achieve these needs, we have developed the Vitess platform, which can solve the expansion challenge and hide the relevant from the application layer complexity.

Vitess is a storage platform for scaling MySQL. It optimizes the efficiency of MySQL on dedicated hardware, just as it is in a cloud architecture. He combines a lot of important features in MySQL and NoSQL database extensibility.

Vitess is an available open source project and can run well in a container environment. If you use Kubernetes and Google Container Engine as your container cluster manager, getting started Vitess will be very simple. We created a single point development deployment configuration for Vitess so that it can run on any Kubernetes supported platform.

In addition, in order to more easily deploy container clusters, Vitess also take full advantage of the container cluster manager brought the benefits, in particular:

  • Horizontal expansion : by running more nodes to increase the ability rather than build a huge node
  • Dynamic layout : Let the cluster manager schedule the Vitess container to any location
  • Declare the specification : describe the final state you want, and then let the container manager go to create it
  • Self -healing components : Automatic machine failure recovery

In this environment, Vitess provides a MySQL storage layer, including enhance the persistence, scalability and manageability.

We are just just starting this integration, but you can already run Vitess on your own Kubernetes. For more questions about Vitess, please check our website, ask questions in the forum, or join the discussion on GitHub. Take a closer look at our review to understand the trade-offs we did with Vitess compared to NoSQL's solution and the full management of MySQL solutions like Google Cloud SQL.

Vitess was actively developed in 2011 and is currently being used as a base part of YouTube's MySQL infrastructure, serving thousands of queries per second per server. If you wish to determine if Vitess is suitable for your project, please visit vitess.io

Source: Scaling MySQL in the cloud with Vitess and Kubernetes (translation: He Wei)

Heads up! This alert needs your attention, but it's not super important.