How does Coursera complete a batch job with Docker?

ECS (Amazon EC2 Container Service) service is AWS specifically for the user to provide container services, ECS services and subdivided into containers, tasks, container instances and clusters four modes, which Tasks mainly for a short period of time, or within a fixed cycle A large number of computing resources for batch operations. The same time as the above-

ETL, picture format conversion, picture adjustment, OCR, PDF generation, weather forecasting, log analysis, feature extraction, automated testing, and search engine reptiles are common batch jobs. This article will take you through Coursera's case to understand this particular container service.

Coursera is an online education platform founded by two computer science professors at Stanford University in the United States. It works with the world's top universities and institutions to provide education for all people in the form of online open classes in the past. Resources.

幻灯片02.png

Coursera currently has more than 13 million registered users, from 190 countries, the course content from programming to writing all-encompassing. In order to cope with such a large amount of users and the amount of courses, Coursera backstage there are many batch operations, such as bulk mail sent, classroom test correction, transcripts, data check and so on.

Initially, Coursera used a PHP framework Cascade to achieve, encountered a lot of problems:

  • Difficult to run, deploy and extend;
  • When a new job arrives, a new thread will be created, each job occupies the memory and CPU resources are not the same, it is difficult to achieve continuous creation;
  • Due to lack of resource isolation, memory limitations of the error, will make the whole application are off

Later, the infrastructure engineer team wanted to implement the micro service architecture through the Docker container, but encountered a lot of problems in the process of using Mesos to manage clusters and containers. Mesos started very complicated, Coursera did not have professional people and time to manage a Mesos cluster The

Finally, Coursera began using ECS ​​services:

  • ECS helps Coursera very simply implement the micro service architecture;
  • Each job is created as a container, managed by Amazon ECS to manage these containers;
  • Amazon ECS provides all of the cluster management and container scheduling services that provide the necessary resource isolation.

ECS significantly reduces the cost of Coursera running batch operations and improves IT efficiency:

  • Easy to use: Amazon ECS simple to start, for the user to shield the details of the bottom of the cluster;
  • Speed ​​and Flexibility: The software update deployment process, from a few hours to a few minutes, at present each team can independently develop and update their own modules, because the application of resources is isolated, there is no interdependence;
  • Elastic expansion: Computational resources can be dynamically applied dynamically;
  • Operation and maintenance efficiency: operation and maintenance personnel no longer need to do the installation of software and maintenance of the cluster work, Amazon ECS done from the cluster management to the container all the things.

Is not that the above scene is very suitable for you?

But ECS not into the Chinese swollen do?

Try the latest batch computing service – the job!

This is the exclusive license for the Chinese users to launch the cloud function, to help you perform a one-time or repeat the short-term operations, such as ETL, picture format conversion, picture adjustment, OCR, PDF generation, weather forecast, log analysis, feature extraction , Automated testing, and search engine reptiles. You can set key parameters for each task, such as environment variables, execution time, execution cycles, and times. Online tutorial: http://docs.alauda.cn/feature/job/intro.html

batchjobwechat.jpg

Here, we open the trial invitation for the latest batch calculation function, you just fill out the questionnaire (no more than two minutes), you can try this new feature. We will be in the feedback to extract 10 to give New Year gift! (As of January 12, 2016)

    Heads up! This alert needs your attention, but it's not super important.