At Airbnb, they move time consuming, resource intensive tasks over to asynchronous background jobs to improve scalability of their web applications. This can also prevent performance issues since potential problems in the background jobs are unlikely to disturb the web servers working with user requests. The job scheduling system has become a very important component and they have therefore built Dynein, a distributed delayed job queueing service, including a highly scalable scheduler. In a blog post Andy Fang, working with cloud infrastructure at Airbnb, describes the background and challenges designing and building Dynein.

Airbnb have been running a centralized cluster of Resque workers on top of Resque Scheduler. Fang notes that this cluster was built for their monolithic application, and while easy to use, it

Read More At Article Source | Article Attribution