Design a Distributed Job Scheduler for Millions of Tasks in Daily Operations

Welcome to the dynamic world of modern computing. Orchestrating millions of daily tasks is essential in our evolving digital landscapes. In this universe of distributed systems, a robust job scheduler is pivotal. Join me as we explore crafting a distributed job scheduler for seamless operations. Sometimes, we need to set up our job execution based on our user's configuration. For example, Imagine we have a web platform where user logs in, create new invoices, and send them to their customers. Our business team has introduced a new feature, allowing users to create invoices and schedule them for later. This process provides the option for configuration, allowing users to choose between a one-time execution or a recurring style. Another example could be scheduled email, push notification, or other processes. In this post, we will design a highly scalable Job Scheduler in a Distributed environment that can execute millions of tasks each day. This architecture is versatile, addressing scheduled job execution for different types of businesses and diverse use cases. For our system let's presume that the given requirements are like this. User actions: Create and delete jobs. Retrieve a list of jobs created by the user. View the execution history for any given job. Job execution style: The system should support both one-time execution, recurring style Number of average estimated daily tasks executions: 50 million tasks each day Task fail strategy: The system should have the capability to support a retry feature Our system is configured to perform task executions every minute

Mayil Bayramov is a Software Engineer II at Microsoft.