Medium
Sign Up/Sign In
All jobs
Report
Design a Distributed Job Scheduler for Millions of Tasks in Daily Operations
Location
remote
About the job
This job is sourced from a job board
Overview
About the role
Welcome to the dynamic world of modern computing. Orchestrating millions of daily tasks is essential in our evolving digital landscapes. In this universe of distributed systems, a robust job scheduler is pivotal. Join me as we explore crafting a distributed job scheduler for seamless operations. Sometimes, we need to set up our job execution based on our user's configuration. For example, Imagine we have a web platform where user logs in, create new invoices, and send them to their customers. Our business team has introduced a new feature, allowing users to create invoices and schedule them for later. This process provides the option for configuration, allowing users to choose between a one-time execution or a recurring style. Another example could be scheduled email, push notification, or other processes. In this post, we will design a highly scalable Job Scheduler in a Distributed environment that can execute millions of tasks each day. This architecture is versatile, addressing scheduled job execution for different types of businesses and diverse use cases. For our system let's presume that the given requirements are like this. User actions: Create and delete jobs. Retrieve a list of jobs created by the user. View the execution history for any given job. Job execution style: The system should support both one-time execution, recurring style Number of average estimated daily tasks executions: 50 million tasks each day Task fail strategy: The system should have the capability to support a retry feature Our system is configured to perform task executions every minute
About the company
Mayil Bayramov is a Software Engineer II at Microsoft.
Skills
distributed systems
job scheduler
task execution
retry feature
Might no longer accept applications
Explore more jobs