In Parts 1 and 2, we covered the basics of YARN resource allocation. In this installment, we’ll provide an overview of cluster scheduling and introduce the Fair Scheduler, one of the scheduler choices available in YARN.
A standalone computer can have several CPU cores, each running a single process, but there can be as many as a few hundred processes running simultaneously. The scheduler is a part of the desktop’s operating system that assigns a process to a CPU core to run for a short period of time.
As described previously in “Part 1: YARN and Cluster Basics”, an application consists of multiple tasks (usually on different hosts) on the cluster. A cluster scheduler essentially has to address:
- Multi-tenancy: On a cluster, many users launch many different applications, on behalf of multiple organizations. A cluster scheduler allows varying workloads to run simultaneously.
- Scalability: A cluster scheduler needs to scale to large clusters running many applications. This means that increasing the size of the cluster should improve overall performance without negatively affecting system latencies.
Scheduling in YARN
The ResourceManager (RM) tracks resources on a cluster, and assigns them to applications that need them. The scheduler is that part of the RM that does this matching honoring organizational policies on sharing resources. Please note that:
- YARN uses queues to share resources among multiple tenants. This will be covered in more detail in “Introducing Queues” below.
- The ApplicationMaster (AM) tracks each task’s resource requirements and coordinates container requests. This approach allows better scaling since the RM/scheduler doesn’t need to track all containers running on the cluster.
The Fair Scheduler is a popular choice (recommended by Cloudera) among the schedulers YARN supports. In its simplest form, it shares resources fairly among all jobs running on the cluster. The next few sections explain scheduler internals in the context of Fair Scheduler and elaborate on the commonly used controls the scheduler offers.
Queues are the organizing structure for YARN schedulers, allowing multiple tenants to share the cluster. As applications are submitted to YARN, they are assigned to a queue by the scheduler. The root queue is the parent of all queues. All other queues are each a child of the
rootqueue or another queue (also calledhierarchical queues). It is common for queues to correspond to users, departments, or priorities.
Figure 1 presents a basic Fair Scheduler example of
fair-scheduler.xml and a graphical representation of each queue’s share of the cluster.
- There are four departmental queues (
- There is a special
adminqueue that that users fred or greg, who are assumed to be administrators, can use.
Figure 1: Example part of
fair-scheduler.xml and corresponding fair shares of each queue
Using Hierarchical Queues
One level of queues allows sharing the cluster along one dimension (e.g. one queue per team), but it is common to share clusters among multiple dimensions (e.g. per-team and priority). Fair Scheduler allows nesting queues to form a hierarchical queue structure, where each level could correspond to a dimension under its parent queue.
For example, Figure 2 below shows resources shared along team and priority dimensions. The first level corresponds to the team (e.g.
root.datascience), and each first-level queue could have a high- and low-priority child queues for jobs from that specific team (e.g.
In the rest of this post and subsequent parts, we refer to queue names in a different font like
root.sales.northamerica. Where unambiguous, we use the shortest version possible, i.e.
Example #1: Weights in a single queue
sales queue, there are two child queues:
europe. Each has a weight of 30.0, so the fair share for each of the child queues within
sales is effectively 50%.
marketing queue, there are two child queues of nonequal weight:
website. This means that jobs in the
reports queue are allocated twice as many resources as jobs in the
website queue. However, together, their weight is still governed by the
marketing queue’s weight.
Queue Weights and Top-Down Scheduling
Queue weights are used to determine the fair share for a queue. The Fair Scheduler starts from the
rootqueue and looks at the weights of all immediate child queues to determine their fair share. Each child queue’s fair share is further evaluated for its set of child queues.
Example #2: Top-down view of weights
marketing queue has a weight of 3.0, the
sales queue has a weight of 4.0, and the
datascience queue has a weight of 13.0. So, the allocation from the root will be 15% to marketing, 20% to sales, and 65% to
Of the share that goes to
datascience, all of the queue’s allocation goes to the
short_jobsqueue. If there are no jobs assigned to the
short_jobs queue, then the jobs in the
best_effort_jobs queue are allocated resources.
Figure 2: Example part of
fair-scheduler.xml and corresponding fair shares of hierarchical queues
To get more information about Fair Scheduler, take a look at the onlinedocumentation.
- The scheduler is a part of a computer operating system that allocates resources to active processes as needed.
- A cluster scheduler allocates resources to an application running on the cluster. The cluster scheduler is designed for multi-tenancy and scalability.
- YARN allows you to choose from a set of schedulers. Fair Scheduler is widely used. In its simplest form, it shares resources fairly among all jobs running on the cluster.
- Fair Scheduler assigns applications to queues. You can set properties on queues to adjust scheduling behavior, based on application and departmental needs. Queue weights are one way to control the fair share for applications in the queue. Shares are allocated top-down from the
root, evaluated one level at a time.