Untangling Apache Hadoop YARN, Part 3

Part 3: Scheduler Concepts

By Ray Chiang and Dennis Dawson

In Parts 1 and 2, we covered the basics of YARN resource allocation. In this installment, we’ll provide an overview of cluster scheduling and introduce the Fair Scheduler, one of the scheduler choices available in YARN.

A standalone computer can have several CPU cores, each running a single process, but there can be as many as a few hundred processes running simultaneously. The scheduler is a part of the desktop’s operating system that assigns a process to a CPU core to run for a short period of time.

As described previously in “Part 1: YARN and Cluster Basics”, an application consists of multiple tasks (usually on different hosts) on the cluster. A cluster scheduler essentially has to address:

  • Multi-tenancy: On a cluster, many users launch many different applications, on behalf of multiple organizations. A cluster scheduler allows varying workloads to run simultaneously.
  • Scalability: A cluster scheduler needs to scale to large clusters running many applications. This means that increasing the size of the cluster should improve overall performance without negatively affecting system latencies.

Scheduling in YARN

The ResourceManager (RM) tracks resources on a cluster, and assigns them to applications that need them. The scheduler is that part of the RM that does this matching honoring organizational policies on sharing resources. Please note that:

  • YARN uses queues to share resources among multiple tenants. This will be covered in more detail in “Introducing Queues” below.
  • The ApplicationMaster (AM) tracks each task’s resource requirements and coordinates container requests. This approach allows better scaling since the RM/scheduler doesn’t need to track all containers running on the cluster.

Fair Scheduler

The Fair Scheduler is a popular choice (recommended by Cloudera) among the schedulers YARN supports. In its simplest form, it shares resources fairly among all jobs running on the cluster. The next few sections explain scheduler internals in the context of Fair Scheduler and elaborate on the commonly used controls the scheduler offers.

Introducing Queues

Queues are the organizing structure for YARN schedulers, allowing multiple tenants to share the cluster. As applications are submitted to YARN, they are assigned to a queue by the scheduler. The root queue is the parent of all queues. All other queues are each a child of the rootqueue or another queue (also calledhierarchical queues). It is common for queues to correspond to users, departments, or priorities.

Figure 1 presents a basic Fair Scheduler example of fair-scheduler.xml and a graphical representation of each queue’s share of the cluster.

  • There are four departmental queues (marketing, sales,datascience, and admin).
  • There is a special admin queue that that users fred or greg, who are assumed to be administrators, can use.

untangling-yarn-3-f1

Figure 1: Example part of fair-scheduler.xml and corresponding fair shares of each queue

Using Hierarchical Queues

One level of queues allows sharing the cluster along one dimension (e.g. one queue per team), but it is common to share clusters among multiple dimensions (e.g. per-team and priority). Fair Scheduler allows nesting queues to form a hierarchical queue structure, where each level could correspond to a dimension under its parent queue.

For example, Figure 2 below shows resources shared along team and priority dimensions. The first level corresponds to the team (e.g.root.datascience), and each first-level queue could have a high- and low-priority child queues for jobs from that specific team (e.g.root.datascience.short_jobs androot.datascience.best_effort_jobs).

Queue Naming

In the rest of this post and subsequent parts, we refer to queue names in a different font like root orroot.sales.northamerica. Where unambiguous, we use the shortest version possible, i.e. northamericainstead ofroot.sales.northamerica.

Example #1: Weights in a single queue

In the sales queue, there are two child queues: northamerica andeurope. Each has a weight of 30.0, so the fair share for each of the child queues within sales is effectively 50%.

In the marketing queue, there are two child queues of nonequal weight:reports and website. This means that jobs in the reports queue are allocated twice as many resources as jobs in the website queue. However, together, their weight is still governed by the marketing queue’s weight.

Queue Weights and Top-Down Scheduling

Queue weights are used to determine the fair share for a queue. The Fair Scheduler starts from the rootqueue and looks at the weights of all immediate child queues to determine their fair share. Each child queue’s fair share is further evaluated for its set of child queues.

Example #2: Top-down view of weights

The marketing queue has a weight of 3.0, the sales queue has a weight of 4.0, and the datascience queue has a weight of 13.0. So, the allocation from the root will be 15% to marketing, 20% to sales, and 65% todatascience.

Of the share that goes todatascience, all of the queue’s allocation goes to the short_jobsqueue. If there are no jobs assigned to the short_jobs queue, then the jobs in the best_effort_jobs queue are allocated resources.

untangling-yarn-3-f2

Figure 2: Example part of fair-scheduler.xml and corresponding fair shares of hierarchical queues

Extra Reading

To get more information about Fair Scheduler, take a look at the onlinedocumentation.

Conclusion

  • The scheduler is a part of a computer operating system that allocates resources to active processes as needed.
  • A cluster scheduler allocates resources to an application running on the cluster. The cluster scheduler is designed for multi-tenancy and scalability.
  • YARN allows you to choose from a set of schedulers. Fair Scheduler is widely used. In its simplest form, it shares resources fairly among all jobs running on the cluster.
  • Fair Scheduler assigns applications to queues. You can set properties on queues to adjust scheduling behavior, based on application and departmental needs. Queue weights are one way to control the fair share for applications in the queue. Shares are allocated top-down from theroot, evaluated one level at a time.
Advertisements

2 thoughts on “Untangling Apache Hadoop YARN, Part 3

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s