Configure Hadoop YARN CapacityScheduler on Amazon EMR on Amazon EC2 for multi-tenant heterogeneous workloads

Apache Hadoop YARN (But One other Useful resource Negotiator) is a cluster useful resource supervisor accountable for assigning computational assets (CPU, reminiscence, I/O), and scheduling and monitoring jobs submitted to a Hadoop cluster. This generic framework permits for efficient administration of cluster assets for distributed information processing frameworks, akin to Apache Spark, Apache MapReduce, and Apache Hive. When supported by the framework, Amazon EMR by default makes use of Hadoop YARN. Please word that not all frameworks provided by Amazon EMR use Hadoop YARN, akin to Trino/Presto and Apache HBase.

On this publish, we focus on numerous elements of Hadoop YARN, and perceive how elements work together with one another to allocate assets, schedule purposes, and monitor purposes. We dive deep into the particular configurations to customise Hadoop YARN’s CapacityScheduler to extend cluster effectivity by allocating assets in a well timed and safe method in a multi-tenant cluster. We take an opinionated have a look at the configurations for CapacityScheduler and configure them on Amazon EMR on Amazon Elastic Compute Cloud (Amazon EC2) to resolve for the frequent useful resource allocation, useful resource rivalry, and job scheduling challenges in a multi-tenant cluster.

We dive deep into CapacityScheduler as a result of Amazon EMR makes use of CapacityScheduler by default, and CapacityScheduler has advantages over different schedulers for operating workloads with heterogeneous useful resource consumption.

Answer overview

Trendy information platforms usually run purposes on Amazon EMR with the next traits:

  • Heterogeneous useful resource consumption patterns by jobs, akin to computation-bound jobs, I/O-bound jobs, or memory-bound jobs
  • A number of groups operating jobs with an expectation to obtain an agreed-upon share of cluster assets and full jobs in a well timed method
  • Cluster admins usually must cater to one-time requests for operating jobs with out impacting scheduled jobs
  • Cluster admins need to guarantee customers are utilizing their assigned capability and never utilizing others
  • Cluster admins need to make the most of the assets effectively and allocate all out there assets to at present operating jobs, however need to retain the flexibility to reclaim assets routinely ought to there be a declare for the agreed-upon cluster assets from different jobs

As an instance these use circumstances, let’s contemplate the next state of affairs:

  • user1 and user2 don’t belong to any crew and use cluster assets periodically on an advert hoc foundation
  • An information platform and analytics program has two groups:
    • A data_engineering crew, containing user3
    • A data_science crew, containing user4
  • user5 and user6 (and plenty of different customers) sporadically use cluster assets to run jobs

Primarily based on this state of affairs, the scheduler queue might seem like the next diagram. Be aware of the frequent configurations utilized to all queues, the overrides, and the person/groups-to-queue mappings.

Capacity Scheduler Queue Setup

Within the subsequent sections, we’ll perceive the high-level elements of Hadoop YARN, focus on the assorted kinds of schedulers out there in Hadoop YARN, evaluate the core ideas of CapacityScheduler, and showcase how one can implement this CapacityScheduler queue setup on Amazon EMR (on Amazon EC2). You may skip to Code walkthrough part in case you are already conversant in Hadoop YARN and CapacityScheduler.

Overview of Hadoop YARN

At a excessive stage, Hadoop YARN consists of three fundamental elements:

  • ResourceManager (one per main node)
  • ApplicationMaster (one per utility)
  • NodeManager (one per node)

The next diagram exhibits the primary elements and their interplay with one another.

Apache Hadoop Yarn Architecture Diagram1

Earlier than diving additional, let’s make clear what Hadoop YARN’s ResourceContainer (or container) is. A ResourceContainer represents a group of bodily computational assets. It’s an abstraction used to bundle assets into distinct, allocatable unit.


The ResourceManager is accountable for useful resource administration and making allocation choices. It’s the ResourceManager’s accountability to establish and allocate assets to a job upon submission to Hadoop YARN. The ResourceManager has two fundamental elements:

  • ApplicationsManager (to not be confused with ApplicationMaster)
  • Scheduler


The ApplicationsManager is accountable for accepting job submissions, negotiating the primary container for operating ApplicationMaster, and offering the service for restarting the ApplicationMaster on failure.


The Scheduler is accountable for scheduling allocation of assets to the roles. The Scheduler performs its scheduling operate primarily based on the useful resource necessities of the roles. The Scheduler is a pluggable interface. Hadoop YARN at present offers three implementations:

  • CapacityScheduler – A pluggable scheduler for Hadoop that permits for a number of tenants to securely share a cluster such that jobs are allotted assets in a well timed method below constraints of allotted capacities. The implementation is offered on GitHub. The Java concrete class is org.apache.hadoop.yarn.server.resourcemanager.scheduler.capability.CapacityScheduler. On this publish, we primarily concentrate on CapacityScheduler, which is the default scheduler on Amazon EMR (on Amazon EC2).
  • FairScheduler – A pluggable scheduler for Hadoop that permits Hadoop YARN purposes to share assets in clusters pretty. The implementation is offered on GitHub. The Java concrete class is org.apache.hadoop.yarn.server.resourcemanager.scheduler.truthful.FairScheduler.
  • FifoScheduler – A pluggable scheduler for Hadoop that permits Hadoop YARN purposes share assets in clusters in a first-in-first-out foundation. The implementation is offered on GitHub. The Java concrete class is org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.


Upon negotiating the primary container by ApplicationsManager, the per-application ApplicationMaster has the accountability of negotiating the remainder of the suitable assets from the Scheduler, monitoring their standing, and monitoring progress.


The NodeManager is accountable for launching and managing containers on a node.

Hadoop YARN on Amazon EMR

By default, Amazon EMR (on Amazon EC2) makes use of Hadoop YARN for cluster administration for the distributed information processing frameworks that help Hadoop YARN as a useful resource supervisor, like Apache Spark, Apache MapReduce, and Apache Hive. Amazon EMR offers a number of smart default settings that work for many situations. Nonetheless, each information platform is completely different and has particular wants. Amazon EMR offers the flexibility to customise the setting at cluster creation utilizing configuration classifications . You can even reconfigure Amazon EMR cluster purposes and specify further configuration classifications for every occasion group in a operating cluster utilizing AWS Command Line Interface (AWS CLI), or the AWS SDK.


CapacityScheduler relies on ResourceCalculator to establish the out there assets and calculate the allocation of the assets to ApplicationMaster. The ResourceCalculator is an summary Java class. Hadoop YARN at present offers two implementations:

  • DefaultResourceCalculator – In DefaultResourceCalculator, assets are calculated primarily based on reminiscence alone.
  • DominantResourceCalculatorDominantResourceCalculator is predicated on the Dominant Useful resource Equity (DRF) mannequin of useful resource allocation. The paper Dominant Useful resource Equity: Truthful Allocation of A number of Useful resource Sorts, Ghodsi et al. [2011] describes DRF as follows: “DRF computes the share of every useful resource allotted to that person. The utmost amongst all shares of a person is known as that person’s dominant share, and the useful resource comparable to the dominant share is known as the dominant useful resource. Totally different customers might have completely different dominant assets. For instance, the dominant useful resource of a person operating a computation-bound job is CPU, whereas the dominant useful resource of a person operating an I/O-bound job is bandwidth. DRF merely applies max-min equity throughout customers’ dominant shares. That’s, DRF seeks to maximise the smallest dominant share within the system, then the second-smallest, and so forth.”

Due to DRF, DominantResourceCalculator is a greater ResourceCalculator for information processing environments operating heterogeneous workloads. By default, Amazon EMR makes use of DefaultResourceCalculator for CapacityScheduler. This may be verified by checking the worth of yarn.scheduler.capability.resource-calculator parameter in /and so forth/hadoop/conf/capacity-scheduler.xml.

Code walkthrough

CapacityScheduler offers a number of parameters to customise the scheduling habits to satisfy particular wants. For a listing of accessible parameters, discuss with Hadoop: CapacityScheduler.

Seek advice from the configurations part in cloudformation/templates/emr.yaml to evaluate all of the CapacityScheduler parameters set as a part of this publish. On this instance, we use two classifiers of Amazon EMR (on Amazon EC2):

  • yarn-site – The classification to replace yarn-site.xml
  • capacity-scheduler – The classification to replace capacity-scheduler.xml

For numerous kinds of classification out there in Amazon EMR, discuss with Customizing cluster and utility configuration with earlier AMI variations of Amazon EMR.

Within the AWS CloudFormation template, now we have modified the ResourceCalculator of CapacityScheduler from the defaults, DefaultResourceCalculator to DominantResourceCalculator. Knowledge processing environments tends to run completely different sorts of jobs, for instance, computation-bound jobs consuming heavy CPU, I/O-bound jobs consuming heavy bandwidth, and memory-bound jobs consuming heavy reminiscence. As beforehand acknowledged, DominantResourceCalculator is best suited to such environments as a consequence of its Dominant Useful resource Equity mannequin of useful resource allocation. In case your information processing surroundings solely runs memory-bound jobs, then modifying this parameter isn’t vital.

You could find the codebase within the AWS Samples GitHub repository.


For deploying the answer, it is best to have the next stipulations:

Deploy the answer

To deploy the answer, full the next steps:

  • Obtain the supply code from the AWS Samples GitHub repository:
    git clone [email protected]:aws-samples/amazon-emr-yarn-capacity-scheduler.git

  • Create an Amazon Easy Storage Service (Amazon S3) bucket:
    aws s3api create-bucket --bucket emr-yarn-capacity-scheduler-<AWS_ACCOUNT_ID>-<AWS_REGION> --region <AWS_REGION>

  • Copy the cloned repository to the Amazon S3 bucket:
    aws s3 cp --recursive amazon-emr-yarn-capacity-scheduler s3://emr-yarn-capacity-scheduler-<AWS_ACCOUNT_ID>-<AWS_REGION>/

    1. ArtifactsS3Repository – The S3 bucket title that was created within the earlier step (emr-yarn-capacity-scheduler-<AWS_ACCOUNT_ID>-<AWS_REGION>).
    2. emrKeyName – An present EC2 key title. Should you don’t have an present key and need to create a brand new key, discuss with Use an Amazon EC2 key pair for SSH credentials.
    3. clientCIDR – The CIDR vary of the shopper machine for accessing the EMR cluster through SSH. You may run the next command to establish the IP of the shopper machine: echo "$(curl -s"
  • Deploy the AWS CloudFormation templates:
    aws cloudformation create-stack 
    --stack-name emr-yarn-capacity-scheduler 
    --template-url https://emr-yarn-capacity-scheduler-<AWS_ACCOUNT_ID>-<AWS_REGION> 
    --parameters file://amazon-emr-yarn-capacity-scheduler/cloudformation/parameters/parameters.json 
    --capabilities CAPABILITY_NAMED_IAM 
    --region <AWS_REGION>

  • On the AWS CloudFormation console, examine for the profitable deployment of the next stacks.

AWS CloudFormation Stack Deployment

  • On the Amazon EMR console, examine for the profitable creation of the emr-cluster-capacity-scheduler cluster.
  • Select the cluster and on the Configurations tab, evaluate the properties below the capacity-scheduler and yarn-site classification labels.

AWS EMR Configurations

Apache Hadoop YARN UI

    • /and so forth/hadoop/conf/yarn-site.xml
    • /and so forth/hadoop/conf/capacity-scheduler.xml

All of the parameters set utilizing the yarn-site and capacity-scheduler classifiers are mirrored in these information. If an admin desires to replace CapacityScheduler configs, they’ll immediately replace capacity-scheduler.xml and run the next command to use the adjustments with out interrupting any operating jobs and providers:

yarn rmadmin -resfreshQueues

Modifications to yarn-site.xml require the ResourceManager service to be restarted, which interrupts the operating jobs. As a greatest observe, chorus from handbook modifications and use model management for change administration.

The CloudFormation template provides a bootstrap motion to create check customers (user1, user2, user3, user4, user5 and user6) on all of the nodes and provides a step script to create HDFS directories for the check customers.

Customers can SSH into the  main node, sudo as completely different customers and submit Spark jobs to confirm the job submission and CapacityScheduler habits:

[[email protected] ~]$ sudo su - user1
[[email protected] ~]$ spark-submit --master yarn --deploy-mode cluster 
--class org.apache.spark.examples.SparkPi /usr/lib/spark/examples/jars/spark-examples.jar

You may validate the outcomes from the useful resource supervisor internet UI.

Apache Hadoop YARN Jobs List

Clear up

To keep away from incurring future expenses, delete the assets you created.

  • Delete the CloudFormation stack:
    aws cloudformation delete-stack --stack-name emr-yarn-capacity-scheduler

  • Delete the S3 bucket:
    aws s3 rb s3://emr-yarn-capacity-scheduler-<AWS_ACCOUNT_ID>-<AWS_REGION> --force

The command deletes the bucket and all information beneath it. The information might not be recoverable after deletion.


On this publish, we mentioned Apache Hadoop YARN and its numerous elements. We mentioned the kinds of schedulers out there in Hadoop YARN. We dived deep in to the specifics of Hadoop YARN CapacityScheduler and using Dominant Useful resource Equity to effectively allocate assets to submitted jobs. We additionally showcased how one can implement the mentioned ideas utilizing AWS CloudFormation.

We encourage you to make use of this publish as a place to begin to implement CapacityScheduler on Amazon EMR (on Amazon EC2) and customise the answer to satisfy your particular information platform objectives.

In regards to the authors

Suvojit Dasgupta is a Sr. Lakehouse Architect at Amazon Internet Providers. He works with clients to design and construct information options on AWS.

Bharat Gamini is a Knowledge Architect targeted on huge information and analytics at Amazon Internet Providers. He helps clients architect and construct extremely scalable, strong, and safe cloud-based analytical options on AWS.

Leave a Reply

Your email address will not be published.