Re: How to set up yarn queue name

Chris Riccomini Mon, 08 Dec 2014 10:33:54 -0800

Hey Jae,

I've opened:

  https://issues.apache.org/jira/browse/SAMZA-491

To track this. Could you please post your patch there?

Cheers,
Chris

From: Chris Riccomini <[email protected]<mailto:[email protected]>>
Date: Monday, December 8, 2014 10:30 AM
To: "[email protected]<mailto:[email protected]>" 
<[email protected]<mailto:[email protected]>>
Subject: Re: How to set up yarn queue name

Hey Jae,

I don't believe Samza supports this right now. We've always run in the default 
queue. This is definitely a feature we should add. Would you be willing to try 
to patch your Samza library to fix it?

The place to fix it is in ClientHelper.scala class. The submitApplicaiton() 
method has an appCtx class. I believe you'll need to set the queue there using 
ApplicationSubmissionContext.setQueue.

The steps are:

1) Add a `yarn.queue` field to YarnConfig.
2) Update YarnJob.scala's submit() method to get the yarn queue, and use it in 
the client.submitApplication() call.
3) Update ClientHelper.scala's submitApplication() method to take an optional 
queue, and if it's set, add it to the ApplicationSubmissionContext, as I 
described above.

Cheers,
Chris

From: <Bae>, Jae Hyeon <[email protected]<mailto:[email protected]>>
Reply-To: 
"[email protected]<mailto:[email protected]>" 
<[email protected]<mailto:[email protected]>>
Date: Monday, December 8, 2014 10:15 AM
To: "[email protected]<mailto:[email protected]>" 
<[email protected]<mailto:[email protected]>>
Subject: How to set up yarn queue name

Hello

I tried submit Samza job in our EMR hadoop cluster but I got the following 
error:

[Inline image 1]

So, I asked our company hadoop admin and I got advised to set up the queue 
name. I added the following line in Samza job property but it didn't work:

mapreduce.job.queuename=root.adhoc
yarn.scheduler.fair.user-as-default-queue=false

Do you know how to set up yarn queue name?

FYI, I copied our hadoop cluster fair scheduler policy. We disabled default 
queue.

<?xml version="1.0"?>
<allocations>
    <queue name="adhoc" type="parent">
        <weight>10</weight>
        <maxRunningApps>300</maxRunningApps>
        <queue name="nsharma">
           <weight>5</weight>
        </queue>
    </queue>

    <queue name="default">
        <weight>0.0</weight>
    </queue>

    <queue name="emergencyuseonly">
        <weight>0.0</weight>
        <minResources>0 mb, 0 vcores</minResources>
        <minSharePreemptionTimeout>10000</minSharePreemptionTimeout>
    </queue>

    <queuePlacementPolicy>
        <rule name="nestedUserQueue">
            <rule name="specified" create="false" />
        </rule>
        <rule name="specified" create="false" />
        <rule name="reject" />
    </queuePlacementPolicy>

    <userMaxAppsDefault>20</userMaxAppsDefault>
    <fairSharePreemptionTimeout>60</fairSharePreemptionTimeout>
</allocations>

Thank you
Best, Jae

Re: How to set up yarn queue name

Reply via email to