Re: Hadoop fair scheduler doubt: allocate jobs to pool
From the fairscheduler docs I assume the following should work:
property
namemapred.fairscheduler.poolnameproperty/name
valuepool.name/value
/property
property
namepool.name/name
value${mapreduce.job.group.name}/value
/property
which means that the default pool will be the group of the user that has
submitted the job. In your case I think that allocations.xml is correct. If
you want to explicitly define a job to specific pool from your
allocation.xml file you can define it as follows:
Configuration conf3 = conf;
conf3.set(pool.name, pool3); // conf.set(propriety.name, value)
Let me know if it works..
On 29 February 2012 14:18, Austin Chungath austi...@gmail.com wrote:
How can I set the fair scheduler such that all jobs submitted from a
particular user group go to a pool with the group name?
I have setup fair scheduler and I have two users: A and B (belonging to the
user group hadoop)
When these users submit hadoop jobs, the jobs from A got to a pool named A
and the jobs from B go to a pool named B.
I want them to go to a pool with their group name, So I tried adding the
following to mapred-site.xml:
property
namemapred.fairscheduler.poolnameproperty/name
valuegroup.name/value
/property
But instead the jobs now go to the default pool.
I want the jobs submitted by A and B to go to the pool named hadoop. How
do I do that?
also how can I explicity set a job to any specified pool?
I have set the allocation file (fair-scheduler.xml) like this:
allocations
pool name=hadoop
minMaps1/minMaps
minReduces1/minReduces
maxMaps3/maxMaps
maxReduces3/maxReduces
/pool
userMaxJobsDefault5/userMaxJobsDefault
/allocations
Any help is greatly appreciated.
Thanks,
Austin
RE: Hadoop fair scheduler doubt: allocate jobs to pool
I've just started playing with the Fair Scheduler. To specify the pool at job
submission time you set the mapred.fairscheduler.pool property on the Job
Conf to the name of the pool you want the job to use.
Dave
-Original Message-
From: Merto Mertek [mailto:masmer...@gmail.com]
Sent: Thursday, March 01, 2012 9:33 AM
To: common-user@hadoop.apache.org
Subject: Re: Hadoop fair scheduler doubt: allocate jobs to pool
From the fairscheduler docs I assume the following should work:
property
namemapred.fairscheduler.poolnameproperty/name
valuepool.name/value
/property
property
namepool.name/name
value${mapreduce.job.group.name}/value
/property
which means that the default pool will be the group of the user that has
submitted the job. In your case I think that allocations.xml is correct. If you
want to explicitly define a job to specific pool from your allocation.xml file
you can define it as follows:
Configuration conf3 = conf;
conf3.set(pool.name, pool3); // conf.set(propriety.name, value)
Let me know if it works..
On 29 February 2012 14:18, Austin Chungath austi...@gmail.com wrote:
How can I set the fair scheduler such that all jobs submitted from a
particular user group go to a pool with the group name?
I have setup fair scheduler and I have two users: A and B (belonging
to the user group hadoop)
When these users submit hadoop jobs, the jobs from A got to a pool
named A and the jobs from B go to a pool named B.
I want them to go to a pool with their group name, So I tried adding
the following to mapred-site.xml:
property
namemapred.fairscheduler.poolnameproperty/name
valuegroup.name/value
/property
But instead the jobs now go to the default pool.
I want the jobs submitted by A and B to go to the pool named hadoop.
How do I do that?
also how can I explicity set a job to any specified pool?
I have set the allocation file (fair-scheduler.xml) like this:
allocations
pool name=hadoop
minMaps1/minMaps
minReduces1/minReduces
maxMaps3/maxMaps
maxReduces3/maxReduces
/pool
userMaxJobsDefault5/userMaxJobsDefault
/allocations
Any help is greatly appreciated.
Thanks,
Austin
The information contained in this email message is considered confidential and
proprietary to the sender and is intended solely for review and use by the
named recipient. Any unauthorized review, use or distribution is strictly
prohibited. If you have received this message in error, please advise the
sender by reply email and delete the message.
Re: Hadoop fair scheduler doubt: allocate jobs to pool
Thanks,
I will be trying the suggestions and will get back to you soon.
On Thu, Mar 1, 2012 at 8:09 PM, Dave Shine
dave.sh...@channelintelligence.com wrote:
I've just started playing with the Fair Scheduler. To specify the pool at
job submission time you set the mapred.fairscheduler.pool property on the
Job Conf to the name of the pool you want the job to use.
Dave
-Original Message-
From: Merto Mertek [mailto:masmer...@gmail.com]
Sent: Thursday, March 01, 2012 9:33 AM
To: common-user@hadoop.apache.org
Subject: Re: Hadoop fair scheduler doubt: allocate jobs to pool
From the fairscheduler docs I assume the following should work:
property
namemapred.fairscheduler.poolnameproperty/name
valuepool.name/value
/property
property
namepool.name/name
value${mapreduce.job.group.name}/value
/property
which means that the default pool will be the group of the user that has
submitted the job. In your case I think that allocations.xml is correct. If
you want to explicitly define a job to specific pool from your
allocation.xml file you can define it as follows:
Configuration conf3 = conf;
conf3.set(pool.name, pool3); // conf.set(propriety.name, value)
Let me know if it works..
On 29 February 2012 14:18, Austin Chungath austi...@gmail.com wrote:
How can I set the fair scheduler such that all jobs submitted from a
particular user group go to a pool with the group name?
I have setup fair scheduler and I have two users: A and B (belonging
to the user group hadoop)
When these users submit hadoop jobs, the jobs from A got to a pool
named A and the jobs from B go to a pool named B.
I want them to go to a pool with their group name, So I tried adding
the following to mapred-site.xml:
property
namemapred.fairscheduler.poolnameproperty/name
valuegroup.name/value
/property
But instead the jobs now go to the default pool.
I want the jobs submitted by A and B to go to the pool named hadoop.
How do I do that?
also how can I explicity set a job to any specified pool?
I have set the allocation file (fair-scheduler.xml) like this:
allocations
pool name=hadoop
minMaps1/minMaps
minReduces1/minReduces
maxMaps3/maxMaps
maxReduces3/maxReduces
/pool
userMaxJobsDefault5/userMaxJobsDefault
/allocations
Any help is greatly appreciated.
Thanks,
Austin
The information contained in this email message is considered confidential
and proprietary to the sender and is intended solely for review and use by
the named recipient. Any unauthorized review, use or distribution is
strictly prohibited. If you have received this message in error, please
advise the sender by reply email and delete the message.
Re: Hadoop fair scheduler doubt: allocate jobs to pool
Hi,
I tried what you had said. I added the following to mapred-site.xml:
property
namemapred.fairscheduler.poolnameproperty/name
valuepool.name/value
/property
property
namepool.name/name
value${mapreduce.job.group.name}/value
/property
Funny enough it created a pool with the name ${mapreduce.job.group.name}
so I tried ${mapred.job.group.name} and ${group.name} all to the same
effect.
But when I did ${user.name} it worked! and created a pool with the user
name.
On Thu, Mar 1, 2012 at 8:03 PM, Merto Mertek masmer...@gmail.com wrote:
From the fairscheduler docs I assume the following should work:
property
namemapred.fairscheduler.poolnameproperty/name
valuepool.name/value
/property
property
namepool.name/name
value${mapreduce.job.group.name}/value
/property
which means that the default pool will be the group of the user that has
submitted the job. In your case I think that allocations.xml is correct. If
you want to explicitly define a job to specific pool from your
allocation.xml file you can define it as follows:
Configuration conf3 = conf;
conf3.set(pool.name, pool3); // conf.set(propriety.name, value)
Let me know if it works..
On 29 February 2012 14:18, Austin Chungath austi...@gmail.com wrote:
How can I set the fair scheduler such that all jobs submitted from a
particular user group go to a pool with the group name?
I have setup fair scheduler and I have two users: A and B (belonging to
the
user group hadoop)
When these users submit hadoop jobs, the jobs from A got to a pool named
A
and the jobs from B go to a pool named B.
I want them to go to a pool with their group name, So I tried adding the
following to mapred-site.xml:
property
namemapred.fairscheduler.poolnameproperty/name
valuegroup.name/value
/property
But instead the jobs now go to the default pool.
I want the jobs submitted by A and B to go to the pool named hadoop.
How
do I do that?
also how can I explicity set a job to any specified pool?
I have set the allocation file (fair-scheduler.xml) like this:
allocations
pool name=hadoop
minMaps1/minMaps
minReduces1/minReduces
maxMaps3/maxMaps
maxReduces3/maxReduces
/pool
userMaxJobsDefault5/userMaxJobsDefault
/allocations
Any help is greatly appreciated.
Thanks,
Austin
