Re: Capacity Scheduler capacity vs. maximum-capacity

2.0 onwards. See > https://issues.apache.org/jira/browse/YARN-569. > > Arun > > On Feb 21, 2014, at 8:28 AM, ricky l wrote: > > Does Hadoop capacity scheduler support preemption in this scenario? > Based on what Vinod says, the preemption seems to be supported by > configuration. If so, ca

Re: meaning or usage of reserved containers in YARN Capacity scheduler

s available... else, > this application would be 'starved' if other apps came in and asked for 1G > or 2G and would hence, 4G would never be freed up. > > Hope that helps. > > Arun > > On Feb 7, 2014, at 7:58 AM, ricky l wrote: > > Hi all, > > I have a qu

Re: Capacity Scheduler capacity vs. maximum-capacity

Does Hadoop capacity scheduler support preemption in this scenario? Based on what Vinod says, the preemption seems to be supported by configuration. If so, can someone point me an instruction to do that? The preemption would really be helpful for my use-case. thanks. On Fri, Feb 21, 2014 at 12:39

Re: change the Yarn application container memory size when it is running

not to overallocate with Xmx in different processes at >> the same time. Can do weird things when it starts using swap. >> >> >> best >> /th >> >> >> >> >> On Mon, 2014-02-10 at 12:32 -0500, ricky l wrote: >> > Hi All, >> >

change the Yarn application container memory size when it is running

Hi All, Can I change the allocated memory size of a container after a container has started (running)? For example, in the beginning, I want to allocate 1GB of memory to a container A and later I want to allocate 2GB, and later 512MB. Is it a possible scenario? thx.

meaning or usage of reserved containers in YARN Capacity scheduler

Hi all, I have a question about reserved containers in the YARN capacity scheduler. After reading the source code and related document, it is not very clear. What is the purpose or practical usage of the reserved container? thx.

making yarn-site.xml file modification effect without restarting daemons

Hi all, After changing the configuration value of yarn-site.xml, can I make the modifications in effect without restarting all daemons? More specifically, after I modify "yarn.nodemanager.resource.memory-mb" value, I want to make the change in-effect without restarting node manager. In addition, c

Re: expressing job anti-affinity in Yarn.

Thanks. I checked the issue and it seems still "unresolved". Can I assume the feature is not supported yet? On Thu, Jan 9, 2014 at 12:00 PM, Ted Yu wrote: > See: > YARN-1042 add ability to specify affinity/anti-affinity in container > requests > > > On Thu, Jan

expressing job anti-affinity in Yarn.

Hi all, Is it possible to express the job anti-affinity in the Yarn-based hadoop? I have a job that is very IO-intensive, and I want to spread the tasks across all available machines. In a default Yarn RM scheduler, it seems many tasks are scheduled in one machine while others are idle. thanks.

enabling priority jobs to Hadoop with Yarn

Hi, Can someone give advice for the following scenario? I have a MR pool with Yarn; the total number of available containers is 100. At a given time, a lower priority job is submitted and consumes all 100 containers. Later, I submit a higher priority job and it is not running as the all containers

using yarn.nodemanager.container-monitor.resource-calculator.class option in Yarn

Hi all, In the hadoop-3.0.0-SNAPSHOT I set the below option hoping that it will throttle a container that over-utilize its resources. yarn.nodemanager.container-monitor.resource-calculator.class org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin If I start a nodemanager with the a

Re: In YARN, how does a task tracker knows the address of a job tracker?

h all mapper, reducer tasks as well as the client. > > John > > > > > > *From:* ricky l [mailto:rickylee0...@gmail.com] > *Sent:* Thursday, November 21, 2013 12:36 PM > *To:* user@hadoop.apache.org > *Subject:* Re: In YARN, how does a task tracker knows the address

Re: In YARN, how does a task tracker knows the address of a job tracker?

lly gets executed. > > Thanks, > Omkar Joshi > *Hortonworks Inc.* <http://www.hortonworks.com> > > > On Thu, Nov 21, 2013 at 7:52 AM, ricky l wrote: > >> Hi all, >> >> I have a question of how a task tracker identifies job tracker address >> when I submit

In YARN, how does a task tracker knows the address of a job tracker?

Hi all, I have a question of how a task tracker identifies job tracker address when I submit MR job through YARN. As far as I know, both job tracker and task trackers are launched through application master and I am curious about the details about job and task tracker launch sequence. thanks.

Re: question about preserving data locality in MapReduce with Yarn

Hi Sandy, thank you very much for the information. It is good to know that MapReduce AM considers the block location information. BTW, I am not very familiar with the concept of splits. Is it specific to MR jobs? If possible, code location would be very helpful for reference as I am trying to imple