RE: Pin Map/Reduce tasks to specific cores

Hi George, The current implementation present in YARN using Cgroups supports CPU isolation but not by pinning to specific cores (Cgroup CPUsets) but based on cpu cycles (quota & Period). Admin is provided with an option of specifying how much percentage of CPU can be used by YARN containers. An

RE: Pin Map/Reduce tasks to specific cores

Hi George In MRV2, YARN supports CGroups implementation. Using CGroup it is possible to run containers in specific cores. For your detailed reference, some of the useful links http://dev.hortonworks.com.s3.amazonaws.com/HDPDocuments/HDP2/HDP-2-trunk/bk_system-admin-guide/content/ch_cgroups.html

答复: Yarn container out of memory when using large memory mapped file

mapreduce.reduce.memory.mb means physical memory, not JVM heap. The large mapped files (about 8G total) is more than 4G(mapreduce.reduce.memory.mb=4096), so you got the error. 发件人: Yao, York [mailto:york@here.com] 发送时间: 2015年4月5日 6:36 收件人: user@hadoop.apache.org 主题: Yarn container out of mem

Why replication of Under-Replicated blocks in decommissioned datanodes is so slow

version: hadoop-2.2.0 There were 13 nodes in our hdfs cluster. We wanted to decommission 7 nodes. We used two methods as follow: Method 1: At the beginning, we set the dfs.hosts.exclude parameter and successfully decommissioned 7 nodes, so there were many Under-Replicated blocks need to replic

Pin Map/Reduce tasks to specific cores

Hello. My question, which can be found on *Stack Overflow * as well, regards pinning map/reduce tasks to specific cores, either on hadoop v.1.2.1 or hadoop v.2. In specific, I would like to know if the end-user can ha

YARN resource manager in ubuntu 14.04: Problem in starting http server. Server handlers failed

Hi all, I am trying to get YARN running in ubuntu 14.04 with not a lot of luck, specially the Resource Manager. See error at the end of the email. I am using CDH 5.3 that supports ubuntu 14.04 The same settings (using the same salt formulas as configuration management) on ubuntu 12.04 are workin

Re: Question about log files

This depends on your OS. When you "delete" a file on Linux, you merely unlink the entry from the directory. The file does not actually get deleted until until the last reference (open handle) goes away. Note that this could lead to an interesting way to fill up a disk. You should be able to see

unsubscribe

Kindly Send email to user-unsubscr...@hadoop.apache.org Subject: Re: Hadoop 2.6 issue To: user@hadoop.apache.org From: rapen...@in.ibm.com Date: Thu, 2 Apr 2015 09:02:09 +0530 Please un subscribe me from this list. Regards, Ravi Prasad Pentakota India Software Lab, IBM Software Group Phone

Re: Question about log files

I noticed that too, I think Hadoop keeps the file open all the time and when you delete it it is just no more able to write on it and doesn't try to recreate it. Not sure if it's a Log4j problem or an Hadoop one... yanghaogn, which is the *correct* way to delete the Hadoop logs? I didn't find anyth

Re: File permission of Hadoop staging directory getting changed time to time

try running with sudo hive. On Mon, Apr 6, 2015 at 1:40 PM, Inosh Goonewardena wrote: > Hi All, > > Really appreciate any input on this issue. > > Thanks and Regards, > Inosh > > On Tue, Jan 20, 2015 at 10:29 PM, Inosh Goonewardena > wrote: > >> Hi All, >> >> We have setup a Hadoop cluster usin

Tracking job failure using APIs

Hi All, How can I track a job failure on node or list of nodes, using YARN apis. I could get the list of long running jobs, using yarn client API, but need to go further to AM, NM, task attempts for map or reduce. Say, I have a job running for long,(about 4hours), might be caused of some task fai

Re: File permission of Hadoop staging directory getting changed time to time

Hi All, Really appreciate any input on this issue. Thanks and Regards, Inosh On Tue, Jan 20, 2015 at 10:29 PM, Inosh Goonewardena wrote: > Hi All, > > We have setup a Hadoop cluster using hadoop 1.0.4 and we use Hive to > submit map/reduce jobs. But we have noticed that these map/reduce job >

Re: Question about log files

I think the log information has lost. the hadoop is not designed for that you deleted these files incorrectly 2015-04-02 11:45 GMT+08:00 煜 韦 : > Hi there, > If log files are deleted without restarting service, it seems that the > logs is to be lost for later operation. For example, on namenode,