what stage dir in yarn framework use for?

hi,mailist: i see yarn.app.mapreduce.am.staging-dir in doc ,and i do not know it use for ,and i also want to know if the content in this dir can be clean, and if it can be set auto clean?

RE: Hadoop SAN Storage reuse

Great, Thanks. Thanks and Regards Prabakaran.N aka NP nsn, Bangalore When "I" is replaced by "We" - even Illness becomes "Wellness" From: ext John Lilley [mailto:john.lil...@redpoint.net] Sent: Friday, June 13, 2014 3:34 AM To: user@hadoop.apache.org Subject: RE: Hadoop SAN Storage reuse Hadoo

Compare Yarn with V1

Is there a good resource that draws similarity and compares Yarn's resource manager, application manager etc. with job tracker, task tracker etc.?

hdfs over http error

Hello, Apache Hadoop 0.20.203.0 A colleague is using a SPARK shell on a remote host using HDFS protocol attempting to run a job on our Hadoop cluster, but the job errors out before finishing with the following noted in the namenode log. 2014-06-11 16:13:24,958 WARN org.apache.hadoop.se

monitor mapper running execution times with hadoop 2.2.0

Hi, I would like to monitor the average execution time of mappers and reducers or something better to check the hadoop throughput. I configured the hadoop metrics2 as follows: *.sink.ganglia.period=10 *.sink.ganglia.supportsparse=true *.sink.ganglia.servers=GANGLIA_SERVER_IP:8649 mrappmaster.sin

RE: Hadoop SAN Storage reuse

Hadoop performs best when each node has exclusive use of its disk. Therefore, if you have a choice, try to provision exclusive use of individual spindles on your SAN and map each one to a separate mount on your Hadoop nodes. Anything other than that will tend to produce poor performance due to

Re: Building hadoop 2.2.1 from source code

Thanks Tomas, but this can't help me now : ) Now i try change my mvn local repo via export MAVEN_OPTS="-Dmaven.repo.local=`pwd`/../.m2" mvn clean package -DskipTests produce new error: [INFO] BUILD FAILURE [INFO] [INFO] To

Re: Building hadoop 2.2.1 from source code

Hi, If you are able to read Spanish, in https://bitacoras.citius.usc.es/tecnologia/2014/06/05/hadoop-para-64-bits/ it's described step by step how to compile Hadoop 2.4 in a CentOS 6.5 64 bits. Cheers Tomas En 12/06/14 18:37, Lukas Drbal escribiu: > Hi all, > > i have a problem with build hado

Re: Building hadoop 2.2.1 from source code

Hi Ted, clean and package ending with same error. One of them: [ERROR] /home/lestr/data/sbks-deps/hadoop2.2/hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServicesContainers

Re: Building hadoop 2.2.1 from source code

You may need to add the 'install' target the first time you build (and every time you build clean thereafter). Your 'java -version' and 'mvn -version' report different versions of Java. Check your JAVA_HOME. On Thu, Jun 12, 2014 at 9:49 AM, Ted Yu wrote: > Can you run the following command

Re: Building hadoop 2.2.1 from source code

Can you run the following command first ? mvn clean package -DskipTests Here is the version of maven I use: mvn -version Apache Maven 3.1.1 (0728685237757ffbf44136acec0402957f723d9a; 2013-09-17 15:22:22+) Cheers On Thu, Jun 12, 2014 at 9:37 AM, Lukas Drbal wrote: > Hi all, > > i have a

Building hadoop 2.2.1 from source code

Hi all, i have a problem with build hadoop from source code. I take git from https://github.com/apache/hadoop-common and branch-2.2.1 and try mvn package -Pdist,native -DskipTests -Dtar but it return a lot errors. Here is log from mvn https://gist.github.com/anonymous/052b6d45f64be01dab43 My en

How to overwrite Key in RecordReader function

DA, We are trying to write a UDF to read an XML which contains some unbounded tags. For repeated tags, new row has to be generated. Please let us know how to ovewrite the default key with the new key in the Record Reader function (where we do for loop to make multiple rows). *Sample XML:*

Re: Counters in MapReduce

yes rectified that error But after 1 st iteration when it enters to second iteration showing java.io.FileNotFoundException: for *Path out1 = new Path(CL);* *Why is it so .* *Normally that should be in this way only the o/p folder should not exist* * //other configuration* * job1.setMapperCl

Re: Why resource requests are normalized in RM ?

I believe they are normalized to be multiples of yarn.scheduler.increment-allocation-mb. yarn.scheduler.minimum-allocation-mb can be set to as low as zero. Llama does this. As to why normalization, I think it is to make sure there is no external fragmentation. It is similar to why memory is paged.

Re: Yarn HA - Zookeeper ACLs

Hi Manoj Firstly, one can choose to leave that config alone. If not set, the ACLs are automatically generated such that all RMs have shared admin access but exclusive create-delete access. For the exclusive create-delete access, the RMs use username:password where the username is yarn.resourcemana