Error while building Hadoop-Yarn
Hi, I am using apache-maven-3.0.3 and i have set LD_LIBRARY_PATH=/usr/local/lib which has google protocbuf library. I am getting following error while building hadoop-yarn using mvn clean install -DskipTests=true [INFO] hadoop-yarn-api ... SUCCESS [14.904s] [INFO] hadoop-yarn-common SUCCESS [8.787s] [INFO] hadoop-yarn-server-common . SUCCESS [4.691s] [INFO] hadoop-yarn-server-nodemanager FAILURE [6.051s] [INFO] hadoop-yarn-server-resourcemanager SKIPPED [INFO] hadoop-yarn-server-tests .. SKIPPED [INFO] hadoop-yarn-server SKIPPED [INFO] hadoop-yarn ... SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 34.870s [INFO] Finished at: Fri Aug 19 11:48:22 IST 2011 [INFO] Final Memory: 44M/107M [ERROR] Failed to execute goal org.codehaus.mojo:make-maven-plugin:1.0-beta-1:autoreconf (autoreconf) on project hadoop-yarn-server-nodemanager: autoreconf command returned an exit value != 0. Aborting build; see debug output for more information. - [Help 1] Thanks in advance Thanks Regards Rajesh Putta
Re: Error while building Hadoop-Yarn
Thanks Arun, Now its working fine Thanks Regards Rajesh Putta Development Engineer Pramati Technologies On Fri, Aug 19, 2011 at 12:25 PM, Arun Murthy a...@hortonworks.com wrote: That means you don't have the autotool chain necessary for build the native code. For now pass -P-cbuild to skip them. Arun Sent from my iPhone On Aug 18, 2011, at 11:26 PM, rajesh putta rajesh.p...@gmail.com wrote: Hi, I am using apache-maven-3.0.3 and i have set LD_LIBRARY_PATH=/usr/local/lib which has google protocbuf library. I am getting following error while building hadoop-yarn using mvn clean install -DskipTests=true [INFO] hadoop-yarn-api ... SUCCESS [14.904s] [INFO] hadoop-yarn-common SUCCESS [8.787s] [INFO] hadoop-yarn-server-common . SUCCESS [4.691s] [INFO] hadoop-yarn-server-nodemanager FAILURE [6.051s] [INFO] hadoop-yarn-server-resourcemanager SKIPPED [INFO] hadoop-yarn-server-tests .. SKIPPED [INFO] hadoop-yarn-server SKIPPED [INFO] hadoop-yarn ... SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 34.870s [INFO] Finished at: Fri Aug 19 11:48:22 IST 2011 [INFO] Final Memory: 44M/107M [ERROR] Failed to execute goal org.codehaus.mojo:make-maven-plugin:1.0-beta-1:autoreconf (autoreconf) on project hadoop-yarn-server-nodemanager: autoreconf command returned an exit value != 0. Aborting build; see debug output for more information. - [Help 1] Thanks in advance Thanks Regards Rajesh Putta
RE: Wonky reduce progress
Please check the defect in MAPREDUCE jira https://issues.apache.org/jira/browse/MAPREDUCE-2264 This is because the compression is enabled for map outputs and statistics are taken on compressed data instead of original data. -Original Message- From: Joey Echeverria [mailto:j...@cloudera.com] Sent: Friday, August 19, 2011 3:07 AM To: d...@hive.apache.org Subject: Wonky reduce progress I'm seeing really weird numbers in the JobTracker when monitoring jobs launched by hives. Some jobs who above 100% completion while running and others show less than 100% after the job completed. I tried searching for JIRAs but didn't see any matches. Has anyone else run into this? -Joey -- Joseph Echeverria Cloudera, Inc. 443.305.9434
SequenceFileInputFormat doesn't return whole records
Hi all, I am having issues using SequenceFileInputFormat to retrieve whole records I have 1 job that is used to write to a SequenceFile SequenceFileOutputFormat.setOutputPath(job, new Path(out/data)); SequenceFileOutputFormat.setOutputCompressionType(job, SequenceFile.CompressionType.NONE); I then have a second job that is ment to read the file for processing SequenceFileInputFormat.addInputPath(job, new Path(out/data)); However, the values that i get as the arguments to the Map part of my job only seems to contain parts of the record. I am sure that i am missing something rather fundamental as to how Hadoop splits inputs to the Mapper, but can't seem to find a way to stop the records being split. Any help (or a pointer to a specific page in the doc) would be greatly appreciated Regards, Tim
Re: SequenceFileInputFormat doesn't return whole records
Tim, Do you also set your I/O formats explicitly to SequenceFileInputFormat and SequenceFileOutputFormat? Via job.setInputFormat/setOutputFormat I mean. Hadoop should not be splitting records across maps/mappers. There are specific test cases that ensure this does not happen, so it would seem strange if it does this. On Fri, Aug 19, 2011 at 6:01 PM, Tim Fletcher zigomu...@gmail.com wrote: Hi all, I am having issues using SequenceFileInputFormat to retrieve whole records I have 1 job that is used to write to a SequenceFile SequenceFileOutputFormat.setOutputPath(job, new Path(out/data)); SequenceFileOutputFormat.setOutputCompressionType(job, SequenceFile.CompressionType.NONE); I then have a second job that is ment to read the file for processing SequenceFileInputFormat.addInputPath(job, new Path(out/data)); However, the values that i get as the arguments to the Map part of my job only seems to contain parts of the record. I am sure that i am missing something rather fundamental as to how Hadoop splits inputs to the Mapper, but can't seem to find a way to stop the records being split. Any help (or a pointer to a specific page in the doc) would be greatly appreciated Regards, Tim -- Harsh J
Re: SequenceFileInputFormat doesn't return whole records
Harsh, that was exactly the issue! Thanks very much for your help Tim On 19 August 2011 15:15, Harsh J ha...@cloudera.com wrote: Tim, Do you also set your I/O formats explicitly to SequenceFileInputFormat and SequenceFileOutputFormat? Via job.setInputFormat/setOutputFormat I mean. Hadoop should not be splitting records across maps/mappers. There are specific test cases that ensure this does not happen, so it would seem strange if it does this. On Fri, Aug 19, 2011 at 6:01 PM, Tim Fletcher zigomu...@gmail.com wrote: Hi all, I am having issues using SequenceFileInputFormat to retrieve whole records I have 1 job that is used to write to a SequenceFile SequenceFileOutputFormat.setOutputPath(job, new Path(out/data)); SequenceFileOutputFormat.setOutputCompressionType(job, SequenceFile.CompressionType.NONE); I then have a second job that is ment to read the file for processing SequenceFileInputFormat.addInputPath(job, new Path(out/data)); However, the values that i get as the arguments to the Map part of my job only seems to contain parts of the record. I am sure that i am missing something rather fundamental as to how Hadoop splits inputs to the Mapper, but can't seem to find a way to stop the records being split. Any help (or a pointer to a specific page in the doc) would be greatly appreciated Regards, Tim -- Harsh J
Re: Wonky reduce progress
Yup, that would do it. Thanks. -Joey On Fri, Aug 19, 2011 at 4:14 AM, Vinayakumar B vinayakuma...@huawei.com wrote: Please check the defect in MAPREDUCE jira https://issues.apache.org/jira/browse/MAPREDUCE-2264 This is because the compression is enabled for map outputs and statistics are taken on compressed data instead of original data. -Original Message- From: Joey Echeverria [mailto:j...@cloudera.com] Sent: Friday, August 19, 2011 3:07 AM To: d...@hive.apache.org Subject: Wonky reduce progress I'm seeing really weird numbers in the JobTracker when monitoring jobs launched by hives. Some jobs who above 100% completion while running and others show less than 100% after the job completed. I tried searching for JIRAs but didn't see any matches. Has anyone else run into this? -Joey -- Joseph Echeverria Cloudera, Inc. 443.305.9434 -- Joseph Echeverria Cloudera, Inc. 443.305.9434
