[jira] [Created] (MAPREDUCE-3086) Supporting range scan using TFile, TotalOrderPartitioner and partition index
Supporting range scan using TFile, TotalOrderPartitioner and partition index Key: MAPREDUCE-3086 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3086 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Binglin Chang Fix For: 0.20.204.0, 0.23.0 Hive/HBase already has similar or more powerful functionality, but using hive/hbase is overkill or inconvenient for some cases, so add some lightweight utility classes to only support range scan should be reasonable. The utility classes include: # InputFormat supporting range scan: Indexed(Text|Binary)InputFormat The input directory for IndexInputFormat should contain one partition index and many tfiles, each tfile store a certain range of keys, not overlapping with other tfiles, the boundaries are stored in partition index. Add 4 jobconfs: mapred.indexed(text|binary)inputformat.key.(start|end), indicate range scan parameters. For a mapreduce job using IndexedInputFormat, IndexedInputFormat.getSplits filter out tfiles which are not in the scan range using partition index IndexedInputFormat do not support multi directory & splitting in single file, these can be added in future. # Tool to convert data of other format into IndexedInputForamt: TotalOrderIndexBuilder If the input data is already total order partitioned and is tfile format, just add partition index to input directory Or run InputSampler to generate partiton index, then run mapreduce job with TotalOrder partitioner to generate tfile backed data, finally move partition index to output directory. # Client tool to scan/search indexed data directory -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Hadoop-Mapreduce-trunk-Commit - Build # 968 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/968/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 14048 lines...] [junit] Test org.apache.hadoop.mapred.TestMapRed FAILED [junit] Running org.apache.hadoop.mapred.TestMiniMRDFSCaching [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 67.559 sec [junit] Test org.apache.hadoop.mapred.TestMiniMRDFSCaching FAILED [junit] Running org.apache.hadoop.mapred.TestQueueAclsForCurrentUser [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.581 sec [junit] Running org.apache.hadoop.mapred.TestRackAwareTaskPlacement [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.528 sec [junit] Running org.apache.hadoop.mapred.TestReduceFetchFromPartialMem [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 29.345 sec [junit] Running org.apache.hadoop.mapred.TestReduceTask [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.614 sec [junit] Running org.apache.hadoop.mapred.TestSequenceFileAsBinaryInputFormat [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.777 sec [junit] Running org.apache.hadoop.mapred.TestSequenceFileAsBinaryOutputFormat [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 1.096 sec [junit] Running org.apache.hadoop.mapred.TestSequenceFileInputFormat [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 5.96 sec [junit] Running org.apache.hadoop.mapred.TestSeveral [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 47.252 sec [junit] Running org.apache.hadoop.mapred.TestSpeculativeExecution [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 4.007 sec [junit] Running org.apache.hadoop.mapred.TestTaskLimits [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 3.894 sec [junit] Running org.apache.hadoop.mapred.TestTaskTrackerBlacklisting [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 1.743 sec [junit] Running org.apache.hadoop.mapred.TestTextInputFormat [junit] Tests run: 8, Failures: 0, Errors: 0, Time elapsed: 87.269 sec [junit] Running org.apache.hadoop.mapred.TestTextOutputFormat [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.177 sec [junit] Running org.apache.hadoop.mapred.TestTrackerBlacklistAcrossJobs [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 43.564 sec [junit] Running org.apache.hadoop.mapreduce.TestCounters [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.344 sec [junit] Running org.apache.hadoop.mapreduce.TestMapCollection [junit] Tests run: 11, Failures: 0, Errors: 11, Time elapsed: 0.617 sec [junit] Test org.apache.hadoop.mapreduce.TestMapCollection FAILED [junit] Running org.apache.hadoop.mapreduce.TestMapReduceLocal [junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 47.973 sec [junit] Test org.apache.hadoop.mapreduce.TestMapReduceLocal FAILED [junit] Running org.apache.hadoop.mapreduce.lib.input.TestFileInputFormat [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.951 sec [junit] Running org.apache.hadoop.mapreduce.lib.output.TestFileOutputCommitter [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.581 sec checkfailure: [touch] Creating /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/testsfailed BUILD FAILED /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:792: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:755: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:816: Tests failed! Total time: 8 minutes 41 seconds Build step 'Execute shell' marked build as failure Recording test results Updating MAPREDUCE-2990 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## All tests passed
Hadoop-Mapreduce-trunk-Commit - Build # 969 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/969/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 14030 lines...] [junit] Running org.apache.hadoop.mapred.TestMapRed [junit] Tests run: 5, Failures: 2, Errors: 3, Time elapsed: 1.327 sec [junit] Test org.apache.hadoop.mapred.TestMapRed FAILED [junit] Running org.apache.hadoop.mapred.TestMiniMRDFSCaching [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 30.02 sec [junit] Running org.apache.hadoop.mapred.TestQueueAclsForCurrentUser [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.617 sec [junit] Running org.apache.hadoop.mapred.TestRackAwareTaskPlacement [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.523 sec [junit] Running org.apache.hadoop.mapred.TestReduceFetchFromPartialMem [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 48.126 sec [junit] Running org.apache.hadoop.mapred.TestReduceTask [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.616 sec [junit] Running org.apache.hadoop.mapred.TestSequenceFileAsBinaryInputFormat [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.766 sec [junit] Running org.apache.hadoop.mapred.TestSequenceFileAsBinaryOutputFormat [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 1.021 sec [junit] Running org.apache.hadoop.mapred.TestSequenceFileInputFormat [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.774 sec [junit] Running org.apache.hadoop.mapred.TestSeveral [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 43.467 sec [junit] Running org.apache.hadoop.mapred.TestSpeculativeExecution [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 4.148 sec [junit] Running org.apache.hadoop.mapred.TestTaskLimits [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 3.892 sec [junit] Running org.apache.hadoop.mapred.TestTaskTrackerBlacklisting [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 1.719 sec [junit] Running org.apache.hadoop.mapred.TestTextInputFormat [junit] Tests run: 8, Failures: 0, Errors: 0, Time elapsed: 65.382 sec [junit] Running org.apache.hadoop.mapred.TestTextOutputFormat [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.182 sec [junit] Running org.apache.hadoop.mapred.TestTrackerBlacklistAcrossJobs [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 41.625 sec [junit] Running org.apache.hadoop.mapreduce.TestCounters [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.349 sec [junit] Running org.apache.hadoop.mapreduce.TestMapCollection [junit] Tests run: 11, Failures: 0, Errors: 11, Time elapsed: 0.627 sec [junit] Test org.apache.hadoop.mapreduce.TestMapCollection FAILED [junit] Running org.apache.hadoop.mapreduce.TestMapReduceLocal [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 28.252 sec [junit] Running org.apache.hadoop.mapreduce.lib.input.TestFileInputFormat [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.937 sec [junit] Running org.apache.hadoop.mapreduce.lib.output.TestFileOutputCommitter [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.586 sec checkfailure: [touch] Creating /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/testsfailed BUILD FAILED /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:792: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:755: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:816: Tests failed! Total time: 6 minutes 11 seconds Build step 'Execute shell' marked build as failure Recording test results Updating MAPREDUCE-3053 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## All tests passed
Hadoop-Mapreduce-0.23-Build - Build # 25 - Failure
See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/25/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 18137 lines...] Running org.apache.hadoop.yarn.util.TestYarnVersionInfo Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec Running org.apache.hadoop.yarn.util.TestLinuxResourceCalculatorPlugin Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec Results : Failed tests: testUnknownCall(org.apache.hadoop.yarn.TestRPC): null expected:<...icationId called on []org.apache.hadoop.ya...> but was:<...icationId called on [interface ]org.apache.hadoop.ya...> Tests run: 65, Failures: 1, Errors: 0, Skipped: 0 [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-yarn-api ... SUCCESS [8.756s] [INFO] hadoop-yarn-common FAILURE [6.076s] [INFO] hadoop-yarn-server-common . SKIPPED [INFO] hadoop-yarn-server-nodemanager SKIPPED [INFO] hadoop-yarn-server-resourcemanager SKIPPED [INFO] hadoop-yarn-server-tests .. SKIPPED [INFO] hadoop-yarn-server SKIPPED [INFO] hadoop-yarn ... SKIPPED [INFO] hadoop-mapreduce-client-core .. SKIPPED [INFO] hadoop-mapreduce-client-common SKIPPED [INFO] hadoop-mapreduce-client-shuffle ... SKIPPED [INFO] hadoop-mapreduce-client-app ... SKIPPED [INFO] hadoop-mapreduce-client-hs SKIPPED [INFO] hadoop-mapreduce-client-jobclient . SKIPPED [INFO] hadoop-mapreduce-client ... SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 15.231s [INFO] Finished at: Sun Sep 25 13:11:17 UTC 2011 [INFO] Final Memory: 24M/378M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.7.2:test (default-test) on project hadoop-yarn-common: There are test failures. [ERROR] [ERROR] Please refer to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Build/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/target/surefire-reports for the individual test results. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hadoop-yarn-common Build step 'Execute shell' marked build as failure Archiving artifacts Updating MAPREDUCE-2961 Updating MAPREDUCE-3053 Updating HDFS-2290 Updating HADOOP-7663 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
Re: Jobs not running after MAPREDUCE-2880
Ravi, the mailing list gulps in any attachments. Can you create a ticket and attach your logs there? That helps for better tracking of the issue too. Thanks, +Vinod On Sat, Sep 24, 2011 at 1:09 AM, Ravi Prakash wrote: > Hi Arun, > > Unsecure single node. > > I'm attaching the classpath I grepped | sort | uniq from the two task.sh > files I got (one from the working version and the other from the notworking > version). Looks like the classpath which worked had some other jars not > present in the new classpath. > > I'm guessing as part of the simplification for CLASSPATH maybe we missed > something that was being included earlier? > > Thanks > Ravi > > > > On Fri, Sep 23, 2011 at 12:18 PM, Arun Murthy wrote: > >> This is secure mode or unsecured? Cluster or single node? Tx >> >> Sent from my iPhone >> >> On Sep 23, 2011, at 8:37 AM, Ravi Prakash wrote: >> >> > Hi Arun/Vinod, >> > >> > After commit d4dca4eabf83a97d158f1e1caa4801020679d5e2 >> > Date: Wed Sep 21 18:52:27 2011 + >> > MAPREDUCE-2880. svn merge -c r1173783 --ignore-ancestry ../../trunk/ >> > git-svn-id: >> > >> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.23@117379213f79535-47bb-0310-9956-ffa450edef68 >> > >> > My mapreduce jobs are failing >> > 2011-09-23 10:27:08,534 INFO ipc.HadoopYarnRPC >> > (HadoopYarnProtoRPC.java:getProxy(49)) - Creating a HadoopYarnProtoRpc >> proxy >> > for protocol interface >> org.apache.hadoop.mapreduce.v2.api.MRClientProtocol >> > 2011-09-23 10:27:08,634 INFO mapreduce.Job >> > (Job.java:monitorAndPrintJob(1209)) - Running job: >> job_1316791524705_0002 >> > 2011-09-23 10:27:09,653 INFO mapreduce.Job >> > (Job.java:monitorAndPrintJob(1229)) - map 0% reduce 0% >> > 2011-09-23 10:27:16,739 INFO mapreduce.Job >> > (Job.java:monitorAndPrintJob(1242)) - Job job_1316791524705_0002 failed >> with >> > state FAILED >> > 2011-09-23 10:27:16,786 INFO mapreduce.Job >> > (Job.java:monitorAndPrintJob(1246)) - Counters: 0 >> > >> > Digging into the stderr logs: I see this single line >> > Exception in thread "main" java.lang.NoClassDefFoundError: >> > org/apache/hadoop/mapreduce/v2/app/MRAppMaster >> > >> > What do I need to add to my environment / config so that the magic >> happens >> > again? >> > >> > Thanks >> > Ravi. >> > >
Hadoop-Mapreduce-trunk-Commit - Build # 970 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/970/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 14035 lines...] [junit] Running org.apache.hadoop.mapred.TestMapRed [junit] Tests run: 5, Failures: 2, Errors: 3, Time elapsed: 1.317 sec [junit] Test org.apache.hadoop.mapred.TestMapRed FAILED [junit] Running org.apache.hadoop.mapred.TestMiniMRDFSCaching [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 32.778 sec [junit] Running org.apache.hadoop.mapred.TestQueueAclsForCurrentUser [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.58 sec [junit] Running org.apache.hadoop.mapred.TestRackAwareTaskPlacement [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.518 sec [junit] Running org.apache.hadoop.mapred.TestReduceFetchFromPartialMem [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 32.076 sec [junit] Running org.apache.hadoop.mapred.TestReduceTask [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.609 sec [junit] Running org.apache.hadoop.mapred.TestSequenceFileAsBinaryInputFormat [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.749 sec [junit] Running org.apache.hadoop.mapred.TestSequenceFileAsBinaryOutputFormat [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 1.018 sec [junit] Running org.apache.hadoop.mapred.TestSequenceFileInputFormat [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 5.289 sec [junit] Running org.apache.hadoop.mapred.TestSeveral [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 42.92 sec [junit] Running org.apache.hadoop.mapred.TestSpeculativeExecution [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 4.142 sec [junit] Running org.apache.hadoop.mapred.TestTaskLimits [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 3.891 sec [junit] Running org.apache.hadoop.mapred.TestTaskTrackerBlacklisting [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 1.725 sec [junit] Running org.apache.hadoop.mapred.TestTextInputFormat [junit] Tests run: 8, Failures: 0, Errors: 0, Time elapsed: 84.316 sec [junit] Running org.apache.hadoop.mapred.TestTextOutputFormat [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.182 sec [junit] Running org.apache.hadoop.mapred.TestTrackerBlacklistAcrossJobs [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 47.045 sec [junit] Running org.apache.hadoop.mapreduce.TestCounters [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.349 sec [junit] Running org.apache.hadoop.mapreduce.TestMapCollection [junit] Tests run: 11, Failures: 0, Errors: 11, Time elapsed: 0.625 sec [junit] Test org.apache.hadoop.mapreduce.TestMapCollection FAILED [junit] Running org.apache.hadoop.mapreduce.TestMapReduceLocal [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 28.346 sec [junit] Running org.apache.hadoop.mapreduce.lib.input.TestFileInputFormat [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.951 sec [junit] Running org.apache.hadoop.mapreduce.lib.output.TestFileOutputCommitter [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.583 sec checkfailure: [touch] Creating /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/testsfailed BUILD FAILED /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:792: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:755: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:816: Tests failed! Total time: 6 minutes 22 seconds Build step 'Execute shell' marked build as failure Recording test results Updating MAPREDUCE-2952 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## All tests passed
Calculations of the InputSplits
Hi, There was a query in StackOverflow regarding high CPU on the client after submitting jobs (upto 200 jobs in batch and 150MB jar file size). Calculation of the InputSplit may be one of the reason for the high CPU on the client. Why should the calculation of the InputSplit happen on the client? JobTracker is a high-end machine, can't the calculation happen on the JobTracker? http://stackoverflow.com/questions/7546064/hadoop-high-cpu-load-on-client-side-after-committing-jobs Thanks, Praveen
[jira] [Created] (MAPREDUCE-3087) CLASSPATH not the same after MAPREDUCE-2880
CLASSPATH not the same after MAPREDUCE-2880 --- Key: MAPREDUCE-3087 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3087 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Ravi Prakash After MAPREDUCE-2880, my classpath was missing key jar files. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Jobs not running after MAPREDUCE-2880
Thank Vinod, Filed https://issues.apache.org/jira/browse/MAPREDUCE-3087 Regards Ravi. On Sun, Sep 25, 2011 at 9:51 AM, Vinod Kumar Vavilapalli < vino...@hortonworks.com> wrote: > Ravi, the mailing list gulps in any attachments. Can you create a ticket > and > attach your logs there? That helps for better tracking of the issue too. > > Thanks, > +Vinod > > > On Sat, Sep 24, 2011 at 1:09 AM, Ravi Prakash > wrote: > > > Hi Arun, > > > > Unsecure single node. > > > > I'm attaching the classpath I grepped | sort | uniq from the two task.sh > > files I got (one from the working version and the other from the > notworking > > version). Looks like the classpath which worked had some other jars not > > present in the new classpath. > > > > I'm guessing as part of the simplification for CLASSPATH maybe we missed > > something that was being included earlier? > > > > Thanks > > Ravi > > > > > > > > On Fri, Sep 23, 2011 at 12:18 PM, Arun Murthy > wrote: > > > >> This is secure mode or unsecured? Cluster or single node? Tx > >> > >> Sent from my iPhone > >> > >> On Sep 23, 2011, at 8:37 AM, Ravi Prakash wrote: > >> > >> > Hi Arun/Vinod, > >> > > >> > After commit d4dca4eabf83a97d158f1e1caa4801020679d5e2 > >> > Date: Wed Sep 21 18:52:27 2011 + > >> > MAPREDUCE-2880. svn merge -c r1173783 --ignore-ancestry ../../trunk/ > >> > git-svn-id: > >> > > >> > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.23@117379213f79535-47bb-0310-9956-ffa450edef68 > >> > > >> > My mapreduce jobs are failing > >> > 2011-09-23 10:27:08,534 INFO ipc.HadoopYarnRPC > >> > (HadoopYarnProtoRPC.java:getProxy(49)) - Creating a HadoopYarnProtoRpc > >> proxy > >> > for protocol interface > >> org.apache.hadoop.mapreduce.v2.api.MRClientProtocol > >> > 2011-09-23 10:27:08,634 INFO mapreduce.Job > >> > (Job.java:monitorAndPrintJob(1209)) - Running job: > >> job_1316791524705_0002 > >> > 2011-09-23 10:27:09,653 INFO mapreduce.Job > >> > (Job.java:monitorAndPrintJob(1229)) - map 0% reduce 0% > >> > 2011-09-23 10:27:16,739 INFO mapreduce.Job > >> > (Job.java:monitorAndPrintJob(1242)) - Job job_1316791524705_0002 > failed > >> with > >> > state FAILED > >> > 2011-09-23 10:27:16,786 INFO mapreduce.Job > >> > (Job.java:monitorAndPrintJob(1246)) - Counters: 0 > >> > > >> > Digging into the stderr logs: I see this single line > >> > Exception in thread "main" java.lang.NoClassDefFoundError: > >> > org/apache/hadoop/mapreduce/v2/app/MRAppMaster > >> > > >> > What do I need to add to my environment / config so that the magic > >> happens > >> > again? > >> > > >> > Thanks > >> > Ravi. > >> > > > > >
Hadoop-Mapreduce-0.23-Build - Build # 26 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/26/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 18137 lines...] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.12 sec Running org.apache.hadoop.yarn.util.TestCompositeService Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018 sec Running org.apache.hadoop.yarn.util.TestYarnVersionInfo Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec Running org.apache.hadoop.yarn.util.TestLinuxResourceCalculatorPlugin Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec Results : Failed tests: testUnknownCall(org.apache.hadoop.yarn.TestRPC): null expected:<...icationId called on []org.apache.hadoop.ya...> but was:<...icationId called on [interface ]org.apache.hadoop.ya...> Tests run: 65, Failures: 1, Errors: 0, Skipped: 0 [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-yarn-api ... SUCCESS [9.786s] [INFO] hadoop-yarn-common FAILURE [6.151s] [INFO] hadoop-yarn-server-common . SKIPPED [INFO] hadoop-yarn-server-nodemanager SKIPPED [INFO] hadoop-yarn-server-resourcemanager SKIPPED [INFO] hadoop-yarn-server-tests .. SKIPPED [INFO] hadoop-yarn-server SKIPPED [INFO] hadoop-yarn ... SKIPPED [INFO] hadoop-mapreduce-client-core .. SKIPPED [INFO] hadoop-mapreduce-client-common SKIPPED [INFO] hadoop-mapreduce-client-shuffle ... SKIPPED [INFO] hadoop-mapreduce-client-app ... SKIPPED [INFO] hadoop-mapreduce-client-hs SKIPPED [INFO] hadoop-mapreduce-client-jobclient . SKIPPED [INFO] hadoop-mapreduce-client ... SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 16.334s [INFO] Finished at: Sun Sep 25 17:48:38 UTC 2011 [INFO] Final Memory: 24M/317M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.7.2:test (default-test) on project hadoop-yarn-common: There are test failures. [ERROR] [ERROR] Please refer to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Build/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/target/surefire-reports for the individual test results. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hadoop-yarn-common Build step 'Execute shell' marked build as failure Archiving artifacts Updating MAPREDUCE-2952 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
Hadoop-Mapreduce-trunk-Commit - Build # 971 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/971/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 9214 lines...]
create-c++-utils-configure:
create-c++-pipes-configure:
create-c++-examples-pipes-configure:
create-c++-configure:
ivy-download:
[get] Getting:
http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.2.0/ivy-2.2.0.jar
[get] To:
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/ivy/ivy-2.2.0.jar
[get] Not modified - so not downloaded
ivy-init-dirs:
ivy-probe-antlib:
ivy-init-antlib:
ivy-init:
ivy-resolve-common:
ivy-retrieve-common:
ivy-resolve-mapred:
ivy-retrieve-mapred:
init:
[touch] Creating /tmp/null1134386612
[delete] Deleting: /tmp/null1134386612
[unzip] Expanding:
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/ivy/lib/Hadoop/common/hadoop-hdfs-0.24.0-SNAPSHOT.jar
into
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build
avro-generate:
compile-mapred-classes:
[jsp-compile] log4j:WARN No appenders could be found for logger
(org.apache.jasper.JspC).
[jsp-compile] log4j:WARN Please initialize the log4j system properly.
[jsp-compile] log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[javac] Compiling 48 source files to
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/classes
[javac]
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobTrackerClientProtocolProvider.java:40:
cannot find symbol
[javac] symbol : variable CLASSIC_FRAMEWORK_NAME
[javac] location: interface org.apache.hadoop.mapreduce.MRConfig
[javac] if (!MRConfig.CLASSIC_FRAMEWORK_NAME.equals(framework)) {
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 1 error
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:398:
Compile failed; see the compiler error output for details.
Total time: 14 seconds
Build step 'Execute shell' marked build as failure
Recording test results
Updating MAPREDUCE-3064
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
Re: Calculations of the InputSplits
Hello Praveen, That is a valid point. Besides, it can even be a task that computes the splits (Safer this way, instead of running _inside_ the JobTracker). Lets continue the discussion on https://issues.apache.org/jira/browse/MAPREDUCE-207 which seems very relevant to this. On Sun, Sep 25, 2011 at 10:12 PM, Praveen Sripati wrote: > Hi, > > There was a query in StackOverflow regarding high CPU on the client after > submitting jobs (upto 200 jobs in batch and 150MB jar file size). > Calculation of the InputSplit may be one of the reason for the high CPU on > the client. Why should the calculation of the InputSplit happen on the > client? JobTracker is a high-end machine, can't the calculation happen on > the JobTracker? > > http://stackoverflow.com/questions/7546064/hadoop-high-cpu-load-on-client-side-after-committing-jobs > > Thanks, > Praveen > -- Harsh J
Re: Calculations of the InputSplits
The reason it isn't done in JobTracker is to not run user-code within the framework - InputSplit.getSplits() is user code. In MRv1 is was highly complicated - in MRv2 it's trivial to do it the MR ApplicationMaster, I'll get to it some wknd soon - patches welcome! :) Arun On Sep 25, 2011, at 9:42 AM, Praveen Sripati wrote: > Hi, > > There was a query in StackOverflow regarding high CPU on the client after > submitting jobs (upto 200 jobs in batch and 150MB jar file size). > Calculation of the InputSplit may be one of the reason for the high CPU on > the client. Why should the calculation of the InputSplit happen on the > client? JobTracker is a high-end machine, can't the calculation happen on > the JobTracker? > > http://stackoverflow.com/questions/7546064/hadoop-high-cpu-load-on-client-side-after-committing-jobs > > Thanks, > Praveen
Hadoop-Mapreduce-22-branch - Build # 74 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/74/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 2241 lines...] A src/examples/org/apache/hadoop/examples/pi/TaskResult.java A src/examples/org/apache/hadoop/examples/pi/Container.java A src/examples/org/apache/hadoop/examples/pi/SummationWritable.java A src/examples/org/apache/hadoop/examples/pi/DistBbp.java A src/examples/org/apache/hadoop/examples/pi/DistSum.java A src/examples/org/apache/hadoop/examples/pi/math A src/examples/org/apache/hadoop/examples/pi/math/LongLong.java A src/examples/org/apache/hadoop/examples/pi/math/Montgomery.java A src/examples/org/apache/hadoop/examples/pi/math/Modular.java A src/examples/org/apache/hadoop/examples/pi/math/ArithmeticProgression.java A src/examples/org/apache/hadoop/examples/pi/math/Bellard.java A src/examples/org/apache/hadoop/examples/pi/math/Summation.java A src/examples/org/apache/hadoop/examples/pi/math/package.html A src/examples/org/apache/hadoop/examples/pi/Combinable.java A src/examples/org/apache/hadoop/examples/pi/Parser.java A src/examples/org/apache/hadoop/examples/pi/Util.java A src/examples/org/apache/hadoop/examples/pi/package.html A src/examples/org/apache/hadoop/examples/AggregateWordCount.java A src/examples/org/apache/hadoop/examples/Grep.java A bin A bin/mapred-config.sh AUbin/stop-mapred.sh AUbin/mapred AUbin/start-mapred.sh A build-utils.xml A build.xml U. Fetching 'https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/common/src/test/bin' at -1 into '/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-22-branch/trunk/src/test/bin' AUsrc/test/bin/test-patch.sh A src/test/bin/test-patch.properties At revision 1175628 At revision 1175628 Checking out http://svn.apache.org/repos/asf/hadoop/nightly AUtar-munge A commitBuild.sh A hudsonEnv.sh A jenkinsSetup A jenkinsSetup/installTools.sh AUhudsonBuildHadoopNightly.sh A buildMR-279Branch.sh AUhudsonBuildHadoopPatch.sh AUhudsonBuildHadoopRelease.sh AUprocessHadoopPatchEmailRemote.sh AUhudsonPatchQueueAdmin.sh AUprocessHadoopPatchEmail.sh A README.txt A test-patch A test-patch/test-patch.sh At revision 1175628 SCM check out aborted [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Publishing Clover coverage report... No Clover report will be published due to a Build Failure Recording test results Publishing Javadoc Recording fingerprints Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Created] (MAPREDUCE-3088) Clover 2.4.3 breaks build for 0.22 branch
Clover 2.4.3 breaks build for 0.22 branch - Key: MAPREDUCE-3088 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3088 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Fix For: 0.22.0 Due to known bug in Clover 2.4.3 build for 0.22 branch is broken. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3052) Maintain consistency in naming appIDs, jobIDs and attemptIDs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-3052. -- Resolution: Duplicate Fix Version/s: (was: 0.24.0) (was: 0.23.0) Duplicate of MAPREDUCE-2793 > Maintain consistency in naming appIDs, jobIDs and attemptIDs > > > Key: MAPREDUCE-3052 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3052 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Jeffrey Naisbitt >Assignee: Jeffrey Naisbitt > > Currently, the appIDs, jobIDs and attempt/container ids are not consistently > named in the logs, console and > UI. > Some recent jiras have fixed the inconsistencies with the appID. > For jobID > On the RM UI: job_1308259676864_5_5 > JHS UI: job_1308259676864_5_5 > Console/logs: job_1308259676864_0005 > mapred-local dirs are named as: No jobID > I am planning on changing the jobID to match job_1308259676864_0005 in the RM > UI and the JHS UI. > For attemptID > On the RM UI: attempt_1308259676864_5_5_m_24_0 > JHS attempt_1308259676864_5_5_m_24_0 > Console/logs: attempt_1308259676864_0005_m_24_0 > mapred-local dirs are named as: container_1308259676864_0005_24 > I'm not sure the best way to handle the attempt and container IDs, but I'm > making them more consistent. (If any of you have preferences, let me know) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3089) Re-enable TestRMContainerAllocator
Re-enable TestRMContainerAllocator -- Key: MAPREDUCE-3089 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3089 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Priority: Blocker Fix For: 0.23.0 Re-enable TestRMContainerAllocator to ensure we have good coverage. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
