[jira] [Created] (MAPREDUCE-5952) LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex
Gera Shegalov created MAPREDUCE-5952:
Summary: LocalContainerLauncher#renameMapOutputForReduce
incorrectly assumes a single dir for mapOutIndex
Key: MAPREDUCE-5952
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5952
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 2.3.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Priority: Blocker
The javadoc comment for {{renameMapOutputForReduce}} incorrectly refers to a
single map output directory, whereas this depends on LOCAL_DIRS.
mapOutIndex should be set to subMapOutputFile.getOutputIndexFile()
{code}
2014-06-30 14:48:35,574 WARN [uber-SubtaskRunner]
org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local
(uberized) 'child' : java.io.FileNotFoundException: File
/Users/gshegalov/workspace/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobs/org.apache.hadoop.mapreduce.v2.
TestMRJobs-localDir-nm-2_3/usercache/gshegalov/appcache/application_1404164272885_0001/output/file.out.index
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:517)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:726)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:507)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
at
org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:334)
at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:504)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.renameMapOutputForReduce(LocalContainerLauncher.java:471)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:292)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:178)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:221)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (MAPREDUCE-5951) Add support for the YARN Shared Cache
Chris Trezzo created MAPREDUCE-5951: --- Summary: Add support for the YARN Shared Cache Key: MAPREDUCE-5951 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Chris Trezzo Assignee: Chris Trezzo Implement the necessary changes so that the MapReduce application can leverage the new YARN shared cache (i.e. YARN-1492). -- This message was sent by Atlassian JIRA (v6.2#6252)
Hadoop-Mapreduce-trunk - Build # 1817 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1817/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 35840 lines...] Running org.apache.hadoop.mapred.TestLocalContainerLauncher Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.717 sec - in org.apache.hadoop.mapred.TestLocalContainerLauncher Running org.apache.hadoop.mapred.TestTaskAttemptListenerImpl Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.87 sec - in org.apache.hadoop.mapred.TestTaskAttemptListenerImpl Results : Failed tests: TestCommitterEventHandler.testFailure:314 null TestCommitterEventHandler.testBasic:268 /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/test-staging/jenkins/.staging/job_123456789_0001/COMMIT_FAIL Tests run: 277, Failures: 2, Errors: 0, Skipped: 0 [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-mapreduce-client ... SUCCESS [3.903s] [INFO] hadoop-mapreduce-client-core .. SUCCESS [1:11.573s] [INFO] hadoop-mapreduce-client-common SUCCESS [33.874s] [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [5.930s] [INFO] hadoop-mapreduce-client-app ... FAILURE [8:11.633s] [INFO] hadoop-mapreduce-client-hs SKIPPED [INFO] hadoop-mapreduce-client-jobclient . SKIPPED [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 10:08.621s [INFO] Finished at: Mon Jun 30 13:47:13 UTC 2014 [INFO] Final Memory: 26M/378M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project hadoop-mapreduce-client-app: There are test failures. [ERROR] [ERROR] Please refer to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/surefire-reports for the individual test results. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hadoop-mapreduce-client-app Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Sending artifact delta relative to Hadoop-Mapreduce-trunk #1765 Archived 2 artifacts Archive block size is 32768 Received 0 blocks and 16957779 bytes Compression is 0.0% Took 14 sec Updating HDFS-6418 Updating YARN-2052 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
