[jira] [Commented] (MAPREDUCE-4355) Add JobStatus getJobStatus(JobID) to JobClient.
[
https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398230#comment-13398230
]
Devaraj K commented on MAPREDUCE-4355:
--
It is good to have this API. Thanks for providing patch.
There are few minor comments about the patch
1. Can we get the JobStatus directly from Cluster instance instead of getting
the Job instance and getting the JobStatus from Job instance? In Cluster.java,
getJob(JobID jobId) API is getting the JobStatus and creating the Job instance
using the JobStatus, again you are getting the JobStatus from Job instance.
2. I don't see any use of having the jobid as final.
{code:xml}
+ public JobStatus getJobStatus(final JobID jobid) throws IOException {
{code}
3. Can you rename the jobid variable name to jobId, anyway it is minor you can
take the final decision.
Add JobStatus getJobStatus(JobID) to JobClient.
---
Key: MAPREDUCE-4355
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4355
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: mrv1, mrv2
Affects Versions: 1.0.3, 2.0.0-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: MR-4355_mr1.patch, MR-4355_mr2.patch
To read the start-time of a particular job, one should not need to
getAllJobs() and iterate through them.
getJob(JobID) returns RunningJob, which doesn't hold the job's start time.
Hence, we need to add getJobStatus(JobID) to the API.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4356) Provide access to ParsedTask.obtainTaskAttempts()
[ https://issues.apache.org/jira/browse/MAPREDUCE-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398235#comment-13398235 ] Hudson commented on MAPREDUCE-4356: --- Integrated in Hadoop-Mapreduce-trunk-Commit #2394 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2394/]) MAPREDUCE-4356. [Rumen] Provide access to the method ParsedTask.obtainTaskAttempts(). (ravigummadi) (Revision 1352415) Result = FAILURE ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1352415 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/ParsedTask.java Provide access to ParsedTask.obtainTaskAttempts() - Key: MAPREDUCE-4356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4356 Project: Hadoop Map/Reduce Issue Type: Bug Components: tools/rumen Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 4356.patch Change the access modifier of obtainTaskAttempts() in ParsedTask.java from default to public sothat it is accessible for everyone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4358) Reducers are assigned containers before all maps are assigned containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398341#comment-13398341 ] Harsh J commented on MAPREDUCE-4358: It may also be related to Jason's earlier report/fix at MAPREDUCE-4228. Reducers are assigned containers before all maps are assigned containers Key: MAPREDUCE-4358 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4358 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.0-alpha Reporter: Ahmed Radwan Reducers start to get containers before all maps are. We have seen this issue and it is problematic since if there is no avaialable resources for the remaining maps, the job will just stall where reducers are waiting for mappers which are unable to start because there is no containers available. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4356) Provide access to ParsedTask.obtainTaskAttempts()
[ https://issues.apache.org/jira/browse/MAPREDUCE-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398358#comment-13398358 ] Hudson commented on MAPREDUCE-4356: --- Integrated in Hadoop-Hdfs-trunk #1083 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1083/]) MAPREDUCE-4356. [Rumen] Provide access to the method ParsedTask.obtainTaskAttempts(). (ravigummadi) (Revision 1352415) Result = FAILURE ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1352415 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/ParsedTask.java Provide access to ParsedTask.obtainTaskAttempts() - Key: MAPREDUCE-4356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4356 Project: Hadoop Map/Reduce Issue Type: Bug Components: tools/rumen Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 4356.patch Change the access modifier of obtainTaskAttempts() in ParsedTask.java from default to public sothat it is accessible for everyone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3889) job client tries to use /tasklog interface, but that doesn't exist anymore
[ https://issues.apache.org/jira/browse/MAPREDUCE-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398359#comment-13398359 ] Hudson commented on MAPREDUCE-3889: --- Integrated in Hadoop-Hdfs-trunk #1083 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1083/]) MAPREDUCE-3889. job client tries to use /tasklog interface, but that doesn't exist anymore (Devaraj K via bobby) (Revision 1352330) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1352330 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java job client tries to use /tasklog interface, but that doesn't exist anymore -- Key: MAPREDUCE-3889 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3889 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 2.0.1-alpha, 3.0.0 Reporter: Thomas Graves Assignee: Devaraj K Priority: Critical Fix For: 0.23.3, 2.0.1-alpha, 3.0.0 Attachments: MAPREDUCE-3889.patch, MAPREDUCE-3889.patch if you specify -Dmapreduce.client.output.filter=SUCCEEDED option when running a job it tries to fetch task logs to print out on the client side from a url like: http://nodemanager:8080/tasklog?plaintext=trueattemptid=attempt_1329857083014_0003_r_00_0filter=stdout It always errors on this request with: Required param job, map and reduce We saw this error when using distcp and the distcp failed. I'm not sure if it is mandatory for distcp or just informational purposes. I'm guessing the latter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3889) job client tries to use /tasklog interface, but that doesn't exist anymore
[ https://issues.apache.org/jira/browse/MAPREDUCE-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398394#comment-13398394 ] Hudson commented on MAPREDUCE-3889: --- Integrated in Hadoop-Hdfs-0.23-Build #293 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/293/]) svn merge -c 1352330. FIXES: MAPREDUCE-3889. job client tries to use /tasklog interface, but that doesn't exist anymore (Devaraj K via bobby) (Revision 1352342) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1352342 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java job client tries to use /tasklog interface, but that doesn't exist anymore -- Key: MAPREDUCE-3889 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3889 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 2.0.1-alpha, 3.0.0 Reporter: Thomas Graves Assignee: Devaraj K Priority: Critical Fix For: 0.23.3, 2.0.1-alpha, 3.0.0 Attachments: MAPREDUCE-3889.patch, MAPREDUCE-3889.patch if you specify -Dmapreduce.client.output.filter=SUCCEEDED option when running a job it tries to fetch task logs to print out on the client side from a url like: http://nodemanager:8080/tasklog?plaintext=trueattemptid=attempt_1329857083014_0003_r_00_0filter=stdout It always errors on this request with: Required param job, map and reduce We saw this error when using distcp and the distcp failed. I'm not sure if it is mandatory for distcp or just informational purposes. I'm guessing the latter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4358) Reducers are assigned containers before all maps are assigned containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398398#comment-13398398 ] Sharad Agarwal commented on MAPREDUCE-4358: --- This is already handled by actively looking at the available headroom for the job and ramp down (pre-empt) the reduces if needed. Are you seeing this issue in your cluster ? Reducers are assigned containers before all maps are assigned containers Key: MAPREDUCE-4358 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4358 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.0-alpha Reporter: Ahmed Radwan Reducers start to get containers before all maps are. We have seen this issue and it is problematic since if there is no avaialable resources for the remaining maps, the job will just stall where reducers are waiting for mappers which are unable to start because there is no containers available. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4351) Make ContainersMonitor pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4351: - Attachment: MAPREDUCE-4351-v5.patch reattaching same patch to kick jenkins. sorry looks like I missed that box. I made sure its installed on all the slaves for mapreduce now. Make ContainersMonitor pluggable Key: MAPREDUCE-4351 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4351 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, nodemanager Reporter: Andrew Ferguson Assignee: Andrew Ferguson Attachments: MAPREDUCE-4351-v1.patch, MAPREDUCE-4351-v2.patch, MAPREDUCE-4351-v3.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v5.patch, MAPREDUCE-4351-v5.patch Make the existing ContainersManager pluggable, just as the ContainerExecutor is currently. This will allow us to add container resource enforcement using other techniques (such as cgroups) in an extensible fashion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4357) Snappy Codec does not load properly when m/r job is run in uber mode
[ https://issues.apache.org/jira/browse/MAPREDUCE-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398414#comment-13398414 ] Harsh J commented on MAPREDUCE-4357: I believe this is cause the MR AM never really starts with the jobConf default of mapred.child.env that has LD_LIBRARY_PATH set to the right native lib area. Without that, I don't think the AM container really gets the proper native paths on its runtime? Snappy Codec does not load properly when m/r job is run in uber mode -- Key: MAPREDUCE-4357 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4357 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Jeff Lord # sudo -u hdfs hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.0.0-cdh4.0.0-tests.jar TestDFSIO -write 12/06/01 18:17:11 INFO fs.TestDFSIO: TestDFSIO.0.0.6 12/06/01 18:17:11 INFO fs.TestDFSIO: nrFiles = 1 12/06/01 18:17:11 INFO fs.TestDFSIO: fileSize (MB) = 1.0 12/06/01 18:17:11 INFO fs.TestDFSIO: bufferSize = 100 12/06/01 18:17:11 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO 12/06/01 18:17:11 INFO fs.TestDFSIO: creating control file: 1048576 bytes, 1 files 12/06/01 18:17:12 INFO fs.TestDFSIO: created control files for: 1 files 12/06/01 18:17:12 INFO mapred.FileInputFormat: Total input paths to process : 1 12/06/01 18:17:12 INFO mapreduce.JobSubmitter: number of splits:1 12/06/01 18:17:12 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar 12/06/01 18:17:12 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces 12/06/01 18:17:12 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 12/06/01 18:17:12 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name 12/06/01 18:17:12 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 12/06/01 18:17:12 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 12/06/01 18:17:12 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 12/06/01 18:17:12 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 12/06/01 18:17:12 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 12/06/01 18:17:12 INFO mapred.ResourceMgrDelegate: Submitted application application_1338599410922_0004 to ResourceManager at /0.0.0.0:8032 12/06/01 18:17:12 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1338599410922_0004/ 12/06/01 18:17:12 INFO mapreduce.Job: Running job: job_1338599410922_0004 12/06/01 18:17:17 INFO mapreduce.Job: Job job_1338599410922_0004 running in uber mode : true 12/06/01 18:17:17 INFO mapreduce.Job: map 0% reduce 0% 12/06/01 18:17:17 INFO mapreduce.Job: Job job_1338599410922_0004 failed with state FAILED due to: 12/06/01 18:17:17 INFO mapreduce.Job: Counters: 11 Job Counters Failed map tasks=1 Failed reduce tasks=1 Launched map tasks=1 Launched reduce tasks=1 Other local map tasks=1 Total time spent by all maps in occupied slots (ms)=2456 Total time spent by all reduces in occupied slots (ms)=136 TOTAL_LAUNCHED_UBERTASKS=2 NUM_UBER_SUBMAPS=1 NUM_UBER_SUBREDUCES=1 NUM_FAILED_UBERTASKS=2 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:883) at org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:340) at org.apache.hadoop.fs.TestDFSIO.writeTest(TestDFSIO.java:321) at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:520) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:445) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:112) at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:120) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at
[jira] [Commented] (MAPREDUCE-4351) Make ContainersMonitor pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398417#comment-13398417 ] Hadoop QA commented on MAPREDUCE-4351: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532868/MAPREDUCE-4351-v5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 13 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2490//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2490//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2490//console This message is automatically generated. Make ContainersMonitor pluggable Key: MAPREDUCE-4351 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4351 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, nodemanager Reporter: Andrew Ferguson Assignee: Andrew Ferguson Attachments: MAPREDUCE-4351-v1.patch, MAPREDUCE-4351-v2.patch, MAPREDUCE-4351-v3.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v5.patch, MAPREDUCE-4351-v5.patch Make the existing ContainersManager pluggable, just as the ContainerExecutor is currently. This will allow us to add container resource enforcement using other techniques (such as cgroups) in an extensible fashion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4356) Provide access to ParsedTask.obtainTaskAttempts()
[ https://issues.apache.org/jira/browse/MAPREDUCE-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398443#comment-13398443 ] Hudson commented on MAPREDUCE-4356: --- Integrated in Hadoop-Mapreduce-trunk #1116 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1116/]) MAPREDUCE-4356. [Rumen] Provide access to the method ParsedTask.obtainTaskAttempts(). (ravigummadi) (Revision 1352415) Result = FAILURE ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1352415 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/ParsedTask.java Provide access to ParsedTask.obtainTaskAttempts() - Key: MAPREDUCE-4356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4356 Project: Hadoop Map/Reduce Issue Type: Bug Components: tools/rumen Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 4356.patch Change the access modifier of obtainTaskAttempts() in ParsedTask.java from default to public sothat it is accessible for everyone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3889) job client tries to use /tasklog interface, but that doesn't exist anymore
[ https://issues.apache.org/jira/browse/MAPREDUCE-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398444#comment-13398444 ] Hudson commented on MAPREDUCE-3889: --- Integrated in Hadoop-Mapreduce-trunk #1116 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1116/]) MAPREDUCE-3889. job client tries to use /tasklog interface, but that doesn't exist anymore (Devaraj K via bobby) (Revision 1352330) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1352330 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java job client tries to use /tasklog interface, but that doesn't exist anymore -- Key: MAPREDUCE-3889 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3889 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 2.0.1-alpha, 3.0.0 Reporter: Thomas Graves Assignee: Devaraj K Priority: Critical Fix For: 0.23.3, 2.0.1-alpha, 3.0.0 Attachments: MAPREDUCE-3889.patch, MAPREDUCE-3889.patch if you specify -Dmapreduce.client.output.filter=SUCCEEDED option when running a job it tries to fetch task logs to print out on the client side from a url like: http://nodemanager:8080/tasklog?plaintext=trueattemptid=attempt_1329857083014_0003_r_00_0filter=stdout It always errors on this request with: Required param job, map and reduce We saw this error when using distcp and the distcp failed. I'm not sure if it is mandatory for distcp or just informational purposes. I'm guessing the latter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4320) gridmix mainClass wrong in pom.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4320: - Resolution: Fixed Fix Version/s: 0.23.3 Status: Resolved (was: Patch Available) thanks for the review Bobby. I've committed this to trunk, branch-2, and branch-0.23. gridmix mainClass wrong in pom.xml -- Key: MAPREDUCE-4320 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4320 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 0.23.3 Reporter: Thomas Graves Assignee: Thomas Graves Fix For: 0.23.3 Attachments: MAPREDUCE-4320.patch, MAPREDUCE-4320.patch, MAPREDUCE-4320.patch when trying to run gridmix its actually trying to run org.apache.hadoop.tools.HadoopArchives. the pom.xml needs to be fixed to have correct mainClass: org.apache.hadoop.mapred.gridmix.Gridmix -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4359) Potential deadlock in Counters
[ https://issues.apache.org/jira/browse/MAPREDUCE-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-4359: --- Attachment: 1_jcarder_result_0.dot.png Potential deadlock in Counters -- Key: MAPREDUCE-4359 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4359 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.0.3 Reporter: Todd Lipcon Attachments: 1_jcarder_result_0.dot.png jcarder identified this deadlock in branch-1 (though it may also be present in trunk): - Counters.size() is synchronized and locks Counters before Group - Counters.Group.getCounterForName() is synchronized and calls through to Counters.size() This creates a potential cycle which could cause a deadlock (though probably quite rare in practice) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4031) Node Manager hangs on shut down
[ https://issues.apache.org/jira/browse/MAPREDUCE-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-4031: -- Status: Open (was: Patch Available) Node Manager hangs on shut down --- Key: MAPREDUCE-4031 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4031 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.2, 2.0.1-alpha, 3.0.0 Reporter: Devaraj K Assignee: Devaraj K Priority: Critical Attachments: MAPREDUCE-4031.patch, MAPREDUCE-4031.patch, MAPREDUCE-4031.patch, MAPREDUCE-4031.patch, MAPREDUCE-4031.patch, nm-threaddump.out I have the MAPREDUCE-3862 changes which fixed this issue earlier and yarn.nodemanager.delete.debug-delay-sec set to default value but still getting this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4031) Node Manager hangs on shut down
[ https://issues.apache.org/jira/browse/MAPREDUCE-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-4031: -- Attachment: MAPREDUCE-4031.patch Node Manager hangs on shut down --- Key: MAPREDUCE-4031 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4031 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.2, 2.0.1-alpha, 3.0.0 Reporter: Devaraj K Assignee: Devaraj K Priority: Critical Attachments: MAPREDUCE-4031.patch, MAPREDUCE-4031.patch, MAPREDUCE-4031.patch, MAPREDUCE-4031.patch, MAPREDUCE-4031.patch, nm-threaddump.out I have the MAPREDUCE-3862 changes which fixed this issue earlier and yarn.nodemanager.delete.debug-delay-sec set to default value but still getting this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4031) Node Manager hangs on shut down
[ https://issues.apache.org/jira/browse/MAPREDUCE-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-4031: -- Status: Patch Available (was: Open) Node Manager hangs on shut down --- Key: MAPREDUCE-4031 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4031 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.2, 2.0.1-alpha, 3.0.0 Reporter: Devaraj K Assignee: Devaraj K Priority: Critical Attachments: MAPREDUCE-4031.patch, MAPREDUCE-4031.patch, MAPREDUCE-4031.patch, MAPREDUCE-4031.patch, MAPREDUCE-4031.patch, nm-threaddump.out I have the MAPREDUCE-3862 changes which fixed this issue earlier and yarn.nodemanager.delete.debug-delay-sec set to default value but still getting this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4295) RM crashes due to DNS issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4295: - Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the review Bobby! I've committed this to trunk, branch-2, and branch-0.23 RM crashes due to DNS issue --- Key: MAPREDUCE-4295 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4295 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, resourcemanager Reporter: Thomas Graves Assignee: Thomas Graves Priority: Critical Fix For: 0.23.3 Attachments: MAPREDUCE-4295.patch, MAPREDUCE-4295.patch, MAPREDUCE-4295.patch, MAPREDUCE-4295.patch we had a DNS outage and the RM crashed with the following backtrace: 2012-05-29 19:17:34,492 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type NODE_UPDATE to the scheduler java.lang.IllegalArgumentException: java.net.UnknownHostException: host.comat org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:430) at org.apache.hadoop.yarn.util.BuilderUtils.newContainerToken(BuilderUtils.java:261) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1184) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1167) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1213) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4031) Node Manager hangs on shut down
[ https://issues.apache.org/jira/browse/MAPREDUCE-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398699#comment-13398699 ] Hadoop QA commented on MAPREDUCE-4031: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532916/MAPREDUCE-4031.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 13 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays org.apache.hadoop.io.file.tfile.TestTFileByteArrays +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2492//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2492//console This message is automatically generated. Node Manager hangs on shut down --- Key: MAPREDUCE-4031 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4031 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.2, 2.0.1-alpha, 3.0.0 Reporter: Devaraj K Assignee: Devaraj K Priority: Critical Attachments: MAPREDUCE-4031.patch, MAPREDUCE-4031.patch, MAPREDUCE-4031.patch, MAPREDUCE-4031.patch, MAPREDUCE-4031.patch, nm-threaddump.out I have the MAPREDUCE-3862 changes which fixed this issue earlier and yarn.nodemanager.delete.debug-delay-sec set to default value but still getting this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4295) RM crashes due to DNS issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398709#comment-13398709 ] Hudson commented on MAPREDUCE-4295: --- Integrated in Hadoop-Common-trunk-Commit #2377 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2377/]) MAPREDUCE-4295. RM crashes due to DNS issue (tgraves) (Revision 1352638) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1352638 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java RM crashes due to DNS issue --- Key: MAPREDUCE-4295 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4295 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, resourcemanager Reporter: Thomas Graves Assignee: Thomas Graves Priority: Critical Fix For: 0.23.3 Attachments: MAPREDUCE-4295.patch, MAPREDUCE-4295.patch, MAPREDUCE-4295.patch, MAPREDUCE-4295.patch we had a DNS outage and the RM crashed with the following backtrace: 2012-05-29 19:17:34,492 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type NODE_UPDATE to the scheduler java.lang.IllegalArgumentException: java.net.UnknownHostException: host.comat org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:430) at org.apache.hadoop.yarn.util.BuilderUtils.newContainerToken(BuilderUtils.java:261) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1184) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1167) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1213) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4357) Snappy Codec does not load properly when m/r job is run in uber mode
[ https://issues.apache.org/jira/browse/MAPREDUCE-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398718#comment-13398718 ] Harsh J commented on MAPREDUCE-4357: Or well the reason I say that is the AMLauncher does not seem to be populating a LD_LIBRARY_PATH for natives, pre-launch. But I see NM launching with a proper java.library.path at least (no LD_LIBRARY_PATH though), but unsure if that is inflicted automatically onto the AM env? Snappy Codec does not load properly when m/r job is run in uber mode -- Key: MAPREDUCE-4357 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4357 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Jeff Lord # sudo -u hdfs hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.0.0-cdh4.0.0-tests.jar TestDFSIO -write 12/06/01 18:17:11 INFO fs.TestDFSIO: TestDFSIO.0.0.6 12/06/01 18:17:11 INFO fs.TestDFSIO: nrFiles = 1 12/06/01 18:17:11 INFO fs.TestDFSIO: fileSize (MB) = 1.0 12/06/01 18:17:11 INFO fs.TestDFSIO: bufferSize = 100 12/06/01 18:17:11 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO 12/06/01 18:17:11 INFO fs.TestDFSIO: creating control file: 1048576 bytes, 1 files 12/06/01 18:17:12 INFO fs.TestDFSIO: created control files for: 1 files 12/06/01 18:17:12 INFO mapred.FileInputFormat: Total input paths to process : 1 12/06/01 18:17:12 INFO mapreduce.JobSubmitter: number of splits:1 12/06/01 18:17:12 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar 12/06/01 18:17:12 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces 12/06/01 18:17:12 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 12/06/01 18:17:12 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name 12/06/01 18:17:12 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 12/06/01 18:17:12 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 12/06/01 18:17:12 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 12/06/01 18:17:12 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 12/06/01 18:17:12 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 12/06/01 18:17:12 INFO mapred.ResourceMgrDelegate: Submitted application application_1338599410922_0004 to ResourceManager at /0.0.0.0:8032 12/06/01 18:17:12 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1338599410922_0004/ 12/06/01 18:17:12 INFO mapreduce.Job: Running job: job_1338599410922_0004 12/06/01 18:17:17 INFO mapreduce.Job: Job job_1338599410922_0004 running in uber mode : true 12/06/01 18:17:17 INFO mapreduce.Job: map 0% reduce 0% 12/06/01 18:17:17 INFO mapreduce.Job: Job job_1338599410922_0004 failed with state FAILED due to: 12/06/01 18:17:17 INFO mapreduce.Job: Counters: 11 Job Counters Failed map tasks=1 Failed reduce tasks=1 Launched map tasks=1 Launched reduce tasks=1 Other local map tasks=1 Total time spent by all maps in occupied slots (ms)=2456 Total time spent by all reduces in occupied slots (ms)=136 TOTAL_LAUNCHED_UBERTASKS=2 NUM_UBER_SUBMAPS=1 NUM_UBER_SUBREDUCES=1 NUM_FAILED_UBERTASKS=2 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:883) at org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:340) at org.apache.hadoop.fs.TestDFSIO.writeTest(TestDFSIO.java:321) at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:520) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:445) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:112) at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:120) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at
[jira] [Commented] (MAPREDUCE-3825) MR should not be getting duplicate tokens for a MR Job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398726#comment-13398726 ] Sanjay Radia commented on MAPREDUCE-3825: - Summary: # Solution 4 with following changes ** FileSystem#addDelegationTokens returns the newly added tokens ** non-static method and also not-final ** add similar method to AbstractFileSystem # in trunk, and 2.0 remove addDelegationTokens - it was added in 0.23. Some customers are testing 0.23 - and hence we could remove this later from 0.23 # Add convenience method - FileUtil:AddTokens(renewer, path[] ps, credentials) - this can use filesystem or filecontext in its impl. MR should not be getting duplicate tokens for a MR Job. --- Key: MAPREDUCE-3825 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3825 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Affects Versions: 0.23.1, 0.24.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: MAPREDUCE-3825.patch, TokenCache.pdf, solution4.patch This is the counterpart to HADOOP-7967. MR gets tokens for all input, output and the default filesystem when a MR job is submitted. The APIs in FileSystem make it challenging to avoid duplicate tokens when there are file systems that have embedded filesystems. Here is the original description that Daryn wrote: The token cache currently tries to assume a filesystem's token service key. The assumption generally worked while there was a one to one mapping of filesystem to token. With the advent of multi-token filesystems like viewfs, the token cache will try to use a service key (ie. for viewfs) that will never exist (because it really gets the mounted fs tokens). The descriop -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4295) RM crashes due to DNS issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398730#comment-13398730 ] Hudson commented on MAPREDUCE-4295: --- Integrated in Hadoop-Hdfs-trunk-Commit #2447 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2447/]) MAPREDUCE-4295. RM crashes due to DNS issue (tgraves) (Revision 1352638) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1352638 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java RM crashes due to DNS issue --- Key: MAPREDUCE-4295 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4295 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, resourcemanager Reporter: Thomas Graves Assignee: Thomas Graves Priority: Critical Fix For: 0.23.3 Attachments: MAPREDUCE-4295.patch, MAPREDUCE-4295.patch, MAPREDUCE-4295.patch, MAPREDUCE-4295.patch we had a DNS outage and the RM crashed with the following backtrace: 2012-05-29 19:17:34,492 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type NODE_UPDATE to the scheduler java.lang.IllegalArgumentException: java.net.UnknownHostException: host.comat org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:430) at org.apache.hadoop.yarn.util.BuilderUtils.newContainerToken(BuilderUtils.java:261) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1184) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1167) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1213) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4355) Add JobStatus getJobStatus(JobID) to JobClient.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-4355: Attachment: MR-4355_mr2.patch Add JobStatus getJobStatus(JobID) to JobClient. --- Key: MAPREDUCE-4355 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4355 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv1, mrv2 Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: MR-4355_mr1.patch, MR-4355_mr2.patch, MR-4355_mr2.patch To read the start-time of a particular job, one should not need to getAllJobs() and iterate through them. getJob(JobID) returns RunningJob, which doesn't hold the job's start time. Hence, we need to add getJobStatus(JobID) to the API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4355) Add JobStatus getJobStatus(JobID) to JobClient.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398752#comment-13398752 ] Karthik Kambatla commented on MAPREDUCE-4355: - Devaraj, Thanks for your comments. I uploaded another patch with the suggested changes, and updated getJob(JobID) also accordingly. Add JobStatus getJobStatus(JobID) to JobClient. --- Key: MAPREDUCE-4355 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4355 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv1, mrv2 Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: MR-4355_mr1.patch, MR-4355_mr2.patch, MR-4355_mr2.patch To read the start-time of a particular job, one should not need to getAllJobs() and iterate through them. getJob(JobID) returns RunningJob, which doesn't hold the job's start time. Hence, we need to add getJobStatus(JobID) to the API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4358) Reducers are assigned containers before all maps are assigned containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398757#comment-13398757 ] Ahmed Radwan commented on MAPREDUCE-4358: - @Robert, I think mapreduce.job.reduce.slowstart.completedmaps is related but different. The issue here is not to wait for a % of mappers to totally complete before start allocating containers to reducers, but the issue is to prevent reducers from occupying containers while these containers are still needed by mappers. @Jason, watching headroom and preempting reducers should be sufficient to address this issue, but this doesn't seem to work in our case. It is using Fifo. @Harsh, MAPREDUCE-4228 seems to address a bug with the behavior of mapreduce.job.reduce.slowstart.completedmaps, which as I mentioned above is different. @Sharad, Yes we are seeing this in a customer cluster. Reducers are assigned containers before all maps are assigned containers Key: MAPREDUCE-4358 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4358 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.0-alpha Reporter: Ahmed Radwan Reducers start to get containers before all maps are. We have seen this issue and it is problematic since if there is no avaialable resources for the remaining maps, the job will just stall where reducers are waiting for mappers which are unable to start because there is no containers available. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4355) Add JobStatus getJobStatus(JobID) to JobClient.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398763#comment-13398763 ] Hadoop QA commented on MAPREDUCE-4355: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532926/MR-4355_mr2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 15 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2493//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2493//console This message is automatically generated. Add JobStatus getJobStatus(JobID) to JobClient. --- Key: MAPREDUCE-4355 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4355 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv1, mrv2 Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: MR-4355_mr1.patch, MR-4355_mr2.patch, MR-4355_mr2.patch To read the start-time of a particular job, one should not need to getAllJobs() and iterate through them. getJob(JobID) returns RunningJob, which doesn't hold the job's start time. Hence, we need to add getJobStatus(JobID) to the API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3868) Reenable Raid
[ https://issues.apache.org/jira/browse/MAPREDUCE-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398770#comment-13398770 ] Jason Lowe commented on MAPREDUCE-3868: --- I updated the patch in HDFS-3549 to fix the findbugs build failures, and apparently there are 31 new findbugs warnings being reported. Also TestRaidNode is consistently failing, were the raid unit tests passing before this was committed? Reenable Raid - Key: MAPREDUCE-3868 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3868 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/raid Reporter: Scott Chen Assignee: Weiyan Wang Attachments: MAPREDUCE-3868-1.patch, MAPREDUCE-3868-2.patch, MAPREDUCE-3868-3.patch, MAPREDUCE-3868.patch, MAPREDUCE-3868v1.patch, MAPREDUCE-3868v1.sh Currently Raid is outdated and not compiled. Make it compile. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4295) RM crashes due to DNS issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398784#comment-13398784 ] Hudson commented on MAPREDUCE-4295: --- Integrated in Hadoop-Mapreduce-trunk-Commit #2396 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2396/]) MAPREDUCE-4295. RM crashes due to DNS issue (tgraves) (Revision 1352638) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1352638 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java RM crashes due to DNS issue --- Key: MAPREDUCE-4295 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4295 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, resourcemanager Reporter: Thomas Graves Assignee: Thomas Graves Priority: Critical Fix For: 0.23.3 Attachments: MAPREDUCE-4295.patch, MAPREDUCE-4295.patch, MAPREDUCE-4295.patch, MAPREDUCE-4295.patch we had a DNS outage and the RM crashed with the following backtrace: 2012-05-29 19:17:34,492 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type NODE_UPDATE to the scheduler java.lang.IllegalArgumentException: java.net.UnknownHostException: host.comat org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:430) at org.apache.hadoop.yarn.util.BuilderUtils.newContainerToken(BuilderUtils.java:261) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1184) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1167) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1213) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4358) Reducers are assigned containers before all maps are assigned containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398815#comment-13398815 ] Ahmed Radwan commented on MAPREDUCE-4358: - Will try to see if the capacity scheduler solves this issue. Will update. Reducers are assigned containers before all maps are assigned containers Key: MAPREDUCE-4358 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4358 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.0-alpha Reporter: Ahmed Radwan Reducers start to get containers before all maps are. We have seen this issue and it is problematic since if there is no avaialable resources for the remaining maps, the job will just stall where reducers are waiting for mappers which are unable to start because there is no containers available. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4334) Add support for CPU isolation/monitoring of containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ferguson updated MAPREDUCE-4334: --- Attachment: MAPREDUCE-4334-pre1.patch This is a preliminary patch to add support for using cgroups to do resource isolation and enforcement. It requires MAPREDUCE-4351, which provides pluggable ContainersMonitors. This patch currently assumes that the memory cgroups controller is mounted at /cgroups/mem and that a /cgroups/mem/hadoop-yarn group exists which is writable by the Hadoop user (this is configurable by yarn-site.xml). I will fix these assumptions shortly, but wanted to get the preliminary patch out for discussion. thanks! Add support for CPU isolation/monitoring of containers -- Key: MAPREDUCE-4334 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Arun C Murthy Assignee: Arun C Murthy Attachments: MAPREDUCE-4334-pre1.patch Once we get in MAPREDUCE-4327, it will be important to actually enforce limits on CPU consumption of containers. Several options spring to mind: # taskset (RHEL5+) # cgroups (RHEL6+) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4360) Capacity Scheduler Hierarchical leaf queue does not honur the max capacity of container queue
Mayank Bansal created MAPREDUCE-4360: Summary: Capacity Scheduler Hierarchical leaf queue does not honur the max capacity of container queue Key: MAPREDUCE-4360 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4360 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.22.1, trunk Reporter: Mayank Bansal Assignee: Mayank Bansal -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4360) Capacity Scheduler Hierarchical leaf queue does not honur the max capacity of container queue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398892#comment-13398892 ] Mayank Bansal commented on MAPREDUCE-4360: -- Maximum capacity set on a container queue is not getting honored. For ex: If Queue A has Queue A1 and Queue A2 as child queues (leaf queues), and if max capacity is not set for the child queues and set only to the container queue (Queue A), then all the slots occupied by the jobs submitted to Queue A1 and Queue A2 should not exceed the maximum capacity configured for Queue A. However, it is not the case now. Tasks gets scheduled beyond Queue A's configured max capacity. Capacity Scheduler Hierarchical leaf queue does not honur the max capacity of container queue - Key: MAPREDUCE-4360 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4360 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.22.1, trunk Reporter: Mayank Bansal Assignee: Mayank Bansal -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4355) Add JobStatus getJobStatus(JobID) to JobClient.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-4355: Attachment: (was: MR-4355_mr2.patch) Add JobStatus getJobStatus(JobID) to JobClient. --- Key: MAPREDUCE-4355 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4355 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv1, mrv2 Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: MR-4355_mr1.patch, MR-4355_mr2.patch To read the start-time of a particular job, one should not need to getAllJobs() and iterate through them. getJob(JobID) returns RunningJob, which doesn't hold the job's start time. Hence, we need to add getJobStatus(JobID) to the API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4360) Capacity Scheduler Hierarchical leaf queue does not honur the max capacity of container queue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-4360: - Attachment: MAPREDUCE-4360-22.patch Attaching the initial patch for 22. Thanks, Mayank Capacity Scheduler Hierarchical leaf queue does not honur the max capacity of container queue - Key: MAPREDUCE-4360 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4360 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.22.1, trunk Reporter: Mayank Bansal Assignee: Mayank Bansal Attachments: MAPREDUCE-4360-22.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4332) Add a yarn-client module
[ https://issues.apache.org/jira/browse/MAPREDUCE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-4332: --- Component/s: mrv2 client Affects Version/s: 2.0.0-alpha Fix Version/s: 2.0.1-alpha Add a yarn-client module Key: MAPREDUCE-4332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4332 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 2.0.0-alpha Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 2.0.1-alpha I see that we are duplicating (some) code for talking to RM via client API. In this light, a yarn-client module will be useful so that clients of all frameworks can use/extend it. And that same module can be the destination for all the YARN's command line tools. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4228) mapreduce.job.reduce.slowstart.completedmaps is not working properly to delay the scheduling of the reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-4228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4228: -- Attachment: MAPREDUCE-4228.patch Upmerged patch to latest changes on trunk. mapreduce.job.reduce.slowstart.completedmaps is not working properly to delay the scheduling of the reduce tasks Key: MAPREDUCE-4228 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4228 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.1 Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-4228.patch, MAPREDUCE-4228.patch, MAPREDUCE-4228.patch If no more map tasks need to be scheduled but not all have completed, the ApplicationMaster will start scheduling reducers even if the number of completed maps has not met the mapreduce.job.reduce.slowstart.completedmaps threshold. For example, if the property is set to 1.0 all maps should complete before any reducers are scheduled. However the reducers are scheduled as soon as the last map task is assigned to a container. For a job with very long-running maps, a cluster with enough capacity to launch all map tasks could cause reducers to launch prematurely and waste cluster resources. Thanks to Phil Su for discovering this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4361) Fix detailed metrics for protobuf-based RPC on 0.23
Jason Lowe created MAPREDUCE-4361: - Summary: Fix detailed metrics for protobuf-based RPC on 0.23 Key: MAPREDUCE-4361 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4361 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Jason Lowe RPC detailed metrics for any protobuf-based RPC ports are always zero. ProtoOverHadoopRpcEngine needs the same detailed metric logic as in WritableRpcEngine. This is effectively the same change as in HADOOP-8085 except tailored for branch-0.23 which didn't take the full protobuf branch changes that went into branch-2 and trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4361) Fix detailed metrics for protobuf-based RPC on 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe reassigned MAPREDUCE-4361: - Assignee: Jason Lowe Fix detailed metrics for protobuf-based RPC on 0.23 --- Key: MAPREDUCE-4361 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4361 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Jason Lowe Assignee: Jason Lowe RPC detailed metrics for any protobuf-based RPC ports are always zero. ProtoOverHadoopRpcEngine needs the same detailed metric logic as in WritableRpcEngine. This is effectively the same change as in HADOOP-8085 except tailored for branch-0.23 which didn't take the full protobuf branch changes that went into branch-2 and trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4228) mapreduce.job.reduce.slowstart.completedmaps is not working properly to delay the scheduling of the reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-4228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398968#comment-13398968 ] Hadoop QA commented on MAPREDUCE-4228: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532955/MAPREDUCE-4228.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 13 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2494//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2494//console This message is automatically generated. mapreduce.job.reduce.slowstart.completedmaps is not working properly to delay the scheduling of the reduce tasks Key: MAPREDUCE-4228 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4228 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.1 Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-4228.patch, MAPREDUCE-4228.patch, MAPREDUCE-4228.patch If no more map tasks need to be scheduled but not all have completed, the ApplicationMaster will start scheduling reducers even if the number of completed maps has not met the mapreduce.job.reduce.slowstart.completedmaps threshold. For example, if the property is set to 1.0 all maps should complete before any reducers are scheduled. However the reducers are scheduled as soon as the last map task is assigned to a container. For a job with very long-running maps, a cluster with enough capacity to launch all map tasks could cause reducers to launch prematurely and waste cluster resources. Thanks to Phil Su for discovering this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4228) mapreduce.job.reduce.slowstart.completedmaps is not working properly to delay the scheduling of the reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-4228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398975#comment-13398975 ] Jason Lowe commented on MAPREDUCE-4228: --- Javadoc warnings are a known issue. See HDFS-3550. mapreduce.job.reduce.slowstart.completedmaps is not working properly to delay the scheduling of the reduce tasks Key: MAPREDUCE-4228 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4228 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.1 Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-4228.patch, MAPREDUCE-4228.patch, MAPREDUCE-4228.patch If no more map tasks need to be scheduled but not all have completed, the ApplicationMaster will start scheduling reducers even if the number of completed maps has not met the mapreduce.job.reduce.slowstart.completedmaps threshold. For example, if the property is set to 1.0 all maps should complete before any reducers are scheduled. However the reducers are scheduled as soon as the last map task is assigned to a container. For a job with very long-running maps, a cluster with enough capacity to launch all map tasks could cause reducers to launch prematurely and waste cluster resources. Thanks to Phil Su for discovering this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4332) Add a yarn-client module
[ https://issues.apache.org/jira/browse/MAPREDUCE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-4332: --- Attachment: MAPREDUCE-4332-20120621.txt Here we go. Adds a yarn-client module, a client interface and its implementation. MR's client as well as the distributed-shell's client use this now to talk to RM. Fixed a few existing tests, more tests to come after MAPREDUCE-4155. Add a yarn-client module Key: MAPREDUCE-4332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4332 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 2.0.0-alpha Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 2.0.1-alpha Attachments: MAPREDUCE-4332-20120621.txt I see that we are duplicating (some) code for talking to RM via client API. In this light, a yarn-client module will be useful so that clients of all frameworks can use/extend it. And that same module can be the destination for all the YARN's command line tools. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4332) Add a yarn-client module
[ https://issues.apache.org/jira/browse/MAPREDUCE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-4332: --- Status: Patch Available (was: Open) Add a yarn-client module Key: MAPREDUCE-4332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4332 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 2.0.0-alpha Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 2.0.1-alpha Attachments: MAPREDUCE-4332-20120621.txt I see that we are duplicating (some) code for talking to RM via client API. In this light, a yarn-client module will be useful so that clients of all frameworks can use/extend it. And that same module can be the destination for all the YARN's command line tools. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4332) Add a yarn-client module
[ https://issues.apache.org/jira/browse/MAPREDUCE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398986#comment-13398986 ] Hadoop QA commented on MAPREDUCE-4332: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532963/MAPREDUCE-4332-20120621.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified test files. -1 javac. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2495//console This message is automatically generated. Add a yarn-client module Key: MAPREDUCE-4332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4332 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 2.0.0-alpha Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 2.0.1-alpha Attachments: MAPREDUCE-4332-20120621.txt I see that we are duplicating (some) code for talking to RM via client API. In this light, a yarn-client module will be useful so that clients of all frameworks can use/extend it. And that same module can be the destination for all the YARN's command line tools. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4334) Add support for CPU isolation/monitoring of containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ferguson updated MAPREDUCE-4334: --- Attachment: MAPREDUCE-4334-pre2.patch This version is ready for testing. It has the following requires: 1) Apply patch in MAPREDUCE-4351. This allows you to set yarn.nodemanager.containers-monitor.class to o.a.h.yarn.server.nodemanager.containermanager.monitor.CgroupsContainersMonitor 2) Mount the cgroups memory controller at a path of your choosing. For example: $ sudo mount -t cgroup -o memory none /cgroups/mem The NodeManager will detect where you have mounted the cgroups. 3) Create a cgroups hierarchy which Hadoop can use. This is most easily done with: $ sudo cgcreate -a hadoop_user_name -g memory:hadoop-yarn hadoop-yarn is the default hierarchy the NodeManager expects; this can be configured with yarn.nodemanager.cgroups.path. that's it! I have tested that it enforces memory limits, and reacts appropriately when the kernel kills processes, or when they complete successfully. It also notifies the user when cgroups have been mis-configured (for example, if the Hadoop user does not have write access to the cgroup hierarchy). Currently, it only enforces memory limits, as per the trunk code. I am planning to augment the patch in MAPREDUCE-4327 to provide CPU limits to the ContainersMonitor. It is easy to extend this patch to any other cgroup controller. thank you, Andrew Add support for CPU isolation/monitoring of containers -- Key: MAPREDUCE-4334 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Arun C Murthy Assignee: Arun C Murthy Attachments: MAPREDUCE-4334-pre1.patch, MAPREDUCE-4334-pre2.patch Once we get in MAPREDUCE-4327, it will be important to actually enforce limits on CPU consumption of containers. Several options spring to mind: # taskset (RHEL5+) # cgroups (RHEL6+) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4332) Add a yarn-client module
[ https://issues.apache.org/jira/browse/MAPREDUCE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-4332: --- Status: Open (was: Patch Available) Missed the fact that cross-project builds are supported after HADOOP-8308. Let me upload the common changes too. Add a yarn-client module Key: MAPREDUCE-4332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4332 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 2.0.0-alpha Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 2.0.1-alpha Attachments: MAPREDUCE-4332-20120621.txt I see that we are duplicating (some) code for talking to RM via client API. In this light, a yarn-client module will be useful so that clients of all frameworks can use/extend it. And that same module can be the destination for all the YARN's command line tools. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3868) Reenable Raid
[ https://issues.apache.org/jira/browse/MAPREDUCE-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13399016#comment-13399016 ] Scott Chen commented on MAPREDUCE-3868: --- Yes, the tests passed before checked in. Maybe it is something non-deterministic. Reenable Raid - Key: MAPREDUCE-3868 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3868 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/raid Reporter: Scott Chen Assignee: Weiyan Wang Attachments: MAPREDUCE-3868-1.patch, MAPREDUCE-3868-2.patch, MAPREDUCE-3868-3.patch, MAPREDUCE-3868.patch, MAPREDUCE-3868v1.patch, MAPREDUCE-3868v1.sh Currently Raid is outdated and not compiled. Make it compile. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4332) Add a yarn-client module
[ https://issues.apache.org/jira/browse/MAPREDUCE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-4332: --- Attachment: MAPREDUCE-4332-20120621-with-common-changes.txt Add a yarn-client module Key: MAPREDUCE-4332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4332 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 2.0.0-alpha Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 2.0.1-alpha Attachments: MAPREDUCE-4332-20120621-with-common-changes.txt, MAPREDUCE-4332-20120621.txt I see that we are duplicating (some) code for talking to RM via client API. In this light, a yarn-client module will be useful so that clients of all frameworks can use/extend it. And that same module can be the destination for all the YARN's command line tools. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4332) Add a yarn-client module
[ https://issues.apache.org/jira/browse/MAPREDUCE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-4332: --- Status: Patch Available (was: Open) Add a yarn-client module Key: MAPREDUCE-4332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4332 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 2.0.0-alpha Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 2.0.1-alpha Attachments: MAPREDUCE-4332-20120621-with-common-changes.txt, MAPREDUCE-4332-20120621.txt I see that we are duplicating (some) code for talking to RM via client API. In this light, a yarn-client module will be useful so that clients of all frameworks can use/extend it. And that same module can be the destination for all the YARN's command line tools. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4362) If possible, we should get back the feature of propagating task logs bag to JobClient
Vinod Kumar Vavilapalli created MAPREDUCE-4362: -- Summary: If possible, we should get back the feature of propagating task logs bag to JobClient Key: MAPREDUCE-4362 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4362 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 2.0.1-alpha Reporter: Vinod Kumar Vavilapalli MAPREDUCE-3889 removed the code which was trying to pull from /tasklog. We should see if it is possible to get back the feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3889) job client tries to use /tasklog interface, but that doesn't exist anymore
[ https://issues.apache.org/jira/browse/MAPREDUCE-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13399020#comment-13399020 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3889: Created MAPREDUCE-4362 to see if it is possible to get back the feature of propagating task logs back to JobClient. job client tries to use /tasklog interface, but that doesn't exist anymore -- Key: MAPREDUCE-3889 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3889 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 2.0.1-alpha, 3.0.0 Reporter: Thomas Graves Assignee: Devaraj K Priority: Critical Fix For: 0.23.3, 2.0.1-alpha, 3.0.0 Attachments: MAPREDUCE-3889.patch, MAPREDUCE-3889.patch if you specify -Dmapreduce.client.output.filter=SUCCEEDED option when running a job it tries to fetch task logs to print out on the client side from a url like: http://nodemanager:8080/tasklog?plaintext=trueattemptid=attempt_1329857083014_0003_r_00_0filter=stdout It always errors on this request with: Required param job, map and reduce We saw this error when using distcp and the distcp failed. I'm not sure if it is mandatory for distcp or just informational purposes. I'm guessing the latter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4362) If possible, we should get back the feature of propagating task logs bag to JobClient
[ https://issues.apache.org/jira/browse/MAPREDUCE-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13399019#comment-13399019 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-4362: As I noted one MAPREDUCE-3889 bq. But the problem is that the client doesn't have a way of mapping TaskAttemptId to the ContainerIDs so that the log-urls can be constructed properly. So may be for the shorter term, we can disable the feature completely. May be we should add an API to ClientProtocol to get this? Thoughts? If possible, we should get back the feature of propagating task logs bag to JobClient - Key: MAPREDUCE-4362 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4362 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 2.0.1-alpha Reporter: Vinod Kumar Vavilapalli MAPREDUCE-3889 removed the code which was trying to pull from /tasklog. We should see if it is possible to get back the feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4362) If possible, we should get back the feature of propagating task logs back to JobClient
[ https://issues.apache.org/jira/browse/MAPREDUCE-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-4362: --- Summary: If possible, we should get back the feature of propagating task logs back to JobClient (was: If possible, we should get back the feature of propagating task logs bag to JobClient) If possible, we should get back the feature of propagating task logs back to JobClient -- Key: MAPREDUCE-4362 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4362 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 2.0.1-alpha Reporter: Vinod Kumar Vavilapalli MAPREDUCE-3889 removed the code which was trying to pull from /tasklog. We should see if it is possible to get back the feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4351) Make ContainersMonitor pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13399029#comment-13399029 ] Andrew Ferguson commented on MAPREDUCE-4351: For an example of how this is used, please see my patch in MAPREDUCE-4334, which provides an alternative ContainersMonitor that enforces limits using cgroups. re: the current patch. I can't tell if the javadoc warnings are real; perhaps someone more experienced can take a look? The findbugs warning is spurious. thanks! Make ContainersMonitor pluggable Key: MAPREDUCE-4351 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4351 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, nodemanager Reporter: Andrew Ferguson Assignee: Andrew Ferguson Attachments: MAPREDUCE-4351-v1.patch, MAPREDUCE-4351-v2.patch, MAPREDUCE-4351-v3.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v5.patch, MAPREDUCE-4351-v5.patch Make the existing ContainersManager pluggable, just as the ContainerExecutor is currently. This will allow us to add container resource enforcement using other techniques (such as cgroups) in an extensible fashion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4363) Hadoop 1.X does not build on Fedora 17
Bruno Mahé created MAPREDUCE-4363: - Summary: Hadoop 1.X does not build on Fedora 17 Key: MAPREDUCE-4363 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4363 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, pipes Affects Versions: 1.0.3 Reporter: Bruno Mahé Assignee: Bruno Mahé Fix For: 1.2.0 I upgraded my machine to the latest Fedora 17 and now Apache Hadoop is failing to build. This seems related to the bump in version of gcc to 4.7.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4363) Hadoop 1.X does not build on Fedora 17
[ https://issues.apache.org/jira/browse/MAPREDUCE-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Mahé updated MAPREDUCE-4363: -- Labels: bigtop (was: ) Hadoop 1.X does not build on Fedora 17 -- Key: MAPREDUCE-4363 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4363 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, pipes Affects Versions: 1.0.3 Reporter: Bruno Mahé Assignee: Bruno Mahé Labels: bigtop Fix For: 1.2.0 I upgraded my machine to the latest Fedora 17 and now Apache Hadoop is failing to build. This seems related to the bump in version of gcc to 4.7.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4363) Hadoop 1.X does not build on Fedora 17
[ https://issues.apache.org/jira/browse/MAPREDUCE-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Mahé updated MAPREDUCE-4363: -- Status: Patch Available (was: Open) Here is a patch Hadoop 1.X does not build on Fedora 17 -- Key: MAPREDUCE-4363 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4363 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, pipes Affects Versions: 1.0.3 Reporter: Bruno Mahé Assignee: Bruno Mahé Fix For: 1.2.0 I upgraded my machine to the latest Fedora 17 and now Apache Hadoop is failing to build. This seems related to the bump in version of gcc to 4.7.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4363) Hadoop 1.X does not build on Fedora 17
[ https://issues.apache.org/jira/browse/MAPREDUCE-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Mahé updated MAPREDUCE-4363: -- Attachment: MAPREDUCE-4363.patch Please find attached a patch which unblock me. autoconf already look for unistd.h. So this patch shouldn't have any impact besides making Apache Hadoop build on Fedora 17. I would also like to petition for including that patch in the upcoming Apache Hadoop 1.X release so Apache Bigtop (incubating) can update its Apache Hadoop 1.X version Hadoop 1.X does not build on Fedora 17 -- Key: MAPREDUCE-4363 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4363 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, pipes Affects Versions: 1.0.3 Reporter: Bruno Mahé Assignee: Bruno Mahé Labels: bigtop Fix For: 1.2.0 Attachments: MAPREDUCE-4363.patch I upgraded my machine to the latest Fedora 17 and now Apache Hadoop is failing to build. This seems related to the bump in version of gcc to 4.7.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4363) Hadoop 1.X does not build on Fedora 17
[ https://issues.apache.org/jira/browse/MAPREDUCE-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13399119#comment-13399119 ] Todd Lipcon commented on MAPREDUCE-4363: Do we need this fix in trunk as well, or does the cmake-based build take care of it somehow? Hadoop 1.X does not build on Fedora 17 -- Key: MAPREDUCE-4363 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4363 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, pipes Affects Versions: 1.0.3 Reporter: Bruno Mahé Assignee: Bruno Mahé Labels: bigtop Fix For: 1.2.0 Attachments: MAPREDUCE-4363.patch I upgraded my machine to the latest Fedora 17 and now Apache Hadoop is failing to build. This seems related to the bump in version of gcc to 4.7.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4363) Hadoop 1.X does not build on Fedora 17
[ https://issues.apache.org/jira/browse/MAPREDUCE-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13399123#comment-13399123 ] Bruno Mahé commented on MAPREDUCE-4363: --- I am in the process of trying and will update the ticket as soon as I get the answer Hadoop 1.X does not build on Fedora 17 -- Key: MAPREDUCE-4363 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4363 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, pipes Affects Versions: 1.0.3 Reporter: Bruno Mahé Assignee: Bruno Mahé Labels: bigtop Fix For: 1.2.0 Attachments: MAPREDUCE-4363.patch I upgraded my machine to the latest Fedora 17 and now Apache Hadoop is failing to build. This seems related to the bump in version of gcc to 4.7.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4363) Hadoop 1.X and trunk (and probably 2.X) do not build on Fedora 17
[ https://issues.apache.org/jira/browse/MAPREDUCE-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Mahé updated MAPREDUCE-4363: -- Affects Version/s: trunk Fix Version/s: (was: 1.2.0) 2.0.1-alpha 1.1.0 Summary: Hadoop 1.X and trunk (and probably 2.X) do not build on Fedora 17 (was: Hadoop 1.X does not build on Fedora 17) Hadoop 1.X and trunk (and probably 2.X) do not build on Fedora 17 - Key: MAPREDUCE-4363 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4363 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, pipes Affects Versions: 1.0.3, trunk Reporter: Bruno Mahé Assignee: Bruno Mahé Labels: bigtop Fix For: 1.1.0, 2.0.1-alpha Attachments: MAPREDUCE-4363.patch I upgraded my machine to the latest Fedora 17 and now Apache Hadoop is failing to build. This seems related to the bump in version of gcc to 4.7.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4363) Hadoop 1.X and trunk (and probably 2.X) do not build on Fedora 17
[ https://issues.apache.org/jira/browse/MAPREDUCE-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13399127#comment-13399127 ] Bruno Mahé commented on MAPREDUCE-4363: --- So I confirm this applies to trunk as well Hadoop 1.X and trunk (and probably 2.X) do not build on Fedora 17 - Key: MAPREDUCE-4363 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4363 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, pipes Affects Versions: 1.0.3, trunk Reporter: Bruno Mahé Assignee: Bruno Mahé Labels: bigtop Fix For: 1.1.0, 2.0.1-alpha Attachments: MAPREDUCE-4363.patch I upgraded my machine to the latest Fedora 17 and now Apache Hadoop is failing to build. This seems related to the bump in version of gcc to 4.7.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4363) Hadoop 1.X, 2.X and trunk do not build on Fedora 17
[ https://issues.apache.org/jira/browse/MAPREDUCE-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13399128#comment-13399128 ] Bruno Mahé commented on MAPREDUCE-4363: --- And also branch-2 Hadoop 1.X, 2.X and trunk do not build on Fedora 17 --- Key: MAPREDUCE-4363 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4363 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, pipes Affects Versions: 1.0.3, trunk Reporter: Bruno Mahé Assignee: Bruno Mahé Labels: bigtop Fix For: 1.1.0, 2.0.1-alpha Attachments: MAPREDUCE-4363.patch I upgraded my machine to the latest Fedora 17 and now Apache Hadoop is failing to build. This seems related to the bump in version of gcc to 4.7.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4363) Hadoop 1.X, 2.X and trunk do not build on Fedora 17
[ https://issues.apache.org/jira/browse/MAPREDUCE-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Mahé updated MAPREDUCE-4363: -- Summary: Hadoop 1.X, 2.X and trunk do not build on Fedora 17 (was: Hadoop 1.X and trunk (and probably 2.X) do not build on Fedora 17) Hadoop 1.X, 2.X and trunk do not build on Fedora 17 --- Key: MAPREDUCE-4363 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4363 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, pipes Affects Versions: 1.0.3, trunk Reporter: Bruno Mahé Assignee: Bruno Mahé Labels: bigtop Fix For: 1.1.0, 2.0.1-alpha Attachments: MAPREDUCE-4363.patch I upgraded my machine to the latest Fedora 17 and now Apache Hadoop is failing to build. This seems related to the bump in version of gcc to 4.7.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
