[jira] [Updated] (MAPREDUCE-3020) Node link in reduce task attempt page is not working [Job History Page]
[ https://issues.apache.org/jira/browse/MAPREDUCE-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chackaravarthy updated MAPREDUCE-3020: -- Attachment: MAPREDUCE-3020.patch Node link in reduce task attempt page is not working [Job History Page] --- Key: MAPREDUCE-3020 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3020 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 0.23.0, 0.24.0 Reporter: chackaravarthy Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-3020.patch RM UI - Applications - Application(Job History) - Reduce Tasks - Task ID - Node link is not working hostname for ReduceAttemptFinishedEvent is coming wrong when loading from history file. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3064) 27 unit test failures with Invalid mapreduce.jobtracker.address configuration value for JobTracker: local
[
https://issues.apache.org/jira/browse/MAPREDUCE-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114488#comment-13114488
]
Venu Gopala Rao commented on MAPREDUCE-3064:
Hi Mahadev, After committing this the Jenkins is failing with a Compilation
error MRConfig does not have CLASSIC_FRAMEWORK_NAME. But in the code trunk I
see the Constant is there. Also my local builds are passing. Am I missing some
thing?
{code:xml}
[javac] Compiling 48 source files to
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/classes
[javac]
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobTrackerClientProtocolProvider.java:40:
cannot find symbol
[javac] symbol : variable CLASSIC_FRAMEWORK_NAME
[javac] location: interface org.apache.hadoop.mapreduce.MRConfig
[javac] if (!MRConfig.CLASSIC_FRAMEWORK_NAME.equals(framework)) {
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 1 error
{code}
27 unit test failures with Invalid mapreduce.jobtracker.address
configuration value for JobTracker: local
--
Key: MAPREDUCE-3064
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3064
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: Thomas Graves
Assignee: Venu Gopala Rao
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-3064.1.patch, MAPREDUCE-3064.patch
unit test failure here:
https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/946/
Test Result (27 failures / +27)
org.apache.hadoop.mapred.TestCollect.testCollect
org.apache.hadoop.mapred.TestComparators.testDefaultMRComparator
org.apache.hadoop.mapred.TestComparators.testUserMRComparator
org.apache.hadoop.mapred.TestComparators.testUserValueGroupingComparator
org.apache.hadoop.mapred.TestComparators.testAllUserComparators
org.apache.hadoop.mapred.TestFileOutputFormat.testCustomFile
org.apache.hadoop.mapred.TestJavaSerialization.testMapReduceJob
org.apache.hadoop.mapred.TestJavaSerialization.testWriteToSequencefile
org.apache.hadoop.mapred.TestMapOutputType.testKeyMismatch
org.apache.hadoop.mapred.TestMapOutputType.testValueMismatch
org.apache.hadoop.mapred.TestMapOutputType.testNoMismatch
org.apache.hadoop.mapred.TestMapRed.testMapred
org.apache.hadoop.mapred.TestMapRed.testNullKeys
org.apache.hadoop.mapred.TestMapRed.testCompression
org.apache.hadoop.mapred.TestMapRed.testSmallInput
org.apache.hadoop.mapred.TestMapRed.testBiggerInput
org.apache.hadoop.mapreduce.TestMapCollection.testValLastByte
org.apache.hadoop.mapreduce.TestMapCollection.testLargeRecords
org.apache.hadoop.mapreduce.TestMapCollection.testSpillPer2B
org.apache.hadoop.mapreduce.TestMapCollection.testZeroVal
org.apache.hadoop.mapreduce.TestMapCollection.testSingleRecord
org.apache.hadoop.mapreduce.TestMapCollection.testLowSpill
org.apache.hadoop.mapreduce.TestMapCollection.testSplitMetaSpill
org.apache.hadoop.mapreduce.TestMapCollection.testPostSpillMeta
org.apache.hadoop.mapreduce.TestMapCollection.testLargeRecConcurrent
org.apache.hadoop.mapreduce.TestMapCollection.testRandom
org.apache.hadoop.mapreduce.TestMapCollection.testRandomCompress
All of them have similar stack traces of:
java.io.IOException: Invalid mapreduce.jobtracker.address configuration
value for JobTracker: local
at
org.apache.hadoop.mapred.JobTrackerClientProtocolProvider.create(JobTrackerClientProtocolProvider.java:47)
at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:74)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:459)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:438)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:777)
at
org.apache.hadoop.mapred.TestCollect.testCollect(TestCollect.java:133)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3020) Node link in reduce task attempt page is not working [Job History Page]
[ https://issues.apache.org/jira/browse/MAPREDUCE-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114490#comment-13114490 ] chackaravarthy commented on MAPREDUCE-3020: --- The issue was : when creating ReduceAttemptFinishedEvent, instead of passing nodeHostName for hostname, containerMgrAddress was passed.. But creation of MapAttemptFinishedEvent is done in right way.Hence changed the code with respect to that. Node link in reduce task attempt page is not working [Job History Page] --- Key: MAPREDUCE-3020 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3020 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 0.23.0, 0.24.0 Reporter: chackaravarthy Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-3020.patch RM UI - Applications - Application(Job History) - Reduce Tasks - Task ID - Node link is not working hostname for ReduceAttemptFinishedEvent is coming wrong when loading from history file. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3090) Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere
Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere --- Key: MAPREDUCE-3090 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3090 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.0 Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere, particularly after MAPREDUCE-3055 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3020) Node link in reduce task attempt page is not working [Job History Page]
[ https://issues.apache.org/jira/browse/MAPREDUCE-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114498#comment-13114498 ] Hadoop QA commented on MAPREDUCE-3020: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12496441/MAPREDUCE-3020.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/856//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/856//console This message is automatically generated. Node link in reduce task attempt page is not working [Job History Page] --- Key: MAPREDUCE-3020 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3020 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 0.23.0, 0.24.0 Reporter: chackaravarthy Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-3020.patch RM UI - Applications - Application(Job History) - Reduce Tasks - Task ID - Node link is not working hostname for ReduceAttemptFinishedEvent is coming wrong when loading from history file. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3090) Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere
[ https://issues.apache.org/jira/browse/MAPREDUCE-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3090: - Status: Patch Available (was: Open) Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere --- Key: MAPREDUCE-3090 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3090 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.0 Attachments: MAPREDUCE-3090.patch Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere, particularly after MAPREDUCE-3055 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3090) Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere
[ https://issues.apache.org/jira/browse/MAPREDUCE-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3090: - Attachment: MAPREDUCE-3090.patch Straight-fwd patch. Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere --- Key: MAPREDUCE-3090 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3090 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.0 Attachments: MAPREDUCE-3090.patch Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere, particularly after MAPREDUCE-3055 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3064) 27 unit test failures with Invalid mapreduce.jobtracker.address configuration value for JobTracker: local
[ https://issues.apache.org/jira/browse/MAPREDUCE-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114505#comment-13114505 ] Venu Gopala Rao commented on MAPREDUCE-3064: Just took a clean workspace and built every thing from scratch. My local build is passing. Is there any possibility that Jenkins has an outdated jar in classpath? 27 unit test failures with Invalid mapreduce.jobtracker.address configuration value for JobTracker: local -- Key: MAPREDUCE-3064 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3064 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Venu Gopala Rao Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3064.1.patch, MAPREDUCE-3064.patch unit test failure here: https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/946/ Test Result (27 failures / +27) org.apache.hadoop.mapred.TestCollect.testCollect org.apache.hadoop.mapred.TestComparators.testDefaultMRComparator org.apache.hadoop.mapred.TestComparators.testUserMRComparator org.apache.hadoop.mapred.TestComparators.testUserValueGroupingComparator org.apache.hadoop.mapred.TestComparators.testAllUserComparators org.apache.hadoop.mapred.TestFileOutputFormat.testCustomFile org.apache.hadoop.mapred.TestJavaSerialization.testMapReduceJob org.apache.hadoop.mapred.TestJavaSerialization.testWriteToSequencefile org.apache.hadoop.mapred.TestMapOutputType.testKeyMismatch org.apache.hadoop.mapred.TestMapOutputType.testValueMismatch org.apache.hadoop.mapred.TestMapOutputType.testNoMismatch org.apache.hadoop.mapred.TestMapRed.testMapred org.apache.hadoop.mapred.TestMapRed.testNullKeys org.apache.hadoop.mapred.TestMapRed.testCompression org.apache.hadoop.mapred.TestMapRed.testSmallInput org.apache.hadoop.mapred.TestMapRed.testBiggerInput org.apache.hadoop.mapreduce.TestMapCollection.testValLastByte org.apache.hadoop.mapreduce.TestMapCollection.testLargeRecords org.apache.hadoop.mapreduce.TestMapCollection.testSpillPer2B org.apache.hadoop.mapreduce.TestMapCollection.testZeroVal org.apache.hadoop.mapreduce.TestMapCollection.testSingleRecord org.apache.hadoop.mapreduce.TestMapCollection.testLowSpill org.apache.hadoop.mapreduce.TestMapCollection.testSplitMetaSpill org.apache.hadoop.mapreduce.TestMapCollection.testPostSpillMeta org.apache.hadoop.mapreduce.TestMapCollection.testLargeRecConcurrent org.apache.hadoop.mapreduce.TestMapCollection.testRandom org.apache.hadoop.mapreduce.TestMapCollection.testRandomCompress All of them have similar stack traces of: java.io.IOException: Invalid mapreduce.jobtracker.address configuration value for JobTracker: local at org.apache.hadoop.mapred.JobTrackerClientProtocolProvider.create(JobTrackerClientProtocolProvider.java:47) at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:74) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:459) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:438) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:777) at org.apache.hadoop.mapred.TestCollect.testCollect(TestCollect.java:133) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3090) Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere
[ https://issues.apache.org/jira/browse/MAPREDUCE-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114522#comment-13114522 ] Hadoop QA commented on MAPREDUCE-3090: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12496447/MAPREDUCE-3090.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/857//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/857//console This message is automatically generated. Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere --- Key: MAPREDUCE-3090 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3090 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.0 Attachments: MAPREDUCE-3090.patch Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere, particularly after MAPREDUCE-3055 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3020) Node link in reduce task attempt page is not working [Job History Page]
[ https://issues.apache.org/jira/browse/MAPREDUCE-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114527#comment-13114527 ] chackaravarthy commented on MAPREDUCE-3020: --- This is trivial change. In Job History Page (Reduce Tasks - Task ID - Node link) Earlier it was redirecting to wrong page (http://containerMgrAddress). It should be http://hostname Node link in reduce task attempt page is not working [Job History Page] --- Key: MAPREDUCE-3020 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3020 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 0.23.0, 0.24.0 Reporter: chackaravarthy Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-3020.patch RM UI - Applications - Application(Job History) - Reduce Tasks - Task ID - Node link is not working hostname for ReduceAttemptFinishedEvent is coming wrong when loading from history file. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-207) Computing Input Splits on the MR Cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-207: Attachment: MAPREDUCE-207.patch As foretold, here is a trivial, preliminary patch to move computation of input-splits inside the cluster - something we've craved for a very long time, as evinced by the interest in this jira and the number of times it comes up on user lists. This is huge, because it's a significant step towards various improvements such as HTTP-based job submission etc. Shameless plug for MRv2 - it took me 15 mins on a Sunday night to get this done... glory to MRv2! *smile* It needs a tad more work to get delegation tokens on the client side, but it's nearly there. Computing Input Splits on the MR Cluster Key: MAPREDUCE-207 URL: https://issues.apache.org/jira/browse/MAPREDUCE-207 Project: Hadoop Map/Reduce Issue Type: New Feature Components: applicationmaster, mrv2 Reporter: Philip Zeyliger Assignee: Arun C Murthy Attachments: MAPREDUCE-207.patch Instead of computing the input splits as part of job submission, Hadoop could have a separate job task type that computes the input splits, therefore allowing that computation to happen on the cluster. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3090) Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere
[ https://issues.apache.org/jira/browse/MAPREDUCE-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114537#comment-13114537 ] Hudson commented on MAPREDUCE-3090: --- Integrated in Hadoop-Common-trunk-Commit #954 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/954/]) MAPREDUCE-3090. Fix MR AM to use ApplicationAttemptId rather than (ApplicationId, startCount) consistently. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1175718 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere --- Key: MAPREDUCE-3090 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3090 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.0 Attachments: MAPREDUCE-3090.patch Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere, particularly after MAPREDUCE-3055 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3090) Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere
[ https://issues.apache.org/jira/browse/MAPREDUCE-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114539#comment-13114539 ] Hudson commented on MAPREDUCE-3090: --- Integrated in Hadoop-Hdfs-trunk-Commit #1032 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1032/]) MAPREDUCE-3090. Fix MR AM to use ApplicationAttemptId rather than (ApplicationId, startCount) consistently. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1175718 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere --- Key: MAPREDUCE-3090 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3090 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.0 Attachments: MAPREDUCE-3090.patch Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere, particularly after MAPREDUCE-3055 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3090) Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere
[ https://issues.apache.org/jira/browse/MAPREDUCE-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114543#comment-13114543 ] Hudson commented on MAPREDUCE-3090: --- Integrated in Hadoop-Mapreduce-trunk-Commit #972 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/972/]) MAPREDUCE-3090. Fix MR AM to use ApplicationAttemptId rather than (ApplicationId, startCount) consistently. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1175718 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere --- Key: MAPREDUCE-3090 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3090 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.0 Attachments: MAPREDUCE-3090.patch Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere, particularly after MAPREDUCE-3055 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2861) Modify version_control.html to reflect new path to mapreduce trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114559#comment-13114559 ] Devaraj K commented on MAPREDUCE-2861: -- It can be closed since it is already updated in http://hadoop.apache.org/mapreduce/version_control.html. Modify version_control.html to reflect new path to mapreduce trunk -- Key: MAPREDUCE-2861 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2861 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Vinod Kumar Vavilapalli We need to modify http://hadoop.apache.org/mapreduce/version_control.html to reflect the change of mapreduce trunk to trunk/hadoop-mapreduce. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3091) Health-Report column of NodePage should display more information.
Health-Report column of NodePage should display more information. --- Key: MAPREDUCE-3091 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3091 Project: Hadoop Map/Reduce Issue Type: Improvement Components: nodemanager, resourcemanager Affects Versions: 0.23.0 Reporter: Subroto Sanyal Assignee: Subroto Sanyal Fix For: 0.23.0 The Health-Checker script of the Nodes can run and generate some output, error and exit code. These information is not available in the GUI. It is possible the Health-Checker script generates some statistics about node. The same can displayed to GUI user. I suggest we display the information in pop-up balloon(using CSS/Javascript)? Any suggestions -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2984) Throwing NullPointerException when we open the container page
[
https://issues.apache.org/jira/browse/MAPREDUCE-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-2984:
-
Attachment: MAPREDUCE-2984-1.patch
Throwing NullPointerException when we open the container page
-
Key: MAPREDUCE-2984
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2984
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager
Affects Versions: 0.23.0, 0.24.0
Reporter: Devaraj K
Assignee: Devaraj K
Fix For: 0.23.0, 0.24.0
Attachments: MAPREDUCE-2984-1.patch, MAPREDUCE-2984.patch
{code:xml}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.yarn.api.records.ContainerId.compareTo(ContainerId.java:97)
at
org.apache.hadoop.yarn.api.records.ContainerId.compareTo(ContainerId.java:23)
at
java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:819)
at
java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1640)
at
org.apache.hadoop.yarn.server.nodemanager.webapp.ContainerPage$ContainerBlock.render(ContainerPage.java:70)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:64)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:74)
at org.apache.hadoop.yarn.webapp.View.render(View.java:210)
{code}
{code:xml}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.nodemanager.webapp.ContainerPage$ContainerBlock.render(ContainerPage.java:71)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:64)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:74)
at org.apache.hadoop.yarn.webapp.View.render(View.java:210)
at
org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:47)
at
org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$TD._(Hamlet.java:843)
at
org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:54)
at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:80)
at org.apache.hadoop.yarn.webapp.Controller.render(Controller.java:210)
at
org.apache.hadoop.yarn.server.nodemanager.webapp.NMController.container(NMController.java:62)
... 30 more
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2984) Throwing NullPointerException when we open the container page
[
https://issues.apache.org/jira/browse/MAPREDUCE-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114598#comment-13114598
]
Hadoop QA commented on MAPREDUCE-2984:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12496463/MAPREDUCE-2984-1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/858//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/858//console
This message is automatically generated.
Throwing NullPointerException when we open the container page
-
Key: MAPREDUCE-2984
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2984
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager
Affects Versions: 0.23.0, 0.24.0
Reporter: Devaraj K
Assignee: Devaraj K
Fix For: 0.23.0, 0.24.0
Attachments: MAPREDUCE-2984-1.patch, MAPREDUCE-2984.patch
{code:xml}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.yarn.api.records.ContainerId.compareTo(ContainerId.java:97)
at
org.apache.hadoop.yarn.api.records.ContainerId.compareTo(ContainerId.java:23)
at
java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:819)
at
java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1640)
at
org.apache.hadoop.yarn.server.nodemanager.webapp.ContainerPage$ContainerBlock.render(ContainerPage.java:70)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:64)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:74)
at org.apache.hadoop.yarn.webapp.View.render(View.java:210)
{code}
{code:xml}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.nodemanager.webapp.ContainerPage$ContainerBlock.render(ContainerPage.java:71)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:64)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:74)
at org.apache.hadoop.yarn.webapp.View.render(View.java:210)
at
org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:47)
at
org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$TD._(Hamlet.java:843)
at
org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:54)
at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:80)
at org.apache.hadoop.yarn.webapp.Controller.render(Controller.java:210)
at
org.apache.hadoop.yarn.server.nodemanager.webapp.NMController.container(NMController.java:62)
... 30 more
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3064) 27 unit test failures with Invalid mapreduce.jobtracker.address configuration value for JobTracker: local
[ https://issues.apache.org/jira/browse/MAPREDUCE-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114636#comment-13114636 ] Hudson commented on MAPREDUCE-3064: --- Integrated in Hadoop-Hdfs-0.23-Build #21 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/21/]) MAPREDUCE-3064. 27 unit test failures with Invalid mapreduce.jobtracker.address configuration value for JobTracker: local (Venu Gopala Rao via mahadev) - Merging r1175449 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1175450 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobTrackerClientProtocolProvider.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/MiniMRCluster.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/QueueManagerTestUtils.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestMiniMRClasspath.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSpecialCharactersInOutputPath.java 27 unit test failures with Invalid mapreduce.jobtracker.address configuration value for JobTracker: local -- Key: MAPREDUCE-3064 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3064 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Venu Gopala Rao Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3064.1.patch, MAPREDUCE-3064.patch unit test failure here: https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/946/ Test Result (27 failures / +27) org.apache.hadoop.mapred.TestCollect.testCollect org.apache.hadoop.mapred.TestComparators.testDefaultMRComparator org.apache.hadoop.mapred.TestComparators.testUserMRComparator org.apache.hadoop.mapred.TestComparators.testUserValueGroupingComparator org.apache.hadoop.mapred.TestComparators.testAllUserComparators org.apache.hadoop.mapred.TestFileOutputFormat.testCustomFile org.apache.hadoop.mapred.TestJavaSerialization.testMapReduceJob org.apache.hadoop.mapred.TestJavaSerialization.testWriteToSequencefile org.apache.hadoop.mapred.TestMapOutputType.testKeyMismatch org.apache.hadoop.mapred.TestMapOutputType.testValueMismatch org.apache.hadoop.mapred.TestMapOutputType.testNoMismatch org.apache.hadoop.mapred.TestMapRed.testMapred org.apache.hadoop.mapred.TestMapRed.testNullKeys org.apache.hadoop.mapred.TestMapRed.testCompression org.apache.hadoop.mapred.TestMapRed.testSmallInput org.apache.hadoop.mapred.TestMapRed.testBiggerInput org.apache.hadoop.mapreduce.TestMapCollection.testValLastByte org.apache.hadoop.mapreduce.TestMapCollection.testLargeRecords org.apache.hadoop.mapreduce.TestMapCollection.testSpillPer2B org.apache.hadoop.mapreduce.TestMapCollection.testZeroVal org.apache.hadoop.mapreduce.TestMapCollection.testSingleRecord org.apache.hadoop.mapreduce.TestMapCollection.testLowSpill org.apache.hadoop.mapreduce.TestMapCollection.testSplitMetaSpill org.apache.hadoop.mapreduce.TestMapCollection.testPostSpillMeta org.apache.hadoop.mapreduce.TestMapCollection.testLargeRecConcurrent org.apache.hadoop.mapreduce.TestMapCollection.testRandom org.apache.hadoop.mapreduce.TestMapCollection.testRandomCompress All of them have similar stack traces of: java.io.IOException: Invalid mapreduce.jobtracker.address configuration value for JobTracker: local at org.apache.hadoop.mapred.JobTrackerClientProtocolProvider.create(JobTrackerClientProtocolProvider.java:47) at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:74) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:459) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:438) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:777) at org.apache.hadoop.mapred.TestCollect.testCollect(TestCollect.java:133) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2952) Application failure diagnostics are not consumed in a couple of cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114634#comment-13114634 ] Hudson commented on MAPREDUCE-2952: --- Integrated in Hadoop-Hdfs-trunk #812 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/812/]) MAPREDUCE-2952. Fixed ResourceManager/MR-client to consume diagnostics for AM failures in a couple of corner cases. Contributed by Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1175403 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/JobReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/impl/pb/JobReportPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/proto/mr_protos.proto * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobStatus.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobStatus.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationReportPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/MockApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/ApplicationMasterLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppFailedAttemptEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java *
[jira] [Commented] (MAPREDUCE-2952) Application failure diagnostics are not consumed in a couple of cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114637#comment-13114637 ] Hudson commented on MAPREDUCE-2952: --- Integrated in Hadoop-Hdfs-0.23-Build #21 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/21/]) MAPREDUCE-2952. svn merge -c r1175403 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1175408 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/JobReport.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/impl/pb/JobReportPBImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/proto/mr_protos.proto * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobStatus.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobStatus.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationReportPBImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/MockApps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/ApplicationMasterLauncher.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppFailedAttemptEvent.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java *
[jira] [Commented] (MAPREDUCE-3090) Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere
[ https://issues.apache.org/jira/browse/MAPREDUCE-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114643#comment-13114643 ] Hudson commented on MAPREDUCE-3090: --- Integrated in Hadoop-Mapreduce-0.23-Build #27 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/27/]) Merge -r 1175717:1175718 from trunk to branch-0.23 to fix MAPREDUCE-3090. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1175719 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere --- Key: MAPREDUCE-3090 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3090 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.0 Attachments: MAPREDUCE-3090.patch Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere, particularly after MAPREDUCE-3055 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3064) 27 unit test failures with Invalid mapreduce.jobtracker.address configuration value for JobTracker: local
[ https://issues.apache.org/jira/browse/MAPREDUCE-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114644#comment-13114644 ] Hudson commented on MAPREDUCE-3064: --- Integrated in Hadoop-Mapreduce-0.23-Build #27 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/27/]) MAPREDUCE-3064. 27 unit test failures with Invalid mapreduce.jobtracker.address configuration value for JobTracker: local (Venu Gopala Rao via mahadev) - Merging r1175449 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1175450 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobTrackerClientProtocolProvider.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/MiniMRCluster.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/QueueManagerTestUtils.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestMiniMRClasspath.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSpecialCharactersInOutputPath.java 27 unit test failures with Invalid mapreduce.jobtracker.address configuration value for JobTracker: local -- Key: MAPREDUCE-3064 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3064 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Venu Gopala Rao Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3064.1.patch, MAPREDUCE-3064.patch unit test failure here: https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/946/ Test Result (27 failures / +27) org.apache.hadoop.mapred.TestCollect.testCollect org.apache.hadoop.mapred.TestComparators.testDefaultMRComparator org.apache.hadoop.mapred.TestComparators.testUserMRComparator org.apache.hadoop.mapred.TestComparators.testUserValueGroupingComparator org.apache.hadoop.mapred.TestComparators.testAllUserComparators org.apache.hadoop.mapred.TestFileOutputFormat.testCustomFile org.apache.hadoop.mapred.TestJavaSerialization.testMapReduceJob org.apache.hadoop.mapred.TestJavaSerialization.testWriteToSequencefile org.apache.hadoop.mapred.TestMapOutputType.testKeyMismatch org.apache.hadoop.mapred.TestMapOutputType.testValueMismatch org.apache.hadoop.mapred.TestMapOutputType.testNoMismatch org.apache.hadoop.mapred.TestMapRed.testMapred org.apache.hadoop.mapred.TestMapRed.testNullKeys org.apache.hadoop.mapred.TestMapRed.testCompression org.apache.hadoop.mapred.TestMapRed.testSmallInput org.apache.hadoop.mapred.TestMapRed.testBiggerInput org.apache.hadoop.mapreduce.TestMapCollection.testValLastByte org.apache.hadoop.mapreduce.TestMapCollection.testLargeRecords org.apache.hadoop.mapreduce.TestMapCollection.testSpillPer2B org.apache.hadoop.mapreduce.TestMapCollection.testZeroVal org.apache.hadoop.mapreduce.TestMapCollection.testSingleRecord org.apache.hadoop.mapreduce.TestMapCollection.testLowSpill org.apache.hadoop.mapreduce.TestMapCollection.testSplitMetaSpill org.apache.hadoop.mapreduce.TestMapCollection.testPostSpillMeta org.apache.hadoop.mapreduce.TestMapCollection.testLargeRecConcurrent org.apache.hadoop.mapreduce.TestMapCollection.testRandom org.apache.hadoop.mapreduce.TestMapCollection.testRandomCompress All of them have similar stack traces of: java.io.IOException: Invalid mapreduce.jobtracker.address configuration value for JobTracker: local at org.apache.hadoop.mapred.JobTrackerClientProtocolProvider.create(JobTrackerClientProtocolProvider.java:47) at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:74) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:459) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:438) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:777) at org.apache.hadoop.mapred.TestCollect.testCollect(TestCollect.java:133) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3090) Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere
[ https://issues.apache.org/jira/browse/MAPREDUCE-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114650#comment-13114650 ] Hudson commented on MAPREDUCE-3090: --- Integrated in Hadoop-Mapreduce-trunk #842 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/842/]) MAPREDUCE-3090. Fix MR AM to use ApplicationAttemptId rather than (ApplicationId, startCount) consistently. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1175718 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere --- Key: MAPREDUCE-3090 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3090 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.0 Attachments: MAPREDUCE-3090.patch Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere, particularly after MAPREDUCE-3055 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2952) Application failure diagnostics are not consumed in a couple of cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114652#comment-13114652 ] Hudson commented on MAPREDUCE-2952: --- Integrated in Hadoop-Mapreduce-trunk #842 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/842/]) MAPREDUCE-2952. Fixed ResourceManager/MR-client to consume diagnostics for AM failures in a couple of corner cases. Contributed by Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1175403 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/JobReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/impl/pb/JobReportPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/proto/mr_protos.proto * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobStatus.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobStatus.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationReportPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/MockApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/ApplicationMasterLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppFailedAttemptEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java *
[jira] [Updated] (MAPREDUCE-2646) MR-279: AM with same sized maps and reduces hangs in presence of failing maps
[ https://issues.apache.org/jira/browse/MAPREDUCE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-2646: --- Component/s: applicationmaster Affects Version/s: 0.23.0 Fix Version/s: 0.23.0 Hadoop Flags: [Reviewed] The patch looks good. +1. MR-279: AM with same sized maps and reduces hangs in presence of failing maps - Key: MAPREDUCE-2646 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2646 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Critical Fix For: 0.23.0 Attachments: MAPREDUCE-2646.patch, mr-2646.patch Currently AM can assign a container given by RM to any map or reduce. However RM allocates for a particular priority. This leads to AM and RM data structures going out of sync. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2646) MR-279: AM with same sized maps and reduces hangs in presence of failing maps
[ https://issues.apache.org/jira/browse/MAPREDUCE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114654#comment-13114654 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-2646: We need to write a test for this, but the central test for this, TestRMContainerAllocator is disabled for now. I'll add a test for this as part of MAPREDUCE-3089. MR-279: AM with same sized maps and reduces hangs in presence of failing maps - Key: MAPREDUCE-2646 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2646 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Critical Fix For: 0.23.0 Attachments: MAPREDUCE-2646.patch, mr-2646.patch Currently AM can assign a container given by RM to any map or reduce. However RM allocates for a particular priority. This leads to AM and RM data structures going out of sync. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3089) Re-enable TestRMContainerAllocator
[ https://issues.apache.org/jira/browse/MAPREDUCE-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli reassigned MAPREDUCE-3089: -- Assignee: Vinod Kumar Vavilapalli We need to add a test for MAPREDUCE-3089 once this test is enabled. Re-enable TestRMContainerAllocator -- Key: MAPREDUCE-3089 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3089 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.0 Re-enable TestRMContainerAllocator to ensure we have good coverage. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2646) MR-279: AM with same sized maps and reduces hangs in presence of failing maps
[ https://issues.apache.org/jira/browse/MAPREDUCE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-2646: --- Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this trunk and branch-0.23. Thanks Sharad and Arun! MR-279: AM with same sized maps and reduces hangs in presence of failing maps - Key: MAPREDUCE-2646 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2646 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Critical Fix For: 0.23.0 Attachments: MAPREDUCE-2646.patch, mr-2646.patch Currently AM can assign a container given by RM to any map or reduce. However RM allocates for a particular priority. This leads to AM and RM data structures going out of sync. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2646) MR-279: AM with same sized maps and reduces hangs in presence of failing maps
[ https://issues.apache.org/jira/browse/MAPREDUCE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114664#comment-13114664 ] Hudson commented on MAPREDUCE-2646: --- Integrated in Hadoop-Common-trunk-Commit #955 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/955/]) MAPREDUCE-2646. Fixed AMRMProtocol to return containers based on priority. Contributed by Sharad Agarwal and Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1175859 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Container.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/NodeManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java MR-279: AM with same sized maps and reduces hangs in presence of failing maps - Key: MAPREDUCE-2646 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2646 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Critical Fix For: 0.23.0 Attachments: MAPREDUCE-2646.patch, mr-2646.patch Currently AM can assign a container given by RM to any map or reduce. However RM allocates for a particular priority. This leads to AM and RM data structures going out of sync. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2646) MR-279: AM with same sized maps and reduces hangs in presence of failing maps
[ https://issues.apache.org/jira/browse/MAPREDUCE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114665#comment-13114665 ] Hudson commented on MAPREDUCE-2646: --- Integrated in Hadoop-Hdfs-trunk-Commit #1033 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1033/]) MAPREDUCE-2646. Fixed AMRMProtocol to return containers based on priority. Contributed by Sharad Agarwal and Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1175859 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Container.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/NodeManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java MR-279: AM with same sized maps and reduces hangs in presence of failing maps - Key: MAPREDUCE-2646 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2646 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Critical Fix For: 0.23.0 Attachments: MAPREDUCE-2646.patch, mr-2646.patch Currently AM can assign a container given by RM to any map or reduce. However RM allocates for a particular priority. This leads to AM and RM data structures going out of sync. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2646) MR-279: AM with same sized maps and reduces hangs in presence of failing maps
[ https://issues.apache.org/jira/browse/MAPREDUCE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114670#comment-13114670 ] Hudson commented on MAPREDUCE-2646: --- Integrated in Hadoop-Mapreduce-trunk-Commit #973 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/973/]) MAPREDUCE-2646. Fixed AMRMProtocol to return containers based on priority. Contributed by Sharad Agarwal and Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1175859 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Container.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/NodeManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java MR-279: AM with same sized maps and reduces hangs in presence of failing maps - Key: MAPREDUCE-2646 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2646 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Critical Fix For: 0.23.0 Attachments: MAPREDUCE-2646.patch, mr-2646.patch Currently AM can assign a container given by RM to any map or reduce. However RM allocates for a particular priority. This leads to AM and RM data structures going out of sync. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3050) YarnScheduler needs to expose Resource Usage Information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114679#comment-13114679 ] Eric Payne commented on MAPREDUCE-3050: --- @Bobby, Would it be possible to change the SchedulerApp constructor to take a queue name instead of a Queue instance? I'm trying to call the SchedulerApp() constructor from the RMAppImpl class, and there's currently not a Scheduler agnostic way to get the Queue without knowing whether the scheduler is Fifo or Capacity. YarnScheduler needs to expose Resource Usage Information Key: MAPREDUCE-3050 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3050 Project: Hadoop Map/Reduce Issue Type: Bug Components: resourcemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Priority: Blocker Fix For: 0.23.0, 0.24.0 Attachments: MR-3050.txt, MR-3050.txt Before the recent refactor The nodes had information in them about how much resources they were using. This information is not hidden inside SchedulerNode. Similarly resource usage information about an application, or in aggregate is only available through the Scheduler and there is not interface to pull it out. We need to expose APIs to get Resource and Container information from the scheduler, in aggregate across the entire cluster, per application, per node, and ideally also per queue if applicable (although there are no JIRAs I am aware of that need this right now). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-3064) 27 unit test failures with Invalid mapreduce.jobtracker.address configuration value for JobTracker: local
[ https://issues.apache.org/jira/browse/MAPREDUCE-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli reopened MAPREDUCE-3064: This patch broke the ant build for MRV1, I cannot compile a test. (ant veryclean all-jars -Dresolvers=internal) 27 unit test failures with Invalid mapreduce.jobtracker.address configuration value for JobTracker: local -- Key: MAPREDUCE-3064 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3064 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Venu Gopala Rao Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3064.1.patch, MAPREDUCE-3064.patch unit test failure here: https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/946/ Test Result (27 failures / +27) org.apache.hadoop.mapred.TestCollect.testCollect org.apache.hadoop.mapred.TestComparators.testDefaultMRComparator org.apache.hadoop.mapred.TestComparators.testUserMRComparator org.apache.hadoop.mapred.TestComparators.testUserValueGroupingComparator org.apache.hadoop.mapred.TestComparators.testAllUserComparators org.apache.hadoop.mapred.TestFileOutputFormat.testCustomFile org.apache.hadoop.mapred.TestJavaSerialization.testMapReduceJob org.apache.hadoop.mapred.TestJavaSerialization.testWriteToSequencefile org.apache.hadoop.mapred.TestMapOutputType.testKeyMismatch org.apache.hadoop.mapred.TestMapOutputType.testValueMismatch org.apache.hadoop.mapred.TestMapOutputType.testNoMismatch org.apache.hadoop.mapred.TestMapRed.testMapred org.apache.hadoop.mapred.TestMapRed.testNullKeys org.apache.hadoop.mapred.TestMapRed.testCompression org.apache.hadoop.mapred.TestMapRed.testSmallInput org.apache.hadoop.mapred.TestMapRed.testBiggerInput org.apache.hadoop.mapreduce.TestMapCollection.testValLastByte org.apache.hadoop.mapreduce.TestMapCollection.testLargeRecords org.apache.hadoop.mapreduce.TestMapCollection.testSpillPer2B org.apache.hadoop.mapreduce.TestMapCollection.testZeroVal org.apache.hadoop.mapreduce.TestMapCollection.testSingleRecord org.apache.hadoop.mapreduce.TestMapCollection.testLowSpill org.apache.hadoop.mapreduce.TestMapCollection.testSplitMetaSpill org.apache.hadoop.mapreduce.TestMapCollection.testPostSpillMeta org.apache.hadoop.mapreduce.TestMapCollection.testLargeRecConcurrent org.apache.hadoop.mapreduce.TestMapCollection.testRandom org.apache.hadoop.mapreduce.TestMapCollection.testRandomCompress All of them have similar stack traces of: java.io.IOException: Invalid mapreduce.jobtracker.address configuration value for JobTracker: local at org.apache.hadoop.mapred.JobTrackerClientProtocolProvider.create(JobTrackerClientProtocolProvider.java:47) at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:74) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:459) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:438) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:777) at org.apache.hadoop.mapred.TestCollect.testCollect(TestCollect.java:133) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2861) Modify version_control.html to reflect new path to mapreduce trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-2861. Resolution: Invalid Someone fixed this. Not valid anymore. Modify version_control.html to reflect new path to mapreduce trunk -- Key: MAPREDUCE-2861 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2861 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Vinod Kumar Vavilapalli We need to modify http://hadoop.apache.org/mapreduce/version_control.html to reflect the change of mapreduce trunk to trunk/hadoop-mapreduce. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3064) 27 unit test failures with Invalid mapreduce.jobtracker.address configuration value for JobTracker: local
[ https://issues.apache.org/jira/browse/MAPREDUCE-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-3064. Resolution: Fixed Never mind, ran into the stale jars issue. 27 unit test failures with Invalid mapreduce.jobtracker.address configuration value for JobTracker: local -- Key: MAPREDUCE-3064 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3064 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Venu Gopala Rao Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3064.1.patch, MAPREDUCE-3064.patch unit test failure here: https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/946/ Test Result (27 failures / +27) org.apache.hadoop.mapred.TestCollect.testCollect org.apache.hadoop.mapred.TestComparators.testDefaultMRComparator org.apache.hadoop.mapred.TestComparators.testUserMRComparator org.apache.hadoop.mapred.TestComparators.testUserValueGroupingComparator org.apache.hadoop.mapred.TestComparators.testAllUserComparators org.apache.hadoop.mapred.TestFileOutputFormat.testCustomFile org.apache.hadoop.mapred.TestJavaSerialization.testMapReduceJob org.apache.hadoop.mapred.TestJavaSerialization.testWriteToSequencefile org.apache.hadoop.mapred.TestMapOutputType.testKeyMismatch org.apache.hadoop.mapred.TestMapOutputType.testValueMismatch org.apache.hadoop.mapred.TestMapOutputType.testNoMismatch org.apache.hadoop.mapred.TestMapRed.testMapred org.apache.hadoop.mapred.TestMapRed.testNullKeys org.apache.hadoop.mapred.TestMapRed.testCompression org.apache.hadoop.mapred.TestMapRed.testSmallInput org.apache.hadoop.mapred.TestMapRed.testBiggerInput org.apache.hadoop.mapreduce.TestMapCollection.testValLastByte org.apache.hadoop.mapreduce.TestMapCollection.testLargeRecords org.apache.hadoop.mapreduce.TestMapCollection.testSpillPer2B org.apache.hadoop.mapreduce.TestMapCollection.testZeroVal org.apache.hadoop.mapreduce.TestMapCollection.testSingleRecord org.apache.hadoop.mapreduce.TestMapCollection.testLowSpill org.apache.hadoop.mapreduce.TestMapCollection.testSplitMetaSpill org.apache.hadoop.mapreduce.TestMapCollection.testPostSpillMeta org.apache.hadoop.mapreduce.TestMapCollection.testLargeRecConcurrent org.apache.hadoop.mapreduce.TestMapCollection.testRandom org.apache.hadoop.mapreduce.TestMapCollection.testRandomCompress All of them have similar stack traces of: java.io.IOException: Invalid mapreduce.jobtracker.address configuration value for JobTracker: local at org.apache.hadoop.mapred.JobTrackerClientProtocolProvider.create(JobTrackerClientProtocolProvider.java:47) at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:74) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:459) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:438) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:777) at org.apache.hadoop.mapred.TestCollect.testCollect(TestCollect.java:133) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3050) YarnScheduler needs to expose Resource Usage Information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3050: --- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Open (was: Patch Available) Canceling the patch to address Eric's request. YarnScheduler needs to expose Resource Usage Information Key: MAPREDUCE-3050 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3050 Project: Hadoop Map/Reduce Issue Type: Bug Components: resourcemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Priority: Blocker Fix For: 0.23.0, 0.24.0 Attachments: MR-3050.txt, MR-3050.txt Before the recent refactor The nodes had information in them about how much resources they were using. This information is not hidden inside SchedulerNode. Similarly resource usage information about an application, or in aggregate is only available through the Scheduler and there is not interface to pull it out. We need to expose APIs to get Resource and Container information from the scheduler, in aggregate across the entire cluster, per application, per node, and ideally also per queue if applicable (although there are no JIRAs I am aware of that need this right now). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3054) Unable to kill submitted jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-3054:
---
Status: Open (was: Patch Available)
Looked at the patch. Some questions and comments on the approach itself.
- With the patch, YarnRunner sends a kill to AM and also a killJob signal to
RM in *all* cases. So in all the cases, AM may not be able to gracefully kill
itself and write the history file. Absence of history file for all the killed
jobs is a broken user experience IMHO. I think we should force-kill on the RM
side only when needed.
-- The implementation could be something like this: Check the Application
state on RM, if it is running, send a kill-signal to the AM otherwise
force-terminate on the RM. Sure, there are corner cases where the client
obtains a non-running state, but by the time it sends a force-terminate signal,
the app may become running, but losing the (little) information in those rare
cases should be bearable.
- Also jobs which are in the queue and get killed have no record in the
system( ! ) (except for the fleeting cache in RM). I can't think of any
solution other than MAPREDUCE-3061.
- I am tending to rename _ClientRMProtocol.finishApplication()_ to
_ClientRMProtocol.forceTerminateApplication()_ to better reflect its semantics.
Thoughts?
Unable to kill submitted jobs
-
Key: MAPREDUCE-3054
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3054
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-3054.patch, MAPREDUCE-3054.patch,
MAPREDUCE-3054.patch, MAPREDUCE-3054.patch
Found by Philip Su
The mapred job -kill command
appears to succeed, but listing the jobs again shows that the job supposedly
killed is still there.
{code}
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
mapred job -kill job_1316203984216_0002
Killed job job_1316203984216_0002
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3092) Remove JOB_ID_COMPARATOR usage in JobHistory.java
Remove JOB_ID_COMPARATOR usage in JobHistory.java - Key: MAPREDUCE-3092 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3092 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0 Reporter: Devaraj K Assignee: Devaraj K Priority: Minor As part of the defect MAPREDUCE-2965, JobId.compareTo() has been implemented. Usage of JOB_ID_COMPARATOR in JobHistory.java can be removed because comparison is handling by JobId itself. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3064) 27 unit test failures with Invalid mapreduce.jobtracker.address configuration value for JobTracker: local
[ https://issues.apache.org/jira/browse/MAPREDUCE-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114711#comment-13114711 ] Venu Gopala Rao commented on MAPREDUCE-3064: Vinod, any idea why Jenkins is getting compilation issue? 27 unit test failures with Invalid mapreduce.jobtracker.address configuration value for JobTracker: local -- Key: MAPREDUCE-3064 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3064 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Venu Gopala Rao Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3064.1.patch, MAPREDUCE-3064.patch unit test failure here: https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/946/ Test Result (27 failures / +27) org.apache.hadoop.mapred.TestCollect.testCollect org.apache.hadoop.mapred.TestComparators.testDefaultMRComparator org.apache.hadoop.mapred.TestComparators.testUserMRComparator org.apache.hadoop.mapred.TestComparators.testUserValueGroupingComparator org.apache.hadoop.mapred.TestComparators.testAllUserComparators org.apache.hadoop.mapred.TestFileOutputFormat.testCustomFile org.apache.hadoop.mapred.TestJavaSerialization.testMapReduceJob org.apache.hadoop.mapred.TestJavaSerialization.testWriteToSequencefile org.apache.hadoop.mapred.TestMapOutputType.testKeyMismatch org.apache.hadoop.mapred.TestMapOutputType.testValueMismatch org.apache.hadoop.mapred.TestMapOutputType.testNoMismatch org.apache.hadoop.mapred.TestMapRed.testMapred org.apache.hadoop.mapred.TestMapRed.testNullKeys org.apache.hadoop.mapred.TestMapRed.testCompression org.apache.hadoop.mapred.TestMapRed.testSmallInput org.apache.hadoop.mapred.TestMapRed.testBiggerInput org.apache.hadoop.mapreduce.TestMapCollection.testValLastByte org.apache.hadoop.mapreduce.TestMapCollection.testLargeRecords org.apache.hadoop.mapreduce.TestMapCollection.testSpillPer2B org.apache.hadoop.mapreduce.TestMapCollection.testZeroVal org.apache.hadoop.mapreduce.TestMapCollection.testSingleRecord org.apache.hadoop.mapreduce.TestMapCollection.testLowSpill org.apache.hadoop.mapreduce.TestMapCollection.testSplitMetaSpill org.apache.hadoop.mapreduce.TestMapCollection.testPostSpillMeta org.apache.hadoop.mapreduce.TestMapCollection.testLargeRecConcurrent org.apache.hadoop.mapreduce.TestMapCollection.testRandom org.apache.hadoop.mapreduce.TestMapCollection.testRandomCompress All of them have similar stack traces of: java.io.IOException: Invalid mapreduce.jobtracker.address configuration value for JobTracker: local at org.apache.hadoop.mapred.JobTrackerClientProtocolProvider.create(JobTrackerClientProtocolProvider.java:47) at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:74) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:459) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:438) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:777) at org.apache.hadoop.mapred.TestCollect.testCollect(TestCollect.java:133) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3064) 27 unit test failures with Invalid mapreduce.jobtracker.address configuration value for JobTracker: local
[ https://issues.apache.org/jira/browse/MAPREDUCE-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114712#comment-13114712 ] Mahadev konar commented on MAPREDUCE-3064: -- @venu, Jenkins is running into some stale jars with ivy cache. Ill fix that sometime today. 27 unit test failures with Invalid mapreduce.jobtracker.address configuration value for JobTracker: local -- Key: MAPREDUCE-3064 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3064 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Venu Gopala Rao Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3064.1.patch, MAPREDUCE-3064.patch unit test failure here: https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/946/ Test Result (27 failures / +27) org.apache.hadoop.mapred.TestCollect.testCollect org.apache.hadoop.mapred.TestComparators.testDefaultMRComparator org.apache.hadoop.mapred.TestComparators.testUserMRComparator org.apache.hadoop.mapred.TestComparators.testUserValueGroupingComparator org.apache.hadoop.mapred.TestComparators.testAllUserComparators org.apache.hadoop.mapred.TestFileOutputFormat.testCustomFile org.apache.hadoop.mapred.TestJavaSerialization.testMapReduceJob org.apache.hadoop.mapred.TestJavaSerialization.testWriteToSequencefile org.apache.hadoop.mapred.TestMapOutputType.testKeyMismatch org.apache.hadoop.mapred.TestMapOutputType.testValueMismatch org.apache.hadoop.mapred.TestMapOutputType.testNoMismatch org.apache.hadoop.mapred.TestMapRed.testMapred org.apache.hadoop.mapred.TestMapRed.testNullKeys org.apache.hadoop.mapred.TestMapRed.testCompression org.apache.hadoop.mapred.TestMapRed.testSmallInput org.apache.hadoop.mapred.TestMapRed.testBiggerInput org.apache.hadoop.mapreduce.TestMapCollection.testValLastByte org.apache.hadoop.mapreduce.TestMapCollection.testLargeRecords org.apache.hadoop.mapreduce.TestMapCollection.testSpillPer2B org.apache.hadoop.mapreduce.TestMapCollection.testZeroVal org.apache.hadoop.mapreduce.TestMapCollection.testSingleRecord org.apache.hadoop.mapreduce.TestMapCollection.testLowSpill org.apache.hadoop.mapreduce.TestMapCollection.testSplitMetaSpill org.apache.hadoop.mapreduce.TestMapCollection.testPostSpillMeta org.apache.hadoop.mapreduce.TestMapCollection.testLargeRecConcurrent org.apache.hadoop.mapreduce.TestMapCollection.testRandom org.apache.hadoop.mapreduce.TestMapCollection.testRandomCompress All of them have similar stack traces of: java.io.IOException: Invalid mapreduce.jobtracker.address configuration value for JobTracker: local at org.apache.hadoop.mapred.JobTrackerClientProtocolProvider.create(JobTrackerClientProtocolProvider.java:47) at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:74) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:459) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:438) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:777) at org.apache.hadoop.mapred.TestCollect.testCollect(TestCollect.java:133) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3054) Unable to kill submitted jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114720#comment-13114720
]
Mahadev konar commented on MAPREDUCE-3054:
--
I think most of the client commands should be fire and forget. Longer term I
think the NodeManager should be sending a SIGTERM, and wait for a while before
sending SIGKILL, to let AM/others do some work before getting a SIGKILL.
We always have to force terminate, I am not sure how you can get away from
that. We should always be sending a KILL SIGNAL to RM, else we cannot confirm
an AM is KILLED. How do you intend to enforce that an AM is dead?
Unable to kill submitted jobs
-
Key: MAPREDUCE-3054
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3054
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-3054.patch, MAPREDUCE-3054.patch,
MAPREDUCE-3054.patch, MAPREDUCE-3054.patch
Found by Philip Su
The mapred job -kill command
appears to succeed, but listing the jobs again shows that the job supposedly
killed is still there.
{code}
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
mapred job -kill job_1316203984216_0002
Killed job job_1316203984216_0002
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3093) Write tests for data locality in MRv2.
Write tests for data locality in MRv2. -- Key: MAPREDUCE-3093 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3093 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: Mahadev konar Priority: Blocker Fix For: 0.23.0 We should add tests to make sure data locality is in place in MRv2 (with respect to the capacity scheduler and also the matching/ask of containers in the MR AM). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3093) Write additional tests for data locality in MRv2.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3093: - Summary: Write additional tests for data locality in MRv2. (was: Write tests for data locality in MRv2.) Write additional tests for data locality in MRv2. - Key: MAPREDUCE-3093 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3093 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: Mahadev konar Priority: Blocker Fix For: 0.23.0 We should add tests to make sure data locality is in place in MRv2 (with respect to the capacity scheduler and also the matching/ask of containers in the MR AM). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2843) [MR-279] Node entries on the RM UI are not sortable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhijit Suresh Shingate updated MAPREDUCE-2843: --- Attachment: (was: MAPREDUCE-2843.2.patch) [MR-279] Node entries on the RM UI are not sortable --- Key: MAPREDUCE-2843 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2843 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Ramya Sunil Assignee: Abhijit Suresh Shingate Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2843- scr shot.jpg, MAPREDUCE-2843.1.patch, MAPREDUCE-2843.patch, MAPREDUCE-2843.patch The nodemanager entries on the RM UI is not sortable unlike the other web pages. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2843) [MR-279] Node entries on the RM UI are not sortable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhijit Suresh Shingate updated MAPREDUCE-2843: --- Attachment: MAPREDUCE-2843.2.patch [MR-279] Node entries on the RM UI are not sortable --- Key: MAPREDUCE-2843 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2843 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Ramya Sunil Assignee: Abhijit Suresh Shingate Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2843- scr shot.jpg, MAPREDUCE-2843.1.patch, MAPREDUCE-2843.patch, MAPREDUCE-2843.patch The nodemanager entries on the RM UI is not sortable unlike the other web pages. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2843) [MR-279] Node entries on the RM UI are not sortable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhijit Suresh Shingate updated MAPREDUCE-2843: --- Attachment: MAPREDUCE-2843.2.patch [MR-279] Node entries on the RM UI are not sortable --- Key: MAPREDUCE-2843 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2843 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Ramya Sunil Assignee: Abhijit Suresh Shingate Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2843- scr shot.jpg, MAPREDUCE-2843.1.patch, MAPREDUCE-2843.2.patch, MAPREDUCE-2843.patch, MAPREDUCE-2843.patch The nodemanager entries on the RM UI is not sortable unlike the other web pages. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3054) Unable to kill submitted jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114749#comment-13114749
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3054:
That's a little different. Cleaning up of containers(or AM)' process tree is
independent and can be triggered once it is clear for the components in
question that they are done with their work - via TaskUmbilicalProtocol.done()
for the task and AMRMProtocol.unregister() for the AM case.
For the killed jobs, once AM gets the kill-signal, it can gracefully shut down
by writing history-file, sending notification url etc, then it can proceed to
deregistering with the RM. Once the deregistration comes through, the RM can
then direct the NM to make sure the AM is dead.
Unable to kill submitted jobs
-
Key: MAPREDUCE-3054
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3054
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-3054.patch, MAPREDUCE-3054.patch,
MAPREDUCE-3054.patch, MAPREDUCE-3054.patch
Found by Philip Su
The mapred job -kill command
appears to succeed, but listing the jobs again shows that the job supposedly
killed is still there.
{code}
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
mapred job -kill job_1316203984216_0002
Killed job job_1316203984216_0002
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3054) Unable to kill submitted jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114754#comment-13114754
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3054:
bq. Also jobs which are in the queue and get killed have no record in the
system( ! ) (except for the fleeting cache in RM). I can't think of any
solution other than MAPREDUCE-3061.
One idea just stuck me - we can make the JobClient write out the history-file
for these jobs which are killed when idling in the queue.
Unable to kill submitted jobs
-
Key: MAPREDUCE-3054
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3054
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-3054.patch, MAPREDUCE-3054.patch,
MAPREDUCE-3054.patch, MAPREDUCE-3054.patch
Found by Philip Su
The mapred job -kill command
appears to succeed, but listing the jobs again shows that the job supposedly
killed is still there.
{code}
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
mapred job -kill job_1316203984216_0002
Killed job job_1316203984216_0002
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3054) Unable to kill submitted jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114759#comment-13114759
]
Mahadev konar commented on MAPREDUCE-3054:
--
@Vinod, Its probably not different but that argument is for another day :).
For the AM getting KILL signal, we can never be sure that the AM will call
finishapplicationMaster. How would you enforce that? I dont think we can get
away from not sending kill to the RM at *all* times. I dont think we can rely
on AM to be responsive and do the right work. Thoughts?
On the renaming of the finishAPplication, I do agree we should change that.
Unable to kill submitted jobs
-
Key: MAPREDUCE-3054
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3054
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-3054.patch, MAPREDUCE-3054.patch,
MAPREDUCE-3054.patch, MAPREDUCE-3054.patch
Found by Philip Su
The mapred job -kill command
appears to succeed, but listing the jobs again shows that the job supposedly
killed is still there.
{code}
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
mapred job -kill job_1316203984216_0002
Killed job job_1316203984216_0002
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2843) [MR-279] Node entries on the RM UI are not sortable
[
https://issues.apache.org/jira/browse/MAPREDUCE-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114764#comment-13114764
]
Luke Lu commented on MAPREDUCE-2843:
Why is a custom writer/validator necessary? It seems:
{code}
verify(writer, times(numHeaderCells).print(th);
verify(writer, times(numDataCells).print(td);
{code}
would suffice.
[MR-279] Node entries on the RM UI are not sortable
---
Key: MAPREDUCE-2843
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2843
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Ramya Sunil
Assignee: Abhijit Suresh Shingate
Fix For: 0.23.0, 0.24.0
Attachments: MAPREDUCE-2843- scr shot.jpg, MAPREDUCE-2843.1.patch,
MAPREDUCE-2843.2.patch, MAPREDUCE-2843.patch, MAPREDUCE-2843.patch
The nodemanager entries on the RM UI is not sortable unlike the other web
pages.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3031) Job Client goes into infinite loop when we kill AM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3031: --- Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this to trunk and branch-0.23. Thanks Sid! Job Client goes into infinite loop when we kill AM -- Key: MAPREDUCE-3031 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3031 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Karam Singh Assignee: Siddharth Seth Priority: Blocker Fix For: 0.23.0 Attachments: MR3031_v1.patch Started a cluster. Submitted a sleep job with around 1 maps and 1000 reduces. Killed AM with kill -9 by which time already 7000 thousands maps got completed. On the RM webUI, Application is stuck in Application.RUNNING state. And JobClient goes into an infinite loop as RM keeps telling the client that the application is running. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3084) race when KILL_CONTAINER is received for a LOCALIZED container
[
https://issues.apache.org/jira/browse/MAPREDUCE-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114830#comment-13114830
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3084:
Good catch, Sid!
While we are at it, we need to fix a couple of missing transitions from the
LOCALIZATION_FAILED state.
race when KILL_CONTAINER is received for a LOCALIZED container
--
Key: MAPREDUCE-3084
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3084
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Priority: Blocker
Depending on when ContainersLaunch starts a container, {{KILL_CONTAINER}}
when container state is {{LOCALIZED}} ({{LAUNCH_CONTAINER}} event already
sent) can end up generating a {{CONTAINER_LAUNCHED}} event - which isn't
handled by ContainerState: {{KILLING}}. Also, the launched container won't be
killed since {{CLEANUP_CONTAINER}} would have already been processed.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3031) Job Client goes into infinite loop when we kill AM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114831#comment-13114831 ] Hudson commented on MAPREDUCE-3031: --- Integrated in Hadoop-Common-trunk-Commit #956 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/956/]) MAPREDUCE-3031. Proper handling of killed containers to prevent stuck containers/AMs on an external kill signal. Contributed by Siddharth Seth. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1175960 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java Job Client goes into infinite loop when we kill AM -- Key: MAPREDUCE-3031 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3031 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Karam Singh Assignee: Siddharth Seth Priority: Blocker Fix For: 0.23.0 Attachments: MR3031_v1.patch Started a cluster. Submitted a sleep job with around 1 maps and 1000 reduces. Killed AM with kill -9 by which time already 7000 thousands maps got completed. On the RM webUI, Application is stuck in Application.RUNNING state. And JobClient goes into an infinite loop as RM keeps telling the client that the application is running. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3050) YarnScheduler needs to expose Resource Usage Information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3050: --- Attachment: MR-3050.txt Uploading a new patch. I upmerged and changed getRootQueue to getRootQueueMetrics, because I did not like exposing the queue in that way. @Eric I looked more closely at your comments and I think you will want to use the API getSchedulerApp instead of creating a SchedulerApp yourself. YarnScheduler needs to expose Resource Usage Information Key: MAPREDUCE-3050 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3050 Project: Hadoop Map/Reduce Issue Type: Bug Components: resourcemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Priority: Blocker Fix For: 0.23.0, 0.24.0 Attachments: MR-3050.txt, MR-3050.txt, MR-3050.txt Before the recent refactor The nodes had information in them about how much resources they were using. This information is not hidden inside SchedulerNode. Similarly resource usage information about an application, or in aggregate is only available through the Scheduler and there is not interface to pull it out. We need to expose APIs to get Resource and Container information from the scheduler, in aggregate across the entire cluster, per application, per node, and ideally also per queue if applicable (although there are no JIRAs I am aware of that need this right now). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3050) YarnScheduler needs to expose Resource Usage Information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3050: --- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Patch Available (was: Open) YarnScheduler needs to expose Resource Usage Information Key: MAPREDUCE-3050 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3050 Project: Hadoop Map/Reduce Issue Type: Bug Components: resourcemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Priority: Blocker Fix For: 0.23.0, 0.24.0 Attachments: MR-3050.txt, MR-3050.txt, MR-3050.txt Before the recent refactor The nodes had information in them about how much resources they were using. This information is not hidden inside SchedulerNode. Similarly resource usage information about an application, or in aggregate is only available through the Scheduler and there is not interface to pull it out. We need to expose APIs to get Resource and Container information from the scheduler, in aggregate across the entire cluster, per application, per node, and ideally also per queue if applicable (although there are no JIRAs I am aware of that need this right now). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3031) Job Client goes into infinite loop when we kill AM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114845#comment-13114845 ] Hudson commented on MAPREDUCE-3031: --- Integrated in Hadoop-Mapreduce-trunk-Commit #974 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/974/]) MAPREDUCE-3031. Proper handling of killed containers to prevent stuck containers/AMs on an external kill signal. Contributed by Siddharth Seth. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1175960 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java Job Client goes into infinite loop when we kill AM -- Key: MAPREDUCE-3031 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3031 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Karam Singh Assignee: Siddharth Seth Priority: Blocker Fix For: 0.23.0 Attachments: MR3031_v1.patch Started a cluster. Submitted a sleep job with around 1 maps and 1000 reduces. Killed AM with kill -9 by which time already 7000 thousands maps got completed. On the RM webUI, Application is stuck in Application.RUNNING state. And JobClient goes into an infinite loop as RM keeps telling the client that the application is running. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3056) Jobs are failing when those are submitted by other users
[
https://issues.apache.org/jira/browse/MAPREDUCE-3056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-3056:
---
Component/s: applicationmaster
Fix Version/s: 0.23.0
I ran into this today myself. This happens with DefaultContainerExecutor case,
which most of us are not using (surprise :) )
Devaraj (K), can you confirm?
The problem is the default staging dir use of ${user.name} system property,
which is the job-submitter on the client side and the system user on the server
side.
Jobs are failing when those are submitted by other users
Key: MAPREDUCE-3056
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3056
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
Fix For: 0.23.0
MR cluster is started by the user 'root'. If any other users other than
'root' submit a job, it is failing always.
Find the conatiner logs in the comments section.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
RE: [jira] [Updated] (MAPREDUCE-3056) Jobs are failing when those are submitted by other users
Hi Vinod,
I faced the problem exaclty as you explained.
The problem is the default staging dir use of ${user.name} system property,
which is the job-submitter on the client side and the system user on the server
side.
I started working on this issue, I will submit patch soon.
Thanks
Devaraj
From: Vinod Kumar Vavilapalli (JIRA) [j...@apache.org]
Sent: Monday, September 26, 2011 11:21 PM
To: mapreduce-issues@hadoop.apache.org
Subject: [jira] [Updated] (MAPREDUCE-3056) Jobs are failing when those are
submitted by other users
[
https://issues.apache.org/jira/browse/MAPREDUCE-3056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-3056:
---
Component/s: applicationmaster
Fix Version/s: 0.23.0
I ran into this today myself. This happens with DefaultContainerExecutor case,
which most of us are not using (surprise :) )
Devaraj (K), can you confirm?
The problem is the default staging dir use of ${user.name} system property,
which is the job-submitter on the client side and the system user on the server
side.
Jobs are failing when those are submitted by other users
Key: MAPREDUCE-3056
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3056
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
Fix For: 0.23.0
MR cluster is started by the user 'root'. If any other users other than
'root' submit a job, it is failing always.
Find the conatiner logs in the comments section.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3095) fairscheduler ivy including wrong version for hdfs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated MAPREDUCE-3095: --- Target Version/s: 0.23.0, 0.24.0 fairscheduler ivy including wrong version for hdfs -- Key: MAPREDUCE-3095 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3095 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0, 0.24.0 Reporter: John George Assignee: John George Attachments: mapreduce-3095.patch fairscheduler ivy.xml includes the common version for hdfs dependency. This could break builds that have different common and hdfs version numbers. The reason we dont see it on the jenkins build is because we use the same version number for common and hdfs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3095) fairscheduler ivy including wrong version for hdfs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated MAPREDUCE-3095: --- Attachment: mapreduce-3095.patch fairscheduler ivy including wrong version for hdfs -- Key: MAPREDUCE-3095 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3095 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0, 0.24.0 Reporter: John George Assignee: John George Attachments: mapreduce-3095.patch fairscheduler ivy.xml includes the common version for hdfs dependency. This could break builds that have different common and hdfs version numbers. The reason we dont see it on the jenkins build is because we use the same version number for common and hdfs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3095) fairscheduler ivy including wrong version for hdfs
fairscheduler ivy including wrong version for hdfs -- Key: MAPREDUCE-3095 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3095 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0, 0.24.0 Reporter: John George Assignee: John George Attachments: mapreduce-3095.patch fairscheduler ivy.xml includes the common version for hdfs dependency. This could break builds that have different common and hdfs version numbers. The reason we dont see it on the jenkins build is because we use the same version number for common and hdfs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3095) fairscheduler ivy including wrong version for hdfs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated MAPREDUCE-3095: --- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Tags: mr2 mrv2 fairscheduler ivy including wrong version for hdfs -- Key: MAPREDUCE-3095 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3095 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0, 0.24.0 Reporter: John George Assignee: John George Attachments: mapreduce-3095.patch, mapreduce-branch-0.23.patch fairscheduler ivy.xml includes the common version for hdfs dependency. This could break builds that have different common and hdfs version numbers. The reason we dont see it on the jenkins build is because we use the same version number for common and hdfs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3095) fairscheduler ivy including wrong version for hdfs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated MAPREDUCE-3095: --- Attachment: mapreduce-branch-0.23.patch attaching the same patch for branch 23. fairscheduler ivy including wrong version for hdfs -- Key: MAPREDUCE-3095 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3095 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0, 0.24.0 Reporter: John George Assignee: John George Attachments: mapreduce-3095.patch, mapreduce-branch-0.23.patch fairscheduler ivy.xml includes the common version for hdfs dependency. This could break builds that have different common and hdfs version numbers. The reason we dont see it on the jenkins build is because we use the same version number for common and hdfs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3095) fairscheduler ivy including wrong version for hdfs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated MAPREDUCE-3095: --- Component/s: mrv2 Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Tags: (was: mr2 mrv2) fairscheduler ivy including wrong version for hdfs -- Key: MAPREDUCE-3095 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3095 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: John George Assignee: John George Attachments: mapreduce-3095.patch, mapreduce-branch-0.23.patch fairscheduler ivy.xml includes the common version for hdfs dependency. This could break builds that have different common and hdfs version numbers. The reason we dont see it on the jenkins build is because we use the same version number for common and hdfs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3096) Add a good way to control the number of map/reduce tasks per node
Add a good way to control the number of map/reduce tasks per node - Key: MAPREDUCE-3096 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3096 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Arsen Zahray Fix For: 0.20.204.0 Currently, controlling the number of map/reduce tasks is a hell. I've tried for it many times, and it doesn't work right. Also, I am not the only one person, who seems to have this problem. There must be a better way to do it. Here's my proposal: add following functions to Job: setNumberOfMappersPerNode(int); setNumberOfReducersPerNode(int); setMaxMemoryPerMapper(int); setMaxMemoryPerReducer(int); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2984) Throwing NullPointerException when we open the container page
[
https://issues.apache.org/jira/browse/MAPREDUCE-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-2984:
-
Priority: Minor (was: Major)
Affects Version/s: (was: 0.24.0)
Fix Version/s: (was: 0.24.0)
Throwing NullPointerException when we open the container page
-
Key: MAPREDUCE-2984
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2984
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager
Affects Versions: 0.23.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Minor
Fix For: 0.23.0
Attachments: MAPREDUCE-2984-1.patch, MAPREDUCE-2984.patch
{code:xml}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.yarn.api.records.ContainerId.compareTo(ContainerId.java:97)
at
org.apache.hadoop.yarn.api.records.ContainerId.compareTo(ContainerId.java:23)
at
java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:819)
at
java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1640)
at
org.apache.hadoop.yarn.server.nodemanager.webapp.ContainerPage$ContainerBlock.render(ContainerPage.java:70)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:64)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:74)
at org.apache.hadoop.yarn.webapp.View.render(View.java:210)
{code}
{code:xml}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.nodemanager.webapp.ContainerPage$ContainerBlock.render(ContainerPage.java:71)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:64)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:74)
at org.apache.hadoop.yarn.webapp.View.render(View.java:210)
at
org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:47)
at
org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$TD._(Hamlet.java:843)
at
org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:54)
at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:80)
at org.apache.hadoop.yarn.webapp.Controller.render(Controller.java:210)
at
org.apache.hadoop.yarn.server.nodemanager.webapp.NMController.container(NMController.java:62)
... 30 more
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2984) Throwing NullPointerException when we open the container page
[
https://issues.apache.org/jira/browse/MAPREDUCE-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-2984:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Devaraj!
Throwing NullPointerException when we open the container page
-
Key: MAPREDUCE-2984
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2984
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager
Affects Versions: 0.23.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Minor
Fix For: 0.23.0
Attachments: MAPREDUCE-2984-1.patch, MAPREDUCE-2984.patch
{code:xml}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.yarn.api.records.ContainerId.compareTo(ContainerId.java:97)
at
org.apache.hadoop.yarn.api.records.ContainerId.compareTo(ContainerId.java:23)
at
java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:819)
at
java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1640)
at
org.apache.hadoop.yarn.server.nodemanager.webapp.ContainerPage$ContainerBlock.render(ContainerPage.java:70)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:64)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:74)
at org.apache.hadoop.yarn.webapp.View.render(View.java:210)
{code}
{code:xml}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.nodemanager.webapp.ContainerPage$ContainerBlock.render(ContainerPage.java:71)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:64)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:74)
at org.apache.hadoop.yarn.webapp.View.render(View.java:210)
at
org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:47)
at
org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$TD._(Hamlet.java:843)
at
org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:54)
at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:80)
at org.apache.hadoop.yarn.webapp.Controller.render(Controller.java:210)
at
org.apache.hadoop.yarn.server.nodemanager.webapp.NMController.container(NMController.java:62)
... 30 more
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3095) fairscheduler ivy including wrong version for hdfs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114913#comment-13114913 ] Mahadev konar commented on MAPREDUCE-3095: -- +1 looks good to me. Nice catch John! fairscheduler ivy including wrong version for hdfs -- Key: MAPREDUCE-3095 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3095 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: John George Assignee: John George Fix For: 0.23.0 Attachments: mapreduce-3095.patch, mapreduce-branch-0.23.patch fairscheduler ivy.xml includes the common version for hdfs dependency. This could break builds that have different common and hdfs version numbers. The reason we dont see it on the jenkins build is because we use the same version number for common and hdfs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2984) Throwing NullPointerException when we open the container page
[
https://issues.apache.org/jira/browse/MAPREDUCE-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114915#comment-13114915
]
Hudson commented on MAPREDUCE-2984:
---
Integrated in Hadoop-Common-trunk-Commit #957 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/957/])
MAPREDUCE-2984. Better error message for displaying completed containers.
Contributed by Devaraj K.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1176045
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerPage.java
Throwing NullPointerException when we open the container page
-
Key: MAPREDUCE-2984
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2984
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager
Affects Versions: 0.23.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Minor
Fix For: 0.23.0
Attachments: MAPREDUCE-2984-1.patch, MAPREDUCE-2984.patch
{code:xml}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.yarn.api.records.ContainerId.compareTo(ContainerId.java:97)
at
org.apache.hadoop.yarn.api.records.ContainerId.compareTo(ContainerId.java:23)
at
java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:819)
at
java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1640)
at
org.apache.hadoop.yarn.server.nodemanager.webapp.ContainerPage$ContainerBlock.render(ContainerPage.java:70)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:64)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:74)
at org.apache.hadoop.yarn.webapp.View.render(View.java:210)
{code}
{code:xml}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.nodemanager.webapp.ContainerPage$ContainerBlock.render(ContainerPage.java:71)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:64)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:74)
at org.apache.hadoop.yarn.webapp.View.render(View.java:210)
at
org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:47)
at
org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$TD._(Hamlet.java:843)
at
org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:54)
at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:80)
at org.apache.hadoop.yarn.webapp.Controller.render(Controller.java:210)
at
org.apache.hadoop.yarn.server.nodemanager.webapp.NMController.container(NMController.java:62)
... 30 more
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2984) Throwing NullPointerException when we open the container page
[
https://issues.apache.org/jira/browse/MAPREDUCE-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114923#comment-13114923
]
Hudson commented on MAPREDUCE-2984:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #975 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/975/])
MAPREDUCE-2984. Better error message for displaying completed containers.
Contributed by Devaraj K.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1176045
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerPage.java
Throwing NullPointerException when we open the container page
-
Key: MAPREDUCE-2984
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2984
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager
Affects Versions: 0.23.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Minor
Fix For: 0.23.0
Attachments: MAPREDUCE-2984-1.patch, MAPREDUCE-2984.patch
{code:xml}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.yarn.api.records.ContainerId.compareTo(ContainerId.java:97)
at
org.apache.hadoop.yarn.api.records.ContainerId.compareTo(ContainerId.java:23)
at
java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:819)
at
java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1640)
at
org.apache.hadoop.yarn.server.nodemanager.webapp.ContainerPage$ContainerBlock.render(ContainerPage.java:70)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:64)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:74)
at org.apache.hadoop.yarn.webapp.View.render(View.java:210)
{code}
{code:xml}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.nodemanager.webapp.ContainerPage$ContainerBlock.render(ContainerPage.java:71)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:64)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:74)
at org.apache.hadoop.yarn.webapp.View.render(View.java:210)
at
org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:47)
at
org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$TD._(Hamlet.java:843)
at
org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:54)
at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:80)
at org.apache.hadoop.yarn.webapp.Controller.render(Controller.java:210)
at
org.apache.hadoop.yarn.server.nodemanager.webapp.NMController.container(NMController.java:62)
... 30 more
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3056) Jobs are failing when those are submitted by other users
[
https://issues.apache.org/jira/browse/MAPREDUCE-3056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114932#comment-13114932
]
Aaron T. Myers commented on MAPREDUCE-3056:
---
FWIW, basically the same bug exists in 0.20.x when ${user.name} is used in the
value of the mapred.local.dir with the {{LinuxTaskController}}. I'd personally
like to completely abandon support for expanding ${user.name} in configs for
exactly this reason (HADOOP-7522) but in the absence of that, removing it from
default values is certainly a step in the right direction.
Jobs are failing when those are submitted by other users
Key: MAPREDUCE-3056
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3056
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
Fix For: 0.23.0
MR cluster is started by the user 'root'. If any other users other than
'root' submit a job, it is failing always.
Find the conatiner logs in the comments section.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3088) Clover 2.4.3 breaks build for 0.22 branch
[ https://issues.apache.org/jira/browse/MAPREDUCE-3088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik reassigned MAPREDUCE-3088: - Assignee: Konstantin Shvachko Clover 2.4.3 breaks build for 0.22 branch - Key: MAPREDUCE-3088 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3088 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.22.0 Attachments: nightly.patch Due to known bug in Clover 2.4.3 build for 0.22 branch is broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens
[
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114946#comment-13114946
]
Daryn Sharp commented on MAPREDUCE-2764:
Is there an estimate for when this jira's trunk patch will be available?
HADOOP-7510's trunk patch will depend on this jira's trunk patch. Thanks!
Fix renewal of dfs delegation tokens
Key: MAPREDUCE-2764
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Daryn Sharp
Assignee: Owen O'Malley
Fix For: 0.20.205.0
Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch,
MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch,
token-renew.patch, token-renew.patch
The JT may have issues renewing hftp tokens which disrupt long distcp jobs.
The problem is the JT's delegation token renewal code is built on brittle
assumptions. The token's service field contains only the ip:port pair.
The renewal process assumes that the scheme must be hdfs. If that fails due
to a {{VersionMismatchException}}, it tries https based on another assumption
that it must be hftp if it's not hdfs. A number of other exceptions, most
commonly {{IOExceptions}}, can be generated which fouls up the renewal since
it won't fallback to https.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3067) Container exit status not set properly to launched process's exit code on successful completion of process
[ https://issues.apache.org/jira/browse/MAPREDUCE-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3067: --- Attachment: MR-3067.1.patch Trivial fix with modified unit tests. Container exit status not set properly to launched process's exit code on successful completion of process -- Key: MAPREDUCE-3067 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3067 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.0 Attachments: MR-3067.1.patch When testing the distributed shell sample app master, the container exit status was being returned incorrectly. 11/09/21 11:32:58 INFO DistributedShell.ApplicationMaster: Got container status for containerID= container_1316629955324_0001_01_02, state=COMPLETE, exitStatus=-1000, diagnostics= -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3067) Container exit status not set properly to launched process's exit code on successful completion of process
[ https://issues.apache.org/jira/browse/MAPREDUCE-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3067: --- Status: Patch Available (was: Open) Container exit status not set properly to launched process's exit code on successful completion of process -- Key: MAPREDUCE-3067 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3067 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.0 Attachments: MR-3067.1.patch When testing the distributed shell sample app master, the container exit status was being returned incorrectly. 11/09/21 11:32:58 INFO DistributedShell.ApplicationMaster: Got container status for containerID= container_1316629955324_0001_01_02, state=COMPLETE, exitStatus=-1000, diagnostics= -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3098) Report Application status as well as ApplicationMaster status in GetApplicationReportResponse
[ https://issues.apache.org/jira/browse/MAPREDUCE-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah reassigned MAPREDUCE-3098: -- Assignee: Hitesh Shah Report Application status as well as ApplicationMaster status in GetApplicationReportResponse -- Key: MAPREDUCE-3098 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3098 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Fix For: 0.23.0 Currently, an application report received by the client from the RM/ASM for a given application returns the status of the application master. It does not return the status of the application i.e. whether that particular job succeeded or failed. The AM status would be one of FINISHED (SUCCEEDED should be renamed to FINISHED as AM state does not indicate overall success/failure), FAILED or KILLED. The final state sent by the AM to the RM in the FinishApplicationMasterRequest should be exposed to the client as ApplicationState. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3067) Container exit status not set properly to launched process's exit code on successful completion of process
[ https://issues.apache.org/jira/browse/MAPREDUCE-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3067: --- Status: Open (was: Patch Available) Minor copy-paste fix pending Container exit status not set properly to launched process's exit code on successful completion of process -- Key: MAPREDUCE-3067 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3067 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.0 Attachments: MR-3067.1.patch When testing the distributed shell sample app master, the container exit status was being returned incorrectly. 11/09/21 11:32:58 INFO DistributedShell.ApplicationMaster: Got container status for containerID= container_1316629955324_0001_01_02, state=COMPLETE, exitStatus=-1000, diagnostics= -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3098) Report Application status as well as ApplicationMaster status in GetApplicationReportResponse
Report Application status as well as ApplicationMaster status in GetApplicationReportResponse -- Key: MAPREDUCE-3098 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3098 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Fix For: 0.23.0 Currently, an application report received by the client from the RM/ASM for a given application returns the status of the application master. It does not return the status of the application i.e. whether that particular job succeeded or failed. The AM status would be one of FINISHED (SUCCEEDED should be renamed to FINISHED as AM state does not indicate overall success/failure), FAILED or KILLED. The final state sent by the AM to the RM in the FinishApplicationMasterRequest should be exposed to the client as ApplicationState. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3067) Container exit status not set properly to launched process's exit code on successful completion of process
[ https://issues.apache.org/jira/browse/MAPREDUCE-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114971#comment-13114971 ] Hadoop QA commented on MAPREDUCE-3067: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12496571/MR-3067.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/860//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/860//console This message is automatically generated. Container exit status not set properly to launched process's exit code on successful completion of process -- Key: MAPREDUCE-3067 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3067 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.0 Attachments: MR-3067.1.patch When testing the distributed shell sample app master, the container exit status was being returned incorrectly. 11/09/21 11:32:58 INFO DistributedShell.ApplicationMaster: Got container status for containerID= container_1316629955324_0001_01_02, state=COMPLETE, exitStatus=-1000, diagnostics= -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3067) Container exit status not set properly to launched process's exit code on successful completion of process
[ https://issues.apache.org/jira/browse/MAPREDUCE-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3067: --- Attachment: MR-3067.2.patch Fixed error in LCE tests. Container exit status not set properly to launched process's exit code on successful completion of process -- Key: MAPREDUCE-3067 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3067 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.0 Attachments: MR-3067.1.patch, MR-3067.2.patch When testing the distributed shell sample app master, the container exit status was being returned incorrectly. 11/09/21 11:32:58 INFO DistributedShell.ApplicationMaster: Got container status for containerID= container_1316629955324_0001_01_02, state=COMPLETE, exitStatus=-1000, diagnostics= -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3067) Container exit status not set properly to launched process's exit code on successful completion of process
[ https://issues.apache.org/jira/browse/MAPREDUCE-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3067: --- Status: Patch Available (was: Open) Container exit status not set properly to launched process's exit code on successful completion of process -- Key: MAPREDUCE-3067 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3067 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.0 Attachments: MR-3067.1.patch, MR-3067.2.patch When testing the distributed shell sample app master, the container exit status was being returned incorrectly. 11/09/21 11:32:58 INFO DistributedShell.ApplicationMaster: Got container status for containerID= container_1316629955324_0001_01_02, state=COMPLETE, exitStatus=-1000, diagnostics= -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3054) Unable to kill submitted jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar updated MAPREDUCE-3054:
-
Attachment: MAPREDUCE-3054.patch
Ok, here's a compromise. We cant change the job api's to add one to force kill
the job for 0.23. So in my patch, the kill command will send a kill to RM if
the job is not running, if RUNNING, it will send a kill to AM and wait for 10
seconds for it to get killed. If it doesnt get killed, it will send a force
kill to RM.
Also, made a couple of other changes:
- added a little more testing
- changed the finishApplication calls to forceKillApplication in RM
- also changed the response/request names to be clear on what they are doing.
Seems reasonable?
Unable to kill submitted jobs
-
Key: MAPREDUCE-3054
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3054
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-3054.patch, MAPREDUCE-3054.patch,
MAPREDUCE-3054.patch, MAPREDUCE-3054.patch, MAPREDUCE-3054.patch
Found by Philip Su
The mapred job -kill command
appears to succeed, but listing the jobs again shows that the job supposedly
killed is still there.
{code}
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
mapred job -kill job_1316203984216_0002
Killed job job_1316203984216_0002
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3054) Unable to kill submitted jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar updated MAPREDUCE-3054:
-
Status: Patch Available (was: Open)
Unable to kill submitted jobs
-
Key: MAPREDUCE-3054
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3054
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-3054.patch, MAPREDUCE-3054.patch,
MAPREDUCE-3054.patch, MAPREDUCE-3054.patch, MAPREDUCE-3054.patch
Found by Philip Su
The mapred job -kill command
appears to succeed, but listing the jobs again shows that the job supposedly
killed is still there.
{code}
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
mapred job -kill job_1316203984216_0002
Killed job job_1316203984216_0002
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3054) Unable to kill submitted jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar updated MAPREDUCE-3054:
-
Status: Open (was: Patch Available)
Fixing javadoc. For the test failure, it looks like test-patch is running mvn
tests without applying the patch. Will open a jira for that.
Unable to kill submitted jobs
-
Key: MAPREDUCE-3054
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3054
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-3054.patch, MAPREDUCE-3054.patch,
MAPREDUCE-3054.patch, MAPREDUCE-3054.patch, MAPREDUCE-3054.patch
Found by Philip Su
The mapred job -kill command
appears to succeed, but listing the jobs again shows that the job supposedly
killed is still there.
{code}
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
mapred job -kill job_1316203984216_0002
Killed job job_1316203984216_0002
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3067) Container exit status not set properly to launched process's exit code on successful completion of process
[ https://issues.apache.org/jira/browse/MAPREDUCE-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115009#comment-13115009 ] Hadoop QA commented on MAPREDUCE-3067: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12496578/MR-3067.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/861//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/861//console This message is automatically generated. Container exit status not set properly to launched process's exit code on successful completion of process -- Key: MAPREDUCE-3067 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3067 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.0 Attachments: MR-3067.1.patch, MR-3067.2.patch When testing the distributed shell sample app master, the container exit status was being returned incorrectly. 11/09/21 11:32:58 INFO DistributedShell.ApplicationMaster: Got container status for containerID= container_1316629955324_0001_01_02, state=COMPLETE, exitStatus=-1000, diagnostics= -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3054) Unable to kill submitted jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115013#comment-13115013
]
Hadoop QA commented on MAPREDUCE-3054:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12496581/MAPREDUCE-3054.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 15 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 2 warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.TestRPC
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/862//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/862//console
This message is automatically generated.
Unable to kill submitted jobs
-
Key: MAPREDUCE-3054
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3054
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-3054.patch, MAPREDUCE-3054.patch,
MAPREDUCE-3054.patch, MAPREDUCE-3054.patch, MAPREDUCE-3054.patch
Found by Philip Su
The mapred job -kill command
appears to succeed, but listing the jobs again shows that the job supposedly
killed is still there.
{code}
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
mapred job -kill job_1316203984216_0002
Killed job job_1316203984216_0002
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3054) Unable to kill submitted jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar updated MAPREDUCE-3054:
-
Attachment: MAPREDUCE-3054.patch
Fixed javadoc warnings.
Unable to kill submitted jobs
-
Key: MAPREDUCE-3054
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3054
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-3054.patch, MAPREDUCE-3054.patch,
MAPREDUCE-3054.patch, MAPREDUCE-3054.patch, MAPREDUCE-3054.patch,
MAPREDUCE-3054.patch
Found by Philip Su
The mapred job -kill command
appears to succeed, but listing the jobs again shows that the job supposedly
killed is still there.
{code}
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
mapred job -kill job_1316203984216_0002
Killed job job_1316203984216_0002
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3054) Unable to kill submitted jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar updated MAPREDUCE-3054:
-
Status: Patch Available (was: Open)
Unable to kill submitted jobs
-
Key: MAPREDUCE-3054
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3054
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-3054.patch, MAPREDUCE-3054.patch,
MAPREDUCE-3054.patch, MAPREDUCE-3054.patch, MAPREDUCE-3054.patch,
MAPREDUCE-3054.patch
Found by Philip Su
The mapred job -kill command
appears to succeed, but listing the jobs again shows that the job supposedly
killed is still there.
{code}
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
mapred job -kill job_1316203984216_0002
Killed job job_1316203984216_0002
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3054) Unable to kill submitted jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115035#comment-13115035
]
Arun C Murthy commented on MAPREDUCE-3054:
--
+1
Unable to kill submitted jobs
-
Key: MAPREDUCE-3054
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3054
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-3054.patch, MAPREDUCE-3054.patch,
MAPREDUCE-3054.patch, MAPREDUCE-3054.patch, MAPREDUCE-3054.patch,
MAPREDUCE-3054.patch
Found by Philip Su
The mapred job -kill command
appears to succeed, but listing the jobs again shows that the job supposedly
killed is still there.
{code}
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
mapred job -kill job_1316203984216_0002
Killed job job_1316203984216_0002
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2980) Fetch failures and other related issues in Jetty 6.1.26
[ https://issues.apache.org/jira/browse/MAPREDUCE-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115034#comment-13115034 ] Rob Weltman commented on MAPREDUCE-2980: Arun - we have customers asking for this. Is it OK to apply the NIO changes to get around the connection dropping until jetty 1.6.27 is out? Fetch failures and other related issues in Jetty 6.1.26 --- Key: MAPREDUCE-2980 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2980 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.205.0, 0.23.0 Reporter: Todd Lipcon Priority: Critical Since upgrading Jetty from 6.1.14 to 6.1.26 we've had a ton of HTTP-related issues, including: - Much higher incidence of fetch failures - A few strange file-descriptor related bugs (eg MAPREDUCE-2389) - A few unexplained issues where long fscks on the NameNode drop out halfway through with a ClosedChannelException Stress tests with 1Map x 1Reduce sleep jobs reliably reproduce fetch failures at a rate of about 1 per million on a 25 node test cluster. These problems are all new since the upgrade from 6.1.14. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3081) Change the name format for hadoop core and vaidya jar to be hadoop-{core/vaidya}-{version}.jar in vaidya.sh
[
https://issues.apache.org/jira/browse/MAPREDUCE-3081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115037#comment-13115037
]
Giridharan Kesavan commented on MAPREDUCE-3081:
---
+1 looks good
Change the name format for hadoop core and vaidya jar to be
hadoop-{core/vaidya}-{version}.jar in vaidya.sh
---
Key: MAPREDUCE-3081
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3081
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/vaidya
Affects Versions: 0.20.1, 0.20.205.0, 0.20.206.0
Reporter: vitthal (Suhas) Gogate
Attachments: HADOOP-3081-trunk.patch, MAPREDUCE-3081-2.patch
Vaidya script is broken due to change in the naming convention for hadoop
core jar and vaidya jar.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3054) Unable to kill submitted jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115046#comment-13115046
]
Hadoop QA commented on MAPREDUCE-3054:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12496588/MAPREDUCE-3054.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 15 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.TestRPC
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/863//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/863//console
This message is automatically generated.
Unable to kill submitted jobs
-
Key: MAPREDUCE-3054
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3054
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-3054.patch, MAPREDUCE-3054.patch,
MAPREDUCE-3054.patch, MAPREDUCE-3054.patch, MAPREDUCE-3054.patch,
MAPREDUCE-3054.patch
Found by Philip Su
The mapred job -kill command
appears to succeed, but listing the jobs again shows that the job supposedly
killed is still there.
{code}
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
mapred job -kill job_1316203984216_0002
Killed job job_1316203984216_0002
mapred job -list
Total jobs:2
JobId State StartTime UserNameQueue Priority
SchedulingInfo
job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3096) Add a good way to control the number of map/reduce tasks per node
[ https://issues.apache.org/jira/browse/MAPREDUCE-3096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115045#comment-13115045 ] Aaron T. Myers commented on MAPREDUCE-3096: --- Hi Arsen, to be completely clear, do you want to be able to limit the maximum number of concurrent map/reduce tasks *from a single job* that run on a given node? Or the maximum number of concurrent map/reduce tasks that run on a given node *across all jobs*? Add a good way to control the number of map/reduce tasks per node - Key: MAPREDUCE-3096 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3096 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Arsen Zahray Fix For: 0.20.204.0 Currently, controlling the number of map/reduce tasks is a hell. I've tried for it many times, and it doesn't work right. Also, I am not the only one person, who seems to have this problem. There must be a better way to do it. Here's my proposal: add following functions to Job: setNumberOfMappersPerNode(int); setNumberOfReducersPerNode(int); setMaxMemoryPerMapper(int); setMaxMemoryPerReducer(int); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3081) Change the name format for hadoop core and vaidya jar to be hadoop-{core/vaidya}-{version}.jar in vaidya.sh
[
https://issues.apache.org/jira/browse/MAPREDUCE-3081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115050#comment-13115050
]
Giridharan Kesavan commented on MAPREDUCE-3081:
---
Thanks Suhas, I just committed this patch to 20-security and 205 branch.
Change the name format for hadoop core and vaidya jar to be
hadoop-{core/vaidya}-{version}.jar in vaidya.sh
---
Key: MAPREDUCE-3081
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3081
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/vaidya
Affects Versions: 0.20.1, 0.20.205.0, 0.20.206.0
Reporter: vitthal (Suhas) Gogate
Attachments: HADOOP-3081-trunk.patch, MAPREDUCE-3081-2.patch
Vaidya script is broken due to change in the naming convention for hadoop
core jar and vaidya jar.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3041) Enhance YARN Client-RM protocol to provide access to information such as cluster's Min/Max Resource capabilities similar to that of AM-RM protocol
[ https://issues.apache.org/jira/browse/MAPREDUCE-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3041: --- Attachment: MR-3041.1.patch Updated GetNewApplicationIdResponse to now also return the min/max resource capabilities of the cluster. Modified MockRM to add a getNewAppId() api to be able to test the modifications. Enhance YARN Client-RM protocol to provide access to information such as cluster's Min/Max Resource capabilities similar to that of AM-RM protocol -- Key: MAPREDUCE-3041 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3041 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.0 Attachments: MR-3041.1.patch To request a container to launch an application master, the client needs to know the min/max resource capabilities so as to be able to make a proper resource request when submitting a new application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3041) Enhance YARN Client-RM protocol to provide access to information such as cluster's Min/Max Resource capabilities similar to that of AM-RM protocol
[ https://issues.apache.org/jira/browse/MAPREDUCE-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3041: --- Status: Patch Available (was: Open) Enhance YARN Client-RM protocol to provide access to information such as cluster's Min/Max Resource capabilities similar to that of AM-RM protocol -- Key: MAPREDUCE-3041 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3041 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.0 Attachments: MR-3041.1.patch To request a container to launch an application master, the client needs to know the min/max resource capabilities so as to be able to make a proper resource request when submitting a new application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2980) Fetch failures and other related issues in Jetty 6.1.26
[ https://issues.apache.org/jira/browse/MAPREDUCE-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115121#comment-13115121 ] Rajiv Chittajallu commented on MAPREDUCE-2980: -- This is an long standing issue that is affecting production installation. Without significant operations monitoring, I wouldn't call 0.20.2xx a stable release for production. Fetch failures and other related issues in Jetty 6.1.26 --- Key: MAPREDUCE-2980 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2980 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.205.0, 0.23.0 Reporter: Todd Lipcon Priority: Critical Since upgrading Jetty from 6.1.14 to 6.1.26 we've had a ton of HTTP-related issues, including: - Much higher incidence of fetch failures - A few strange file-descriptor related bugs (eg MAPREDUCE-2389) - A few unexplained issues where long fscks on the NameNode drop out halfway through with a ClosedChannelException Stress tests with 1Map x 1Reduce sleep jobs reliably reproduce fetch failures at a rate of about 1 per million on a 25 node test cluster. These problems are all new since the upgrade from 6.1.14. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3041) Enhance YARN Client-RM protocol to provide access to information such as cluster's Min/Max Resource capabilities similar to that of AM-RM protocol
[ https://issues.apache.org/jira/browse/MAPREDUCE-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115126#comment-13115126 ] Hadoop QA commented on MAPREDUCE-3041: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12496605/MR-3041.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/864//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/864//console This message is automatically generated. Enhance YARN Client-RM protocol to provide access to information such as cluster's Min/Max Resource capabilities similar to that of AM-RM protocol -- Key: MAPREDUCE-3041 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3041 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.0 Attachments: MR-3041.1.patch To request a container to launch an application master, the client needs to know the min/max resource capabilities so as to be able to make a proper resource request when submitting a new application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3092) Remove JOB_ID_COMPARATOR usage in JobHistory.java
[ https://issues.apache.org/jira/browse/MAPREDUCE-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3092: - Attachment: MAPREDUCE-3092.patch Remove JOB_ID_COMPARATOR usage in JobHistory.java - Key: MAPREDUCE-3092 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3092 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0 Reporter: Devaraj K Assignee: Devaraj K Priority: Minor Attachments: MAPREDUCE-3092.patch As part of the defect MAPREDUCE-2965, JobId.compareTo() has been implemented. Usage of JOB_ID_COMPARATOR in JobHistory.java can be removed because comparison is handling by JobId itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
