[jira] [Updated] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5536: - Status: Patch Available (was: Open) > mapreduce.jobhistory.webapp.https.address property is not respected > --- > > Key: MAPREDUCE-5536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Yesha Vora >Assignee: Omkar Vinit Joshi >Priority: Blocker > Attachments: MAPREDUCE-5536.20131027.1.patch, > MAPREDUCE-5536.20131030.1.patch, MAPREDUCE-5536.20131030.1.patch, > MAPREDUCE-5536.20131030.3.patch, YARN-1240.20131025.1.patch > > > The jobhistory server starts on port defined by > mapreduce.jobhistory.webapp.address property instead > mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5536: - Attachment: MAPREDUCE-5536.20131030.3.patch > mapreduce.jobhistory.webapp.https.address property is not respected > --- > > Key: MAPREDUCE-5536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Yesha Vora >Assignee: Omkar Vinit Joshi >Priority: Blocker > Attachments: MAPREDUCE-5536.20131027.1.patch, > MAPREDUCE-5536.20131030.1.patch, MAPREDUCE-5536.20131030.1.patch, > MAPREDUCE-5536.20131030.3.patch, YARN-1240.20131025.1.patch > > > The jobhistory server starts on port defined by > mapreduce.jobhistory.webapp.address property instead > mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5536: - Status: Open (was: Patch Available) > mapreduce.jobhistory.webapp.https.address property is not respected > --- > > Key: MAPREDUCE-5536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Yesha Vora >Assignee: Omkar Vinit Joshi >Priority: Blocker > Attachments: MAPREDUCE-5536.20131027.1.patch, > MAPREDUCE-5536.20131030.1.patch, MAPREDUCE-5536.20131030.1.patch, > MAPREDUCE-5536.20131030.3.patch, YARN-1240.20131025.1.patch > > > The jobhistory server starts on port defined by > mapreduce.jobhistory.webapp.address property instead > mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5536: - Status: Open (was: Patch Available) > mapreduce.jobhistory.webapp.https.address property is not respected > --- > > Key: MAPREDUCE-5536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Yesha Vora >Assignee: Omkar Vinit Joshi >Priority: Blocker > Attachments: MAPREDUCE-5536.20131027.1.patch, > MAPREDUCE-5536.20131030.1.patch, MAPREDUCE-5536.20131030.1.patch, > YARN-1240.20131025.1.patch > > > The jobhistory server starts on port defined by > mapreduce.jobhistory.webapp.address property instead > mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782584#comment-13782584 ] Hadoop QA commented on MAPREDUCE-5536: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12606013/MAPREDUCE-5536.20131030.1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.v2.hs.webapp.TestMapReduceTrackingUriPlugin org.apache.hadoop.mapred.TestJobCleanup The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.v2.TestUberAM {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4068//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4068//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs-plugins.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4068//console This message is automatically generated. > mapreduce.jobhistory.webapp.https.address property is not respected > --- > > Key: MAPREDUCE-5536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Yesha Vora >Assignee: Omkar Vinit Joshi >Priority: Blocker > Attachments: MAPREDUCE-5536.20131027.1.patch, > MAPREDUCE-5536.20131030.1.patch, MAPREDUCE-5536.20131030.1.patch, > YARN-1240.20131025.1.patch > > > The jobhistory server starts on port defined by > mapreduce.jobhistory.webapp.address property instead > mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5533) Speculative execution does not function for reduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated MAPREDUCE-5533: - Status: Patch Available (was: Open) > Speculative execution does not function for reduce > -- > > Key: MAPREDUCE-5533 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5533 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Reporter: Tassapol Athiapinya >Assignee: Xuan Gong > Attachments: MAPREDUCE-5533.1.patch, MAPREDUCE-5533.2.patch, > MAPREDUCE-5533.3.patch, MAPREDUCE-5533.4.patch, MAPREDUCE-5533.5.patch > > > We have sort job where reduce attempt does not send heartbeat in timely > manner to application master. The AM should kick off another attempt to let > job succeeds. What we find is the job fails and there is no speculation > happening. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5533) Speculative execution does not function for reduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated MAPREDUCE-5533: - Attachment: MAPREDUCE-5533.5.patch Add testcases to test the logic > Speculative execution does not function for reduce > -- > > Key: MAPREDUCE-5533 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5533 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Reporter: Tassapol Athiapinya >Assignee: Xuan Gong > Attachments: MAPREDUCE-5533.1.patch, MAPREDUCE-5533.2.patch, > MAPREDUCE-5533.3.patch, MAPREDUCE-5533.4.patch, MAPREDUCE-5533.5.patch > > > We have sort job where reduce attempt does not send heartbeat in timely > manner to application master. The AM should kick off another attempt to let > job succeeds. What we find is the job fails and there is no speculation > happening. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5530: - Status: Open (was: Patch Available) > Binary and source incompatibility in mapred.lib.CombineFileInputFormat > between branch-1 and branch-2 > > > Key: MAPREDUCE-5530 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv1, mrv2 >Affects Versions: 2.1.1-beta >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > Attachments: MAPREDUCE-5530.patch, MAPREDUCE-5530.patch > > > {{mapred.lib.CombineFileInputFormat}} in branch-1 has this method: > {code:java} > protected boolean isSplitable(FileSystem fs, Path file) > {code} > In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of > {{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the > similar method: > {code:java} > protected boolean isSplitable(JobContext context, Path file) > {code} > This means that any code that subclasses > {{mapred.lib.CombineFileInputFormat}} and does not provide its own > implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}} > will not be binary or source compatible if it tries to call {{isSplitable}} > with a {{FileSystem}} argument anywhere (that is, if compiled against > branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2; > also, it won't even compile against branch-2). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782567#comment-13782567 ] Arun C Murthy commented on MAPREDUCE-5530: -- [~rkanter] I agree the patch isn't perfect, but that's the only option we have now... thought I wonder if we should just document this and not fix it... thoughts? IAC, couple of comments: # Please mark org.apache.hadoop.mapred.lib.CombineFileInputFormat.isSplitable(JobContext, Path) as @Private # Also add some documentation to the method with a pointer to this jira explaining why end-users should never override org.apache.hadoop.mapred.lib.CombineFileInputFormat.isSplitable(JobContext, Path). Thanks. > Binary and source incompatibility in mapred.lib.CombineFileInputFormat > between branch-1 and branch-2 > > > Key: MAPREDUCE-5530 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv1, mrv2 >Affects Versions: 2.1.1-beta >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > Attachments: MAPREDUCE-5530.patch, MAPREDUCE-5530.patch > > > {{mapred.lib.CombineFileInputFormat}} in branch-1 has this method: > {code:java} > protected boolean isSplitable(FileSystem fs, Path file) > {code} > In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of > {{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the > similar method: > {code:java} > protected boolean isSplitable(JobContext context, Path file) > {code} > This means that any code that subclasses > {{mapred.lib.CombineFileInputFormat}} and does not provide its own > implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}} > will not be binary or source compatible if it tries to call {{isSplitable}} > with a {{FileSystem}} argument anywhere (that is, if compiled against > branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2; > also, it won't even compile against branch-2). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5551) Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5551: - Resolution: Fixed Fix Version/s: 2.1.2-beta Status: Resolved (was: Patch Available) I just committed this. Thanks [~zjshen]! > Binary Incompatibility of > O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes > > > Key: MAPREDUCE-5551 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5551 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Fix For: 2.1.2-beta > > Attachments: MAPREDUCE-5551.1.patch > > > The non-default constructor is moved to the super class, but it cannot be > inherited. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5551) Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782545#comment-13782545 ] Hudson commented on MAPREDUCE-5551: --- SUCCESS: Integrated in Hadoop-trunk-Commit #4503 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4503/]) MAPREDUCE-5551. Fix compat with hadoop-1 in SequenceFileAsBinaryOutputFormat.WritableValueBytes by re-introducing missing constructors. Contributed by Zhijie Shen. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527848) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SequenceFileAsBinaryOutputFormat.java > Binary Incompatibility of > O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes > > > Key: MAPREDUCE-5551 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5551 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Attachments: MAPREDUCE-5551.1.patch > > > The non-default constructor is moved to the super class, but it cannot be > inherited. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5551) Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782510#comment-13782510 ] Hadoop QA commented on MAPREDUCE-5551: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12606016/MAPREDUCE-5551.1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4067//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4067//console This message is automatically generated. > Binary Incompatibility of > O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes > > > Key: MAPREDUCE-5551 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5551 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Attachments: MAPREDUCE-5551.1.patch > > > The non-default constructor is moved to the super class, but it cannot be > inherited. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5551) Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5551: --- Attachment: MAPREDUCE-5551.1.patch Uploaded a patch to add the missing constructors. Note that zero-param constructor is added as well, because the default constructor is no longer valid once the one-param constructor is added. > Binary Incompatibility of > O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes > > > Key: MAPREDUCE-5551 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5551 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Attachments: MAPREDUCE-5551.1.patch > > > The non-default constructor is moved to the super class, but it cannot be > inherited. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5551) Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5551: --- Status: Patch Available (was: Open) > Binary Incompatibility of > O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes > > > Key: MAPREDUCE-5551 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5551 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Attachments: MAPREDUCE-5551.1.patch > > > The non-default constructor is moved to the super class, but it cannot be > inherited. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5551) Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5551: --- Issue Type: Sub-task (was: Bug) Parent: MAPREDUCE-5108 > Binary Incompatibility of > O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes > > > Key: MAPREDUCE-5551 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5551 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Attachments: MAPREDUCE-5551.1.patch > > > The non-default constructor is moved to the super class, but it cannot be > inherited. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (MAPREDUCE-5524) java.io.IOException: Task process exit with nonzero status of 255. how to fix it?
[ https://issues.apache.org/jira/browse/MAPREDUCE-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved MAPREDUCE-5524. --- Resolution: Invalid Please email u...@hadoop.apache.org with this question. Apache JIRA is for reporting bugs and tracking features/improvements. It's not intended for user-level help. > java.io.IOException: Task process exit with nonzero status of 255. how to > fix it? > --- > > Key: MAPREDUCE-5524 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5524 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: hawkswood > > > Task ..FAILED > > java.lang.Throwable: Child Error > > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) > > Caused by: java.io.IOException: Task process exit with nonzero status of > > 255. > > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5551) Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes
Zhijie Shen created MAPREDUCE-5551: -- Summary: Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes Key: MAPREDUCE-5551 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5551 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker The non-default constructor is moved to the super class, but it cannot be inherited. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5550) Job Status message not shown in UI with Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782472#comment-13782472 ] Omkar Vinit Joshi commented on MAPREDUCE-5550: -- you should check the AM UI? > Job Status message not shown in UI with Hadoop 2.0 > -- > > Key: MAPREDUCE-5550 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5550 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.5-alpha >Reporter: Vrushali C >Assignee: Vrushali C > > Hadoop 1.0 JobTracker UI displays Job Status message when list of mapper or > reduce tasks are listed. This give an idea of how that task is making > progress. > Hadoop 2.0 ResourceManager UI does not have this. It would be good to have > this on RM UI. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5536: - Status: Patch Available (was: Open) > mapreduce.jobhistory.webapp.https.address property is not respected > --- > > Key: MAPREDUCE-5536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Yesha Vora >Assignee: Omkar Vinit Joshi >Priority: Blocker > Attachments: MAPREDUCE-5536.20131027.1.patch, > MAPREDUCE-5536.20131030.1.patch, MAPREDUCE-5536.20131030.1.patch, > YARN-1240.20131025.1.patch > > > The jobhistory server starts on port defined by > mapreduce.jobhistory.webapp.address property instead > mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5536: - Attachment: MAPREDUCE-5536.20131030.1.patch > mapreduce.jobhistory.webapp.https.address property is not respected > --- > > Key: MAPREDUCE-5536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Yesha Vora >Assignee: Omkar Vinit Joshi >Priority: Blocker > Attachments: MAPREDUCE-5536.20131027.1.patch, > MAPREDUCE-5536.20131030.1.patch, MAPREDUCE-5536.20131030.1.patch, > YARN-1240.20131025.1.patch > > > The jobhistory server starts on port defined by > mapreduce.jobhistory.webapp.address property instead > mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782471#comment-13782471 ] Omkar Vinit Joshi commented on MAPREDUCE-5536: -- Thanks vinod.. bq. WebAppUtil.getDefaultJHSWebappPort() and getDefaultJHSWebappURLWithoutScheme() can be private fixed.. bq. getHistoryUrl() -> getApplicationHistoryURLWithoutScheme(). fixed. bq. getJHSWebappURLWithScheme() : No need to have a isSSLEnabledInJHS check. fixed. bq. getJHSSchemePrefix() -> getJHSWebAppSchemePrefix()() fixed. bq. Add a getApplicationHistoryURLWithSchem() and use in MR AM's RMCommunicator? fixed. bq. MR's WebAppUtil -> Mark as Private. - WebAppUtil -> MRWebAppUtil. fixed.. bq. JHAdminConfig.getResolvedMRHistoryWebAppURLWithoutScheme() still takes in a flag - it doesn't need to. And may be move it to MiniMRYarnCluster if that is the only caller. moved it to MiniMRYarnCluster bq. I think instead of WebAppUtil.setSSLEnabledIn*() methods, we should have WebAppUtil.initialize(Configuration conf). That way, every component doesn't need to remember what config to read. fixed. > mapreduce.jobhistory.webapp.https.address property is not respected > --- > > Key: MAPREDUCE-5536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Yesha Vora >Assignee: Omkar Vinit Joshi >Priority: Blocker > Attachments: MAPREDUCE-5536.20131027.1.patch, > MAPREDUCE-5536.20131030.1.patch, MAPREDUCE-5536.20131030.1.patch, > YARN-1240.20131025.1.patch > > > The jobhistory server starts on port defined by > mapreduce.jobhistory.webapp.address property instead > mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-4421) Remove dependency on deployed MR jars
[ https://issues.apache.org/jira/browse/MAPREDUCE-4421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782438#comment-13782438 ] Hitesh Shah commented on MAPREDUCE-4421: Sorry for the delay in the review. Regarding addMRFrameworkToDistributedCache() - one minor question: the code allows for a non-qualified URI. Should we enforce provision of a fully-qualified path always? Minor nit: I believe there should be nothing in the implementation that requires HDFS as the storage for the MR tarball? Documentation needs to change as a result unless you believe there are reasons for not mentioning other filesystems ( except maybe from a testing point of view )? Patch looks good otherwise. Thanks for adding the detailed docs. > Remove dependency on deployed MR jars > - > > Key: MAPREDUCE-4421 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4421 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.0.0-alpha >Reporter: Arun C Murthy >Assignee: Jason Lowe > Attachments: MAPREDUCE-4421-2.patch, MAPREDUCE-4421.patch, > MAPREDUCE-4421.patch > > > Currently MR AM depends on MR jars being deployed on all nodes via implicit > dependency on YARN_APPLICATION_CLASSPATH. > We should stop adding mapreduce jars to YARN_APPLICATION_CLASSPATH and, > probably, just rely on adding a shaded MR jar along with job.jar to the > dist-cache. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782430#comment-13782430 ] Robert Kanter commented on MAPREDUCE-5530: -- I don't think this really adds any instability (if there's a problem getting the FileSystem, we're bound to run into that somewhere else as well anyway), and without this the user might not even be able to compile. > Binary and source incompatibility in mapred.lib.CombineFileInputFormat > between branch-1 and branch-2 > > > Key: MAPREDUCE-5530 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv1, mrv2 >Affects Versions: 2.1.1-beta >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > Attachments: MAPREDUCE-5530.patch, MAPREDUCE-5530.patch > > > {{mapred.lib.CombineFileInputFormat}} in branch-1 has this method: > {code:java} > protected boolean isSplitable(FileSystem fs, Path file) > {code} > In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of > {{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the > similar method: > {code:java} > protected boolean isSplitable(JobContext context, Path file) > {code} > This means that any code that subclasses > {{mapred.lib.CombineFileInputFormat}} and does not provide its own > implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}} > will not be binary or source compatible if it tries to call {{isSplitable}} > with a {{FileSystem}} argument anywhere (that is, if compiled against > branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2; > also, it won't even compile against branch-2). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782401#comment-13782401 ] Hadoop QA commented on MAPREDUCE-5536: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12605975/MAPREDUCE-5536.20131030.1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.v2.hs.webapp.TestMapReduceTrackingUriPlugin org.apache.hadoop.mapred.TestJobCleanup The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.v2.TestUberAM {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4064//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4064//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs-plugins.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4064//console This message is automatically generated. > mapreduce.jobhistory.webapp.https.address property is not respected > --- > > Key: MAPREDUCE-5536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Yesha Vora >Assignee: Omkar Vinit Joshi >Priority: Blocker > Attachments: MAPREDUCE-5536.20131027.1.patch, > MAPREDUCE-5536.20131030.1.patch, YARN-1240.20131025.1.patch > > > The jobhistory server starts on port defined by > mapreduce.jobhistory.webapp.address property instead > mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782383#comment-13782383 ] Hadoop QA commented on MAPREDUCE-3202: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12605989/MAPREDUCE-3202.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 15 new or modified test files. {color:red}-1 javac{color}. The applied patch generated 1536 javac compiler warnings (more than the trunk's current 1534 warnings). {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 12 new Findbugs (version 1.3.9) warnings. {color:red}-1 release audit{color}. The applied patch generated 1 release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-tools-dist hadoop-tools/hadoop-vaidya: org.apache.hadoop.mapreduce.v2.hs.webapp.dao.TestJobInfo {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-vaidya.html Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//console This message is automatically generated. > Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 > > > Key: MAPREDUCE-3202 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobhistoryserver >Affects Versions: 2.0.0-alpha >Reporter: vitthal (Suhas) Gogate >Assignee: Evan (Yiming) Meng > Fix For: trunk > > Attachments: MAPREDUCE-3202.patch > > > Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various > execution inefficiencies and the associated remedies that user can easily > understand and fix. This Jira patch integrates it with Job History UI under > Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5536: --- Status: Open (was: Patch Available) More comments on the patch. This has to be one of the hardest patches I reviewed in recent times. - WebAppUtil.getDefaultJHSWebappPort() and getDefaultJHSWebappURLWithoutScheme() can be private. - getHistoryUrl() -> getApplicationHistoryURLWithoutScheme(). - getJHSWebappURLWithScheme() : No need to have a isSSLEnabledInJHS check. - getJHSSchemePrefix() -> getJHSWebAppSchemePrefix()() - Add a getApplicationHistoryURLWithSchem() and use in MR AM's RMCommunicator? - MR's WebAppUtil -> Mark as Private. - WebAppUtil -> MRWebAppUtil. - JHAdminConfig.getResolvedMRHistoryWebAppURLWithoutScheme() still takes in a flag - it doesn't need to. And may be move it to MiniMRYarnCluster if that is the only caller. I think instead of WebAppUtil.setSSLEnabledIn*() methods, we should have WebAppUtil.initialize(Configuration conf). That way, every component doesn't need to remember what config to read. > mapreduce.jobhistory.webapp.https.address property is not respected > --- > > Key: MAPREDUCE-5536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Yesha Vora >Assignee: Omkar Vinit Joshi >Priority: Blocker > Attachments: MAPREDUCE-5536.20131027.1.patch, > MAPREDUCE-5536.20131030.1.patch, YARN-1240.20131025.1.patch > > > The jobhistory server starts on port defined by > mapreduce.jobhistory.webapp.address property instead > mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5088) MR Client gets an renewer token exception while Oozie is submitting a job
[ https://issues.apache.org/jira/browse/MAPREDUCE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782364#comment-13782364 ] Robert Kanter commented on MAPREDUCE-5088: -- We're seeing an issue where Oozie can't submit to a secure cluster when the JHS is down because it can't get a delegation token from the Job History Server. However, if I submit a job without Oozie (i.e. hadoop jar ) it works fine. When investigating, it looks like this JIRA may be the cause, or is at least related. Any ideas on why this is happening? {noformat} 2013-09-30 13:14:48,544 WARN ActionStartXCommand:542 - USER[testuser] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[055-130929201011702-oozie-oozi-W] ACTION[055-130929201011702-oozie-oozi-W@mr-node] Er ror starting action [mr-node]. ErrorType [TRANSIENT], ErrorCode [ JA006], Message [ JA006: Connection refused] org.apache.oozie.action.ActionExecutorException: JA006: Connection refused at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:412) at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:392) at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:794) at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:948) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:215) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:60) at org.apache.oozie.command.XCommand.call(XCommand.java:280) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:547) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:642) at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399) at org.apache.hadoop.ipc.Client.call(Client.java:1318) at org.apache.hadoop.ipc.Client.call(Client.java:1300) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy31.getDelegationToken(Unknown Source) at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getDelegationToken(MRClientProtocolPBClientImpl.java:211) at org.apache.hadoop.mapred.YARNRunner.getDelegationTokenFromHS(YARNRunner.java:208) at org.apache.hadoop.mapred.YARNRunner.addHistoryToken(YARNRunner.java:195) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:280) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:430) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:552) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:552) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:543) at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:779) ... 10 more {noformat} > MR Client gets an renewer token exception while Oozie is submitting a job > --
[jira] [Commented] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782362#comment-13782362 ] Zhijie Shen commented on MAPREDUCE-5530: [~rkanter], IMHO, it's not a good idea to make isSplitable(JobContext context, Path file) call isSplitable(FileSystem fs, Path file), because, it introduces the step of FileSystem.get(context.getConfiguration()), which may lead to unexpected IOException. Personally, I incline not to increase the instability of Hadoop2 code pass. Thoughts? > Binary and source incompatibility in mapred.lib.CombineFileInputFormat > between branch-1 and branch-2 > > > Key: MAPREDUCE-5530 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv1, mrv2 >Affects Versions: 2.1.1-beta >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > Attachments: MAPREDUCE-5530.patch, MAPREDUCE-5530.patch > > > {{mapred.lib.CombineFileInputFormat}} in branch-1 has this method: > {code:java} > protected boolean isSplitable(FileSystem fs, Path file) > {code} > In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of > {{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the > similar method: > {code:java} > protected boolean isSplitable(JobContext context, Path file) > {code} > This means that any code that subclasses > {{mapred.lib.CombineFileInputFormat}} and does not provide its own > implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}} > will not be binary or source compatible if it tries to call {{isSplitable}} > with a {{FileSystem}} argument anywhere (that is, if compiled against > branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2; > also, it won't even compile against branch-2). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vitthal (Suhas) Gogate updated MAPREDUCE-3202: -- Attachment: (was: vaidya-2.0.6-alpha.patch) > Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 > > > Key: MAPREDUCE-3202 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobhistoryserver >Affects Versions: 2.0.0-alpha >Reporter: vitthal (Suhas) Gogate >Assignee: Evan (Yiming) Meng > Fix For: trunk > > Attachments: MAPREDUCE-3202.patch > > > Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various > execution inefficiencies and the associated remedies that user can easily > understand and fix. This Jira patch integrates it with Job History UI under > Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vitthal (Suhas) Gogate updated MAPREDUCE-3202: -- Fix Version/s: (was: 2.0.6-alpha) trunk Status: Patch Available (was: Open) > Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 > > > Key: MAPREDUCE-3202 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobhistoryserver >Affects Versions: 2.0.0-alpha >Reporter: vitthal (Suhas) Gogate >Assignee: Evan (Yiming) Meng > Fix For: trunk > > Attachments: MAPREDUCE-3202.patch, vaidya-2.0.6-alpha.patch > > > Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various > execution inefficiencies and the associated remedies that user can easily > understand and fix. This Jira patch integrates it with Job History UI under > Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vitthal (Suhas) Gogate updated MAPREDUCE-3202: -- Attachment: MAPREDUCE-3202.patch Submitting patch for trunk. > Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 > > > Key: MAPREDUCE-3202 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobhistoryserver >Affects Versions: 2.0.0-alpha >Reporter: vitthal (Suhas) Gogate >Assignee: Evan (Yiming) Meng > Fix For: 2.0.6-alpha > > Attachments: MAPREDUCE-3202.patch, vaidya-2.0.6-alpha.patch > > > Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various > execution inefficiencies and the associated remedies that user can easily > understand and fix. This Jira patch integrates it with Job History UI under > Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vitthal (Suhas) Gogate updated MAPREDUCE-3202: -- Target Version/s: trunk Status: Open (was: Patch Available) > Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 > > > Key: MAPREDUCE-3202 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobhistoryserver >Affects Versions: 2.0.0-alpha >Reporter: vitthal (Suhas) Gogate >Assignee: Evan (Yiming) Meng > Fix For: 2.0.6-alpha > > Attachments: vaidya-2.0.6-alpha.patch > > > Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various > execution inefficiencies and the associated remedies that user can easily > understand and fix. This Jira patch integrates it with Job History UI under > Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5512) TaskTracker hung after failed reconnect to the JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-5512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Mitic updated MAPREDUCE-5512: -- Attachment: MAPREDUCE-5512.branch-1.patch Attaching the patch. My proposal for the fix is to make the dist cache cleanup thread a daemon. Based on the scan thru the code I think it should be safe to make this change. For the unittest, I added a test that validates the list of non-daemon threads. This is a more general test case but I think it will serve well to protect the codebase against regressions in this area. I was not able to come up with a nice way to simulate the condition from this bug without adding a test hook in the production code, so I moved away from this approach (we would have to start JT, stop JT, start JT again which would tell TT to reinit, and then stop JT, but last JT stop must have the right timing and run before TT#initialize() executes). Slightly orthogonally, looking at the list of threads I had to whitelist, there might be some other candidate threads that could be made daemons, but I'd prefer not to make this change in the context of this Jira. > TaskTracker hung after failed reconnect to the JobTracker > - > > Key: MAPREDUCE-5512 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5512 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Attachments: hadoop-tasktracker-RD00155DD09100.log, > MAPREDUCE-5512.branch-1.patch, tt_Hung.txt > > > TaskTracker hung after failed reconnect to the JobTracker. > This is the problematic piece of code: > {code} > this.distributedCacheManager = new TrackerDistributedCacheManager( > this.fConf, taskController); > this.distributedCacheManager.startCleanupThread(); > > this.jobClient = (InterTrackerProtocol) > UserGroupInformation.getLoginUser().doAs( > new PrivilegedExceptionAction() { > public Object run() throws IOException { > return RPC.waitForProxy(InterTrackerProtocol.class, > InterTrackerProtocol.versionID, > jobTrackAddr, fConf); > } > }); > {code} > In case RPC.waitForProxy() throws, TrackerDistributedCacheManager cleanup > thread will never be stopped, and given that it is a non daemon thread it > will keep TT up forever. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5544) JobClient#getJob loads job conf twice
[ https://issues.apache.org/jira/browse/MAPREDUCE-5544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782302#comment-13782302 ] Hadoop QA commented on MAPREDUCE-5544: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12605974/MAPREDUCE-5544.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4065//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4065//console This message is automatically generated. > JobClient#getJob loads job conf twice > - > > Key: MAPREDUCE-5544 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5544 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5544.patch > > > Calling JobClient#getJob causes the job conf file to be loaded twice, once in > the constructor of JobClient.NetworkedJob and once in Cluster#getJob. We > should remove the former. > MAPREDUCE-5001 was meant to fix a race that was causing problems in Hive > tests, but the problem persists because it only fixed one of the places where > the job conf file is loaded. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5544) JobClient#getJob loads job conf twice
[ https://issues.apache.org/jira/browse/MAPREDUCE-5544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5544: -- Attachment: MAPREDUCE-5544.patch > JobClient#getJob loads job conf twice > - > > Key: MAPREDUCE-5544 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5544 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5544.patch > > > Calling JobClient#getJob causes the job conf file to be loaded twice, once in > the constructor of JobClient.NetworkedJob and once in Cluster#getJob. We > should remove the former. > MAPREDUCE-5001 was meant to fix a race that was causing problems in Hive > tests, but the problem persists because it only fixed one of the places where > the job conf file is loaded. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5544) JobClient#getJob loads job conf twice
[ https://issues.apache.org/jira/browse/MAPREDUCE-5544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5544: -- Status: Patch Available (was: Open) > JobClient#getJob loads job conf twice > - > > Key: MAPREDUCE-5544 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5544 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5544.patch > > > Calling JobClient#getJob causes the job conf file to be loaded twice, once in > the constructor of JobClient.NetworkedJob and once in Cluster#getJob. We > should remove the former. > MAPREDUCE-5001 was meant to fix a race that was causing problems in Hive > tests, but the problem persists because it only fixed one of the places where > the job conf file is loaded. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5536: - Status: Open (was: Patch Available) restructuring the patch. > mapreduce.jobhistory.webapp.https.address property is not respected > --- > > Key: MAPREDUCE-5536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Yesha Vora >Assignee: Omkar Vinit Joshi >Priority: Blocker > Attachments: MAPREDUCE-5536.20131027.1.patch, > MAPREDUCE-5536.20131030.1.patch, YARN-1240.20131025.1.patch > > > The jobhistory server starts on port defined by > mapreduce.jobhistory.webapp.address property instead > mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5536: - Status: Patch Available (was: Open) > mapreduce.jobhistory.webapp.https.address property is not respected > --- > > Key: MAPREDUCE-5536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Yesha Vora >Assignee: Omkar Vinit Joshi >Priority: Blocker > Attachments: MAPREDUCE-5536.20131027.1.patch, > MAPREDUCE-5536.20131030.1.patch, YARN-1240.20131025.1.patch > > > The jobhistory server starts on port defined by > mapreduce.jobhistory.webapp.address property instead > mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5536: - Attachment: MAPREDUCE-5536.20131030.1.patch > mapreduce.jobhistory.webapp.https.address property is not respected > --- > > Key: MAPREDUCE-5536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Yesha Vora >Assignee: Omkar Vinit Joshi >Priority: Blocker > Attachments: MAPREDUCE-5536.20131027.1.patch, > MAPREDUCE-5536.20131030.1.patch, YARN-1240.20131025.1.patch > > > The jobhistory server starts on port defined by > mapreduce.jobhistory.webapp.address property instead > mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5550) Job Status message not shown in UI with Hadoop 2.0
Vrushali C created MAPREDUCE-5550: - Summary: Job Status message not shown in UI with Hadoop 2.0 Key: MAPREDUCE-5550 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5550 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.5-alpha Reporter: Vrushali C Hadoop 1.0 JobTracker UI displays Job Status message when list of mapper or reduce tasks are listed. This give an idea of how that task is making progress. Hadoop 2.0 ResourceManager UI does not have this. It would be good to have this on RM UI. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (MAPREDUCE-5550) Job Status message not shown in UI with Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C reassigned MAPREDUCE-5550: - Assignee: Vrushali C > Job Status message not shown in UI with Hadoop 2.0 > -- > > Key: MAPREDUCE-5550 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5550 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.5-alpha >Reporter: Vrushali C >Assignee: Vrushali C > > Hadoop 1.0 JobTracker UI displays Job Status message when list of mapper or > reduce tasks are listed. This give an idea of how that task is making > progress. > Hadoop 2.0 ResourceManager UI does not have this. It would be good to have > this on RM UI. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782029#comment-13782029 ] Hadoop QA commented on MAPREDUCE-5530: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12605934/MAPREDUCE-5530.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4063//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4063//console This message is automatically generated. > Binary and source incompatibility in mapred.lib.CombineFileInputFormat > between branch-1 and branch-2 > > > Key: MAPREDUCE-5530 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv1, mrv2 >Affects Versions: 2.1.1-beta >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > Attachments: MAPREDUCE-5530.patch, MAPREDUCE-5530.patch > > > {{mapred.lib.CombineFileInputFormat}} in branch-1 has this method: > {code:java} > protected boolean isSplitable(FileSystem fs, Path file) > {code} > In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of > {{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the > similar method: > {code:java} > protected boolean isSplitable(JobContext context, Path file) > {code} > This means that any code that subclasses > {{mapred.lib.CombineFileInputFormat}} and does not provide its own > implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}} > will not be binary or source compatible if it tries to call {{isSplitable}} > with a {{FileSystem}} argument anywhere (that is, if compiled against > branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2; > also, it won't even compile against branch-2). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-5530: - Attachment: MAPREDUCE-5530.patch I've uploaded a new patch that includes the above proposed change > Binary and source incompatibility in mapred.lib.CombineFileInputFormat > between branch-1 and branch-2 > > > Key: MAPREDUCE-5530 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv1, mrv2 >Affects Versions: 2.1.1-beta >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > Attachments: MAPREDUCE-5530.patch, MAPREDUCE-5530.patch > > > {{mapred.lib.CombineFileInputFormat}} in branch-1 has this method: > {code:java} > protected boolean isSplitable(FileSystem fs, Path file) > {code} > In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of > {{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the > similar method: > {code:java} > protected boolean isSplitable(JobContext context, Path file) > {code} > This means that any code that subclasses > {{mapred.lib.CombineFileInputFormat}} and does not provide its own > implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}} > will not be binary or source compatible if it tries to call {{isSplitable}} > with a {{FileSystem}} argument anywhere (that is, if compiled against > branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2; > also, it won't even compile against branch-2). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5549) distcp app should fail if m/r job fails
David Rosenstrauch created MAPREDUCE-5549: - Summary: distcp app should fail if m/r job fails Key: MAPREDUCE-5549 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp, mrv2 Reporter: David Rosenstrauch I run distcpv2 in a scripted manner. The script checks if the distcp step fails and, if so, aborts the rest of the script. However, I ran into an issue today where the distcp job failed, but my calling script went on its merry way. Digging into the code a bit more (at https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java), I think I see the issue: the distcp app is not returning an error exit code to the shell when the distcp job fails. This is a big problem, IMO, as it prevents distcp from being successfully used in a scripted environment. IMO, the code should change like so: Before: {code:title=org.apache.hadoop.tools.DistCp.java} //... public int run(String[] argv) { //... try { execute(); } catch (InvalidInputException e) { LOG.error("Invalid input: ", e); return DistCpConstants.INVALID_ARGUMENT; } catch (DuplicateFileException e) { LOG.error("Duplicate files in input path: ", e); return DistCpConstants.DUPLICATE_INPUT; } catch (Exception e) { LOG.error("Exception encountered ", e); return DistCpConstants.UNKNOWN_ERROR; } return DistCpConstants.SUCCESS; } //... {code} After: {code:title=org.apache.hadoop.tools.DistCp.java} //... public int run(String[] argv) { //... Job job = null; try { job = execute(); } catch (InvalidInputException e) { LOG.error("Invalid input: ", e); return DistCpConstants.INVALID_ARGUMENT; } catch (DuplicateFileException e) { LOG.error("Duplicate files in input path: ", e); return DistCpConstants.DUPLICATE_INPUT; } catch (Exception e) { LOG.error("Exception encountered ", e); return DistCpConstants.UNKNOWN_ERROR; } if (job.isSuccessful()) { return DistCpConstants.SUCCESS; } else { return DistCpConstants.UNKNOWN_ERROR; } } //... {code} -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13781968#comment-13781968 ] Hadoop QA commented on MAPREDUCE-3202: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12605914/vaidya-2.0.6-alpha.patch against trunk revision . {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4062//console This message is automatically generated. > Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 > > > Key: MAPREDUCE-3202 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobhistoryserver >Affects Versions: 2.0.0-alpha >Reporter: vitthal (Suhas) Gogate >Assignee: Evan (Yiming) Meng > Fix For: 2.0.6-alpha > > Attachments: vaidya-2.0.6-alpha.patch > > > Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various > execution inefficiencies and the associated remedies that user can easily > understand and fix. This Jira patch integrates it with Job History UI under > Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13781962#comment-13781962 ] vitthal (Suhas) Gogate commented on MAPREDUCE-3202: --- Will submit the patch for trunk soon. > Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 > > > Key: MAPREDUCE-3202 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobhistoryserver >Affects Versions: 2.0.0-alpha >Reporter: vitthal (Suhas) Gogate > Fix For: 2.0.6-alpha > > Attachments: vaidya-2.0.6-alpha.patch > > > Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various > execution inefficiencies and the associated remedies that user can easily > understand and fix. This Jira patch integrates it with Job History UI under > Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vitthal (Suhas) Gogate updated MAPREDUCE-3202: -- Assignee: Evan (Yiming) Meng (was: vitthal (Suhas) Gogate) > Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 > > > Key: MAPREDUCE-3202 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobhistoryserver >Affects Versions: 2.0.0-alpha >Reporter: vitthal (Suhas) Gogate >Assignee: Evan (Yiming) Meng > Fix For: 2.0.6-alpha > > Attachments: vaidya-2.0.6-alpha.patch > > > Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various > execution inefficiencies and the associated remedies that user can easily > understand and fix. This Jira patch integrates it with Job History UI under > Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vitthal (Suhas) Gogate reassigned MAPREDUCE-3202: - Assignee: vitthal (Suhas) Gogate > Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 > > > Key: MAPREDUCE-3202 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobhistoryserver >Affects Versions: 2.0.0-alpha >Reporter: vitthal (Suhas) Gogate >Assignee: vitthal (Suhas) Gogate > Fix For: 2.0.6-alpha > > Attachments: vaidya-2.0.6-alpha.patch > > > Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various > execution inefficiencies and the associated remedies that user can easily > understand and fix. This Jira patch integrates it with Job History UI under > Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vitthal (Suhas) Gogate updated MAPREDUCE-3202: -- Fix Version/s: 2.0.6-alpha Release Note: Hadoop Vaidya analysis available using Job History Server UI in 2.0 branch. Also ability for users to add new rules is provided. Status: Patch Available (was: Open) Submitting the patch to make Hadoop Vaidya work with 2.0 branch and also integrate it with Job History server providing UI interface analyze M/R jobs using Vaidya tool. Thanks to Evan Meng @ Pivotal Inc [~evanmeng] make this happen. Submitting the patch on his behalf. > Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 > > > Key: MAPREDUCE-3202 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobhistoryserver >Affects Versions: 2.0.0-alpha >Reporter: vitthal (Suhas) Gogate > Fix For: 2.0.6-alpha > > Attachments: vaidya-2.0.6-alpha.patch > > > Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various > execution inefficiencies and the associated remedies that user can easily > understand and fix. This Jira patch integrates it with Job History UI under > Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vitthal (Suhas) Gogate updated MAPREDUCE-3202: -- Attachment: vaidya-2.0.6-alpha.patch > Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 > > > Key: MAPREDUCE-3202 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobhistoryserver >Affects Versions: 2.0.0-alpha >Reporter: vitthal (Suhas) Gogate > Attachments: vaidya-2.0.6-alpha.patch > > > Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various > execution inefficiencies and the associated remedies that user can easily > understand and fix. This Jira patch integrates it with Job History UI under > Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5547) Job history should not be flushed to JHS until AM gets unregistered
[ https://issues.apache.org/jira/browse/MAPREDUCE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13781865#comment-13781865 ] Jason Lowe commented on MAPREDUCE-5547: --- Won't this create a race condition in the client where the job can be lost? Currently the client relies on the job state as reported by the RM to know where to query for job status. If the RM reports the job as running it will try to query the AM, otherwise it will try the history server. If we unregister _before_ copying the history data to the done_intermediate directory then the client could try to query the history server before the AM has had a chance to copy the jhist file. The AM needs to move the file to the done_intermediate directory before unregistering or we create this race. If the concern is that the unregister could fail and the AM could re-upload the jhist file, we could make the AM smarter when it goes to upload (e.g.: check for it already existing in the done_intermediate directory before copying). If the history server has already moved it from done_intermediate to done then the history server could either re-update the history with the new copy in done_intermediate or simply delete the redundant copy in done_intermediate. > Job history should not be flushed to JHS until AM gets unregistered > --- > > Key: MAPREDUCE-5547 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5547 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen > -- This message was sent by Atlassian JIRA (v6.1#6144)