[jira] [Commented] (MAPREDUCE-5547) Job history should not be flushed to JHS until AM gets unregistered
[ https://issues.apache.org/jira/browse/MAPREDUCE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13781865#comment-13781865 ] Jason Lowe commented on MAPREDUCE-5547: --- Won't this create a race condition in the client where the job can be lost? Currently the client relies on the job state as reported by the RM to know where to query for job status. If the RM reports the job as running it will try to query the AM, otherwise it will try the history server. If we unregister _before_ copying the history data to the done_intermediate directory then the client could try to query the history server before the AM has had a chance to copy the jhist file. The AM needs to move the file to the done_intermediate directory before unregistering or we create this race. If the concern is that the unregister could fail and the AM could re-upload the jhist file, we could make the AM smarter when it goes to upload (e.g.: check for it already existing in the done_intermediate directory before copying). If the history server has already moved it from done_intermediate to done then the history server could either re-update the history with the new copy in done_intermediate or simply delete the redundant copy in done_intermediate. Job history should not be flushed to JHS until AM gets unregistered --- Key: MAPREDUCE-5547 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5547 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vitthal (Suhas) Gogate updated MAPREDUCE-3202: -- Attachment: vaidya-2.0.6-alpha.patch Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 Key: MAPREDUCE-3202 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.0.0-alpha Reporter: vitthal (Suhas) Gogate Attachments: vaidya-2.0.6-alpha.patch Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various execution inefficiencies and the associated remedies that user can easily understand and fix. This Jira patch integrates it with Job History UI under Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vitthal (Suhas) Gogate updated MAPREDUCE-3202: -- Fix Version/s: 2.0.6-alpha Release Note: Hadoop Vaidya analysis available using Job History Server UI in 2.0 branch. Also ability for users to add new rules is provided. Status: Patch Available (was: Open) Submitting the patch to make Hadoop Vaidya work with 2.0 branch and also integrate it with Job History server providing UI interface analyze M/R jobs using Vaidya tool. Thanks to Evan Meng @ Pivotal Inc [~evanmeng] make this happen. Submitting the patch on his behalf. Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 Key: MAPREDUCE-3202 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.0.0-alpha Reporter: vitthal (Suhas) Gogate Fix For: 2.0.6-alpha Attachments: vaidya-2.0.6-alpha.patch Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various execution inefficiencies and the associated remedies that user can easily understand and fix. This Jira patch integrates it with Job History UI under Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vitthal (Suhas) Gogate reassigned MAPREDUCE-3202: - Assignee: vitthal (Suhas) Gogate Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 Key: MAPREDUCE-3202 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.0.0-alpha Reporter: vitthal (Suhas) Gogate Assignee: vitthal (Suhas) Gogate Fix For: 2.0.6-alpha Attachments: vaidya-2.0.6-alpha.patch Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various execution inefficiencies and the associated remedies that user can easily understand and fix. This Jira patch integrates it with Job History UI under Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vitthal (Suhas) Gogate updated MAPREDUCE-3202: -- Assignee: Evan (Yiming) Meng (was: vitthal (Suhas) Gogate) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 Key: MAPREDUCE-3202 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.0.0-alpha Reporter: vitthal (Suhas) Gogate Assignee: Evan (Yiming) Meng Fix For: 2.0.6-alpha Attachments: vaidya-2.0.6-alpha.patch Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various execution inefficiencies and the associated remedies that user can easily understand and fix. This Jira patch integrates it with Job History UI under Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13781962#comment-13781962 ] vitthal (Suhas) Gogate commented on MAPREDUCE-3202: --- Will submit the patch for trunk soon. Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 Key: MAPREDUCE-3202 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.0.0-alpha Reporter: vitthal (Suhas) Gogate Fix For: 2.0.6-alpha Attachments: vaidya-2.0.6-alpha.patch Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various execution inefficiencies and the associated remedies that user can easily understand and fix. This Jira patch integrates it with Job History UI under Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[
https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13781968#comment-13781968
]
Hadoop QA commented on MAPREDUCE-3202:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12605914/vaidya-2.0.6-alpha.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4062//console
This message is automatically generated.
Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
Key: MAPREDUCE-3202
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: jobhistoryserver
Affects Versions: 2.0.0-alpha
Reporter: vitthal (Suhas) Gogate
Assignee: Evan (Yiming) Meng
Fix For: 2.0.6-alpha
Attachments: vaidya-2.0.6-alpha.patch
Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various
execution inefficiencies and the associated remedies that user can easily
understand and fix. This Jira patch integrates it with Job History UI under
Hadoop 2.0 branch.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (MAPREDUCE-5549) distcp app should fail if m/r job fails
David Rosenstrauch created MAPREDUCE-5549:
-
Summary: distcp app should fail if m/r job fails
Key: MAPREDUCE-5549
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5549
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, mrv2
Reporter: David Rosenstrauch
I run distcpv2 in a scripted manner. The script checks if the distcp step
fails and, if so, aborts the rest of the script. However, I ran into an issue
today where the distcp job failed, but my calling script went on its merry way.
Digging into the code a bit more (at
https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
I think I see the issue: the distcp app is not returning an error exit code
to the shell when the distcp job fails. This is a big problem, IMO, as it
prevents distcp from being successfully used in a scripted environment. IMO,
the code should change like so:
Before:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
try {
execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
return DistCpConstants.SUCCESS;
}
//...
{code}
After:
{code:title=org.apache.hadoop.tools.DistCp.java}
//...
public int run(String[] argv) {
//...
Job job = null;
try {
job = execute();
} catch (InvalidInputException e) {
LOG.error(Invalid input: , e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error(Duplicate files in input path: , e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (Exception e) {
LOG.error(Exception encountered , e);
return DistCpConstants.UNKNOWN_ERROR;
}
if (job.isSuccessful()) {
return DistCpConstants.SUCCESS;
}
else {
return DistCpConstants.UNKNOWN_ERROR;
}
}
//...
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-5530:
-
Attachment: MAPREDUCE-5530.patch
I've uploaded a new patch that includes the above proposed change
Binary and source incompatibility in mapred.lib.CombineFileInputFormat
between branch-1 and branch-2
Key: MAPREDUCE-5530
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv1, mrv2
Affects Versions: 2.1.1-beta
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Attachments: MAPREDUCE-5530.patch, MAPREDUCE-5530.patch
{{mapred.lib.CombineFileInputFormat}} in branch-1 has this method:
{code:java}
protected boolean isSplitable(FileSystem fs, Path file)
{code}
In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of
{{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the
similar method:
{code:java}
protected boolean isSplitable(JobContext context, Path file)
{code}
This means that any code that subclasses
{{mapred.lib.CombineFileInputFormat}} and does not provide its own
implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}}
will not be binary or source compatible if it tries to call {{isSplitable}}
with a {{FileSystem}} argument anywhere (that is, if compiled against
branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2;
also, it won't even compile against branch-2).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782029#comment-13782029
]
Hadoop QA commented on MAPREDUCE-5530:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12605934/MAPREDUCE-5530.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4063//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4063//console
This message is automatically generated.
Binary and source incompatibility in mapred.lib.CombineFileInputFormat
between branch-1 and branch-2
Key: MAPREDUCE-5530
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv1, mrv2
Affects Versions: 2.1.1-beta
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Attachments: MAPREDUCE-5530.patch, MAPREDUCE-5530.patch
{{mapred.lib.CombineFileInputFormat}} in branch-1 has this method:
{code:java}
protected boolean isSplitable(FileSystem fs, Path file)
{code}
In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of
{{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the
similar method:
{code:java}
protected boolean isSplitable(JobContext context, Path file)
{code}
This means that any code that subclasses
{{mapred.lib.CombineFileInputFormat}} and does not provide its own
implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}}
will not be binary or source compatible if it tries to call {{isSplitable}}
with a {{FileSystem}} argument anywhere (that is, if compiled against
branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2;
also, it won't even compile against branch-2).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (MAPREDUCE-5550) Job Status message not shown in UI with Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C reassigned MAPREDUCE-5550: - Assignee: Vrushali C Job Status message not shown in UI with Hadoop 2.0 -- Key: MAPREDUCE-5550 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5550 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.5-alpha Reporter: Vrushali C Assignee: Vrushali C Hadoop 1.0 JobTracker UI displays Job Status message when list of mapper or reduce tasks are listed. This give an idea of how that task is making progress. Hadoop 2.0 ResourceManager UI does not have this. It would be good to have this on RM UI. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5550) Job Status message not shown in UI with Hadoop 2.0
Vrushali C created MAPREDUCE-5550: - Summary: Job Status message not shown in UI with Hadoop 2.0 Key: MAPREDUCE-5550 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5550 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.5-alpha Reporter: Vrushali C Hadoop 1.0 JobTracker UI displays Job Status message when list of mapper or reduce tasks are listed. This give an idea of how that task is making progress. Hadoop 2.0 ResourceManager UI does not have this. It would be good to have this on RM UI. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5536: - Attachment: MAPREDUCE-5536.20131030.1.patch mapreduce.jobhistory.webapp.https.address property is not respected --- Key: MAPREDUCE-5536 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.1.1-beta Reporter: Yesha Vora Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: MAPREDUCE-5536.20131027.1.patch, MAPREDUCE-5536.20131030.1.patch, YARN-1240.20131025.1.patch The jobhistory server starts on port defined by mapreduce.jobhistory.webapp.address property instead mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5536: - Status: Open (was: Patch Available) restructuring the patch. mapreduce.jobhistory.webapp.https.address property is not respected --- Key: MAPREDUCE-5536 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.1.1-beta Reporter: Yesha Vora Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: MAPREDUCE-5536.20131027.1.patch, MAPREDUCE-5536.20131030.1.patch, YARN-1240.20131025.1.patch The jobhistory server starts on port defined by mapreduce.jobhistory.webapp.address property instead mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5536: - Status: Patch Available (was: Open) mapreduce.jobhistory.webapp.https.address property is not respected --- Key: MAPREDUCE-5536 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.1.1-beta Reporter: Yesha Vora Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: MAPREDUCE-5536.20131027.1.patch, MAPREDUCE-5536.20131030.1.patch, YARN-1240.20131025.1.patch The jobhistory server starts on port defined by mapreduce.jobhistory.webapp.address property instead mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5544) JobClient#getJob loads job conf twice
[ https://issues.apache.org/jira/browse/MAPREDUCE-5544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5544: -- Attachment: MAPREDUCE-5544.patch JobClient#getJob loads job conf twice - Key: MAPREDUCE-5544 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5544 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5544.patch Calling JobClient#getJob causes the job conf file to be loaded twice, once in the constructor of JobClient.NetworkedJob and once in Cluster#getJob. We should remove the former. MAPREDUCE-5001 was meant to fix a race that was causing problems in Hive tests, but the problem persists because it only fixed one of the places where the job conf file is loaded. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5544) JobClient#getJob loads job conf twice
[ https://issues.apache.org/jira/browse/MAPREDUCE-5544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5544: -- Status: Patch Available (was: Open) JobClient#getJob loads job conf twice - Key: MAPREDUCE-5544 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5544 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5544.patch Calling JobClient#getJob causes the job conf file to be loaded twice, once in the constructor of JobClient.NetworkedJob and once in Cluster#getJob. We should remove the former. MAPREDUCE-5001 was meant to fix a race that was causing problems in Hive tests, but the problem persists because it only fixed one of the places where the job conf file is loaded. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5544) JobClient#getJob loads job conf twice
[
https://issues.apache.org/jira/browse/MAPREDUCE-5544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782302#comment-13782302
]
Hadoop QA commented on MAPREDUCE-5544:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12605974/MAPREDUCE-5544.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4065//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4065//console
This message is automatically generated.
JobClient#getJob loads job conf twice
-
Key: MAPREDUCE-5544
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5544
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: MAPREDUCE-5544.patch
Calling JobClient#getJob causes the job conf file to be loaded twice, once in
the constructor of JobClient.NetworkedJob and once in Cluster#getJob. We
should remove the former.
MAPREDUCE-5001 was meant to fix a race that was causing problems in Hive
tests, but the problem persists because it only fixed one of the places where
the job conf file is loaded.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5512) TaskTracker hung after failed reconnect to the JobTracker
[
https://issues.apache.org/jira/browse/MAPREDUCE-5512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Mitic updated MAPREDUCE-5512:
--
Attachment: MAPREDUCE-5512.branch-1.patch
Attaching the patch.
My proposal for the fix is to make the dist cache cleanup thread a daemon.
Based on the scan thru the code I think it should be safe to make this change.
For the unittest, I added a test that validates the list of non-daemon threads.
This is a more general test case but I think it will serve well to protect the
codebase against regressions in this area. I was not able to come up with a
nice way to simulate the condition from this bug without adding a test hook in
the production code, so I moved away from this approach (we would have to start
JT, stop JT, start JT again which would tell TT to reinit, and then stop JT,
but last JT stop must have the right timing and run before TT#initialize()
executes).
Slightly orthogonally, looking at the list of threads I had to whitelist, there
might be some other candidate threads that could be made daemons, but I'd
prefer not to make this change in the context of this Jira.
TaskTracker hung after failed reconnect to the JobTracker
-
Key: MAPREDUCE-5512
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5512
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.3.0
Reporter: Ivan Mitic
Assignee: Ivan Mitic
Attachments: hadoop-tasktracker-RD00155DD09100.log,
MAPREDUCE-5512.branch-1.patch, tt_Hung.txt
TaskTracker hung after failed reconnect to the JobTracker.
This is the problematic piece of code:
{code}
this.distributedCacheManager = new TrackerDistributedCacheManager(
this.fConf, taskController);
this.distributedCacheManager.startCleanupThread();
this.jobClient = (InterTrackerProtocol)
UserGroupInformation.getLoginUser().doAs(
new PrivilegedExceptionActionObject() {
public Object run() throws IOException {
return RPC.waitForProxy(InterTrackerProtocol.class,
InterTrackerProtocol.versionID,
jobTrackAddr, fConf);
}
});
{code}
In case RPC.waitForProxy() throws, TrackerDistributedCacheManager cleanup
thread will never be stopped, and given that it is a non daemon thread it
will keep TT up forever.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vitthal (Suhas) Gogate updated MAPREDUCE-3202: -- Target Version/s: trunk Status: Open (was: Patch Available) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 Key: MAPREDUCE-3202 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.0.0-alpha Reporter: vitthal (Suhas) Gogate Assignee: Evan (Yiming) Meng Fix For: 2.0.6-alpha Attachments: vaidya-2.0.6-alpha.patch Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various execution inefficiencies and the associated remedies that user can easily understand and fix. This Jira patch integrates it with Job History UI under Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vitthal (Suhas) Gogate updated MAPREDUCE-3202: -- Attachment: MAPREDUCE-3202.patch Submitting patch for trunk. Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 Key: MAPREDUCE-3202 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.0.0-alpha Reporter: vitthal (Suhas) Gogate Assignee: Evan (Yiming) Meng Fix For: 2.0.6-alpha Attachments: MAPREDUCE-3202.patch, vaidya-2.0.6-alpha.patch Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various execution inefficiencies and the associated remedies that user can easily understand and fix. This Jira patch integrates it with Job History UI under Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vitthal (Suhas) Gogate updated MAPREDUCE-3202: -- Fix Version/s: (was: 2.0.6-alpha) trunk Status: Patch Available (was: Open) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 Key: MAPREDUCE-3202 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.0.0-alpha Reporter: vitthal (Suhas) Gogate Assignee: Evan (Yiming) Meng Fix For: trunk Attachments: MAPREDUCE-3202.patch, vaidya-2.0.6-alpha.patch Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various execution inefficiencies and the associated remedies that user can easily understand and fix. This Jira patch integrates it with Job History UI under Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vitthal (Suhas) Gogate updated MAPREDUCE-3202: -- Attachment: (was: vaidya-2.0.6-alpha.patch) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 Key: MAPREDUCE-3202 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.0.0-alpha Reporter: vitthal (Suhas) Gogate Assignee: Evan (Yiming) Meng Fix For: trunk Attachments: MAPREDUCE-3202.patch Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various execution inefficiencies and the associated remedies that user can easily understand and fix. This Jira patch integrates it with Job History UI under Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782362#comment-13782362
]
Zhijie Shen commented on MAPREDUCE-5530:
[~rkanter], IMHO, it's not a good idea to make isSplitable(JobContext context,
Path file) call isSplitable(FileSystem fs, Path file), because, it introduces
the step of FileSystem.get(context.getConfiguration()), which may lead to
unexpected IOException. Personally, I incline not to increase the instability
of Hadoop2 code pass. Thoughts?
Binary and source incompatibility in mapred.lib.CombineFileInputFormat
between branch-1 and branch-2
Key: MAPREDUCE-5530
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv1, mrv2
Affects Versions: 2.1.1-beta
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Attachments: MAPREDUCE-5530.patch, MAPREDUCE-5530.patch
{{mapred.lib.CombineFileInputFormat}} in branch-1 has this method:
{code:java}
protected boolean isSplitable(FileSystem fs, Path file)
{code}
In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of
{{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the
similar method:
{code:java}
protected boolean isSplitable(JobContext context, Path file)
{code}
This means that any code that subclasses
{{mapred.lib.CombineFileInputFormat}} and does not provide its own
implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}}
will not be binary or source compatible if it tries to call {{isSplitable}}
with a {{FileSystem}} argument anywhere (that is, if compiled against
branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2;
also, it won't even compile against branch-2).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5088) MR Client gets an renewer token exception while Oozie is submitting a job
[
https://issues.apache.org/jira/browse/MAPREDUCE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782364#comment-13782364
]
Robert Kanter commented on MAPREDUCE-5088:
--
We're seeing an issue where Oozie can't submit to a secure cluster when the JHS
is down because it can't get a delegation token from the Job History Server.
However, if I submit a job without Oozie (i.e. hadoop jar ) it works fine.
When investigating, it looks like this JIRA may be the cause, or is at least
related. Any ideas on why this is happening?
{noformat}
2013-09-30 13:14:48,544 WARN ActionStartXCommand:542 - USER[testuser] GROUP[-]
TOKEN[] APP[map-reduce-wf] JOB[055-130929201011702-oozie-oozi-W]
ACTION[055-130929201011702-oozie-oozi-W@mr-node] Er
ror starting action [mr-node]. ErrorType [TRANSIENT], ErrorCode [ JA006],
Message [ JA006: Connection refused]
org.apache.oozie.action.ActionExecutorException: JA006: Connection refused
at
org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:412)
at
org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:392)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:794)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:948)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:215)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:60)
at org.apache.oozie.command.XCommand.call(XCommand.java:280)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:547)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:642)
at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399)
at org.apache.hadoop.ipc.Client.call(Client.java:1318)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy31.getDelegationToken(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getDelegationToken(MRClientProtocolPBClientImpl.java:211)
at
org.apache.hadoop.mapred.YARNRunner.getDelegationTokenFromHS(YARNRunner.java:208)
at
org.apache.hadoop.mapred.YARNRunner.addHistoryToken(YARNRunner.java:195)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:280)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:430)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:552)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:552)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:543)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:779)
... 10 more
{noformat}
MR Client gets an renewer token exception while Oozie is submitting a job
[jira] [Updated] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5536: --- Status: Open (was: Patch Available) More comments on the patch. This has to be one of the hardest patches I reviewed in recent times. - WebAppUtil.getDefaultJHSWebappPort() and getDefaultJHSWebappURLWithoutScheme() can be private. - getHistoryUrl() - getApplicationHistoryURLWithoutScheme(). - getJHSWebappURLWithScheme() : No need to have a isSSLEnabledInJHS check. - getJHSSchemePrefix() - getJHSWebAppSchemePrefix()() - Add a getApplicationHistoryURLWithSchem() and use in MR AM's RMCommunicator? - MR's WebAppUtil - Mark as Private. - WebAppUtil - MRWebAppUtil. - JHAdminConfig.getResolvedMRHistoryWebAppURLWithoutScheme() still takes in a flag - it doesn't need to. And may be move it to MiniMRYarnCluster if that is the only caller. I think instead of WebAppUtil.setSSLEnabledIn*() methods, we should have WebAppUtil.initialize(Configuration conf). That way, every component doesn't need to remember what config to read. mapreduce.jobhistory.webapp.https.address property is not respected --- Key: MAPREDUCE-5536 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.1.1-beta Reporter: Yesha Vora Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: MAPREDUCE-5536.20131027.1.patch, MAPREDUCE-5536.20131030.1.patch, YARN-1240.20131025.1.patch The jobhistory server starts on port defined by mapreduce.jobhistory.webapp.address property instead mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[
https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782383#comment-13782383
]
Hadoop QA commented on MAPREDUCE-3202:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12605989/MAPREDUCE-3202.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1536 javac
compiler warnings (more than the trunk's current 1534 warnings).
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 12 new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-tools/hadoop-tools-dist hadoop-tools/hadoop-vaidya:
org.apache.hadoop.mapreduce.v2.hs.webapp.dao.TestJobInfo
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-vaidya.html
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4066//console
This message is automatically generated.
Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
Key: MAPREDUCE-3202
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: jobhistoryserver
Affects Versions: 2.0.0-alpha
Reporter: vitthal (Suhas) Gogate
Assignee: Evan (Yiming) Meng
Fix For: trunk
Attachments: MAPREDUCE-3202.patch
Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various
execution inefficiencies and the associated remedies that user can easily
understand and fix. This Jira patch integrates it with Job History UI under
Hadoop 2.0 branch.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[
https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782401#comment-13782401
]
Hadoop QA commented on MAPREDUCE-5536:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12605975/MAPREDUCE-5536.20131030.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapreduce.v2.hs.webapp.TestMapReduceTrackingUriPlugin
org.apache.hadoop.mapred.TestJobCleanup
The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapreduce.v2.TestUberAM
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4064//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4064//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs-plugins.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4064//console
This message is automatically generated.
mapreduce.jobhistory.webapp.https.address property is not respected
---
Key: MAPREDUCE-5536
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.1.1-beta
Reporter: Yesha Vora
Assignee: Omkar Vinit Joshi
Priority: Blocker
Attachments: MAPREDUCE-5536.20131027.1.patch,
MAPREDUCE-5536.20131030.1.patch, YARN-1240.20131025.1.patch
The jobhistory server starts on port defined by
mapreduce.jobhistory.webapp.address property instead
mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782430#comment-13782430
]
Robert Kanter commented on MAPREDUCE-5530:
--
I don't think this really adds any instability (if there's a problem getting
the FileSystem, we're bound to run into that somewhere else as well anyway),
and without this the user might not even be able to compile.
Binary and source incompatibility in mapred.lib.CombineFileInputFormat
between branch-1 and branch-2
Key: MAPREDUCE-5530
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv1, mrv2
Affects Versions: 2.1.1-beta
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Attachments: MAPREDUCE-5530.patch, MAPREDUCE-5530.patch
{{mapred.lib.CombineFileInputFormat}} in branch-1 has this method:
{code:java}
protected boolean isSplitable(FileSystem fs, Path file)
{code}
In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of
{{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the
similar method:
{code:java}
protected boolean isSplitable(JobContext context, Path file)
{code}
This means that any code that subclasses
{{mapred.lib.CombineFileInputFormat}} and does not provide its own
implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}}
will not be binary or source compatible if it tries to call {{isSplitable}}
with a {{FileSystem}} argument anywhere (that is, if compiled against
branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2;
also, it won't even compile against branch-2).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-4421) Remove dependency on deployed MR jars
[ https://issues.apache.org/jira/browse/MAPREDUCE-4421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782438#comment-13782438 ] Hitesh Shah commented on MAPREDUCE-4421: Sorry for the delay in the review. Regarding addMRFrameworkToDistributedCache() - one minor question: the code allows for a non-qualified URI. Should we enforce provision of a fully-qualified path always? Minor nit: I believe there should be nothing in the implementation that requires HDFS as the storage for the MR tarball? Documentation needs to change as a result unless you believe there are reasons for not mentioning other filesystems ( except maybe from a testing point of view )? Patch looks good otherwise. Thanks for adding the detailed docs. Remove dependency on deployed MR jars - Key: MAPREDUCE-4421 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4421 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.0-alpha Reporter: Arun C Murthy Assignee: Jason Lowe Attachments: MAPREDUCE-4421-2.patch, MAPREDUCE-4421.patch, MAPREDUCE-4421.patch Currently MR AM depends on MR jars being deployed on all nodes via implicit dependency on YARN_APPLICATION_CLASSPATH. We should stop adding mapreduce jars to YARN_APPLICATION_CLASSPATH and, probably, just rely on adding a shaded MR jar along with job.jar to the dist-cache. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5536: - Attachment: MAPREDUCE-5536.20131030.1.patch mapreduce.jobhistory.webapp.https.address property is not respected --- Key: MAPREDUCE-5536 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.1.1-beta Reporter: Yesha Vora Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: MAPREDUCE-5536.20131027.1.patch, MAPREDUCE-5536.20131030.1.patch, MAPREDUCE-5536.20131030.1.patch, YARN-1240.20131025.1.patch The jobhistory server starts on port defined by mapreduce.jobhistory.webapp.address property instead mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782471#comment-13782471 ] Omkar Vinit Joshi commented on MAPREDUCE-5536: -- Thanks vinod.. bq. WebAppUtil.getDefaultJHSWebappPort() and getDefaultJHSWebappURLWithoutScheme() can be private fixed.. bq. getHistoryUrl() - getApplicationHistoryURLWithoutScheme(). fixed. bq. getJHSWebappURLWithScheme() : No need to have a isSSLEnabledInJHS check. fixed. bq. getJHSSchemePrefix() - getJHSWebAppSchemePrefix()() fixed. bq. Add a getApplicationHistoryURLWithSchem() and use in MR AM's RMCommunicator? fixed. bq. MR's WebAppUtil - Mark as Private. - WebAppUtil - MRWebAppUtil. fixed.. bq. JHAdminConfig.getResolvedMRHistoryWebAppURLWithoutScheme() still takes in a flag - it doesn't need to. And may be move it to MiniMRYarnCluster if that is the only caller. moved it to MiniMRYarnCluster bq. I think instead of WebAppUtil.setSSLEnabledIn*() methods, we should have WebAppUtil.initialize(Configuration conf). That way, every component doesn't need to remember what config to read. fixed. mapreduce.jobhistory.webapp.https.address property is not respected --- Key: MAPREDUCE-5536 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.1.1-beta Reporter: Yesha Vora Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: MAPREDUCE-5536.20131027.1.patch, MAPREDUCE-5536.20131030.1.patch, MAPREDUCE-5536.20131030.1.patch, YARN-1240.20131025.1.patch The jobhistory server starts on port defined by mapreduce.jobhistory.webapp.address property instead mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5536: - Status: Patch Available (was: Open) mapreduce.jobhistory.webapp.https.address property is not respected --- Key: MAPREDUCE-5536 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.1.1-beta Reporter: Yesha Vora Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: MAPREDUCE-5536.20131027.1.patch, MAPREDUCE-5536.20131030.1.patch, MAPREDUCE-5536.20131030.1.patch, YARN-1240.20131025.1.patch The jobhistory server starts on port defined by mapreduce.jobhistory.webapp.address property instead mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5551) Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes
Zhijie Shen created MAPREDUCE-5551: -- Summary: Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes Key: MAPREDUCE-5551 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5551 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker The non-default constructor is moved to the super class, but it cannot be inherited. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5550) Job Status message not shown in UI with Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782472#comment-13782472 ] Omkar Vinit Joshi commented on MAPREDUCE-5550: -- you should check the AM UI? Job Status message not shown in UI with Hadoop 2.0 -- Key: MAPREDUCE-5550 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5550 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.5-alpha Reporter: Vrushali C Assignee: Vrushali C Hadoop 1.0 JobTracker UI displays Job Status message when list of mapper or reduce tasks are listed. This give an idea of how that task is making progress. Hadoop 2.0 ResourceManager UI does not have this. It would be good to have this on RM UI. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (MAPREDUCE-5524) java.io.IOException: Task process exit with nonzero status of 255. how to fix it?
[ https://issues.apache.org/jira/browse/MAPREDUCE-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved MAPREDUCE-5524. --- Resolution: Invalid Please email u...@hadoop.apache.org with this question. Apache JIRA is for reporting bugs and tracking features/improvements. It's not intended for user-level help. java.io.IOException: Task process exit with nonzero status of 255. how to fix it? --- Key: MAPREDUCE-5524 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5524 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: hawkswood Task ..FAILED java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.io.IOException: Task process exit with nonzero status of 255. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5551) Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5551: --- Attachment: MAPREDUCE-5551.1.patch Uploaded a patch to add the missing constructors. Note that zero-param constructor is added as well, because the default constructor is no longer valid once the one-param constructor is added. Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes Key: MAPREDUCE-5551 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5551 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Attachments: MAPREDUCE-5551.1.patch The non-default constructor is moved to the super class, but it cannot be inherited. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5551) Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5551: --- Status: Patch Available (was: Open) Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes Key: MAPREDUCE-5551 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5551 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Attachments: MAPREDUCE-5551.1.patch The non-default constructor is moved to the super class, but it cannot be inherited. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5551) Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5551: --- Issue Type: Sub-task (was: Bug) Parent: MAPREDUCE-5108 Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes Key: MAPREDUCE-5551 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5551 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Attachments: MAPREDUCE-5551.1.patch The non-default constructor is moved to the super class, but it cannot be inherited. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5551) Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes
[
https://issues.apache.org/jira/browse/MAPREDUCE-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782510#comment-13782510
]
Hadoop QA commented on MAPREDUCE-5551:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12606016/MAPREDUCE-5551.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4067//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4067//console
This message is automatically generated.
Binary Incompatibility of
O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes
Key: MAPREDUCE-5551
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5551
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
Attachments: MAPREDUCE-5551.1.patch
The non-default constructor is moved to the super class, but it cannot be
inherited.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5551) Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782545#comment-13782545 ] Hudson commented on MAPREDUCE-5551: --- SUCCESS: Integrated in Hadoop-trunk-Commit #4503 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4503/]) MAPREDUCE-5551. Fix compat with hadoop-1 in SequenceFileAsBinaryOutputFormat.WritableValueBytes by re-introducing missing constructors. Contributed by Zhijie Shen. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1527848) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SequenceFileAsBinaryOutputFormat.java Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes Key: MAPREDUCE-5551 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5551 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Attachments: MAPREDUCE-5551.1.patch The non-default constructor is moved to the super class, but it cannot be inherited. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5551) Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5551: - Resolution: Fixed Fix Version/s: 2.1.2-beta Status: Resolved (was: Patch Available) I just committed this. Thanks [~zjshen]! Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes Key: MAPREDUCE-5551 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5551 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Fix For: 2.1.2-beta Attachments: MAPREDUCE-5551.1.patch The non-default constructor is moved to the super class, but it cannot be inherited. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5530:
-
Status: Open (was: Patch Available)
Binary and source incompatibility in mapred.lib.CombineFileInputFormat
between branch-1 and branch-2
Key: MAPREDUCE-5530
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv1, mrv2
Affects Versions: 2.1.1-beta
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Attachments: MAPREDUCE-5530.patch, MAPREDUCE-5530.patch
{{mapred.lib.CombineFileInputFormat}} in branch-1 has this method:
{code:java}
protected boolean isSplitable(FileSystem fs, Path file)
{code}
In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of
{{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the
similar method:
{code:java}
protected boolean isSplitable(JobContext context, Path file)
{code}
This means that any code that subclasses
{{mapred.lib.CombineFileInputFormat}} and does not provide its own
implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}}
will not be binary or source compatible if it tries to call {{isSplitable}}
with a {{FileSystem}} argument anywhere (that is, if compiled against
branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2;
also, it won't even compile against branch-2).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782567#comment-13782567
]
Arun C Murthy commented on MAPREDUCE-5530:
--
[~rkanter] I agree the patch isn't perfect, but that's the only option we have
now... thought I wonder if we should just document this and not fix it...
thoughts?
IAC, couple of comments:
# Please mark
org.apache.hadoop.mapred.lib.CombineFileInputFormat.isSplitable(JobContext,
Path) as @Private
# Also add some documentation to the method with a pointer to this jira
explaining why end-users should never override
org.apache.hadoop.mapred.lib.CombineFileInputFormat.isSplitable(JobContext,
Path).
Thanks.
Binary and source incompatibility in mapred.lib.CombineFileInputFormat
between branch-1 and branch-2
Key: MAPREDUCE-5530
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv1, mrv2
Affects Versions: 2.1.1-beta
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Attachments: MAPREDUCE-5530.patch, MAPREDUCE-5530.patch
{{mapred.lib.CombineFileInputFormat}} in branch-1 has this method:
{code:java}
protected boolean isSplitable(FileSystem fs, Path file)
{code}
In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of
{{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the
similar method:
{code:java}
protected boolean isSplitable(JobContext context, Path file)
{code}
This means that any code that subclasses
{{mapred.lib.CombineFileInputFormat}} and does not provide its own
implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}}
will not be binary or source compatible if it tries to call {{isSplitable}}
with a {{FileSystem}} argument anywhere (that is, if compiled against
branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2;
also, it won't even compile against branch-2).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5533) Speculative execution does not function for reduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated MAPREDUCE-5533: - Attachment: MAPREDUCE-5533.5.patch Add testcases to test the logic Speculative execution does not function for reduce -- Key: MAPREDUCE-5533 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5533 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Reporter: Tassapol Athiapinya Assignee: Xuan Gong Attachments: MAPREDUCE-5533.1.patch, MAPREDUCE-5533.2.patch, MAPREDUCE-5533.3.patch, MAPREDUCE-5533.4.patch, MAPREDUCE-5533.5.patch We have sort job where reduce attempt does not send heartbeat in timely manner to application master. The AM should kick off another attempt to let job succeeds. What we find is the job fails and there is no speculation happening. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5533) Speculative execution does not function for reduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated MAPREDUCE-5533: - Status: Patch Available (was: Open) Speculative execution does not function for reduce -- Key: MAPREDUCE-5533 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5533 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Reporter: Tassapol Athiapinya Assignee: Xuan Gong Attachments: MAPREDUCE-5533.1.patch, MAPREDUCE-5533.2.patch, MAPREDUCE-5533.3.patch, MAPREDUCE-5533.4.patch, MAPREDUCE-5533.5.patch We have sort job where reduce attempt does not send heartbeat in timely manner to application master. The AM should kick off another attempt to let job succeeds. What we find is the job fails and there is no speculation happening. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[
https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782584#comment-13782584
]
Hadoop QA commented on MAPREDUCE-5536:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12606013/MAPREDUCE-5536.20131030.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapreduce.v2.hs.webapp.TestMapReduceTrackingUriPlugin
org.apache.hadoop.mapred.TestJobCleanup
The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapreduce.v2.TestUberAM
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4068//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4068//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs-plugins.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4068//console
This message is automatically generated.
mapreduce.jobhistory.webapp.https.address property is not respected
---
Key: MAPREDUCE-5536
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.1.1-beta
Reporter: Yesha Vora
Assignee: Omkar Vinit Joshi
Priority: Blocker
Attachments: MAPREDUCE-5536.20131027.1.patch,
MAPREDUCE-5536.20131030.1.patch, MAPREDUCE-5536.20131030.1.patch,
YARN-1240.20131025.1.patch
The jobhistory server starts on port defined by
mapreduce.jobhistory.webapp.address property instead
mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5536: - Status: Open (was: Patch Available) mapreduce.jobhistory.webapp.https.address property is not respected --- Key: MAPREDUCE-5536 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.1.1-beta Reporter: Yesha Vora Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: MAPREDUCE-5536.20131027.1.patch, MAPREDUCE-5536.20131030.1.patch, MAPREDUCE-5536.20131030.1.patch, YARN-1240.20131025.1.patch The jobhistory server starts on port defined by mapreduce.jobhistory.webapp.address property instead mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
