[jira] [Commented] (MAPREDUCE-5216) While using TextSplitter in DataDrivenDBInputformat, the lower limit (split start) always remains the same, for all splits.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049743#comment-14049743 ] Jack Xu commented on MAPREDUCE-5216: until now,the lastest version is hadoop-2.4.1 ,the problem is still there While using TextSplitter in DataDrivenDBInputformat, the lower limit (split start) always remains the same, for all splits. --- Key: MAPREDUCE-5216 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5216 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Gelesh Attachments: MAPREDUCE-5216.patch Original Estimate: 1h Remaining Estimate: 1h While using TextSplitter in DataDrivenDBInputformat, the lower limit (split start) always remains the same, for all splits. ie, Split 1 Start =A, End = M, Split 2 Start =A, End = P, Split 3 Start =A, End = S, instead of Split 1 Start =A, End = M, Split 2 Start =M, End = P, Split 3 Start =P, End = S, -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5216) While using TextSplitter in DataDrivenDBInputformat, the lower limit (split start) always remains the same, for all splits.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049768#comment-14049768
]
Hadoop QA commented on MAPREDUCE-5216:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582090/MAPREDUCE-5216.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.mapreduce.lib.db.TestSplitters
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4703//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4703//console
This message is automatically generated.
While using TextSplitter in DataDrivenDBInputformat, the lower limit (split
start) always remains the same, for all splits.
---
Key: MAPREDUCE-5216
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5216
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Gelesh
Attachments: MAPREDUCE-5216.patch
Original Estimate: 1h
Remaining Estimate: 1h
While using TextSplitter in DataDrivenDBInputformat, the lower limit (split
start) always remains the same, for all splits.
ie,
Split 1 Start =A, End = M, Split 2 Start =A, End = P, Split 3 Start =A, End =
S,
instead of
Split 1 Start =A, End = M, Split 2 Start =M, End = P, Split 3 Start =P, End =
S,
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-4064) Job History Link in RM UI is redirecting to the URL which contains Job Id twice
[
https://issues.apache.org/jira/browse/MAPREDUCE-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049810#comment-14049810
]
Zhijie Shen commented on MAPREDUCE-4064:
The bug still happens on trunk. I did some investigation:
1. When generating proxyTrackingUrl from originalTrackUrl, the original host
section is replaced by [proxy host]:[port]/proxy/[application id], while the
following sections are kept. For example,
{code}
http://192.168.1.108:19888/jobhistory/job/job_1404265212778_0003
{code}
in translated into
{code}
https://0.0.0.0:8088/proxy/application_1404265212778_0003/jobhistory/job/job_1404265212778_0003
{code}
2. On the other hand, in WebAppProxyServlet, [application id] section is used
to search for originalTrackingURL, and the following part is appended to the
end of originalTrackingURL. For example, we will see
{code}
http://192.168.1.108:19888/jobhistory/job/job_1404265212778_0001/jobhistory/job/job_1404265212778_0001
{code}
Fortunately, JHS will simply ignore the additional sections, such that we won't
see an apparent error.
Therefore, with the current logic, whenever AM provides a tracking url that has
more sections after host:port, it will be finally translated into an url that
has duplicate sections after host:port.
It seems we should either hide all the rest sections in proxyTrackingURL, or
not append them when WebAppProxyServlet recovers the originalTrackingURL.
Job History Link in RM UI is redirecting to the URL which contains Job Id
twice
---
Key: MAPREDUCE-4064
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4064
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4064-1.patch, MAPREDUCE-4064.patch
{code:xml}
http://xx.x.x.x:19888/jobhistory/job/job_1332435449546_0001/jobhistory/job/job_1332435449546_0001
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-2257) distcp can copy blocks in parallel
[ https://issues.apache.org/jira/browse/MAPREDUCE-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049997#comment-14049997 ] liuwei commented on MAPREDUCE-2257: --- since distcp has distcp2, is there a patch exits for distcp2 to copy blocks in parallel? distcp can copy blocks in parallel -- Key: MAPREDUCE-2257 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2257 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: 0.21.0 Reporter: dhruba borthakur Assignee: Mithun Radhakrishnan Attachments: MAPREDUCE-2257.patch The minimum unit of work for a distcp task is a file. We have files that are greater than 1 TB with a block size of 1 GB. If we use distcp to copy these files, the tasks either take a long long long time or finally fails. A better way for distcp would be to copy all the source blocks in parallel, and then stich the blocks back to files at the destination via the HDFS Concat API (HDFS-222) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5957) AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050054#comment-14050054
]
Sangjin Lee commented on MAPREDUCE-5957:
The gist of this issue is regarding the use of Configuration.getClass() and the
use of the thread context classloader (TCCL). Currently
MRApps.setJobClassLoader() sets both the configuration classloader and the TCCL
at the same time. So once setJobClassLoader() is called, it is made available
in both contexts.
MAPREDUCE-5751 was caused because the job classloader was made available *too
early as the TCCL*. This issue is caused because the job classloader is made
available *too late as the configuration classloader*.
The normal classloading scheme (one class initializing another class via normal
use or even Class.forName) is unaffected by this if my understanding is correct.
I see two possible approaches for this:
(1) separate the timing of setting the job classloader as the configuration
classloader and the TCCL
I think while setting the TCCL should be delayed as much as possible (i.e. the
current timing), the job classloader can be installed as the configuration
classloader much earlier. If the configuration loads a user class, that's
precisely what we need. If it loads a system class, the job classloader will
delegate anyhow. I don't think there is harm in setting the configuration
classloader early.
(2) set and unset the job classloader around the code that loads classes from
the configuration
Identify the code points in MRAppMaster where Configuration.getClass() is
needed, and set and unset the job classloader around them. Although this would
also solve this problem, the downside is that one needs to make a determination
that the job classloader is needed and set/unset it. This is potentially
brittle.
I think (1) is a more robust solution to this problem. Do you see an issue with
taking that approach?
I don't think the task (YarnChild) is affected by this.
AM throws ClassNotFoundException with job classloader enabled if custom
output format/committer is used
---
Key: MAPREDUCE-5957
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
With the job classloader enabled, the MR AM throws ClassNotFoundException if
a custom output format class is specified.
{noformat}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
... 8 more
Caused by: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5954) Optional exclusion of counters from getTaskReports
[
https://issues.apache.org/jira/browse/MAPREDUCE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maysam Yabandeh updated MAPREDUCE-5954:
---
Description:
MRClientService#getTaskReports returns the set of map or reduce tasks along
with their counters, which are quite large. For big jobs, the response could be
as large as 0.5 GB. This has a negative impact both on MRAppMaster and the
monitoring tool that invokes getTaskReports. This problem has led Pig users to
entirely disable getTaskReports for big jobs:
https://issues.apache.org/jira/browse/PIG-4043
Many monitoring tools, including ours, do not need the task counters when
invoking getTaskReports. Pig also does not make any use of task counters. Here
are the usages of Tasks in pig:
{code}
protected void getErrorMessages(TaskReport reports[], String type,
String msgs[] = reports[i].getDiagnostics();
if (HadoopShims.isJobFailed(reports[i])) {
{code}
and
{code}
protected long computeTimeSpent(TaskReport[] taskReports) {
long timeSpent = 0;
for (TaskReport r : taskReports) {
timeSpent += (r.getFinishTime() - r.getStartTime());
}
return timeSpent;
}
{code}
GetTaskReportsRequest can be augmented with an optional boolean with which the
monitoring tool can request excluding the counters form the response. This
minor change is very simple and yet makes many existing monitoring tools more
efficient.
was:
MRClientService.getTaskReport returns the set of map or reduce tasks along with
their counters, which are quite large. For big jobs, the response could be as
large as 0.5 GB. This has a negative impact both on MRAppMaster and the
monitoring tool that invokes getTaskReports. This problem has led Pig users to
entirely disable getTaskReports for big jobs:
https://issues.apache.org/jira/browse/PIG-4043
Many monitoring tools, including ours, do not need the task counters when
invoking getTaskReports. Pig also does not make any use of task counters. Here
are the usages of Tasks in pig:
{code}
protected void getErrorMessages(TaskReport reports[], String type,
String msgs[] = reports[i].getDiagnostics();
if (HadoopShims.isJobFailed(reports[i])) {
{code}
and
{code}
protected long computeTimeSpent(TaskReport[] taskReports) {
long timeSpent = 0;
for (TaskReport r : taskReports) {
timeSpent += (r.getFinishTime() - r.getStartTime());
}
return timeSpent;
}
{code}
GetTaskReportsRequest can be augmented with an optional boolean with which the
monitoring tool can request excluding the counters form the response. This
minor change is very simple and yet makes many existing monitoring tools more
efficient.
Optional exclusion of counters from getTaskReports
--
Key: MAPREDUCE-5954
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5954
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Maysam Yabandeh
Assignee: Maysam Yabandeh
MRClientService#getTaskReports returns the set of map or reduce tasks along
with their counters, which are quite large. For big jobs, the response could
be as large as 0.5 GB. This has a negative impact both on MRAppMaster and the
monitoring tool that invokes getTaskReports. This problem has led Pig users
to entirely disable getTaskReports for big jobs:
https://issues.apache.org/jira/browse/PIG-4043
Many monitoring tools, including ours, do not need the task counters when
invoking getTaskReports. Pig also does not make any use of task counters.
Here are the usages of Tasks in pig:
{code}
protected void getErrorMessages(TaskReport reports[], String type,
String msgs[] = reports[i].getDiagnostics();
if (HadoopShims.isJobFailed(reports[i])) {
{code}
and
{code}
protected long computeTimeSpent(TaskReport[] taskReports) {
long timeSpent = 0;
for (TaskReport r : taskReports) {
timeSpent += (r.getFinishTime() - r.getStartTime());
}
return timeSpent;
}
{code}
GetTaskReportsRequest can be augmented with an optional boolean with which
the monitoring tool can request excluding the counters form the response.
This minor change is very simple and yet makes many existing monitoring tools
more efficient.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5957) AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated MAPREDUCE-5957:
---
Target Version/s: 2.5.0
Status: Patch Available (was: Open)
AM throws ClassNotFoundException with job classloader enabled if custom
output format/committer is used
---
Key: MAPREDUCE-5957
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: MAPREDUCE-5957.patch
With the job classloader enabled, the MR AM throws ClassNotFoundException if
a custom output format class is specified.
{noformat}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
... 8 more
Caused by: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5957) AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated MAPREDUCE-5957:
---
Attachment: MAPREDUCE-5957.patch
Patch that implements approach (1)
AM throws ClassNotFoundException with job classloader enabled if custom
output format/committer is used
---
Key: MAPREDUCE-5957
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: MAPREDUCE-5957.patch
With the job classloader enabled, the MR AM throws ClassNotFoundException if
a custom output format class is specified.
{noformat}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
... 8 more
Caused by: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5957) AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050871#comment-14050871
]
Hadoop QA commented on MAPREDUCE-5957:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653711/MAPREDUCE-5957.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4704//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4704//console
This message is automatically generated.
AM throws ClassNotFoundException with job classloader enabled if custom
output format/committer is used
---
Key: MAPREDUCE-5957
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: MAPREDUCE-5957.patch
With the job classloader enabled, the MR AM throws ClassNotFoundException if
a custom output format class is specified.
{noformat}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
... 8 more
Caused by: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (MAPREDUCE-5956) MapReduce AM should not use maxAttempts to determine if this is the last retry
[
https://issues.apache.org/jira/browse/MAPREDUCE-5956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan reassigned MAPREDUCE-5956:
-
Assignee: Wangda Tan (was: Vinod Kumar Vavilapalli)
MapReduce AM should not use maxAttempts to determine if this is the last retry
--
Key: MAPREDUCE-5956
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5956
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: applicationmaster, mrv2
Reporter: Vinod Kumar Vavilapalli
Assignee: Wangda Tan
Priority: Blocker
Found this while reviewing YARN-2074. The problem is that after YARN-2074, we
don't count AM preemption towards AM failures on RM side, but MapReduce AM
itself checks the attempt id against the max-attempt count to determine if
this is the last attempt.
{code}
public void computeIsLastAMRetry() {
isLastAMRetry = appAttemptID.getAttemptId() = maxAppAttempts;
}
{code}
This causes issues w.r.t deletion of staging directory etc..
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Work started] (MAPREDUCE-5956) MapReduce AM should not use maxAttempts to determine if this is the last retry
[
https://issues.apache.org/jira/browse/MAPREDUCE-5956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on MAPREDUCE-5956 started by Wangda Tan.
MapReduce AM should not use maxAttempts to determine if this is the last retry
--
Key: MAPREDUCE-5956
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5956
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: applicationmaster, mrv2
Reporter: Vinod Kumar Vavilapalli
Assignee: Wangda Tan
Priority: Blocker
Found this while reviewing YARN-2074. The problem is that after YARN-2074, we
don't count AM preemption towards AM failures on RM side, but MapReduce AM
itself checks the attempt id against the max-attempt count to determine if
this is the last attempt.
{code}
public void computeIsLastAMRetry() {
isLastAMRetry = appAttemptID.getAttemptId() = maxAppAttempts;
}
{code}
This causes issues w.r.t deletion of staging directory etc..
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5900) Container preemption interpreted as task failures and eventually job failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-5900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5900: - Status: Open (was: Patch Available) Container preemption interpreted as task failures and eventually job failures -- Key: MAPREDUCE-5900 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5900 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mr-am, mrv2 Affects Versions: 2.4.1 Reporter: Mayank Bansal Assignee: Mayank Bansal Attachments: MAPREDUCE-5900-1.patch, MAPREDUCE-5900-branch-241-2.patch, MAPREDUCE-5900-trunk-1.patch, MAPREDUCE-5900-trunk-2.patch, MAPREDUCE-5900-trunk-3.patch We have Added preemption exit code needs to be incorporated MR needs to recognize the special exit code value of -102 and interpret it as a container being killed instead of a container failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5956) MapReduce AM should not use maxAttempts to determine if this is the last retry
[
https://issues.apache.org/jira/browse/MAPREDUCE-5956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050924#comment-14050924
]
Wangda Tan commented on MAPREDUCE-5956:
---
Assigned it to me, already started working on this ..
MapReduce AM should not use maxAttempts to determine if this is the last retry
--
Key: MAPREDUCE-5956
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5956
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: applicationmaster, mrv2
Reporter: Vinod Kumar Vavilapalli
Assignee: Wangda Tan
Priority: Blocker
Found this while reviewing YARN-2074. The problem is that after YARN-2074, we
don't count AM preemption towards AM failures on RM side, but MapReduce AM
itself checks the attempt id against the max-attempt count to determine if
this is the last attempt.
{code}
public void computeIsLastAMRetry() {
isLastAMRetry = appAttemptID.getAttemptId() = maxAppAttempts;
}
{code}
This causes issues w.r.t deletion of staging directory etc..
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5900) Container preemption interpreted as task failures and eventually job failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-5900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5900: - Attachment: MAPREDUCE-5900-trunk-3.patch Thanks [~zjshen] for the review. Update Comments. Thanks, Mayank Container preemption interpreted as task failures and eventually job failures -- Key: MAPREDUCE-5900 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5900 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mr-am, mrv2 Affects Versions: 2.4.1 Reporter: Mayank Bansal Assignee: Mayank Bansal Attachments: MAPREDUCE-5900-1.patch, MAPREDUCE-5900-branch-241-2.patch, MAPREDUCE-5900-trunk-1.patch, MAPREDUCE-5900-trunk-2.patch, MAPREDUCE-5900-trunk-3.patch We have Added preemption exit code needs to be incorporated MR needs to recognize the special exit code value of -102 and interpret it as a container being killed instead of a container failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5900) Container preemption interpreted as task failures and eventually job failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-5900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5900: - Status: Patch Available (was: Open) Container preemption interpreted as task failures and eventually job failures -- Key: MAPREDUCE-5900 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5900 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mr-am, mrv2 Affects Versions: 2.4.1, trunk Reporter: Mayank Bansal Assignee: Mayank Bansal Attachments: MAPREDUCE-5900-1.patch, MAPREDUCE-5900-branch-241-2.patch, MAPREDUCE-5900-trunk-1.patch, MAPREDUCE-5900-trunk-2.patch, MAPREDUCE-5900-trunk-3.patch We have Added preemption exit code needs to be incorporated MR needs to recognize the special exit code value of -102 and interpret it as a container being killed instead of a container failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5900) Container preemption interpreted as task failures and eventually job failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-5900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5900: - Affects Version/s: trunk Container preemption interpreted as task failures and eventually job failures -- Key: MAPREDUCE-5900 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5900 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mr-am, mrv2 Affects Versions: trunk, 2.4.1 Reporter: Mayank Bansal Assignee: Mayank Bansal Attachments: MAPREDUCE-5900-1.patch, MAPREDUCE-5900-branch-241-2.patch, MAPREDUCE-5900-trunk-1.patch, MAPREDUCE-5900-trunk-2.patch, MAPREDUCE-5900-trunk-3.patch We have Added preemption exit code needs to be incorporated MR needs to recognize the special exit code value of -102 and interpret it as a container being killed instead of a container failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5900) Container preemption interpreted as task failures and eventually job failures
[
https://issues.apache.org/jira/browse/MAPREDUCE-5900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050962#comment-14050962
]
Hadoop QA commented on MAPREDUCE-5900:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653731/MAPREDUCE-5900-trunk-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4705//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4705//console
This message is automatically generated.
Container preemption interpreted as task failures and eventually job failures
--
Key: MAPREDUCE-5900
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5900
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: applicationmaster, mr-am, mrv2
Affects Versions: trunk, 2.4.1
Reporter: Mayank Bansal
Assignee: Mayank Bansal
Attachments: MAPREDUCE-5900-1.patch,
MAPREDUCE-5900-branch-241-2.patch, MAPREDUCE-5900-trunk-1.patch,
MAPREDUCE-5900-trunk-2.patch, MAPREDUCE-5900-trunk-3.patch
We have Added preemption exit code needs to be incorporated
MR needs to recognize the special exit code value of -102 and interpret it as
a container being killed instead of a container failure.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5900) Container preemption interpreted as task failures and eventually job failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-5900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050967#comment-14050967 ] Zhijie Shen commented on MAPREDUCE-5900: +1 will commit it Container preemption interpreted as task failures and eventually job failures -- Key: MAPREDUCE-5900 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5900 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mr-am, mrv2 Affects Versions: trunk, 2.4.1 Reporter: Mayank Bansal Assignee: Mayank Bansal Attachments: MAPREDUCE-5900-1.patch, MAPREDUCE-5900-branch-241-2.patch, MAPREDUCE-5900-trunk-1.patch, MAPREDUCE-5900-trunk-2.patch, MAPREDUCE-5900-trunk-3.patch We have Added preemption exit code needs to be incorporated MR needs to recognize the special exit code value of -102 and interpret it as a container being killed instead of a container failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5900) Container preemption interpreted as task failures and eventually job failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-5900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5900: --- Resolution: Fixed Fix Version/s: 2.5.0 Status: Resolved (was: Patch Available) Committed to trunk and branch-2. Thanks Mayank for the patch and Wangda for review! Container preemption interpreted as task failures and eventually job failures -- Key: MAPREDUCE-5900 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5900 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mr-am, mrv2 Affects Versions: trunk, 2.4.1 Reporter: Mayank Bansal Assignee: Mayank Bansal Fix For: 2.5.0 Attachments: MAPREDUCE-5900-1.patch, MAPREDUCE-5900-branch-241-2.patch, MAPREDUCE-5900-trunk-1.patch, MAPREDUCE-5900-trunk-2.patch, MAPREDUCE-5900-trunk-3.patch We have Added preemption exit code needs to be incorporated MR needs to recognize the special exit code value of -102 and interpret it as a container being killed instead of a container failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5900) Container preemption interpreted as task failures and eventually job failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-5900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050989#comment-14050989 ] Hudson commented on MAPREDUCE-5900: --- SUCCESS: Integrated in Hadoop-trunk-Commit #5815 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5815/]) MAPREDUCE-5900. Changed to the interpret container preemption exit code as a task attempt killing event. Contributed by Mayank Bansal. (zjshen: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1607512) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java Container preemption interpreted as task failures and eventually job failures -- Key: MAPREDUCE-5900 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5900 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mr-am, mrv2 Affects Versions: trunk, 2.4.1 Reporter: Mayank Bansal Assignee: Mayank Bansal Fix For: 2.5.0 Attachments: MAPREDUCE-5900-1.patch, MAPREDUCE-5900-branch-241-2.patch, MAPREDUCE-5900-trunk-1.patch, MAPREDUCE-5900-trunk-2.patch, MAPREDUCE-5900-trunk-3.patch We have Added preemption exit code needs to be incorporated MR needs to recognize the special exit code value of -102 and interpret it as a container being killed instead of a container failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5940) Avoid negative elapsed time in JHS/MRAM web UI and services
[ https://issues.apache.org/jira/browse/MAPREDUCE-5940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14051046#comment-14051046 ] Junping Du commented on MAPREDUCE-5940: --- Thanks [~zjshen] to work on this and [~vvasudev] to review! This seems to be an interesting issue. I think a better/clean way to fix it is to update elapsed() method: If System.currentTimeMillis() started, then we can return -1 or 0 instead (and log a warn that clock not getting synchronized). Thoughts? Avoid negative elapsed time in JHS/MRAM web UI and services --- Key: MAPREDUCE-5940 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5940 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mr-am, webapps Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5940.1.patch Recently we observed a rare bug that an elapsed time of a reducer is going to be negative on JHS web UI and via REST APIs. While the real reason for this bug seems to be clock asynchronization on different hosts, the web frontend should have masked the negative values. However, in the current code, *org.apache.hadoop.mapreduce.v2.app.webapp.dao.** only check whether the elapsed time is -1 or not. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5940) Avoid negative elapsed time in JHS/MRAM web UI and services
[ https://issues.apache.org/jira/browse/MAPREDUCE-5940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14051048#comment-14051048 ] Junping Du commented on MAPREDUCE-5940: --- Also adding a test in TestTimes.java could be a good idea. Avoid negative elapsed time in JHS/MRAM web UI and services --- Key: MAPREDUCE-5940 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5940 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mr-am, webapps Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5940.1.patch Recently we observed a rare bug that an elapsed time of a reducer is going to be negative on JHS web UI and via REST APIs. While the real reason for this bug seems to be clock asynchronization on different hosts, the web frontend should have masked the negative values. However, in the current code, *org.apache.hadoop.mapreduce.v2.app.webapp.dao.** only check whether the elapsed time is -1 or not. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5940) Avoid negative elapsed time in JHS/MRAM web UI and services
[ https://issues.apache.org/jira/browse/MAPREDUCE-5940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14051073#comment-14051073 ] Devaraj K commented on MAPREDUCE-5940: -- Silently making the elapsed time as 0 when it is negative may lead to hiding the bugs related to elapsed time. Adding a warning/info message before making it as 0 would help to diagnose/find out the issues if any. Avoid negative elapsed time in JHS/MRAM web UI and services --- Key: MAPREDUCE-5940 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5940 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mr-am, webapps Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5940.1.patch Recently we observed a rare bug that an elapsed time of a reducer is going to be negative on JHS web UI and via REST APIs. While the real reason for this bug seems to be clock asynchronization on different hosts, the web frontend should have masked the negative values. However, in the current code, *org.apache.hadoop.mapreduce.v2.app.webapp.dao.** only check whether the elapsed time is -1 or not. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5957) AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14051080#comment-14051080
]
Sangjin Lee commented on MAPREDUCE-5957:
Tested with a pseudo-distributed cluster by specifying a custom output format
class as well as a custom speculator class. Confirmed that
ClassNotFoundException is thrown without the fix and verified the issue goes
away with the fix.
AM throws ClassNotFoundException with job classloader enabled if custom
output format/committer is used
---
Key: MAPREDUCE-5957
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: MAPREDUCE-5957.patch
With the job classloader enabled, the MR AM throws ClassNotFoundException if
a custom output format class is specified.
{noformat}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
... 8 more
Caused by: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5044) Have AM trigger jstack on task attempts that timeout before killing them
[ https://issues.apache.org/jira/browse/MAPREDUCE-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5044: Target Version/s: 2.6.0 (was: 2.5.0) Have AM trigger jstack on task attempts that timeout before killing them Key: MAPREDUCE-5044 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5044 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Gera Shegalov Attachments: MAPREDUCE-5044.v01.patch, MAPREDUCE-5044.v02.patch, MAPREDUCE-5044.v03.patch, MAPREDUCE-5044.v04.patch, MAPREDUCE-5044.v05.patch, MAPREDUCE-5044.v06.patch, Screen Shot 2013-11-12 at 1.05.32 PM.png, Screen Shot 2013-11-12 at 1.06.04 PM.png When an AM expires a task attempt it would be nice if it triggered a jstack output via SIGQUIT before killing the task attempt. This would be invaluable for helping users debug their hung tasks, especially if they do not have shell access to the nodes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-4758) jobhistory web ui not showing correct # failed reducers
[ https://issues.apache.org/jira/browse/MAPREDUCE-4758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-4758: Target Version/s: 2.6.0 (was: 3.0.0, 2.5.0) jobhistory web ui not showing correct # failed reducers --- Key: MAPREDUCE-4758 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4758 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.4 Reporter: Thomas Graves Priority: Critical we had a job fail due to a reducer failing 4 times. Unfortunately the job history UI didn't show this particular failed reducer which lead to confusion as to why the job failed. This reducer failed to launch all 4 task attempts with a Token Expiration error and the jobhistory file only gets an event when the task attempt transitions to launched. The webapp JobInfo object only counts the task attempts in the jobhistory file to display under the Attempt Type table, so since this task didn't have an attempt with it, it did show it on the UI. We need to reconcile the task list with the task attempts or also shows more stats for the tasks vs task attempts. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5885) build/test/test.mapred.spill causes release audit warnings
[ https://issues.apache.org/jira/browse/MAPREDUCE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5885: Target Version/s: 2.6.0 (was: 2.5.0) build/test/test.mapred.spill causes release audit warnings -- Key: MAPREDUCE-5885 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5885 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: trunk Reporter: Jason Lowe Assignee: Chen He Attachments: MAPREDUCE-5885.patch, MAPREDUCE-5885.patch Multiple unit tests are creating files under hadoop-mapreduce-client-jobclient/build/test/test.mapred.spill which are causing release audit warnings during Jenkins patch precommit builds. In addition to being in a poor location for test output and not cleaning up after the test, there are multiple tests using this location which will cause conflicts if tests are run in parallel. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5906) Inconsistent configuration in property mapreduce.reduce.shuffle.input.buffer.percent
[
https://issues.apache.org/jira/browse/MAPREDUCE-5906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-5906:
Target Version/s: 2.6.0 (was: 2.5.0)
Inconsistent configuration in property
mapreduce.reduce.shuffle.input.buffer.percent
--
Key: MAPREDUCE-5906
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5906
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.4.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Minor
Labels: newbie
Attachments: MAPREDUCE-5906.patch
In MergeManagerImpl.java, the default value of
MRJobConfig.SHUFFLE_INPUT_BUFFER_PERCENT
(=mapreduce.reduce.shuffle.input.buffer.percent) looks 0.90.
{code}
final float maxInMemCopyUse =
jobConf.getFloat(MRJobConfig.SHUFFLE_INPUT_BUFFER_PERCENT, 0.90f);
{code}
However, the actual default value is 0.70 in mapred-default.xml.
{code}
namemapreduce.reduce.shuffle.input.buffer.percent/name
value0.70/value
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-2398) MRBench: setting the baseDir parameter has no effect
[
https://issues.apache.org/jira/browse/MAPREDUCE-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-2398:
Target Version/s: 2.6.0 (was: 1.3.0, 2.5.0)
MRBench: setting the baseDir parameter has no effect
Key: MAPREDUCE-2398
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2398
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: benchmarks
Affects Versions: 0.20.2, 2.3.0
Reporter: Michael Noll
Assignee: Yang Hao
Priority: Minor
Attachments: MAPREDUCE-2398-trunk.patch, MAPREDUCE-2398_0.20.2.patch,
MAPREDUCE-2398_v2-0.20.203.0.patch, MAPREDUCE-2398_v2-trunk.patch
The optional {{-baseDir}} parameter lets user specify the base DFS path for
output/input of MRBench.
However, the two private variables {{INPUT_DIR}} and {{OUTPUT_DIR}}
(MRBench.java) are not updated in the case that the default value of
{{-baseDir}} is actually overwritten by the user. Hence any input and output
is always written to the default locations ({{/benchmarks/MRBench/...}}),
even though the user-supplied location for {{-baseDir}} is created (and
eventually deleted again) on HDFS.
The bug affects at least Hadoop 0.20.2 and the current trunk (r1082703) as of
March 21, 2011.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-4640) jobHistoryEventHandler doesn't fix done_intermediate parent directory permissions
[ https://issues.apache.org/jira/browse/MAPREDUCE-4640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-4640: Target Version/s: 2.6.0 (was: 3.0.0, 2.5.0) jobHistoryEventHandler doesn't fix done_intermediate parent directory permissions - Key: MAPREDUCE-4640 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4640 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 0.23.3 Reporter: Thomas Graves The JobHistoryEventHandler will create the done intermediate directory and its parents if it doesn't exist, but it doesn't properly set the parent directory permissions when the default umask is restrictive - 077. Normally if you start the history server, it properly creates the directories with the right permissions but if you don't start it before you run a job the parent directories get wrong permissions.. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-4191) capacity scheduler: job unexpectedly exceeds queue capacity limit by one task
[ https://issues.apache.org/jira/browse/MAPREDUCE-4191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-4191: Target Version/s: 2.6.0 (was: 3.0.0, 2.5.0) capacity scheduler: job unexpectedly exceeds queue capacity limit by one task - Key: MAPREDUCE-4191 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4191 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, scheduler Affects Versions: 0.23.3 Reporter: Thomas Graves Assignee: Thomas Graves While testing the queue capacity limits, it appears that the job can exceed the queue capacity limit by one task while the user limit factor is 1. It's not clear to me why this is. Here is the steps to reproduce: 1) set yarn.app.mapreduce.am.resource.mb to 2048 (default value) 2) set yarn.scheduler.capacity.root.default.user-limit-factor to 1.0 (default) 3) set yarn.scheduler.capacity.root.default.capacity to 90 (%) 4) For a cluster with capacity of 56G, 90% rounded up is 51. 5) submit a job with large number of tasks, each task using 1G memory. 6) webui shows that the used resource is 52 G, which is 92.9% of the cluster capacity (instead of the expected 90%), and 103.2% of the queue capacity (instead of the expected 100%). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5957) AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-5957:
Target Version/s: 2.6.0 (was: 2.5.0)
AM throws ClassNotFoundException with job classloader enabled if custom
output format/committer is used
---
Key: MAPREDUCE-5957
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: MAPREDUCE-5957.patch
With the job classloader enabled, the MR AM throws ClassNotFoundException if
a custom output format class is specified.
{noformat}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
... 8 more
Caused by: java.lang.ClassNotFoundException: Class
com.foo.test.TestOutputFormat not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5796) Use current version of the archive name in DistributedCacheDeploy document
[
https://issues.apache.org/jira/browse/MAPREDUCE-5796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-5796:
Target Version/s: 2.6.0 (was: 2.5.0)
Use current version of the archive name in DistributedCacheDeploy document
--
Key: MAPREDUCE-5796
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5796
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: documentation
Affects Versions: 2.3.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Minor
Labels: newbie
Attachments: MAPREDUCE-5796.patch
The archive name is {{hadoop-mapreduce-2.1.1.tar.gz}} in
DistributedCacheDeploy document but Hadoop 2.1.1 is not released. It should
be fixed to {{hadoop-mapreduce-$\{project.version\}.tar.gz}} to show the
current version.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5756) CombineFileInputFormat.getSplits() including directories in its results
[
https://issues.apache.org/jira/browse/MAPREDUCE-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-5756:
Target Version/s: 2.6.0 (was: 2.5.0)
CombineFileInputFormat.getSplits() including directories in its results
---
Key: MAPREDUCE-5756
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5756
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Jason Dere
Attachments: MAPREDUCE-5756.1.patch, MAPREDUCE-5756.2.patch
Trying to track down HIVE-6401, where we see some is not a file errors
because getSplits() is giving us directories. I believe the culprit is
FileInputFormat.listStatus():
{code}
if (recursive stat.isDirectory()) {
addInputPathRecursively(result, fs, stat.getPath(),
inputFilter);
} else {
result.add(stat);
}
{code}
Which seems to be allowing directories to be added to the results if
recursive is false. Is this meant to return directories? If not, I think it
should look like this:
{code}
if (stat.isDirectory()) {
if (recursive) {
addInputPathRecursively(result, fs, stat.getPath(),
inputFilter);
}
} else {
result.add(stat);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5889) Deprecate FileInputFormat.setInputPaths(Job, String) and FileInputFormat.addInputPaths(Job, String)
[
https://issues.apache.org/jira/browse/MAPREDUCE-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-5889:
Target Version/s: 2.6.0 (was: 2.5.0)
Deprecate FileInputFormat.setInputPaths(Job, String) and
FileInputFormat.addInputPaths(Job, String)
---
Key: MAPREDUCE-5889
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5889
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Minor
Labels: newbie
Attachments: MAPREDUCE-5889.3.patch, MAPREDUCE-5889.patch,
MAPREDUCE-5889.patch
{{FileInputFormat.setInputPaths(Job job, String commaSeparatedPaths)}} and
{{FileInputFormat.addInputPaths(Job job, String commaSeparatedPaths)}} fail
to parse commaSeparatedPaths if a comma is included in the file path. (e.g.
Path: {{/path/file,with,comma}})
We should deprecate these methods and document to use {{setInputPaths(Job
job, Path... inputPaths)}} and {{addInputPaths(Job job, Path... inputPaths)}}
instead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5439) mapred-default.xml has missing properties
[ https://issues.apache.org/jira/browse/MAPREDUCE-5439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5439: Target Version/s: 2.6.0 (was: 2.5.0) mapred-default.xml has missing properties - Key: MAPREDUCE-5439 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5439 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.1.0-beta Reporter: Siddharth Wagle Properties that need to be added: mapreduce.map.memory.mb mapreduce.map.java.opts mapreduce.reduce.memory.mb mapreduce.reduce.java.opts Properties that need to be fixed: mapred.child.java.opts should not be in mapred-default. yarn.app.mapreduce.am.command-opts description needs fixing -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5848) MapReduce counts forcibly preempted containers as FAILED
[ https://issues.apache.org/jira/browse/MAPREDUCE-5848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5848: Target Version/s: 2.6.0 (was: 2.5.0) MapReduce counts forcibly preempted containers as FAILED Key: MAPREDUCE-5848 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5848 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.1.0-beta Reporter: Carlo Curino Assignee: Subramaniam Venkatraman Krishnan Attachments: MR-5848.patch, MR-5848.patch, YARN-1958.patch The MapReduce AM is considering a forcibly preempted container as FAILED, while I think it should be considered as KILLED (i.e., not count against the maximum number of failures). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5831) Old MR client is not compatible with new MR application
[
https://issues.apache.org/jira/browse/MAPREDUCE-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-5831:
Target Version/s: 2.6.0 (was: 2.5.0)
Old MR client is not compatible with new MR application
---
Key: MAPREDUCE-5831
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5831
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client, mr-am
Affects Versions: 2.2.0, 2.3.0
Reporter: Zhijie Shen
Assignee: Tan, Wangda
Priority: Critical
Recently, we saw the following scenario:
1. The user setup a cluster of hadoop 2.3., which contains YARN 2.3 and MR
2.3.
2. The user client on a machine that MR 2.2 is installed and in the classpath.
Then, when the user submitted a simple wordcount job, he saw the following
message:
{code}
16:00:41,027 INFO main mapreduce.Job:1345 - map 100% reduce 100%
16:00:41,036 INFO main mapreduce.Job:1356 - Job job_1396468045458_0006
completed successfully
16:02:20,535 WARN main mapreduce.JobRunner:212 - Cannot start job
[wordcountJob]
java.lang.IllegalArgumentException: No enum constant
org.apache.hadoop.mapreduce.JobCounter.MB_MILLIS_REDUCES
at java.lang.Enum.valueOf(Enum.java:236)
at
org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.valueOf(FrameworkCounterGroup.java:148)
at
org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.findCounter(FrameworkCounterGroup.java:182)
at
org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:154)
at
org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:240)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:370)
at
org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1289)
. . .
{code}
The problem is that the wordcount job was running on one or more than one
nodes of the YARN cluster, where MR 2.3 libs were installed, and
JobCounter.MB_MILLIS_REDUCES is available in the counters. On the other side,
due to the classpath setting, the client was likely to run with MR 2.2 libs.
After the client retrieved the counters from MR AM, it tried to construct the
Counter object with the received counter name. Unfortunately, the enum didn't
exist in the client's classpath. Therefore, No enum constant exception is
thrown here.
JobCounter.MB_MILLIS_REDUCES is brought to MR2 via MAPREDUCE-5464 since
Hadoop 2.3.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5267) History server should be more robust when cleaning old jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5267: Target Version/s: 2.6.0 (was: 2.5.0) History server should be more robust when cleaning old jobs --- Key: MAPREDUCE-5267 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5267 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobhistoryserver Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Jason Lowe Assignee: Maysam Yabandeh Attachments: MAPREDUCE-5267.patch, MAPREDUCE-5267.patch Ran across a situation where an admin user had accidentally created a directory in one of the date directories under /mapred/history/done/ that was not readable by the historyserver user. That effectively prevented the history server from cleaning any jobs from that date forward, as it hit an IOException trying to scan the directory and that aborted the entire clean process. The history server should localize IOException handling to the directory/file being processed and move on to the next entry in the list rather than aborting the entire cleaning process. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5878) some standard JDK APIs are not part of system classes defaults
[ https://issues.apache.org/jira/browse/MAPREDUCE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5878: Target Version/s: 2.6.0 (was: 3.0.0, 2.5.0) some standard JDK APIs are not part of system classes defaults -- Key: MAPREDUCE-5878 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5878 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.4.0 Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: MAPREDUCE-5878.patch There are some standard JDK APIs that are not part of the mapreduce.job.classloader.system.classes property value. Currently the default value covers only java.,javax. from the JDK. However, there are other APIs that are as well-established as these, such as org.w3c.dom and org.xml.sax. In other similar systems (e.g. OSGi), it is a standard practice to include both of these packages in the system classes. We should add these to the default values. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-4711) Append time elapsed since job-start-time for finished tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-4711: Target Version/s: 2.6.0 (was: 3.0.0, 2.5.0) Append time elapsed since job-start-time for finished tasks --- Key: MAPREDUCE-4711 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4711 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 0.23.3 Reporter: Ravi Prakash Attachments: MAPREDUCE-4711.branch-0.23.patch In 0.20.x/1.x, the analyze job link gave this information bq. The last Map task task_sometask finished at (relative to the Job launch time): 5/10 20:23:10 (1hrs, 27mins, 54sec) The time it took for the last task to finish needs to be calculated mentally in 0.23. I believe we should print it next to the finish time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5800) Use Job#getInstance instead of deprecated constructors
[
https://issues.apache.org/jira/browse/MAPREDUCE-5800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-5800:
Target Version/s: 2.6.0 (was: 2.5.0)
Use Job#getInstance instead of deprecated constructors
--
Key: MAPREDUCE-5800
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5800
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.3.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Minor
Labels: newbie
Attachments: MAPREDUCE-5800.2.patch, MAPREDUCE-5800.patch
There're some methods calling deprecated constructors such as {{new Job()}},
which causes javac warnings.
We should use {{Job.getInstance()}} to get an instance.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-3182) loadgen ignores -m command line when writing random data
[ https://issues.apache.org/jira/browse/MAPREDUCE-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-3182: Target Version/s: 2.6.0 (was: 2.5.0) loadgen ignores -m command line when writing random data Key: MAPREDUCE-3182 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3182 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0, 2.3.0 Reporter: Jonathan Eagles Assignee: Chen He Attachments: MAPREDUCE-3182.patch If no input directories are specified, loadgen goes into a special mode where random data is generated and written. In that mode, setting the number of mappers (-m command line option) is overridden by a calculation. Instead, it should take into consideration the user specified number of mappers and fall back to the calculation. In addition, update the documentation as well to match the new behavior in the code. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5950) incorrect description in distcp2 document
[
https://issues.apache.org/jira/browse/MAPREDUCE-5950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-5950:
Target Version/s: 2.6.0 (was: 2.5.0)
incorrect description in distcp2 document
-
Key: MAPREDUCE-5950
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5950
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: documentation
Affects Versions: 1.2.1, 2.4.0
Reporter: Yongjun Zhang
Assignee: Akira AJISAKA
Labels: newbie
Attachments: HDFS-6502.2.patch, HDFS-6502.3.patch, HDFS-6502.4.patch,
HDFS-6502.patch
In http://hadoop.apache.org/docs/r1.2.1/distcp2.html#UpdateAndOverwrite
The first statement of the Update and Overwrite section says:
{quote}
-update is used to copy files from source that don't exist at the target, or
have different contents. -overwrite overwrites target-files even if they
exist at the source, or have the same contents.
{quote}
The Command Line Options table says :
{quote}
-overwrite: Overwrite destination
-update: Overwrite if src size different from dst size
{quote}
Based on the implementation, making the following modification would be more
accurate:
The first statement of the Update and Overwrite section:
{code}
-update is used to copy files from source that don't exist at the target, or
have different contents. -overwrite overwrites target-files if they exist at
the target.
{code}
The Command Line Options table:
{code}
-overwrite: Overwrite destination
-update: Overwrite destination if source and destination have different
contents
{code}
Thanks.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5866) TestFixedLengthInputFormat fails in windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-5866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5866: Target Version/s: 2.6.0 (was: 3.0.0, 2.5.0) TestFixedLengthInputFormat fails in windows --- Key: MAPREDUCE-5866 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5866 Project: Hadoop Map/Reduce Issue Type: Test Components: client, test Affects Versions: 3.0.0, 2.4.0 Reporter: Varun Vasudev Assignee: Varun Vasudev Attachments: apache-mapreduce-5866.1.patch, apache-yarn-1992.0.patch org.apache.hadoop.mapred.TextFixedLengthInputFormat and org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat tests fail in Windows -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5362) clean up POM dependencies
[ https://issues.apache.org/jira/browse/MAPREDUCE-5362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5362: Target Version/s: 2.6.0 (was: 2.5.0) clean up POM dependencies - Key: MAPREDUCE-5362 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5362 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: MAPREDUCE-5362.patch, mr-5362-0.patch Intermediate 'pom' modules define dependencies inherited by leaf modules. This is causing issues in intellij IDE. We should normalize the leaf modules like in common, hdfs and tools where all dependencies are defined in each leaf module and the intermediate 'pom' module do not define any dependency. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-3550) RM web proxy should handle redirect of web services urls
[ https://issues.apache.org/jira/browse/MAPREDUCE-3550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-3550: Target Version/s: 2.6.0 (was: 3.0.0, 2.5.0) RM web proxy should handle redirect of web services urls Key: MAPREDUCE-3550 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3550 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves the RM web proxy should handle the web services urls added in MAPREDUCE-2863. The proxy does handle passing the web service urls to the AM, it just doesn't handle redirecting it after the AM goes away. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5597) Missing alternatives in javadocs for deprecated constructors in mapreduce.Job
[ https://issues.apache.org/jira/browse/MAPREDUCE-5597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5597: Target Version/s: 2.6.0 (was: 2.5.0) Missing alternatives in javadocs for deprecated constructors in mapreduce.Job - Key: MAPREDUCE-5597 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5597 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, documentation, job submission Affects Versions: 2.2.0 Reporter: Christopher Tubbs Assignee: Akira AJISAKA Priority: Minor Labels: newbie Attachments: MAPREDUCE-5597.2.patch, MAPREDUCE-5597.patch Deprecated API, such as `new Job()` don't have javadocs explaining what the alternatives are. (It'd also help if the new methods had @since tags to help determine if one could safely use that API on older versions at runtime.) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5891) Improved shuffle error handling across NM restarts
[ https://issues.apache.org/jira/browse/MAPREDUCE-5891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5891: Target Version/s: 2.6.0 (was: 2.5.0) Improved shuffle error handling across NM restarts -- Key: MAPREDUCE-5891 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5891 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.5.0 Reporter: Jason Lowe To minimize the number of map fetch failures reported by reducers across an NM restart it would be nice if reducers only reported a fetch failure after trying for at specified period of time to retrieve the data. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5931) Validate SleepJob command line parameters
[
https://issues.apache.org/jira/browse/MAPREDUCE-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-5931:
Target Version/s: 2.6.0 (was: 2.5.0)
Validate SleepJob command line parameters
-
Key: MAPREDUCE-5931
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5931
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 1.2.1, 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Priority: Minor
Attachments: MAPREDUCE-5931.v01.patch, MAPREDUCE-5931.v02.patch,
MAPREDUCE-5931.v03.patch
This is a minor issue per se. I had a typo in my script specifying a negative
number of reducers for the SleepJob. It results in the exception that is far
from the root cause, and appeared as a serious issue with the map-side sort.
{noformat}
2014-06-17 21:42:48,072 INFO [main] org.apache.hadoop.mapred.MapTask:
Ignoring exception during close for
org.apache.hadoop.mapred.MapTask$NewOutputCollector@972141f
java.lang.NullPointerException
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1447)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:700)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1990)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:774)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:173)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1626)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
2014-06-17 21:42:48,075 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.IllegalArgumentException
at java.nio.ByteBuffer.allocate(ByteBuffer.java:330)
at org.apache.hadoop.mapred.SpillRecord.init(SpillRecord.java:51)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1824)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1484)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:700)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:173)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1626)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5883) Total megabyte-seconds in job counters is slightly misleading
[ https://issues.apache.org/jira/browse/MAPREDUCE-5883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5883: Target Version/s: 2.6.0 (was: 3.0.0, 2.5.0) Total megabyte-seconds in job counters is slightly misleading --- Key: MAPREDUCE-5883 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5883 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.4.0 Reporter: Nathan Roberts Assignee: Nathan Roberts Priority: Minor Attachments: MAPREDUCE-5883.patch The following counters are in milliseconds so megabyte-seconds might be better stated as megabyte-milliseconds MB_MILLIS_MAPS.name= Total megabyte-seconds taken by all map tasks MB_MILLIS_REDUCES.name=Total megabyte-seconds taken by all reduce tasks VCORES_MILLIS_MAPS.name= Total vcore-seconds taken by all map tasks VCORES_MILLIS_REDUCES.name=Total vcore-seconds taken by all reduce tasks -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-3483) CapacityScheduler reserves container on same node as AM but can't ever use due to never enough avail memory
[ https://issues.apache.org/jira/browse/MAPREDUCE-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-3483: Target Version/s: 2.6.0 (was: 3.0.0, 2.5.0) CapacityScheduler reserves container on same node as AM but can't ever use due to never enough avail memory --- Key: MAPREDUCE-3483 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3483 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.3.0 Reporter: Thomas Graves Assignee: Arun C Murthy Saw a case where a job was stuck trying to get reducers. The issue is the capacity scheduler reserved a container on the same node as the application master but there wasn't ever enough memory to run the reducer on that node. Node total memory was 8G, Reducer needed 8G, AM was using 2G. This particular job had 10 reducers and it was stuck waiting on the one because the AM + reserved reducer memory was already over the queue limit. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5817) mappers get rescheduled on node transition even after all reducers are completed
[ https://issues.apache.org/jira/browse/MAPREDUCE-5817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5817: Target Version/s: 2.6.0 (was: 2.5.0) mappers get rescheduled on node transition even after all reducers are completed Key: MAPREDUCE-5817 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5817 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.3.0 Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: mapreduce-5817.patch We're seeing a behavior where a job runs long after all reducers were already finished. We found that the job was rescheduling and running a number of mappers beyond the point of reducer completion. In one situation, the job ran for some 9 more hours after all reducers completed! This happens because whenever a node transition (to an unusable state) comes into the app master, it just reschedules all mappers that already ran on the node in all cases. Therefore, if any node transition has a potential to extend the job period. Once this window opens, another node transition can prolong it, and this can happen indefinitely in theory. If there is some instability in the pool (unhealthy, etc.) for a duration, then any big job is severely vulnerable to this problem. If all reducers have been completed, JobImpl.actOnUnusableNode() should not reschedule mapper tasks. If all reducers are completed, the mapper outputs are no longer needed, and there is no need to reschedule mapper tasks as they would not be consumed anyway. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-4065) Add .proto files to built tarball
[ https://issues.apache.org/jira/browse/MAPREDUCE-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-4065: Target Version/s: 2.6.0 (was: 2.5.0) Add .proto files to built tarball - Key: MAPREDUCE-4065 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4065 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.2, 2.4.0 Reporter: Ralph H Castain Assignee: Tsuyoshi OZAWA Attachments: MAPREDUCE-4065.1.patch Please add the .proto files to the built tarball so that users can build 3rd party tools that use protocol buffers without having to do an svn checkout of the source code. Sorry I don't know more about Maven, or I would provide a patch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5363) Fix doc and spelling for TaskCompletionEvent#getTaskStatus and getStatus
[
https://issues.apache.org/jira/browse/MAPREDUCE-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-5363:
Target Version/s: 2.6.0 (was: 3.0.0, 2.5.0)
Fix doc and spelling for TaskCompletionEvent#getTaskStatus and getStatus
Key: MAPREDUCE-5363
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5363
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1, mrv2
Affects Versions: 1.1.2, 2.1.0-beta
Reporter: Sandy Ryza
Assignee: Akira AJISAKA
Priority: Minor
Labels: newbie
Attachments: MAPREDUCE-5363-1.patch, MAPREDUCE-5363-2.patch,
MAPREDUCE-5363-3.patch
The doc for TaskCompletionEvent#get(Task)Status in both MR1 and MR2 is
{code}
Returns enum Status.SUCESS or Status.FAILURE.
@return task tracker status
{code}
The actual values that the Status enum can take are
FAILED, KILLED, SUCCEEDED, OBSOLETE, TIPFAILED
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5905) CountersStrings.toEscapedCompactStrings outputs unnecessary null strings
[ https://issues.apache.org/jira/browse/MAPREDUCE-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5905: Target Version/s: 2.6.0 (was: 2.5.0) CountersStrings.toEscapedCompactStrings outputs unnecessary null strings -- Key: MAPREDUCE-5905 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5905 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.4.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Attachments: MAPREDUCE-5905.patch CountersStrings.toEscapedCompactStrings outputs null strings if a CounterGroup has more than one Counter. That way there are some null strings in MRv1(CDH) job history log. https://issues.cloudera.org/browse/DISTRO-598 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-3838) MapReduce job submission time has increased in 0.23 when compared to 0.20.206
[ https://issues.apache.org/jira/browse/MAPREDUCE-3838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-3838: Target Version/s: 2.6.0 (was: 3.0.0, 2.5.0) MapReduce job submission time has increased in 0.23 when compared to 0.20.206 - Key: MAPREDUCE-3838 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3838 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 0.23.0 Reporter: Amar Kamat Labels: gridmix, job-submit-time, yarn Fix For: 0.23.2 While running Gridmix on 0.23, we found that the job submission time has increased when compared to 0.20.206. Here are some stats: ||Submit-Time||Total number of jobs in YARN|| Total number of jobs in FRED|| | 25secs|3 |1 | | 20secs| 6 | 2 | | 15secs| 14 | 4 | | 10secs| 24 | 4 | | 5secs | 67 | 28| Note that Gridmix was run using the same trace. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-4975) gridmix docs missing
[ https://issues.apache.org/jira/browse/MAPREDUCE-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-4975: Target Version/s: 2.6.0 (was: 3.0.0, 2.5.0) gridmix docs missing Key: MAPREDUCE-4975 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4975 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.6 Reporter: Thomas Graves The docs for hadoop streaming and gridmix weren't moved out of the mrv1 code so don't existing in the 0.23 or 2.x line. ie the 1.X line are http://hadoop.apache.org/docs/r1.1.0/streaming.html and http://hadoop.apache.org/docs/r1.1.0/gridmix.html We should also check for others that are missing. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5932) Provide an option to use a dedicated reduce-side shuffle log
[
https://issues.apache.org/jira/browse/MAPREDUCE-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-5932:
Target Version/s: 2.6.0 (was: 2.5.0)
Provide an option to use a dedicated reduce-side shuffle log
Key: MAPREDUCE-5932
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5932
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: MAPREDUCE-5932.v01.patch, MAPREDUCE-5932.v02.patch
For reducers in large jobs our users cannot easily spot portions of the log
associated with problems with their code. An example reducer with INFO-level
logging generates ~3500 lines / ~700KiB lines per second. 95% of the log is
the client-side of the shuffle {{org.apache.hadoop.mapreduce.task.reduce.*}}
{code}
$ wc syslog
3642 48192 691013 syslog
$ grep task.reduce syslog | wc
3424 46534 659038
$ grep task.reduce.ShuffleScheduler syslog | wc
1521 17745 251458
$ grep task.reduce.Fetcher syslog | wc
1045 15340 223683
$ grep task.reduce.InMemoryMapOutput syslog | wc
4004800 72060
$ grep task.reduce.MergeManagerImpl syslog | wc
4328200 106555
{code}
Byte percentage breakdown:
{code}
Shuffle total: 95%
ShuffleScheduler:36%
Fetcher: 32%
InMemoryMapOutput: 10%
MergeManagerImpl:15%
{code}
While this is information is actually often useful for devops debugging
shuffle performance issues, the job users are often lost.
We propose to have a dedicated syslog.shuffle file.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-3552) add ability to specify the format type (xml|json) of web services when requesting it via url query param
[ https://issues.apache.org/jira/browse/MAPREDUCE-3552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-3552: Target Version/s: 2.6.0 (was: 3.0.0, 2.5.0) add ability to specify the format type (xml|json) of web services when requesting it via url query param Key: MAPREDUCE-3552 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3552 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves add ability to specify the format type (xml|json) of web services when requesting it via url query param. Perhaps ?format=xml or similar. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5945) Update the description of GenericOptionsParser -jt option
[ https://issues.apache.org/jira/browse/MAPREDUCE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5945: Target Version/s: 2.6.0 (was: 2.5.0) Update the description of GenericOptionsParser -jt option - Key: MAPREDUCE-5945 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5945 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: documentation Affects Versions: 2.4.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Labels: newbie Attachments: MAPREDUCE-5945.patch Now -jt option is used to specify the address of ResourceManager but document says -jt option specifies JobTracker. The document should be updated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-3554) add job history/am hostname to web services info output
[ https://issues.apache.org/jira/browse/MAPREDUCE-3554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-3554: Target Version/s: 2.6.0 (was: 3.0.0, 2.5.0) add job history/am hostname to web services info output - Key: MAPREDUCE-3554 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3554 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves It would be useful to add the job history or am hostname to web services info output. history server uri is like host:19888/ws/v1/history/info mapreduce app master uri is something like host:8088/proxy/application_1323191000473_0002/ws/v1/mapreduce/info -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-4877) AM doesn't properly support multiple NNs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-4877: Target Version/s: 2.6.0 (was: 3.0.0, 2.5.0) AM doesn't properly support multiple NNs Key: MAPREDUCE-4877 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4877 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, job submission Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Yarn/MR clusters assume there's a 1-to-1 correspondence between itself and a NN. Certain internal paths like the staging dir, job history, intermediate/intermediate-done dirs are resolved relative to the defaultFS. The JT used the host's conf which ensured the correct/expected NN. However the AM uses the user's job conf, which means the user's defined defaultFS can cause the job to use incorrect paths. Typically the output path's NN is also the yarn cluster's NN. However problems occur when a yarn cluster is servicing multiple NN's (ex. federated clusters). The JHS is assuming the AM will write to NN1, whereas the user's job conf may be using a defaultFS of NN2 or NN3 which influences where the AM writes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5948) org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well
[ https://issues.apache.org/jira/browse/MAPREDUCE-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5948: Target Version/s: 2.6.0 (was: 2.5.0) org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well -- Key: MAPREDUCE-5948 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5948 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.2, 0.23.9, 2.2.0 Environment: CDH3U2 Redhat linux 5.7 Reporter: Kris Geusebroek Assignee: Rushabh S Shah Priority: Critical Attachments: HADOOP-9867.patch, HADOOP-9867.patch, HADOOP-9867.patch, HADOOP-9867.patch Having defined a recorddelimiter of multiple bytes in a new InputFileFormat sometimes has the effect of skipping records from the input. This happens when the input splits are split off just after a recordseparator. Starting point for the next split would be non zero and skipFirstLine would be true. A seek into the file is done to start - 1 and the text until the first recorddelimiter is ignored (due to the presumption that this record is already handled by the previous maptask). Since the re ord delimiter is multibyte the seek only got the last byte of the delimiter into scope and its not recognized as a full delimiter. So the text is skipped until the next delimiter (ignoring a full record!!) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5942) Remove MRv1 commands from CommandsManual.apt.vm
[ https://issues.apache.org/jira/browse/MAPREDUCE-5942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5942: Target Version/s: 2.6.0 (was: 2.5.0) Remove MRv1 commands from CommandsManual.apt.vm --- Key: MAPREDUCE-5942 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5942 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: documentation Affects Versions: 2.2.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Labels: newbie Attachments: MAPREDUCE-5942.patch There're some old commands such as 'hadoop jobtracker' and 'hadoop tasktracker' in CommandsManual.apt.vm. These commands should be removed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-3174) app master UI goes away when app finishes - not very user friendly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-3174: Target Version/s: 2.6.0 (was: 2.5.0) app master UI goes away when app finishes - not very user friendly -- Key: MAPREDUCE-3174 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3174 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves A user can go to the application master UI to see the stats on the app, but as soon as the app finishes that UI goes away and user is left with nothing. A redirect to history server or similar would be much better. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-3418) If map output is not found, shuffle runs in tight loop
[ https://issues.apache.org/jira/browse/MAPREDUCE-3418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-3418: Target Version/s: 2.6.0 (was: 2.5.0) If map output is not found, shuffle runs in tight loop -- Key: MAPREDUCE-3418 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3418 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0, 2.3.0 Reporter: John George Sharad Agarwal bumped into this while simulating fetch failures. Removed the map output directory. Shuffle runs in tight loop throwing : 2011-06-01 09:02:20,511 WARN org.apache.hadoop.mapreduce.task.reduce.Fetcher: Invalid map id java.lang.IllegalArgumentException: TaskAttemptId string : TTP/1.1 500 Internal Server Error Content-Type: text/plain; charset=UTF is not properly formed at org.apache.hadoop.mapreduce.TaskAttemptID.forName(TaskAttemptID.java:174) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:284) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:251) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:149) Fetch failure is not triggered. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5595) Typo in MergeManagerImpl.java
[
https://issues.apache.org/jira/browse/MAPREDUCE-5595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-5595:
Target Version/s: 2.6.0 (was: 3.0.0, 2.5.0)
Typo in MergeManagerImpl.java
-
Key: MAPREDUCE-5595
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5595
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0, 2.2.0
Environment: all
Reporter: Efe Gencer
Assignee: Akira AJISAKA
Priority: Trivial
Labels: newbie
Attachments: MAPREDUCE-5595.patch, MergeManagerImpl.java
Original Estimate: 1m
Remaining Estimate: 1m
There's a typo (Invlaid which should be Invalid) in line 199 of
MergeManagerImpl.java
currently:
if (this.maxSingleShuffleLimit = this.mergeThreshold) {
throw new RuntimeException(Invlaid configuration:
+ maxSingleShuffleLimit should be less than mergeThreshold
+ maxSingleShuffleLimit: + this.maxSingleShuffleLimit
+ mergeThreshold: + this.mergeThreshold);
}
should be:
if (this.maxSingleShuffleLimit = this.mergeThreshold) {
throw new RuntimeException(Invalid configuration:
+ maxSingleShuffleLimit should be less than mergeThreshold
+ maxSingleShuffleLimit: + this.maxSingleShuffleLimit
+ mergeThreshold: + this.mergeThreshold);
}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-3917) Use java.net.preferIPv4Stack to force IPv4 in yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-3917: Target Version/s: 2.6.0 (was: 3.0.0, 2.5.0) Use java.net.preferIPv4Stack to force IPv4 in yarn -- Key: MAPREDUCE-3917 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3917 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves HADOOP-6056 made the changes for hadoop cli to use java.net.preferIPv4Stack to force IPv4. We should do the same things for the yarn commands. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-207) Computing Input Splits on the MR Cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-207: --- Target Version/s: 2.6.0 (was: 2.5.0) Computing Input Splits on the MR Cluster Key: MAPREDUCE-207 URL: https://issues.apache.org/jira/browse/MAPREDUCE-207 Project: Hadoop Map/Reduce Issue Type: New Feature Components: applicationmaster, mrv2 Reporter: Philip Zeyliger Assignee: Gera Shegalov Attachments: MAPREDUCE-207.patch, MAPREDUCE-207.v02.patch, MAPREDUCE-207.v03.patch, MAPREDUCE-207.v05.patch, MAPREDUCE-207.v06.patch Instead of computing the input splits as part of job submission, Hadoop could have a separate job task type that computes the input splits, therefore allowing that computation to happen on the cluster. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-3591) webapps always return html on non-existent URL
[ https://issues.apache.org/jira/browse/MAPREDUCE-3591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-3591: Target Version/s: 2.6.0 (was: 3.0.0, 2.5.0) webapps always return html on non-existent URL -- Key: MAPREDUCE-3591 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3591 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Priority: Minor If the user tries to go to a non-existent url, say rm:8088/cluster/foo, via the web ui or the web service rest api, it returns 404 and it always returns html content. With the addition of the web service rest api it would be nice if it returned what was requested - XML or JSON. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5044) Have AM trigger jstack on task attempts that timeout before killing them
[
https://issues.apache.org/jira/browse/MAPREDUCE-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14051097#comment-14051097
]
Hadoop QA commented on MAPREDUCE-5044:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12645521/MAPREDUCE-5044.v06.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4706//console
This message is automatically generated.
Have AM trigger jstack on task attempts that timeout before killing them
Key: MAPREDUCE-5044
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5044
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mr-am
Affects Versions: 2.1.0-beta
Reporter: Jason Lowe
Assignee: Gera Shegalov
Attachments: MAPREDUCE-5044.v01.patch, MAPREDUCE-5044.v02.patch,
MAPREDUCE-5044.v03.patch, MAPREDUCE-5044.v04.patch, MAPREDUCE-5044.v05.patch,
MAPREDUCE-5044.v06.patch, Screen Shot 2013-11-12 at 1.05.32 PM.png, Screen
Shot 2013-11-12 at 1.06.04 PM.png
When an AM expires a task attempt it would be nice if it triggered a jstack
output via SIGQUIT before killing the task attempt. This would be invaluable
for helping users debug their hung tasks, especially if they do not have
shell access to the nodes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
