[jira] [Commented] (MAPREDUCE-3353) Need a RM-AM channel to inform AMs about faulty/unhealthy/lost nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233248#comment-13233248 ] Hadoop QA commented on MAPREDUCE-3353: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519012/MAPREDUCE-3353-branch-0.23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 17 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 507 javac compiler warnings (more than the trunk's current 505 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2074//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2074//console This message is automatically generated. Need a RM-AM channel to inform AMs about faulty/unhealthy/lost nodes - Key: MAPREDUCE-3353 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3353 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Bikas Saha Fix For: 0.23.2 Attachments: MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch When a node gets lost or turns faulty, AM needs to know about that event so that it can take some action like for e.g. re-executing map tasks whose intermediate output live on that faulty node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4039) Sort Avoidance
Sort Avoidance
--
Key: MAPREDUCE-4039
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4039
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: mrv2
Affects Versions: 0.23.2
Reporter: anty.rao
Priority: Minor
Fix For: 0.23.2
Inspired by
[Tenzing|http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en//pubs/archive/37200.pdf],
in 5.1 MapReduce Enhanceemtns:
{quote}*Sort Avoidance*. Certain operators such as hash join
and hash aggregation require shuffling, but not sorting. The
MapReduce API was enhanced to automatically turn off
sorting for these operations. When sorting is turned off, the
mapper feeds data to the reducer which directly passes the
data to the Reduce() function bypassing the intermediate
sorting step. This makes many SQL operators significantly
more ecient.{quote}
There are a lot of applications which need aggregation only, not sorting.Using
sorting to achieve aggregation is costly and inefficient. Without sorting, up
application can make use of hash table or hash map to do aggregation
efficiently.But application should bear in mind that reduce memory is limited,
itself is committed to manage memory of reduce, guard against out of memory.
Map-side combiner is not supported, you can also do hash aggregation in map
side as a workaround.
the following is the main points of sort avoidance implementation
# add a configuration parameter ??mapreduce.sort.avoidance??, boolean type, to
turn on/off sort avoidance workflow.Two type of workflow are coexist together.
# key/value pairs emitted by map function is sorted by partition only, using a
more efficient sorting algorithm: counting sort.
# map-side merge, use a kind of byte merge, which just concatenate bytes from
generated spills, read in bytes, write out bytes, without overhead of key/value
serialization/deserailization, comparison, which currently version incurs.
# reduce can start up as soon as there is any map output available, in contrast
to sort workflow which must wait until all map outputs are fetched and merged.
# map output in memory can be directly consumed by reduce.When reduce can catch
up with the speed of incoming map output, in-memory merge thread will kick in,
merging in-memory map outputs onto disk.
# sequentially read in on-disk files to feed reduce, in contrast to currently
implementation which read multiple files concurrently, result in many disk
seek. Map output in memory take precedence over on disk files in feeding reduce
function.
I have already implement this feature based on hadoop CDH3U3 and done some
performance evaluation, you can reference to
[https://github.com/hanborq/hadoop] for details. Now,I'm willing to port it
into yarn. Welcome for commenting.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4039) Sort Avoidance
[
https://issues.apache.org/jira/browse/MAPREDUCE-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
anty.rao updated MAPREDUCE-4039:
Description:
Inspired by
[Tenzing|http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en//pubs/archive/37200.pdf],
in 5.1 MapReduce Enhanceemtns:
{quote}*Sort Avoidance*. Certain operators such as hash join
and hash aggregation require shuffling, but not sorting. The
MapReduce API was enhanced to automatically turn off
sorting for these operations. When sorting is turned off, the
mapper feeds data to the reducer which directly passes the
data to the Reduce() function bypassing the intermediate
sorting step. This makes many SQL operators significantly
more ecient.{quote}
There are a lot of applications which need aggregation only, not sorting.Using
sorting to achieve aggregation is costly and inefficient. Without sorting, up
application can make use of hash table or hash map to do aggregation
efficiently.But application should bear in mind that reduce memory is limited,
itself is committed to manage memory of reduce, guard against out of memory.
Map-side combiner is not supported, you can also do hash aggregation in map
side as a workaround.
the following is the main points of sort avoidance implementation
# add a configuration parameter ??mapreduce.sort.avoidance??, boolean type, to
turn on/off sort avoidance workflow.Two type of workflow are coexist together.
# key/value pairs emitted by map function is sorted by partition only, using a
more efficient sorting algorithm: counting sort.
# map-side merge, use a kind of byte merge, which just concatenate bytes from
generated spills, read in bytes, write out bytes, without overhead of key/value
serialization/deserailization, comparison, which current version incurs.
# reduce can start up as soon as there is any map output available, in contrast
to sort workflow which must wait until all map outputs are fetched and merged.
# map output in memory can be directly consumed by reduce.When reduce can't
catch up with the speed of incoming map outputs, in-memory merge thread will
kick in, merging in-memory map outputs onto disk.
# sequentially read in on-disk files to feed reduce, in contrast to currently
implementation which read multiple files concurrently, result in many disk
seek. Map output in memory take precedence over on disk files in feeding reduce
function.
I have already implement this feature based on hadoop CDH3U3 and done some
performance evaluation, you can reference to
[https://github.com/hanborq/hadoop] for details. Now,I'm willing to port it
into yarn. Welcome for commenting.
was:
Inspired by
[Tenzing|http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en//pubs/archive/37200.pdf],
in 5.1 MapReduce Enhanceemtns:
{quote}*Sort Avoidance*. Certain operators such as hash join
and hash aggregation require shuffling, but not sorting. The
MapReduce API was enhanced to automatically turn off
sorting for these operations. When sorting is turned off, the
mapper feeds data to the reducer which directly passes the
data to the Reduce() function bypassing the intermediate
sorting step. This makes many SQL operators significantly
more ecient.{quote}
There are a lot of applications which need aggregation only, not sorting.Using
sorting to achieve aggregation is costly and inefficient. Without sorting, up
application can make use of hash table or hash map to do aggregation
efficiently.But application should bear in mind that reduce memory is limited,
itself is committed to manage memory of reduce, guard against out of memory.
Map-side combiner is not supported, you can also do hash aggregation in map
side as a workaround.
the following is the main points of sort avoidance implementation
# add a configuration parameter ??mapreduce.sort.avoidance??, boolean type, to
turn on/off sort avoidance workflow.Two type of workflow are coexist together.
# key/value pairs emitted by map function is sorted by partition only, using a
more efficient sorting algorithm: counting sort.
# map-side merge, use a kind of byte merge, which just concatenate bytes from
generated spills, read in bytes, write out bytes, without overhead of key/value
serialization/deserailization, comparison, which currently version incurs.
# reduce can start up as soon as there is any map output available, in contrast
to sort workflow which must wait until all map outputs are fetched and merged.
# map output in memory can be directly consumed by reduce.When reduce can catch
up with the speed of incoming map output, in-memory merge thread will kick in,
merging in-memory map outputs onto disk.
# sequentially read in on-disk files to feed reduce, in contrast to currently
implementation which read multiple files concurrently, result in many disk
seek. Map output in memory take precedence over
[jira] [Commented] (MAPREDUCE-4039) Sort Avoidance
[
https://issues.apache.org/jira/browse/MAPREDUCE-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1324#comment-1324
]
Schubert Zhang commented on MAPREDUCE-4039:
---
Nice summary Anty, and expect your patch on 0.23.
The more detailed description and benchmark on this feature, you can refer to
http://www.slideshare.net/hanborq/hanborq-optimizations-on-hadoop-mapreduce-20120216a
or
http://www.slideshare.net/schubertzhang/hanborq-optimizations-on-hadoop-map-reduce-20120221a
Sort Avoidance
--
Key: MAPREDUCE-4039
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4039
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: mrv2
Affects Versions: 0.23.2
Reporter: anty.rao
Priority: Minor
Fix For: 0.23.2
Inspired by
[Tenzing|http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en//pubs/archive/37200.pdf],
in 5.1 MapReduce Enhanceemtns:
{quote}*Sort Avoidance*. Certain operators such as hash join
and hash aggregation require shuffling, but not sorting. The
MapReduce API was enhanced to automatically turn off
sorting for these operations. When sorting is turned off, the
mapper feeds data to the reducer which directly passes the
data to the Reduce() function bypassing the intermediate
sorting step. This makes many SQL operators significantly
more ecient.{quote}
There are a lot of applications which need aggregation only, not
sorting.Using sorting to achieve aggregation is costly and inefficient.
Without sorting, up application can make use of hash table or hash map to do
aggregation efficiently.But application should bear in mind that reduce
memory is limited, itself is committed to manage memory of reduce, guard
against out of memory. Map-side combiner is not supported, you can also do
hash aggregation in map side as a workaround.
the following is the main points of sort avoidance implementation
# add a configuration parameter ??mapreduce.sort.avoidance??, boolean type,
to turn on/off sort avoidance workflow.Two type of workflow are coexist
together.
# key/value pairs emitted by map function is sorted by partition only, using
a more efficient sorting algorithm: counting sort.
# map-side merge, use a kind of byte merge, which just concatenate bytes from
generated spills, read in bytes, write out bytes, without overhead of
key/value serialization/deserailization, comparison, which current version
incurs.
# reduce can start up as soon as there is any map output available, in
contrast to sort workflow which must wait until all map outputs are fetched
and merged.
# map output in memory can be directly consumed by reduce.When reduce can't
catch up with the speed of incoming map outputs, in-memory merge thread will
kick in, merging in-memory map outputs onto disk.
# sequentially read in on-disk files to feed reduce, in contrast to currently
implementation which read multiple files concurrently, result in many disk
seek. Map output in memory take precedence over on disk files in feeding
reduce function.
I have already implement this feature based on hadoop CDH3U3 and done some
performance evaluation, you can reference to
[https://github.com/hanborq/hadoop] for details. Now,I'm willing to port it
into yarn. Welcome for commenting.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3353) Need a RM-AM channel to inform AMs about faulty/unhealthy/lost nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233361#comment-13233361 ] Arun C Murthy commented on MAPREDUCE-3353: -- Bikas, can you pls look at the javac warnings? Thanks. Need a RM-AM channel to inform AMs about faulty/unhealthy/lost nodes - Key: MAPREDUCE-3353 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3353 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Bikas Saha Fix For: 0.23.2 Attachments: MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch When a node gets lost or turns faulty, AM needs to know about that event so that it can take some action like for e.g. re-executing map tasks whose intermediate output live on that faulty node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4040) History links should use hostname rather than IP address.
History links should use hostname rather than IP address.
-
Key: MAPREDUCE-4040
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4040
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 0.23.2
Reporter: Bhallamudi Venkata Siva Kamesh
Priority: Minor
While navigating from web page (eg: */cluster/app/app-id* ) to HS, browser
displays IP address rather than hostname.
{code:title=JobHistoryUtils.java|borderStyle=solid}
if (address.getAddress().isAnyLocalAddress() ||
address.getAddress().isLoopbackAddress()) {
sb.append(InetAddress.getLocalHost().getHostAddress());
}
}
{code}
I *think* it is better to use hostname rather than IP address.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4034) Unable to view task logs on history server with mapreduce.job.acl-view-job=*
[ https://issues.apache.org/jira/browse/MAPREDUCE-4034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233529#comment-13233529 ] Robert Joseph Evans commented on MAPREDUCE-4034: The patch looks good to me too +1. Unable to view task logs on history server with mapreduce.job.acl-view-job=* Key: MAPREDUCE-4034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Attachments: MAPREDUCE-4034.patch With log aggregation enabled, users other than the app owner or admins are sometimes unable to view the task logs on the history server even though they are in the ACL for the app. The same users are able to see the configuration and counters. Sometimes the users can see some task logs but not other task logs for the same application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4034) Unable to view task logs on history server with mapreduce.job.acl-view-job=*
[ https://issues.apache.org/jira/browse/MAPREDUCE-4034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4034: --- Resolution: Fixed Fix Version/s: 0.23.2 Target Version/s: 0.24.0, 0.23.2, 0.23.3 (was: 0.23.3, 0.23.2, 0.24.0) Status: Resolved (was: Patch Available) Thanks Jason and Sid. I just put this into 0.23.2, trunk, and branch-0.23 Unable to view task logs on history server with mapreduce.job.acl-view-job=* Key: MAPREDUCE-4034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-4034.patch With log aggregation enabled, users other than the app owner or admins are sometimes unable to view the task logs on the history server even though they are in the ACL for the app. The same users are able to see the configuration and counters. Sometimes the users can see some task logs but not other task logs for the same application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3353) Need a RM-AM channel to inform AMs about faulty/unhealthy/lost nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233568#comment-13233568 ] Bikas Saha commented on MAPREDUCE-3353: --- They have already been clarified in the first patch submission. Pasting from an earlier comment. The javac warnings are because of events handlers being called in NodesListManager.java and are similar to pre-existing warnings. === [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java:[142,19] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java:[155,21] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java:[244,35] [unchecked] unchecked call to handle(T) as a member of the raw type org.apache.hadoop.yarn.event.EventHandler === Need a RM-AM channel to inform AMs about faulty/unhealthy/lost nodes - Key: MAPREDUCE-3353 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3353 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Bikas Saha Fix For: 0.23.2 Attachments: MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch When a node gets lost or turns faulty, AM needs to know about that event so that it can take some action like for e.g. re-executing map tasks whose intermediate output live on that faulty node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4036) Streaming TestUlimit fails on CentOS 6
[ https://issues.apache.org/jira/browse/MAPREDUCE-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-4036: -- Status: Patch Available (was: Open) Streaming TestUlimit fails on CentOS 6 -- Key: MAPREDUCE-4036 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4036 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.0.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: MAPREDUCE-4036.patch CentOS 6 seems to have higher memory requirements than other distros and together with the new MALLOC library makes the TestUlimit to fail with exit status 134. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1740) NPE in getMatchingLevelForNodes when node locations are variable depth
[ https://issues.apache.org/jira/browse/MAPREDUCE-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233645#comment-13233645 ] Alejandro Abdelnur commented on MAPREDUCE-1740: --- +1 NPE in getMatchingLevelForNodes when node locations are variable depth -- Key: MAPREDUCE-1740 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1740 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.3 Reporter: Todd Lipcon Attachments: MAPREDUCE-1740.patch, MAPREDUCE-1740_branch-1.0.patch, MAPREDUCE-1740_trunk.patch, mapreduce-1740.txt In getMatchingLevelForNodes, we assume that both nodes have the same depth (ie number of path components). If the user provides a topology script that assigns one node a path like /foo/bar/baz and another node a path like /foo/blah, this function will throw an NPE. I'm not sure if there are other places where we assume that all node locations have a constant number of paths. If so we should check the output of the topology script aggressively to be sure this is the case. Otherwise I think we simply need to add n2 != null to the while loop -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1740) NPE in getMatchingLevelForNodes when node locations are variable depth
[ https://issues.apache.org/jira/browse/MAPREDUCE-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-1740: -- Resolution: Fixed Assignee: Ahmed Radwan Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Ahmed. Committed to trunk and branch-1 NPE in getMatchingLevelForNodes when node locations are variable depth -- Key: MAPREDUCE-1740 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1740 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.3 Reporter: Todd Lipcon Assignee: Ahmed Radwan Attachments: MAPREDUCE-1740.patch, MAPREDUCE-1740_branch-1.0.patch, MAPREDUCE-1740_trunk.patch, mapreduce-1740.txt In getMatchingLevelForNodes, we assume that both nodes have the same depth (ie number of path components). If the user provides a topology script that assigns one node a path like /foo/bar/baz and another node a path like /foo/blah, this function will throw an NPE. I'm not sure if there are other places where we assume that all node locations have a constant number of paths. If so we should check the output of the topology script aggressively to be sure this is the case. Otherwise I think we simply need to add n2 != null to the while loop -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1740) NPE in getMatchingLevelForNodes when node locations are variable depth
[ https://issues.apache.org/jira/browse/MAPREDUCE-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233658#comment-13233658 ] Arun C Murthy commented on MAPREDUCE-1740: -- This is nice to fix. However, we should look at HDFS too. Doing it in isolation in MR isn't sufficient. NPE in getMatchingLevelForNodes when node locations are variable depth -- Key: MAPREDUCE-1740 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1740 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.3 Reporter: Todd Lipcon Assignee: Ahmed Radwan Attachments: MAPREDUCE-1740.patch, MAPREDUCE-1740_branch-1.0.patch, MAPREDUCE-1740_trunk.patch, mapreduce-1740.txt In getMatchingLevelForNodes, we assume that both nodes have the same depth (ie number of path components). If the user provides a topology script that assigns one node a path like /foo/bar/baz and another node a path like /foo/blah, this function will throw an NPE. I'm not sure if there are other places where we assume that all node locations have a constant number of paths. If so we should check the output of the topology script aggressively to be sure this is the case. Otherwise I think we simply need to add n2 != null to the while loop -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1109) ConcurrentModificationException in jobtracker.jsp
[ https://issues.apache.org/jira/browse/MAPREDUCE-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-1109: --- Attachment: (was: MAPREDUCE-1109.patch) ConcurrentModificationException in jobtracker.jsp - Key: MAPREDUCE-1109 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1109 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.1 Reporter: dhruba borthakur Assignee: Harsh J Attachments: MAPREDUCE-1109.patch, MAPREDUCE-1109.patch The jobtracker.jsp invoked methods tracker.runningJobs(), tracker.completedJobs() and tracker.failedJobs() but these methods are not synchronized and can cause ConcurrentModificationException if the JT is concurrently changing the contents of these datastructures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1109) ConcurrentModificationException in jobtracker.jsp
[ https://issues.apache.org/jira/browse/MAPREDUCE-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-1109: --- Attachment: MAPREDUCE-1109.patch ConcurrentModificationException in jobtracker.jsp - Key: MAPREDUCE-1109 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1109 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.1 Reporter: dhruba borthakur Assignee: Harsh J Attachments: MAPREDUCE-1109.patch, MAPREDUCE-1109.patch The jobtracker.jsp invoked methods tracker.runningJobs(), tracker.completedJobs() and tracker.failedJobs() but these methods are not synchronized and can cause ConcurrentModificationException if the JT is concurrently changing the contents of these datastructures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3955) Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated MAPREDUCE-3955: Attachment: MAPREDUCE-3955-trunk.patch Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine. Key: MAPREDUCE-3955 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3955 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: MAPREDUCE-3955-trunk.patch We shouldn't have two rpc engines based on protocol buffers. ProtoOverHadoopRpcEngine in hadoop-yarn-common should be replaced by ProtobufRpcEngine in hadoop-common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3955) Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated MAPREDUCE-3955: Target Version/s: 0.24.0, 0.23.3 (was: 0.23.3, 0.24.0) Status: Patch Available (was: Open) Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine. Key: MAPREDUCE-3955 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3955 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: MAPREDUCE-3955-trunk.patch We shouldn't have two rpc engines based on protocol buffers. ProtoOverHadoopRpcEngine in hadoop-yarn-common should be replaced by ProtobufRpcEngine in hadoop-common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4036) Streaming TestUlimit fails on CentOS 6
[ https://issues.apache.org/jira/browse/MAPREDUCE-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233718#comment-13233718 ] Hadoop QA commented on MAPREDUCE-4036: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519020/MAPREDUCE-4036.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2075//console This message is automatically generated. Streaming TestUlimit fails on CentOS 6 -- Key: MAPREDUCE-4036 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4036 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.0.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: MAPREDUCE-4036.patch CentOS 6 seems to have higher memory requirements than other distros and together with the new MALLOC library makes the TestUlimit to fail with exit status 134. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4041) TestMapredGroupMappingServiceRefresh unit test failures
TestMapredGroupMappingServiceRefresh unit test failures --- Key: MAPREDUCE-4041 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4041 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3 Reporter: Thomas Graves On branch-0.23 the following unit tests fail: org.apache.hadoop.security.TestMapredGroupMappingServiceRefresh.testGroupMappingRefresh org.apache.hadoop.security.TestMapredGroupMappingServiceRefresh.testRefreshSuperUserGroupsConfiguration -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4041) TestMapredGroupMappingServiceRefresh unit test failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233738#comment-13233738 ] Thomas Graves commented on MAPREDUCE-4041: -- Note this is an ant test: ant test -Dtestcase=TestMapredGroupMappingServiceRefresh TestMapredGroupMappingServiceRefresh unit test failures --- Key: MAPREDUCE-4041 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4041 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3 Reporter: Thomas Graves On branch-0.23 the following unit tests fail: org.apache.hadoop.security.TestMapredGroupMappingServiceRefresh.testGroupMappingRefresh org.apache.hadoop.security.TestMapredGroupMappingServiceRefresh.testRefreshSuperUserGroupsConfiguration -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4042) unit test TestControlledMapReduceJob.testControlledMapReduceJob fails
unit test TestControlledMapReduceJob.testControlledMapReduceJob fails -- Key: MAPREDUCE-4042 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4042 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3 Reporter: Thomas Graves unit test TestControlledMapReduceJob.testControlledMapReduceJob fails. This is an ant test: ant test -Dtestcase=TestControlledMapReduceJob error: Timeout occurred. Please note the time in the report does not reflect the time until the timeout. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4034) Unable to view task logs on history server with mapreduce.job.acl-view-job=*
[ https://issues.apache.org/jira/browse/MAPREDUCE-4034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233759#comment-13233759 ] Hudson commented on MAPREDUCE-4034: --- Integrated in Hadoop-Mapreduce-0.23-Build #231 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/231/]) svn merge -c 1302980 from trunk to branch 0.23 FIXES MAPREDUCE-4034. Unable to view task logs on history server with mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby) (Revision 1302982) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1302982 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java Unable to view task logs on history server with mapreduce.job.acl-view-job=* Key: MAPREDUCE-4034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-4034.patch With log aggregation enabled, users other than the app owner or admins are sometimes unable to view the task logs on the history server even though they are in the ACL for the app. The same users are able to see the configuration and counters. Sometimes the users can see some task logs but not other task logs for the same application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4025) AM can crash if task attempt reports bogus progress value
[
https://issues.apache.org/jira/browse/MAPREDUCE-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233758#comment-13233758
]
Hudson commented on MAPREDUCE-4025:
---
Integrated in Hadoop-Mapreduce-0.23-Build #231 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/231/])
svn merge -c 1302645 from trunk to branch-0.23 FIXES MAPREDUCE-4025. AM
can crash if task attempt reports bogus progress value (Jason Lowe via bobby)
(Revision 1302646)
Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1302646
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
AM can crash if task attempt reports bogus progress value
-
Key: MAPREDUCE-4025
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4025
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 0.23.2
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Blocker
Fix For: 0.23.2
Attachments: MAPREDUCE-4025.patch
If a task attempt reports a bogus progress value (e.g.: something above 1.0)
then the AM can crash like this:
{noformat}
java.lang.ArrayIndexOutOfBoundsException: 12
at
org.apache.hadoop.mapred.PeriodicStatsAccumulator.extend(PeriodicStatsAccumulator.java:185)
at
org.apache.hadoop.mapred.WrappedPeriodicStatsAccumulator.extend(WrappedPeriodicStatsAccumulator.java:31)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.updateProgressSplits(TaskAttemptImpl.java:1043)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.access$4100(TaskAttemptImpl.java:136)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$StatusUpdater.transition(TaskAttemptImpl.java:1509)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$StatusUpdater.transition(TaskAttemptImpl.java:1490)
at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:931)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:135)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:886)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:878)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
at java.lang.Thread.run(Thread.java:619)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3992) Reduce fetcher doesn't verify HTTP status code of response
[ https://issues.apache.org/jira/browse/MAPREDUCE-3992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233764#comment-13233764 ] Hudson commented on MAPREDUCE-3992: --- Integrated in Hadoop-Mapreduce-0.23-Build #231 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/231/]) MAPREDUCE-3992. Reduce fetcher doesn't verify HTTP status code of response. Contributed by Todd Lipcon. (Revision 1302755) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1302755 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java Reduce fetcher doesn't verify HTTP status code of response -- Key: MAPREDUCE-3992 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3992 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 0.23.1, 0.24.0, 1.0.1 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.24.0, 0.23.3 Attachments: mr-3992-branch-1.txt, mr-3992.txt Currently, the reduce fetch code doesn't check the HTTP status code of the response. This can lead to the following situation: - the map output servlet gets an IOException after setting the headers but before the first call to flush() - this causes it to send a response with a non-OK result code, including the exception text as the response body (response.sendError() does this if the response isn't committed) - it will still include the response headers indicating it's a valid response In the case of a merge-to-memory, the compression codec might then try to interpret the HTML response as compressed data, resulting in either a huge allocation (OOME) or some other nasty error. This bug seems to be present in MR1, but haven't checked trunk/MR2 yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4034) Unable to view task logs on history server with mapreduce.job.acl-view-job=*
[ https://issues.apache.org/jira/browse/MAPREDUCE-4034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233772#comment-13233772 ] Hudson commented on MAPREDUCE-4034: --- Integrated in Hadoop-Common-0.23-Commit #708 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/708/]) svn merge -c 1302980 from trunk to branch 0.23 FIXES MAPREDUCE-4034. Unable to view task logs on history server with mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby) (Revision 1302982) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1302982 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java Unable to view task logs on history server with mapreduce.job.acl-view-job=* Key: MAPREDUCE-4034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-4034.patch With log aggregation enabled, users other than the app owner or admins are sometimes unable to view the task logs on the history server even though they are in the ACL for the app. The same users are able to see the configuration and counters. Sometimes the users can see some task logs but not other task logs for the same application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4034) Unable to view task logs on history server with mapreduce.job.acl-view-job=*
[ https://issues.apache.org/jira/browse/MAPREDUCE-4034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233785#comment-13233785 ] Hudson commented on MAPREDUCE-4034: --- Integrated in Hadoop-Common-trunk-Commit #1907 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1907/]) MAPREDUCE-4034. Unable to view task logs on history server with mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby) (Revision 1302980) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1302980 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java Unable to view task logs on history server with mapreduce.job.acl-view-job=* Key: MAPREDUCE-4034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-4034.patch With log aggregation enabled, users other than the app owner or admins are sometimes unable to view the task logs on the history server even though they are in the ACL for the app. The same users are able to see the configuration and counters. Sometimes the users can see some task logs but not other task logs for the same application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1740) NPE in getMatchingLevelForNodes when node locations are variable depth
[ https://issues.apache.org/jira/browse/MAPREDUCE-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233786#comment-13233786 ] Hudson commented on MAPREDUCE-1740: --- Integrated in Hadoop-Common-trunk-Commit #1907 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1907/]) MAPREDUCE-1740. NPE in getMatchingLevelForNodes when node locations are variable depth (ahmed via tucu) (Revision 1303076) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1303076 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java NPE in getMatchingLevelForNodes when node locations are variable depth -- Key: MAPREDUCE-1740 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1740 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.3 Reporter: Todd Lipcon Assignee: Ahmed Radwan Attachments: MAPREDUCE-1740.patch, MAPREDUCE-1740_branch-1.0.patch, MAPREDUCE-1740_trunk.patch, mapreduce-1740.txt In getMatchingLevelForNodes, we assume that both nodes have the same depth (ie number of path components). If the user provides a topology script that assigns one node a path like /foo/bar/baz and another node a path like /foo/blah, this function will throw an NPE. I'm not sure if there are other places where we assume that all node locations have a constant number of paths. If so we should check the output of the topology script aggressively to be sure this is the case. Otherwise I think we simply need to add n2 != null to the while loop -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4034) Unable to view task logs on history server with mapreduce.job.acl-view-job=*
[ https://issues.apache.org/jira/browse/MAPREDUCE-4034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233795#comment-13233795 ] Hudson commented on MAPREDUCE-4034: --- Integrated in Hadoop-Hdfs-0.23-Commit #699 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/699/]) svn merge -c 1302980 from trunk to branch 0.23 FIXES MAPREDUCE-4034. Unable to view task logs on history server with mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby) (Revision 1302982) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1302982 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java Unable to view task logs on history server with mapreduce.job.acl-view-job=* Key: MAPREDUCE-4034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-4034.patch With log aggregation enabled, users other than the app owner or admins are sometimes unable to view the task logs on the history server even though they are in the ACL for the app. The same users are able to see the configuration and counters. Sometimes the users can see some task logs but not other task logs for the same application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1740) NPE in getMatchingLevelForNodes when node locations are variable depth
[ https://issues.apache.org/jira/browse/MAPREDUCE-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233802#comment-13233802 ] Hudson commented on MAPREDUCE-1740: --- Integrated in Hadoop-Hdfs-trunk-Commit #1981 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1981/]) MAPREDUCE-1740. NPE in getMatchingLevelForNodes when node locations are variable depth (ahmed via tucu) (Revision 1303076) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1303076 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java NPE in getMatchingLevelForNodes when node locations are variable depth -- Key: MAPREDUCE-1740 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1740 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.3 Reporter: Todd Lipcon Assignee: Ahmed Radwan Attachments: MAPREDUCE-1740.patch, MAPREDUCE-1740_branch-1.0.patch, MAPREDUCE-1740_trunk.patch, mapreduce-1740.txt In getMatchingLevelForNodes, we assume that both nodes have the same depth (ie number of path components). If the user provides a topology script that assigns one node a path like /foo/bar/baz and another node a path like /foo/blah, this function will throw an NPE. I'm not sure if there are other places where we assume that all node locations have a constant number of paths. If so we should check the output of the topology script aggressively to be sure this is the case. Otherwise I think we simply need to add n2 != null to the while loop -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4034) Unable to view task logs on history server with mapreduce.job.acl-view-job=*
[ https://issues.apache.org/jira/browse/MAPREDUCE-4034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233801#comment-13233801 ] Hudson commented on MAPREDUCE-4034: --- Integrated in Hadoop-Hdfs-trunk-Commit #1981 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1981/]) MAPREDUCE-4034. Unable to view task logs on history server with mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby) (Revision 1302980) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1302980 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java Unable to view task logs on history server with mapreduce.job.acl-view-job=* Key: MAPREDUCE-4034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-4034.patch With log aggregation enabled, users other than the app owner or admins are sometimes unable to view the task logs on the history server even though they are in the ACL for the app. The same users are able to see the configuration and counters. Sometimes the users can see some task logs but not other task logs for the same application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4025) AM can crash if task attempt reports bogus progress value
[
https://issues.apache.org/jira/browse/MAPREDUCE-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233834#comment-13233834
]
Hudson commented on MAPREDUCE-4025:
---
Integrated in Hadoop-Mapreduce-trunk #1025 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1025/])
MAPREDUCE-4025. AM can crash if task attempt reports bogus progress value
(Jason Lowe via bobby) (Revision 1302645)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1302645
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
AM can crash if task attempt reports bogus progress value
-
Key: MAPREDUCE-4025
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4025
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 0.23.2
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Blocker
Fix For: 0.23.2
Attachments: MAPREDUCE-4025.patch
If a task attempt reports a bogus progress value (e.g.: something above 1.0)
then the AM can crash like this:
{noformat}
java.lang.ArrayIndexOutOfBoundsException: 12
at
org.apache.hadoop.mapred.PeriodicStatsAccumulator.extend(PeriodicStatsAccumulator.java:185)
at
org.apache.hadoop.mapred.WrappedPeriodicStatsAccumulator.extend(WrappedPeriodicStatsAccumulator.java:31)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.updateProgressSplits(TaskAttemptImpl.java:1043)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.access$4100(TaskAttemptImpl.java:136)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$StatusUpdater.transition(TaskAttemptImpl.java:1509)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$StatusUpdater.transition(TaskAttemptImpl.java:1490)
at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:931)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:135)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:886)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:878)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
at java.lang.Thread.run(Thread.java:619)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1740) NPE in getMatchingLevelForNodes when node locations are variable depth
[ https://issues.apache.org/jira/browse/MAPREDUCE-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233836#comment-13233836 ] Hudson commented on MAPREDUCE-1740: --- Integrated in Hadoop-Mapreduce-trunk #1025 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1025/]) MAPREDUCE-1740. NPE in getMatchingLevelForNodes when node locations are variable depth (ahmed via tucu) (Revision 1303076) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1303076 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java NPE in getMatchingLevelForNodes when node locations are variable depth -- Key: MAPREDUCE-1740 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1740 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.3 Reporter: Todd Lipcon Assignee: Ahmed Radwan Attachments: MAPREDUCE-1740.patch, MAPREDUCE-1740_branch-1.0.patch, MAPREDUCE-1740_trunk.patch, mapreduce-1740.txt In getMatchingLevelForNodes, we assume that both nodes have the same depth (ie number of path components). If the user provides a topology script that assigns one node a path like /foo/bar/baz and another node a path like /foo/blah, this function will throw an NPE. I'm not sure if there are other places where we assume that all node locations have a constant number of paths. If so we should check the output of the topology script aggressively to be sure this is the case. Otherwise I think we simply need to add n2 != null to the while loop -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4034) Unable to view task logs on history server with mapreduce.job.acl-view-job=*
[ https://issues.apache.org/jira/browse/MAPREDUCE-4034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233835#comment-13233835 ] Hudson commented on MAPREDUCE-4034: --- Integrated in Hadoop-Mapreduce-trunk #1025 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1025/]) MAPREDUCE-4034. Unable to view task logs on history server with mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby) (Revision 1302980) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1302980 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java Unable to view task logs on history server with mapreduce.job.acl-view-job=* Key: MAPREDUCE-4034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-4034.patch With log aggregation enabled, users other than the app owner or admins are sometimes unable to view the task logs on the history server even though they are in the ACL for the app. The same users are able to see the configuration and counters. Sometimes the users can see some task logs but not other task logs for the same application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3992) Reduce fetcher doesn't verify HTTP status code of response
[ https://issues.apache.org/jira/browse/MAPREDUCE-3992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233841#comment-13233841 ] Hudson commented on MAPREDUCE-3992: --- Integrated in Hadoop-Mapreduce-trunk #1025 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1025/]) MAPREDUCE-3992. Reduce fetcher doesn't verify HTTP status code of response. Contributed by Todd Lipcon. (Revision 1302754) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1302754 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java Reduce fetcher doesn't verify HTTP status code of response -- Key: MAPREDUCE-3992 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3992 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 0.23.1, 0.24.0, 1.0.1 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.24.0, 0.23.3 Attachments: mr-3992-branch-1.txt, mr-3992.txt Currently, the reduce fetch code doesn't check the HTTP status code of the response. This can lead to the following situation: - the map output servlet gets an IOException after setting the headers but before the first call to flush() - this causes it to send a response with a non-OK result code, including the exception text as the response body (response.sendError() does this if the response isn't committed) - it will still include the response headers indicating it's a valid response In the case of a merge-to-memory, the compression codec might then try to interpret the HTML response as compressed data, resulting in either a huge allocation (OOME) or some other nasty error. This bug seems to be present in MR1, but haven't checked trunk/MR2 yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3955) Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233877#comment-13233877 ] Hadoop QA commented on MAPREDUCE-3955: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519118/MAPREDUCE-3955-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapred.TestMiniMRBringup org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapred.TestJobCounters org.apache.hadoop.mapreduce.TestChild org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath org.apache.hadoop.mapreduce.v2.TestRMNMInfo org.apache.hadoop.mapreduce.v2.TestNonExistentJob org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2076//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2076//console This message is automatically generated. Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine. Key: MAPREDUCE-3955 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3955 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: MAPREDUCE-3955-trunk.patch We shouldn't have two rpc engines based on protocol buffers. ProtoOverHadoopRpcEngine in hadoop-yarn-common should be replaced by ProtobufRpcEngine in hadoop-common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4043) Secret keys set in Credentials are not seen by tasks
Secret keys set in Credentials are not seen by tasks Key: MAPREDUCE-4043 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4043 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Priority: Blocker The following scenario works in 0.20.205 but no longer works in 0.23: 1) During job submission, a secret key is set by calling jobConf.getCredentials().addSecretKey(Text, byte[]) 2) A map task retrieves the secret key by calling jobConf.getCredentials().getSecretKey(Text) In 205 the secret key is retrieved successfully but in 0.23 the secret key is missing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4044) YarnClientProtocolProvider does not honor mapred.job.tracker property
YarnClientProtocolProvider does not honor mapred.job.tracker property - Key: MAPREDUCE-4044 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4044 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0, 0.23.3 Reporter: Alejandro Abdelnur Fix For: 0.23.3 The YarnClientProtocolProvider/YARNRunner/ResourceMgrDelegate bootstrap only looks for 'yarn.resourcemanager.address', they ignore 'mapred.job.tracker' This breaks backward compatibility and creates issues in Oozie. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4043) Secret keys set in Credentials are not seen by tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234093#comment-13234093
]
Jason Lowe commented on MAPREDUCE-4043:
---
The tasks can workaround the problem by reading the appTokens file that was
pushed to HDFS when the job was launched. For example:
{code}
Credentials cred = Credentials.readTokenStorageFile(new
Path(jobConf.get(mapreduce.job.dir), appTokens), jobConf);
byte[] secretKey = cred.getSecretKey(SECRET_KEY_ALIAS);
{code}
Secret keys set in Credentials are not seen by tasks
Key: MAPREDUCE-4043
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4043
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.2
Reporter: Jason Lowe
Priority: Blocker
The following scenario works in 0.20.205 but no longer works in 0.23:
1) During job submission, a secret key is set by calling
jobConf.getCredentials().addSecretKey(Text, byte[])
2) A map task retrieves the secret key by calling
jobConf.getCredentials().getSecretKey(Text)
In 205 the secret key is retrieved successfully but in 0.23 the secret key is
missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4043) Secret keys set in Credentials are not seen by tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4043: -- Component/s: (was: contrib/streaming) security Secret keys set in Credentials are not seen by tasks Key: MAPREDUCE-4043 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4043 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, security Affects Versions: 0.23.2 Reporter: Jason Lowe Priority: Blocker The following scenario works in 0.20.205 but no longer works in 0.23: 1) During job submission, a secret key is set by calling jobConf.getCredentials().addSecretKey(Text, byte[]) 2) A map task retrieves the secret key by calling jobConf.getCredentials().getSecretKey(Text) In 205 the secret key is retrieved successfully but in 0.23 the secret key is missing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4045) RM UI - Applications - Application Master Link - Job Link - New Maps/Reduces leads to circular redirect error
RM UI - Applications - Application Master Link - Job Link - New
Maps/Reduces leads to circular redirect error
-
Key: MAPREDUCE-4045
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4045
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.2
Reporter: Devaraj K
Assignee: Devaraj K
{code:xml}
HTTP ERROR 500
Problem accessing
/proxy/application_1332261815858_0002/mapreduce/attempts/job_1332261815858_2_2/m/NEW.
Reason:
Circular redirect to
'http://HOST-192-168-47-207:41992/mapreduce/attempts/job_1332261815858_2_2/m/NEW'
Caused by:
org.apache.commons.httpclient.CircularRedirectException: Circular redirect to
'http://HOST-192-168-47-207:41992/mapreduce/attempts/job_1332261815858_2_2/m/NEW'
at
org.apache.commons.httpclient.HttpMethodDirector.processRedirectResponse(HttpMethodDirector.java:638)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:179)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at
org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.proxyLink(WebAppProxyServlet.java:148)
at
org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:269)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:940)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Powered by Jetty://
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4046) Task Log stdout and stderr don't honor the property mapred.userlog.limit.kb
Task Log stdout and stderr don't honor the property mapred.userlog.limit.kb - Key: MAPREDUCE-4046 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4046 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.0.1 Reporter: Devaraj K Assignee: Devaraj K -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4040) History links should use hostname rather than IP address.
[
https://issues.apache.org/jira/browse/MAPREDUCE-4040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bhallamudi Venkata Siva Kamesh updated MAPREDUCE-4040:
--
Attachment: MAPREDUCE-4040.patch
Pls review this patch
History links should use hostname rather than IP address.
-
Key: MAPREDUCE-4040
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4040
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 0.23.2
Reporter: Bhallamudi Venkata Siva Kamesh
Priority: Minor
Attachments: MAPREDUCE-4040.patch
While navigating from web page (eg: */cluster/app/app-id* ) to HS, browser
displays IP address rather than hostname.
{code:title=JobHistoryUtils.java|borderStyle=solid}
if (address.getAddress().isAnyLocalAddress() ||
address.getAddress().isLoopbackAddress()) {
sb.append(InetAddress.getLocalHost().getHostAddress());
}
}
{code}
I *think* it is better to use hostname rather than IP address.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4040) History links should use hostname rather than IP address.
[
https://issues.apache.org/jira/browse/MAPREDUCE-4040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bhallamudi Venkata Siva Kamesh updated MAPREDUCE-4040:
--
Status: Patch Available (was: Open)
History links should use hostname rather than IP address.
-
Key: MAPREDUCE-4040
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4040
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 0.23.2
Reporter: Bhallamudi Venkata Siva Kamesh
Priority: Minor
Attachments: MAPREDUCE-4040.patch
While navigating from web page (eg: */cluster/app/app-id* ) to HS, browser
displays IP address rather than hostname.
{code:title=JobHistoryUtils.java|borderStyle=solid}
if (address.getAddress().isAnyLocalAddress() ||
address.getAddress().isLoopbackAddress()) {
sb.append(InetAddress.getLocalHost().getHostAddress());
}
}
{code}
I *think* it is better to use hostname rather than IP address.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3955) Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated MAPREDUCE-3955: Target Version/s: 0.24.0, 0.23.3 (was: 0.23.3, 0.24.0) Status: Open (was: Patch Available) Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine. Key: MAPREDUCE-3955 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3955 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: MAPREDUCE-3955-trunk.patch We shouldn't have two rpc engines based on protocol buffers. ProtoOverHadoopRpcEngine in hadoop-yarn-common should be replaced by ProtobufRpcEngine in hadoop-common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3955) Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated MAPREDUCE-3955: Attachment: MAPREDUCE-3955-trunk.patch New patch fixing the test cases. Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine. Key: MAPREDUCE-3955 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3955 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: MAPREDUCE-3955-trunk.patch, MAPREDUCE-3955-trunk.patch We shouldn't have two rpc engines based on protocol buffers. ProtoOverHadoopRpcEngine in hadoop-yarn-common should be replaced by ProtobufRpcEngine in hadoop-common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
