[jira] [Commented] (MAPREDUCE-3317) Rumen TraceBuilder is emiting null as hostname
[ https://issues.apache.org/jira/browse/MAPREDUCE-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13140967#comment-13140967 ] Hudson commented on MAPREDUCE-3317: --- Integrated in Hadoop-Common-trunk-Commit #1220 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1220/]) MAPREDUCE-3317. Rumen TraceBuilder is emiting null as hostname. (Ravi Gummadi via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1195814 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java Rumen TraceBuilder is emiting null as hostname -- Key: MAPREDUCE-3317 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3317 Project: Hadoop Map/Reduce Issue Type: Bug Components: tools/rumen Affects Versions: 0.23.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.23.0 Attachments: 3317.patch Trace generated by Rumen TraceBuilder contains null as hostname even though hostName and rackName are seen in history file. This is after MAPREDUCE-3035. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3317) Rumen TraceBuilder is emiting null as hostname
[ https://issues.apache.org/jira/browse/MAPREDUCE-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13140972#comment-13140972 ] Hudson commented on MAPREDUCE-3317: --- Integrated in Hadoop-Hdfs-trunk-Commit #1296 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1296/]) MAPREDUCE-3317. Rumen TraceBuilder is emiting null as hostname. (Ravi Gummadi via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1195814 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java Rumen TraceBuilder is emiting null as hostname -- Key: MAPREDUCE-3317 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3317 Project: Hadoop Map/Reduce Issue Type: Bug Components: tools/rumen Affects Versions: 0.23.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.23.0 Attachments: 3317.patch Trace generated by Rumen TraceBuilder contains null as hostname even though hostName and rackName are seen in history file. This is after MAPREDUCE-3035. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3246) Make Task extensible to support modifications of Task or even alternate programming paradigms
[ https://issues.apache.org/jira/browse/MAPREDUCE-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-3246: - Attachment: MAPREDUCE-3246-extensible-task.v3.patch Make Task extensible to support modifications of Task or even alternate programming paradigms - Key: MAPREDUCE-3246 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3246 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Affects Versions: 0.23.0 Reporter: Binglin Chang Assignee: Binglin Chang Attachments: MAPREDUCE-3246-extensible-task.patch, MAPREDUCE-3246-extensible-task.v2.patch, MAPREDUCE-3246-extensible-task.v3.patch One of MRv2's goal is to support alternate programming paradigms, but building a application using YARN from the bottom is not trivial. In fact most component of MapReduce can be reused, mostly the scheduler/master side, and we can make changes/extensions only on the task/slave side, such as native tasks, hash-aggregation style combiner/reducer interfaces. The first thing to do I think is to make task/slave side extensible, more specific, the Task in JvmTask should serialized with class name, not simply a boolean isMap, and make task class name configurable in JobConf, there maybe other minor changes. By doing so, developers can at least extends their own MapTask/ReduceTask. I just post my initial thoughts here for opinions. If this change is OK, I can submit a patch, this is just a trivial work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3246) Make Task extensible to support modifications of Task or even alternate programming paradigms
[ https://issues.apache.org/jira/browse/MAPREDUCE-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13140982#comment-13140982 ] Binglin Chang commented on MAPREDUCE-3246: -- Some code just committed(LocalJobRunner) break the build. Make some change to make it comptable with LocalJobRunner Make Task extensible to support modifications of Task or even alternate programming paradigms - Key: MAPREDUCE-3246 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3246 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Affects Versions: 0.23.0 Reporter: Binglin Chang Assignee: Binglin Chang Attachments: MAPREDUCE-3246-extensible-task.patch, MAPREDUCE-3246-extensible-task.v2.patch, MAPREDUCE-3246-extensible-task.v3.patch One of MRv2's goal is to support alternate programming paradigms, but building a application using YARN from the bottom is not trivial. In fact most component of MapReduce can be reused, mostly the scheduler/master side, and we can make changes/extensions only on the task/slave side, such as native tasks, hash-aggregation style combiner/reducer interfaces. The first thing to do I think is to make task/slave side extensible, more specific, the Task in JvmTask should serialized with class name, not simply a boolean isMap, and make task class name configurable in JobConf, there maybe other minor changes. By doing so, developers can at least extends their own MapTask/ReduceTask. I just post my initial thoughts here for opinions. If this change is OK, I can submit a patch, this is just a trivial work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3323) Distributed Cache for Map or Reduce or Both
Distributed Cache for Map or Reduce or Both --- Key: MAPREDUCE-3323 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3323 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Azuryy(Chijiong) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3323) Distributed Cache for Map or Reduce or Both
[ https://issues.apache.org/jira/browse/MAPREDUCE-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Azuryy(Chijiong) updated MAPREDUCE-3323: Description: We put some file into Distributed Cache, but sometimes, only Map or Reduce use thses cached files, not useful for both. but TaskTracker always download cached files from HDFS, if there are some little bit big files in cache, it's time expensive. so, this patch add some new API in the DistributedCache.java as follow: addArchiveToClassPathForMap addArchiveToClassPathForReduce addFileToClassPathForMap addFileToClassPathForReduce addCacheFileForMap addCacheFileForReduce addCacheArchiveForMap addCacheArchiveForReduce New API doesn't affect original interface. but they are specified for only map or reduce, not both of them. But if you do need cache file during both map and reduce, then use original interface. Distributed Cache for Map or Reduce or Both --- Key: MAPREDUCE-3323 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3323 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Azuryy(Chijiong) We put some file into Distributed Cache, but sometimes, only Map or Reduce use thses cached files, not useful for both. but TaskTracker always download cached files from HDFS, if there are some little bit big files in cache, it's time expensive. so, this patch add some new API in the DistributedCache.java as follow: addArchiveToClassPathForMap addArchiveToClassPathForReduce addFileToClassPathForMap addFileToClassPathForReduce addCacheFileForMap addCacheFileForReduce addCacheArchiveForMap addCacheArchiveForReduce New API doesn't affect original interface. but they are specified for only map or reduce, not both of them. But if you do need cache file during both map and reduce, then use original interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3246) Make Task extensible to support modifications of Task or even alternate programming paradigms
[ https://issues.apache.org/jira/browse/MAPREDUCE-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141026#comment-13141026 ] Hadoop QA commented on MAPREDUCE-3246: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501742/MAPREDUCE-3246-extensible-task.v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 1740 javac compiler warnings (more than the trunk's current 1738 warnings). +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1233//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1233//console This message is automatically generated. Make Task extensible to support modifications of Task or even alternate programming paradigms - Key: MAPREDUCE-3246 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3246 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Affects Versions: 0.23.0 Reporter: Binglin Chang Assignee: Binglin Chang Attachments: MAPREDUCE-3246-extensible-task.patch, MAPREDUCE-3246-extensible-task.v2.patch, MAPREDUCE-3246-extensible-task.v3.patch One of MRv2's goal is to support alternate programming paradigms, but building a application using YARN from the bottom is not trivial. In fact most component of MapReduce can be reused, mostly the scheduler/master side, and we can make changes/extensions only on the task/slave side, such as native tasks, hash-aggregation style combiner/reducer interfaces. The first thing to do I think is to make task/slave side extensible, more specific, the Task in JvmTask should serialized with class name, not simply a boolean isMap, and make task class name configurable in JobConf, there maybe other minor changes. By doing so, developers can at least extends their own MapTask/ReduceTask. I just post my initial thoughts here for opinions. If this change is OK, I can submit a patch, this is just a trivial work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3323) Distributed Cache for Map or Reduce or Both
[ https://issues.apache.org/jira/browse/MAPREDUCE-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Azuryy(Chijiong) updated MAPREDUCE-3323: Attachment: dc.patch Distributed Cache for Map or Reduce or Both --- Key: MAPREDUCE-3323 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3323 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Azuryy(Chijiong) Attachments: dc.patch We put some file into Distributed Cache, but sometimes, only Map or Reduce use thses cached files, not useful for both. but TaskTracker always download cached files from HDFS, if there are some little bit big files in cache, it's time expensive. so, this patch add some new API in the DistributedCache.java as follow: addArchiveToClassPathForMap addArchiveToClassPathForReduce addFileToClassPathForMap addFileToClassPathForReduce addCacheFileForMap addCacheFileForReduce addCacheArchiveForMap addCacheArchiveForReduce New API doesn't affect original interface. but they are specified for only map or reduce, not both of them. But if you do need cache file during both map and reduce, then use original interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3323) Distributed Cache for Map or Reduce or Both
[ https://issues.apache.org/jira/browse/MAPREDUCE-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Azuryy(Chijiong) updated MAPREDUCE-3323: Component/s: tasktracker Affects Version/s: 0.20.203.0 Release Note: Tested as follow: 1: Add cache file for map; 2: get cache files in the configure of the map and reduce, then pring some message if map/reduce can get cache file or not conclusion: It does work! Distributed Cache for Map or Reduce or Both --- Key: MAPREDUCE-3323 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3323 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tasktracker Affects Versions: 0.20.203.0 Reporter: Azuryy(Chijiong) Attachments: dc.patch We put some file into Distributed Cache, but sometimes, only Map or Reduce use thses cached files, not useful for both. but TaskTracker always download cached files from HDFS, if there are some little bit big files in cache, it's time expensive. so, this patch add some new API in the DistributedCache.java as follow: addArchiveToClassPathForMap addArchiveToClassPathForReduce addFileToClassPathForMap addFileToClassPathForReduce addCacheFileForMap addCacheFileForReduce addCacheArchiveForMap addCacheArchiveForReduce New API doesn't affect original interface. but they are specified for only map or reduce, not both of them. But if you do need cache file during both map and reduce, then use original interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3323) Distributed Cache for Map or Reduce or Both
[ https://issues.apache.org/jira/browse/MAPREDUCE-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Azuryy(Chijiong) updated MAPREDUCE-3323: Component/s: distributed-cache Distributed Cache for Map or Reduce or Both --- Key: MAPREDUCE-3323 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3323 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distributed-cache, tasktracker Affects Versions: 0.20.203.0 Reporter: Azuryy(Chijiong) Attachments: dc.patch We put some file into Distributed Cache, but sometimes, only Map or Reduce use thses cached files, not useful for both. but TaskTracker always download cached files from HDFS, if there are some little bit big files in cache, it's time expensive. so, this patch add some new API in the DistributedCache.java as follow: addArchiveToClassPathForMap addArchiveToClassPathForReduce addFileToClassPathForMap addFileToClassPathForReduce addCacheFileForMap addCacheFileForReduce addCacheArchiveForMap addCacheArchiveForReduce New API doesn't affect original interface. but they are specified for only map or reduce, not both of them. But if you do need cache file during both map and reduce, then use original interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3323) Distributed Cache for Map or Reduce or Both
[ https://issues.apache.org/jira/browse/MAPREDUCE-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Azuryy(Chijiong) updated MAPREDUCE-3323: Release Note: Tested as follow: 1: Add cache file for map; 2: get cache files in the configure of the map and reduce, then print some messages if map/reduce can get cache file or not. 3: Three test cases: cache for mapred, cache for map, cache for reduce For the first case, both map and reduce can get local files from the distributed cache. The second case, Map Task can get local files from the distributed cache, but reduce can not. You know what happened during the third case. conclusion: It does work well. was: Tested as follow: 1: Add cache file for map; 2: get cache files in the configure of the map and reduce, then pring some message if map/reduce can get cache file or not conclusion: It does work! Distributed Cache for Map or Reduce or Both --- Key: MAPREDUCE-3323 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3323 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distributed-cache, tasktracker Affects Versions: 0.20.203.0 Reporter: Azuryy(Chijiong) Attachments: dc.patch We put some file into Distributed Cache, but sometimes, only Map or Reduce use thses cached files, not useful for both. but TaskTracker always download cached files from HDFS, if there are some little bit big files in cache, it's time expensive. so, this patch add some new API in the DistributedCache.java as follow: addArchiveToClassPathForMap addArchiveToClassPathForReduce addFileToClassPathForMap addFileToClassPathForReduce addCacheFileForMap addCacheFileForReduce addCacheArchiveForMap addCacheArchiveForReduce New API doesn't affect original interface. but they are specified for only map or reduce, not both of them. But if you do need cache file during both map and reduce, then use original interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3246) Make Task extensible to support modifications of Task or even alternate programming paradigms
[ https://issues.apache.org/jira/browse/MAPREDUCE-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141039#comment-13141039 ] Binglin Chang commented on MAPREDUCE-3246: -- The extra warnings are deprecation warnings and I can't make them disappear. org/apache/hadoop/mapred/MapTask.java:[109,36] [deprecation] org.apache.hadoop.mapred.JobConf in org.apache.hadoop.mapred has been deprecated Make Task extensible to support modifications of Task or even alternate programming paradigms - Key: MAPREDUCE-3246 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3246 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Affects Versions: 0.23.0 Reporter: Binglin Chang Assignee: Binglin Chang Attachments: MAPREDUCE-3246-extensible-task.patch, MAPREDUCE-3246-extensible-task.v2.patch, MAPREDUCE-3246-extensible-task.v3.patch One of MRv2's goal is to support alternate programming paradigms, but building a application using YARN from the bottom is not trivial. In fact most component of MapReduce can be reused, mostly the scheduler/master side, and we can make changes/extensions only on the task/slave side, such as native tasks, hash-aggregation style combiner/reducer interfaces. The first thing to do I think is to make task/slave side extensible, more specific, the Task in JvmTask should serialized with class name, not simply a boolean isMap, and make task class name configurable in JobConf, there maybe other minor changes. By doing so, developers can at least extends their own MapTask/ReduceTask. I just post my initial thoughts here for opinions. If this change is OK, I can submit a patch, this is just a trivial work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3323) Distributed Cache for Map or Reduce or Both
[ https://issues.apache.org/jira/browse/MAPREDUCE-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Azuryy(Chijiong) updated MAPREDUCE-3323: Attachment: (was: dc.patch) Distributed Cache for Map or Reduce or Both --- Key: MAPREDUCE-3323 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3323 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distributed-cache, tasktracker Affects Versions: 0.20.203.0 Reporter: Azuryy(Chijiong) Attachments: DistributedCache.patch, TaskTracker.patch We put some file into Distributed Cache, but sometimes, only Map or Reduce use thses cached files, not useful for both. but TaskTracker always download cached files from HDFS, if there are some little bit big files in cache, it's time expensive. so, this patch add some new API in the DistributedCache.java as follow: addArchiveToClassPathForMap addArchiveToClassPathForReduce addFileToClassPathForMap addFileToClassPathForReduce addCacheFileForMap addCacheFileForReduce addCacheArchiveForMap addCacheArchiveForReduce New API doesn't affect original interface. but they are specified for only map or reduce, not both of them. But if you do need cache file during both map and reduce, then use original interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3323) Distributed Cache for Map or Reduce or Both
[ https://issues.apache.org/jira/browse/MAPREDUCE-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Azuryy(Chijiong) updated MAPREDUCE-3323: Attachment: TaskTracker.patch DistributedCache.patch Distributed Cache for Map or Reduce or Both --- Key: MAPREDUCE-3323 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3323 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distributed-cache, tasktracker Affects Versions: 0.20.203.0 Reporter: Azuryy(Chijiong) Attachments: DistributedCache.patch, TaskTracker.patch We put some file into Distributed Cache, but sometimes, only Map or Reduce use thses cached files, not useful for both. but TaskTracker always download cached files from HDFS, if there are some little bit big files in cache, it's time expensive. so, this patch add some new API in the DistributedCache.java as follow: addArchiveToClassPathForMap addArchiveToClassPathForReduce addFileToClassPathForMap addFileToClassPathForReduce addCacheFileForMap addCacheFileForReduce addCacheArchiveForMap addCacheArchiveForReduce New API doesn't affect original interface. but they are specified for only map or reduce, not both of them. But if you do need cache file during both map and reduce, then use original interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3323) Distributed Cache for Map or Reduce or Both
[ https://issues.apache.org/jira/browse/MAPREDUCE-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Azuryy(Chijiong) updated MAPREDUCE-3323: Release Note: Tested as follow: 1: Add cache file for map/reduce; 2: get cache files in the configure of the map and reduce, then print some messages if map/reduce can get cache file or not. 3: Three test cases: cache for mapred, cache for map, cache for reduce For the first case, both map and reduce can get local files from the distributed cache. The second case, Map Task can get local files from the distributed cache, but reduce can not. You know what happened during the third case. conclusion: It does work well. was: Tested as follow: 1: Add cache file for map; 2: get cache files in the configure of the map and reduce, then print some messages if map/reduce can get cache file or not. 3: Three test cases: cache for mapred, cache for map, cache for reduce For the first case, both map and reduce can get local files from the distributed cache. The second case, Map Task can get local files from the distributed cache, but reduce can not. You know what happened during the third case. conclusion: It does work well. Distributed Cache for Map or Reduce or Both --- Key: MAPREDUCE-3323 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3323 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distributed-cache, tasktracker Affects Versions: 0.20.203.0 Reporter: Azuryy(Chijiong) Attachments: DistributedCache.patch, TaskTracker.patch We put some file into Distributed Cache, but sometimes, only Map or Reduce use thses cached files, not useful for both. but TaskTracker always download cached files from HDFS, if there are some little bit big files in cache, it's time expensive. so, this patch add some new API in the DistributedCache.java as follow: addArchiveToClassPathForMap addArchiveToClassPathForReduce addFileToClassPathForMap addFileToClassPathForReduce addCacheFileForMap addCacheFileForReduce addCacheArchiveForMap addCacheArchiveForReduce New API doesn't affect original interface. but they are specified for only map or reduce, not both of them. But if you do need cache file during both map and reduce, then use original interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3321) Disable some failing legacy tests for MRv2 builds to go through
[ https://issues.apache.org/jira/browse/MAPREDUCE-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141121#comment-13141121 ] Hudson commented on MAPREDUCE-3321: --- Integrated in Hadoop-Hdfs-trunk #850 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/850/]) MAPREDUCE-3321. Disabled a few MR tests for 0.23. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1195743 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/MapAttemptFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/ReduceAttemptFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/cli/TestMRCLI.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/conf/TestNoDefaultsJobConf.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestAuditLogger.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSeveral.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestNoJobSetupCleanup.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestTaskContext.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/token/delegation/TestDelegationToken.java Disable some failing legacy tests for MRv2 builds to go through --- Key: MAPREDUCE-3321 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3321 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Minor Fix For: 0.23.0 Attachments: MR-3321.1.patch By-product of MR-3214. Disable tests for the short term until fixes are available for all tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3322) Create a better index.html for maven docs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141120#comment-13141120 ] Hudson commented on MAPREDUCE-3322: --- Integrated in Hadoop-Hdfs-trunk #850 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/850/]) MAPREDUCE-3322. Added a better index.html and an brief overview of YARN architecture. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1195764 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/YARN.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/yarn_architecture.gif * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/site.xml Create a better index.html for maven docs - Key: MAPREDUCE-3322 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3322 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.0 Attachments: MAPREDUCE-3322.patch Create a better index.html for maven docs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3035) MR V2 jobhistory does not contain rack information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141122#comment-13141122 ] Hudson commented on MAPREDUCE-3035: --- Integrated in Hadoop-Hdfs-trunk #850 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/850/]) MAPREDUCE-3035. Fixed MR JobHistory to ensure rack information is present. Contributed by chakravarthy. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1195575 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/TaskAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/MapAttemptFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/ReduceAttemptFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTaskAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTaskPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/HadoopLogsAnalyzer.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobBuilder.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/MapAttempt20LineHistoryEventEmitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ReduceAttempt20LineHistoryEventEmitter.java MR V2 jobhistory does not contain rack information -- Key: MAPREDUCE-3035 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3035 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Karam Singh Assignee: chackaravarthy Priority: Critical Fix For: 0.23.0 Attachments: MAPREDUCE-3035-1.patch, MAPREDUCE-3035-20111031.1.txt, MAPREDUCE-3035.patch When topology.node.switch.mapping.impl is set to enable rack-locality resolution via the topology script, from the RM web-UI, we can see the rack information for each node. Running a job also reveals the information about rack-local map tasks launched at end of job completion on the client side. But the hostname field for attempts in the JobHistory does not contain this rack information. In case of hadoop-0.20 securiy or MRV1, hostname field of job history does contain rackid/hostname whereas in MRV2, hostname field only contains the hostIP. Thus this is a regression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3237) Move LocalJobRunner to hadoop-mapreduce-client-core module
[ https://issues.apache.org/jira/browse/MAPREDUCE-3237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141130#comment-13141130 ] Hudson commented on MAPREDUCE-3237: --- Integrated in Hadoop-Hdfs-trunk #850 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/850/]) MAPREDUCE-3237. Move LocalJobRunner to hadoop-mapreduce-client-core. Contributed by Tom White. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1195792 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalClientProtocolProvider.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunnerMetrics.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF/services * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF/services/org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestFSDownload.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/FSDownload.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestFSDownload.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/META-INF/services/org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/LocalClientProtocolProvider.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/LocalJobRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestMRWithDistributedCache.java Move LocalJobRunner to hadoop-mapreduce-client-core module -- Key: MAPREDUCE-3237 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3237 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Tom White Assignee: Tom White Fix For: 0.23.1 Attachments: MAPREDUCE-3237-consolidated.patch, MAPREDUCE-3237-consolidated.patch, MAPREDUCE-3237-consolidated.patch, MAPREDUCE-3237.patch, MAPREDUCE-3237.patch, MAPREDUCE-3237.patch LocalJobRunner works independently of MR1 (jobtracker and tasktrackers) and MR2 (YARN). The MR1 directory is being kept around only to support unit tests, so LocalJobRunner should be moved out to somewhere more permanent. -- This
[jira] [Commented] (MAPREDUCE-3220) ant test TestCombineOutputCollector failing on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-3220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141129#comment-13141129 ] Hudson commented on MAPREDUCE-3220: --- Integrated in Hadoop-Hdfs-trunk #850 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/850/]) MAPREDUCE-3220. Fixed TestCombineOutputCollector. Contributed by Devaraj K. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1195745 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestCombineOutputCollector.java ant test TestCombineOutputCollector failing on trunk Key: MAPREDUCE-3220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Devaraj K Priority: Minor Fix For: 0.23.1 Attachments: MAPREDUCE-3220.patch Testsuite: org.apache.hadoop.mapred.TestCombineOutputCollector Tests run: 2, Failures: 2, Errors: 0, Time elapsed: 1.591 sec Testcase: testCustomCollect took 0.363 sec FAILED taskReporter.progress(); Never wanted here: - at org.apache.hadoop.mapred.TestCombineOutputCollector.testCustomCollect(TestCombineOutputCollector.java:118) But invoked here: - at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1202) junit.framework.AssertionFailedError: taskReporter.progress(); Never wanted here: - at org.apache.hadoop.mapred.TestCombineOutputCollector.testCustomCollect(TestCombineOutputCollector.java:118) But invoked here: - at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1202) at org.apache.hadoop.mapred.TestCombineOutputCollector.testCustomCollect(TestCombineOutputCollector.java:118) Testcase: testDefaultCollect took 1.211 sec FAILED taskReporter.progress(); Wanted 1 time: - at org.apache.hadoop.mapred.TestCombineOutputCollector.testDefaultCollect(TestCombineOutputCollector.java:139) But was 1 times. Undesired invocation: - at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1202) junit.framework.AssertionFailedError: taskReporter.progress(); Wanted 1 time: - at org.apache.hadoop.mapred.TestCombineOutputCollector.testDefaultCollect(TestCombineOutputCollector.java:139) But was 1 times. Undesired invocation: - at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1202) at org.apache.hadoop.mapred.TestCombineOutputCollector.testDefaultCollect(TestCombineOutputCollector.java:139) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3316) Rebooted link is not working properly
[
https://issues.apache.org/jira/browse/MAPREDUCE-3316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141137#comment-13141137
]
Hudson commented on MAPREDUCE-3316:
---
Integrated in Hadoop-Hdfs-trunk #850 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/850/])
MAPREDUCE-3316. Rebooted link is not working properly. (Bhallamudi Venkata
Siva Kamesh via mahadev)
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1195805
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeState.java
Rebooted link is not working properly
-
Key: MAPREDUCE-3316
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3316
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: resourcemanager
Affects Versions: 0.23.0
Environment: NA
Reporter: Bhallamudi Venkata Siva Kamesh
Assignee: Bhallamudi Venkata Siva Kamesh
Fix For: 0.23.0
Attachments: MapReduce-3316.patch, reboot.png
While clicking on the *Rebooted Nodes* link, it is showing the following
error message
{color:red}Sorry, got error 500{color}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3275) Add docs for WebAppProxy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141136#comment-13141136 ] Hudson commented on MAPREDUCE-3275: --- Integrated in Hadoop-Hdfs-trunk #850 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/850/]) MAPREDUCE-3275. Added documentation for AM WebApp Proxy. Contributed by Robert Evans. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1195579 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm Add docs for WebAppProxy Key: MAPREDUCE-3275 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3275 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation, mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Priority: Critical Fix For: 0.23.0 Attachments: MR-3275.txt, MR-3275.txt In my haste to get the WebAppProxy code in the documentation for it was neglected. This is to fix that. Docs need to be added to ClusterSetup.html about how to configure and use the WebAppProxy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3317) Rumen TraceBuilder is emiting null as hostname
[ https://issues.apache.org/jira/browse/MAPREDUCE-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141150#comment-13141150 ] Amar Kamat commented on MAPREDUCE-3317: --- What should be the expected output if either hostname or rackname is null? We should make sure that Rumen's output is consistent and should match the previous version. Rumen TraceBuilder is emiting null as hostname -- Key: MAPREDUCE-3317 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3317 Project: Hadoop Map/Reduce Issue Type: Bug Components: tools/rumen Affects Versions: 0.23.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.23.0 Attachments: 3317.patch Trace generated by Rumen TraceBuilder contains null as hostname even though hostName and rackName are seen in history file. This is after MAPREDUCE-3035. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3322) Create a better index.html for maven docs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141154#comment-13141154 ] Hudson commented on MAPREDUCE-3322: --- Integrated in Hadoop-Mapreduce-trunk #884 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/884/]) MAPREDUCE-3322. Added a better index.html and an brief overview of YARN architecture. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1195764 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/YARN.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/yarn_architecture.gif * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/site.xml Create a better index.html for maven docs - Key: MAPREDUCE-3322 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3322 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.0 Attachments: MAPREDUCE-3322.patch Create a better index.html for maven docs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3321) Disable some failing legacy tests for MRv2 builds to go through
[ https://issues.apache.org/jira/browse/MAPREDUCE-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141155#comment-13141155 ] Hudson commented on MAPREDUCE-3321: --- Integrated in Hadoop-Mapreduce-trunk #884 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/884/]) MAPREDUCE-3321. Disabled a few MR tests for 0.23. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1195743 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/MapAttemptFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/ReduceAttemptFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/cli/TestMRCLI.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/conf/TestNoDefaultsJobConf.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestAuditLogger.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSeveral.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestNoJobSetupCleanup.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestTaskContext.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/token/delegation/TestDelegationToken.java Disable some failing legacy tests for MRv2 builds to go through --- Key: MAPREDUCE-3321 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3321 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Minor Fix For: 0.23.0 Attachments: MR-3321.1.patch By-product of MR-3214. Disable tests for the short term until fixes are available for all tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3035) MR V2 jobhistory does not contain rack information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141156#comment-13141156 ] Hudson commented on MAPREDUCE-3035: --- Integrated in Hadoop-Mapreduce-trunk #884 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/884/]) MAPREDUCE-3035. Fixed MR JobHistory to ensure rack information is present. Contributed by chakravarthy. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1195575 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/TaskAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/MapAttemptFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/ReduceAttemptFinishedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTaskAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTaskPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/HadoopLogsAnalyzer.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobBuilder.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/MapAttempt20LineHistoryEventEmitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ReduceAttempt20LineHistoryEventEmitter.java MR V2 jobhistory does not contain rack information -- Key: MAPREDUCE-3035 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3035 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Karam Singh Assignee: chackaravarthy Priority: Critical Fix For: 0.23.0 Attachments: MAPREDUCE-3035-1.patch, MAPREDUCE-3035-20111031.1.txt, MAPREDUCE-3035.patch When topology.node.switch.mapping.impl is set to enable rack-locality resolution via the topology script, from the RM web-UI, we can see the rack information for each node. Running a job also reveals the information about rack-local map tasks launched at end of job completion on the client side. But the hostname field for attempts in the JobHistory does not contain this rack information. In case of hadoop-0.20 securiy or MRV1, hostname field of job history does contain rackid/hostname whereas in MRV2, hostname field only contains the hostIP. Thus this is a regression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3237) Move LocalJobRunner to hadoop-mapreduce-client-core module
[ https://issues.apache.org/jira/browse/MAPREDUCE-3237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141164#comment-13141164 ] Hudson commented on MAPREDUCE-3237: --- Integrated in Hadoop-Mapreduce-trunk #884 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/884/]) MAPREDUCE-3237. Move LocalJobRunner to hadoop-mapreduce-client-core. Contributed by Tom White. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1195792 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalClientProtocolProvider.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunnerMetrics.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF/services * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF/services/org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestFSDownload.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/FSDownload.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestFSDownload.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/META-INF/services/org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/LocalClientProtocolProvider.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/LocalJobRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestMRWithDistributedCache.java Move LocalJobRunner to hadoop-mapreduce-client-core module -- Key: MAPREDUCE-3237 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3237 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Tom White Assignee: Tom White Fix For: 0.23.1 Attachments: MAPREDUCE-3237-consolidated.patch, MAPREDUCE-3237-consolidated.patch, MAPREDUCE-3237-consolidated.patch, MAPREDUCE-3237.patch, MAPREDUCE-3237.patch, MAPREDUCE-3237.patch LocalJobRunner works independently of MR1 (jobtracker and tasktrackers) and MR2 (YARN). The MR1 directory is being kept around only to support unit tests, so LocalJobRunner should be moved out to somewhere more permanent. --
[jira] [Commented] (MAPREDUCE-3220) ant test TestCombineOutputCollector failing on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-3220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141163#comment-13141163 ] Hudson commented on MAPREDUCE-3220: --- Integrated in Hadoop-Mapreduce-trunk #884 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/884/]) MAPREDUCE-3220. Fixed TestCombineOutputCollector. Contributed by Devaraj K. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1195745 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestCombineOutputCollector.java ant test TestCombineOutputCollector failing on trunk Key: MAPREDUCE-3220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Devaraj K Priority: Minor Fix For: 0.23.1 Attachments: MAPREDUCE-3220.patch Testsuite: org.apache.hadoop.mapred.TestCombineOutputCollector Tests run: 2, Failures: 2, Errors: 0, Time elapsed: 1.591 sec Testcase: testCustomCollect took 0.363 sec FAILED taskReporter.progress(); Never wanted here: - at org.apache.hadoop.mapred.TestCombineOutputCollector.testCustomCollect(TestCombineOutputCollector.java:118) But invoked here: - at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1202) junit.framework.AssertionFailedError: taskReporter.progress(); Never wanted here: - at org.apache.hadoop.mapred.TestCombineOutputCollector.testCustomCollect(TestCombineOutputCollector.java:118) But invoked here: - at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1202) at org.apache.hadoop.mapred.TestCombineOutputCollector.testCustomCollect(TestCombineOutputCollector.java:118) Testcase: testDefaultCollect took 1.211 sec FAILED taskReporter.progress(); Wanted 1 time: - at org.apache.hadoop.mapred.TestCombineOutputCollector.testDefaultCollect(TestCombineOutputCollector.java:139) But was 1 times. Undesired invocation: - at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1202) junit.framework.AssertionFailedError: taskReporter.progress(); Wanted 1 time: - at org.apache.hadoop.mapred.TestCombineOutputCollector.testDefaultCollect(TestCombineOutputCollector.java:139) But was 1 times. Undesired invocation: - at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1202) at org.apache.hadoop.mapred.TestCombineOutputCollector.testDefaultCollect(TestCombineOutputCollector.java:139) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3103) Implement Job ACLs for MRAppMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141158#comment-13141158 ] Hudson commented on MAPREDUCE-3103: --- Integrated in Hadoop-Mapreduce-trunk #884 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/884/]) MAPREDUCE-3103. Implement Job ACLs for MRAppMaster. (mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1195761 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AppController.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java Implement Job ACLs for MRAppMaster -- Key: MAPREDUCE-3103 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3103 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, security Reporter: Vinod Kumar Vavilapalli Assignee: Mahadev konar Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3103.patch, MAPREDUCE-3103.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3275) Add docs for WebAppProxy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141170#comment-13141170 ] Hudson commented on MAPREDUCE-3275: --- Integrated in Hadoop-Mapreduce-trunk #884 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/884/]) MAPREDUCE-3275. Added documentation for AM WebApp Proxy. Contributed by Robert Evans. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1195579 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm Add docs for WebAppProxy Key: MAPREDUCE-3275 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3275 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation, mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Priority: Critical Fix For: 0.23.0 Attachments: MR-3275.txt, MR-3275.txt In my haste to get the WebAppProxy code in the documentation for it was neglected. This is to fix that. Docs need to be added to ClusterSetup.html about how to configure and use the WebAppProxy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3241) (Rumen)TraceBuilder throws IllegalArgumentException
[
https://issues.apache.org/jira/browse/MAPREDUCE-3241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141174#comment-13141174
]
Hudson commented on MAPREDUCE-3241:
---
Integrated in Hadoop-Mapreduce-trunk #884 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/884/])
MAPREDUCE-3241. [Rumen] Fix Rumen to ignore the AMStartedEvent. (amarrk)
amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1195535
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobBuilder.java
(Rumen)TraceBuilder throws IllegalArgumentException
---
Key: MAPREDUCE-3241
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3241
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.24.0
Reporter: Devaraj K
Assignee: Amar Kamat
Fix For: 0.24.0
Attachments: mapreduce-3241-v1.1.patch
When we run the TraceBuilder, we get this exception. Output of the
TraceBuilder doesn't contain the map and reduce task information.
{code}
2011-10-21 22:07:17,268 WARN rumen.TraceBuilder (TraceBuilder.java:run(272))
- TraceBuilder got an error while processing the [possibly virtual] file
job_1319214405771_0002-1319214846458-root-word+count-1319214871038-1-1-SUCCEEDED.jhist
within Path
hdfs://10.18.52.57:9000/user/root/null/history/done_intermediate/root/job_1319214405771_0002-1319214846458-root-word+count-1319214871038-1-1-SUCCEEDED.jhist
java.lang.IllegalArgumentException: JobBuilder.process(HistoryEvent): unknown
event type
at
org.apache.hadoop.tools.rumen.JobBuilder.process(JobBuilder.java:165)
at
org.apache.hadoop.tools.rumen.TraceBuilder.processJobHistory(TraceBuilder.java:304)
at
org.apache.hadoop.tools.rumen.TraceBuilder.run(TraceBuilder.java:258)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at
org.apache.hadoop.tools.rumen.TraceBuilder.main(TraceBuilder.java:185)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:189)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3316) Rebooted link is not working properly
[
https://issues.apache.org/jira/browse/MAPREDUCE-3316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141171#comment-13141171
]
Hudson commented on MAPREDUCE-3316:
---
Integrated in Hadoop-Mapreduce-trunk #884 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/884/])
MAPREDUCE-3316. Rebooted link is not working properly. (Bhallamudi Venkata
Siva Kamesh via mahadev)
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1195805
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeState.java
Rebooted link is not working properly
-
Key: MAPREDUCE-3316
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3316
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: resourcemanager
Affects Versions: 0.23.0
Environment: NA
Reporter: Bhallamudi Venkata Siva Kamesh
Assignee: Bhallamudi Venkata Siva Kamesh
Fix For: 0.23.0
Attachments: MapReduce-3316.patch, reboot.png
While clicking on the *Rebooted Nodes* link, it is showing the following
error message
{color:red}Sorry, got error 500{color}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3317) Rumen TraceBuilder is emiting null as hostname
[ https://issues.apache.org/jira/browse/MAPREDUCE-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141172#comment-13141172 ] Hudson commented on MAPREDUCE-3317: --- Integrated in Hadoop-Mapreduce-trunk #884 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/884/]) MAPREDUCE-3317. Rumen TraceBuilder is emiting null as hostname. (Ravi Gummadi via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1195814 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java Rumen TraceBuilder is emiting null as hostname -- Key: MAPREDUCE-3317 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3317 Project: Hadoop Map/Reduce Issue Type: Bug Components: tools/rumen Affects Versions: 0.23.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.23.0 Attachments: 3317.patch Trace generated by Rumen TraceBuilder contains null as hostname even though hostName and rackName are seen in history file. This is after MAPREDUCE-3035. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3139) SlivePartitioner generates negative partitions
[
https://issues.apache.org/jira/browse/MAPREDUCE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan reassigned MAPREDUCE-3139:
--
Assignee: Jakob Homan
SlivePartitioner generates negative partitions
--
Key: MAPREDUCE-3139
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3139
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 0.22.0
Reporter: Konstantin Shvachko
Assignee: Jakob Homan
Fix For: 0.24.0
{{SlivePartitioner.getPartition()}} returns negative partition numbers on
some occasions, which is illegal.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141310#comment-13141310 ] Tom White commented on MAPREDUCE-3169: -- Another nice benefit of this approach (either of Todd's 1 or 2) is that we could write a LocalJobRunner implementation of MiniMRCluster and get increased test coverage (e.g. by writing a parameterized test superclass, see http://junit.sourceforge.net/javadoc/org/junit/runners/Parameterized.html). This would be a follow up JIRA. Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2 --- Key: MAPREDUCE-3169 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, mrv2, test Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Ahmed Radwan Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well. This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be recompile-compatible with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3324) HttpServer tools are not accessible through the UI Tools
HttpServer tools are not accessible through the UI Tools Key: MAPREDUCE-3324 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3324 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3325) Improvements to CapacityScheduler doc
[ https://issues.apache.org/jira/browse/MAPREDUCE-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned MAPREDUCE-3325: Assignee: Thomas Graves Improvements to CapacityScheduler doc - Key: MAPREDUCE-3325 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3325 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves I noticed the following issues with the capacity scheduler doc: ./hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm - In overview section, 3rd paragraph, sentence There is an added benefit that an organization can access any excess capacity no being used by others. No should be not. - in overview section, 4th paragraph. dispropotionate misspelled - in features section, under multitenancy - monopolizing is misspelled. - in features section, under operability - it doesn't say if you can delete queues at runtime? I see there is a note at the end but perhaps that can be added into the other sections to since its easy to miss that Note at the very end. - in features section - hierarchy and Hierarchical mispelled. - under configuration section the class to turn on to use capacity scheduler should be: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler - section on setting up queues, 4th sentence - hierarchy misspelled as heirarcy and heirarchy. - I think specifying how a user has to specify the queue when running a job/app would be useful information. Especially with the new hierarchical queues. Does the user have to specify the entire path like a.b.c or can they just specify c. - under Running and Pending Application Limits section, property yarn.scheduler.capacity.maximum-applications, they are referred to them as jobs, I believe that should be applications. - misspelled concurrently as concurently in same section of maximum-applications. - I think it should specify the defaults (if any) for the config vars. Also what format are they specified in - int, float,etc? - might be nice to say it doesn't support preemption. - under admin options yarn.scheduler.capacity.queue-path.state - queues misspelled as queueus - under changing queue configuration it should have yarn in front of the rmadmin -refreshQueues. Similarly a few lines down at $YARN_HOME/bin/rmadmin -refreshQueues -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3325) Improvements to CapacityScheduler doc
Improvements to CapacityScheduler doc - Key: MAPREDUCE-3325 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3325 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves I noticed the following issues with the capacity scheduler doc: ./hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm - In overview section, 3rd paragraph, sentence There is an added benefit that an organization can access any excess capacity no being used by others. No should be not. - in overview section, 4th paragraph. dispropotionate misspelled - in features section, under multitenancy - monopolizing is misspelled. - in features section, under operability - it doesn't say if you can delete queues at runtime? I see there is a note at the end but perhaps that can be added into the other sections to since its easy to miss that Note at the very end. - in features section - hierarchy and Hierarchical mispelled. - under configuration section the class to turn on to use capacity scheduler should be: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler - section on setting up queues, 4th sentence - hierarchy misspelled as heirarcy and heirarchy. - I think specifying how a user has to specify the queue when running a job/app would be useful information. Especially with the new hierarchical queues. Does the user have to specify the entire path like a.b.c or can they just specify c. - under Running and Pending Application Limits section, property yarn.scheduler.capacity.maximum-applications, they are referred to them as jobs, I believe that should be applications. - misspelled concurrently as concurently in same section of maximum-applications. - I think it should specify the defaults (if any) for the config vars. Also what format are they specified in - int, float,etc? - might be nice to say it doesn't support preemption. - under admin options yarn.scheduler.capacity.queue-path.state - queues misspelled as queueus - under changing queue configuration it should have yarn in front of the rmadmin -refreshQueues. Similarly a few lines down at $YARN_HOME/bin/rmadmin -refreshQueues -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3326) RM web UI scheduler link not as useful as should be
RM web UI scheduler link not as useful as should be --- Key: MAPREDUCE-3326 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3326 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Priority: Critical The resource manager web ui page for scheduler doesn't have all the information about the configuration like the jobtracker page used to have. The things it seems to show you are the current queues - each queues used, set, and max percent and then what apps are running in that queue. It doesn't list any of yarn.scheduler.capacity.maximum-applications, yarn.scheduler.capacity.maximum-am-resource-percent, yarn.scheduler.capacity.queue-path.user-limit-factor, yarn.scheduler.capacity.queue-path.minimum-user-limit-percent, queue state, active users and percent used by user -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141402#comment-13141402 ] Ahmed Radwan commented on MAPREDUCE-3169: - Yes, this will be a very nice benefit. I am now going through the tests changing them to try to assess which way is better. Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2 --- Key: MAPREDUCE-3169 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, mrv2, test Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Ahmed Radwan Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well. This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be recompile-compatible with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3215) org.apache.hadoop.mapreduce.TestNoJobSetupCleanup failing on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141416#comment-13141416 ] Hitesh Shah commented on MAPREDUCE-3215: @Devaraj, if you have not already started working on this, I can take this up. org.apache.hadoop.mapreduce.TestNoJobSetupCleanup failing on trunk -- Key: MAPREDUCE-3215 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3215 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Devaraj K Priority: Minor Fix For: 0.24.0 Testcase: testNoJobSetupCleanup took 13.271 sec FAILED Number of part-files is 0 and not 1 junit.framework.AssertionFailedError: Number of part-files is 0 and not 1 at org.apache.hadoop.mapreduce.TestNoJobSetupCleanup.submitAndValidateJob(TestNoJobSetupCleanup.java:60) at org.apache.hadoop.mapreduce.TestNoJobSetupCleanup.testNoJobSetupCleanup(TestNoJobSetupCleanup.java:70) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3324) Not All HttpServer tools links (stacks,logs,config,jmx,metrics) are accessible through all UI servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3324: --- Description: Nodemanager has no tools listed under tools UI. Jobhistory server has no logs tool listed under tools UI. Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Summary: Not All HttpServer tools links (stacks,logs,config,jmx,metrics) are accessible through all UI servers (was: HttpServer tools are not accessible through the UI Tools) Not All HttpServer tools links (stacks,logs,config,jmx,metrics) are accessible through all UI servers - Key: MAPREDUCE-3324 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3324 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Nodemanager has no tools listed under tools UI. Jobhistory server has no logs tool listed under tools UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3215) org.apache.hadoop.mapreduce.TestNoJobSetupCleanup failing on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141483#comment-13141483 ] Hitesh Shah commented on MAPREDUCE-3215: The failure is due to commit job not being called. org.apache.hadoop.mapreduce.TestNoJobSetupCleanup failing on trunk -- Key: MAPREDUCE-3215 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3215 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Devaraj K Priority: Minor Fix For: 0.24.0 Testcase: testNoJobSetupCleanup took 13.271 sec FAILED Number of part-files is 0 and not 1 junit.framework.AssertionFailedError: Number of part-files is 0 and not 1 at org.apache.hadoop.mapreduce.TestNoJobSetupCleanup.submitAndValidateJob(TestNoJobSetupCleanup.java:60) at org.apache.hadoop.mapreduce.TestNoJobSetupCleanup.testNoJobSetupCleanup(TestNoJobSetupCleanup.java:70) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3324) Not All HttpServer tools links (stacks,logs,config,jmx,metrics) are accessible through all UI servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3324: --- Attachment: MAPREDUCE-3324.patch Not All HttpServer tools links (stacks,logs,config,jmx,metrics) are accessible through all UI servers - Key: MAPREDUCE-3324 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3324 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3324.patch Nodemanager has no tools listed under tools UI. Jobhistory server has no logs tool listed under tools UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3324) Not All HttpServer tools links (stacks,logs,config,jmx,metrics) are accessible through all UI servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3324: --- Target Version/s: 0.24.0, 0.23.1 (was: 0.24.0, 0.23.0) Status: Patch Available (was: Open) Not All HttpServer tools links (stacks,logs,config,jmx,metrics) are accessible through all UI servers - Key: MAPREDUCE-3324 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3324 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3324.patch Nodemanager has no tools listed under tools UI. Jobhistory server has no logs tool listed under tools UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3324) Not All HttpServer tools links (stacks,logs,config,jmx,metrics) are accessible through all UI servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141529#comment-13141529 ] Hadoop QA commented on MAPREDUCE-3324: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501815/MAPREDUCE-3324.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1234//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1234//console This message is automatically generated. Not All HttpServer tools links (stacks,logs,config,jmx,metrics) are accessible through all UI servers - Key: MAPREDUCE-3324 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3324 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3324.patch Nodemanager has no tools listed under tools UI. Jobhistory server has no logs tool listed under tools UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3324) Not All HttpServer tools links (stacks,logs,config,jmx,metrics) are accessible through all UI servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141530#comment-13141530 ] Jonathan Eagles commented on MAPREDUCE-3324: No tests needed. Just adding links to web UI. Not All HttpServer tools links (stacks,logs,config,jmx,metrics) are accessible through all UI servers - Key: MAPREDUCE-3324 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3324 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3324.patch Nodemanager has no tools listed under tools UI. Jobhistory server has no logs tool listed under tools UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3327) RM web ui scheduler link doesn't show correct max value for queues
RM web ui scheduler link doesn't show correct max value for queues -- Key: MAPREDUCE-3327 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3327 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Priority: Critical Configure a cluster to use the capacity scheduler and then specifying a maximum-capacity 100% for a queue. If you go to the RM Web UI and hover over the queue, it always shows the max at 100%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3328) mapred queue -list output inconsistent and missing child queues
mapred queue -list output inconsistent and missing child queues --- Key: MAPREDUCE-3328 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3328 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Priority: Critical -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3328) mapred queue -list output inconsistent and missing child queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-3328: - Description: When running mapred queue -list on a 0.23.0 cluster with capacity scheduler configured with child queues. In my case I have queues default, test1, and test2. test1 has subqueues of a1, a2. test2 has subqueues of a3 and a4. - the child queues do not show up - The output of maximum capacity doesn't match the format of the current capacity and capacity. the latter two use float while the maximum is specified as int: Queue Name : default Queue State : running Scheduling Info : queueName: default, capacity: 0.7, maximumCapacity: 90.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test Queue State : running Scheduling Info : queueName: test, capacity: 0.2, maximumCapacity: -1.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test2 Queue State : running Scheduling Info : queueName: test2, capacity: 0.1, maximumCapacity: 5.0, currentCapacity: 0.0, state: Q_RUNNING, == here default is configured to have capacity=70% and maximum capacity = 90% Release Note: (was: When running mapred queue -list on a 0.23.0 cluster with capacity scheduler configured with child queues. In my case I have queues default, test1, and test2. test1 has subqueues of a1, a2. test2 has subqueues of a3 and a4. - the child queues do not show up - The output of maximum capacity doesn't match the format of the current capacity and capacity. the latter two use float while the maximum is specified as int: Queue Name : default Queue State : running Scheduling Info : queueName: default, capacity: 0.7, maximumCapacity: 90.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test Queue State : running Scheduling Info : queueName: test, capacity: 0.2, maximumCapacity: -1.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test2 Queue State : running Scheduling Info : queueName: test2, capacity: 0.1, maximumCapacity: 5.0, currentCapacity: 0.0, state: Q_RUNNING, == here default is configured to have capacity=70% and maximum capacity = 90%) mapred queue -list output inconsistent and missing child queues --- Key: MAPREDUCE-3328 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3328 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Priority: Critical When running mapred queue -list on a 0.23.0 cluster with capacity scheduler configured with child queues. In my case I have queues default, test1, and test2. test1 has subqueues of a1, a2. test2 has subqueues of a3 and a4. - the child queues do not show up - The output of maximum capacity doesn't match the format of the current capacity and capacity. the latter two use float while the maximum is specified as int: Queue Name : default Queue State : running Scheduling Info : queueName: default, capacity: 0.7, maximumCapacity: 90.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test Queue State : running Scheduling Info : queueName: test, capacity: 0.2, maximumCapacity: -1.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test2 Queue State : running Scheduling Info : queueName: test2, capacity: 0.1, maximumCapacity: 5.0, currentCapacity: 0.0, state: Q_RUNNING, == here default is configured to have capacity=70% and maximum capacity = 90% -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3330) resources/capacity-scheduler.xml description of user-limit-factor is wrong
resources/capacity-scheduler.xml description of user-limit-factor is wrong -- Key: MAPREDUCE-3330 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3330 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Priority: Minor The description of user-limit-factor is wrong in hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/capacity-scheduler.xml. Its not a percent. property nameyarn.scheduler.capacity.root.default.user-limit-factor/name value1/value description Default queue user limit a percentage from 0.0 to 1.0. /description /property -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3324) Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3324: --- Target Version/s: 0.24.0, 0.23.1 (was: 0.23.1, 0.24.0) Summary: Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers (was: Not All HttpServer tools links (stacks,logs,config,jmx,metrics) are accessible through all UI servers) Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers - Key: MAPREDUCE-3324 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3324 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3324.patch Nodemanager has no tools listed under tools UI. Jobhistory server has no logs tool listed under tools UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3328) mapred queue -list output inconsistent and missing child queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash reassigned MAPREDUCE-3328: --- Assignee: Ravi Prakash mapred queue -list output inconsistent and missing child queues --- Key: MAPREDUCE-3328 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3328 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Ravi Prakash Priority: Critical When running mapred queue -list on a 0.23.0 cluster with capacity scheduler configured with child queues. In my case I have queues default, test1, and test2. test1 has subqueues of a1, a2. test2 has subqueues of a3 and a4. - the child queues do not show up - The output of maximum capacity doesn't match the format of the current capacity and capacity. the latter two use float while the maximum is specified as int: Queue Name : default Queue State : running Scheduling Info : queueName: default, capacity: 0.7, maximumCapacity: 90.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test Queue State : running Scheduling Info : queueName: test, capacity: 0.2, maximumCapacity: -1.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test2 Queue State : running Scheduling Info : queueName: test2, capacity: 0.1, maximumCapacity: 5.0, currentCapacity: 0.0, state: Q_RUNNING, == here default is configured to have capacity=70% and maximum capacity = 90% -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3324) Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141614#comment-13141614 ] jirapos...@reviews.apache.org commented on MAPREDUCE-3324: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2665/ --- Review request for Tom Graves, Robert Evans and Mark Holderbaugh. Summary --- Nodemanager has no tools listed under tools UI. Jobhistory server has no logs tool listed under tools UI. This addresses bug MAPREDUCE-3324. http://issues.apache.org/jira/browse/MAPREDUCE-3324 Diffs - hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsNavBlock.java 8d3ccff hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java 01ea4aa Diff: https://reviews.apache.org/r/2665/diff Testing --- Manually verified that Tools navigation bar on NM and Job History contain stacks,logs,config,metrics Thanks, Jonathan Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers - Key: MAPREDUCE-3324 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3324 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3324.patch Nodemanager has no tools listed under tools UI. Jobhistory server has no logs tool listed under tools UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3324) Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3324: --- Priority: Critical (was: Major) Target Version/s: 0.24.0, 0.23.1 (was: 0.23.1, 0.24.0) Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers - Key: MAPREDUCE-3324 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3324 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Attachments: MAPREDUCE-3324.patch Nodemanager has no tools listed under tools UI. Jobhistory server has no logs tool listed under tools UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3326) RM web UI scheduler link not as useful as should be
[ https://issues.apache.org/jira/browse/MAPREDUCE-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141644#comment-13141644 ] Thomas Graves commented on MAPREDUCE-3326: -- Also note that when you click on a child queue it doesn't properly show you the applications running in that queue. RM web UI scheduler link not as useful as should be --- Key: MAPREDUCE-3326 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3326 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Priority: Critical The resource manager web ui page for scheduler doesn't have all the information about the configuration like the jobtracker page used to have. The things it seems to show you are the current queues - each queues used, set, and max percent and then what apps are running in that queue. It doesn't list any of yarn.scheduler.capacity.maximum-applications, yarn.scheduler.capacity.maximum-am-resource-percent, yarn.scheduler.capacity.queue-path.user-limit-factor, yarn.scheduler.capacity.queue-path.minimum-user-limit-percent, queue state, active users and percent used by user -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3331) Improvement to single node cluster setup documentation for 0.23
Improvement to single node cluster setup documentation for 0.23 --- Key: MAPREDUCE-3331 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3331 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Anupam Seth Assignee: Anupam Seth Priority: Minor This JIRA is to track some minor corrections and suggestions for improvement for the documentation for the setup of a single node cluster using 0.23 currently available at http://people.apache.org/~acmurthy/hadoop-0.23/hadoop-yarn/hadoop-yarn-site/SingleCluster.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3331) Improvement to single node cluster setup documentation for 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anupam Seth updated MAPREDUCE-3331: --- Attachment: MRV2doc_review.patch Improvement to single node cluster setup documentation for 0.23 --- Key: MAPREDUCE-3331 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3331 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Anupam Seth Assignee: Anupam Seth Priority: Minor Attachments: MRV2doc_review.patch This JIRA is to track some minor corrections and suggestions for improvement for the documentation for the setup of a single node cluster using 0.23 currently available at http://people.apache.org/~acmurthy/hadoop-0.23/hadoop-yarn/hadoop-yarn-site/SingleCluster.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3331) Improvement to single node cluster setup documentation for 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anupam Seth updated MAPREDUCE-3331: --- Status: Patch Available (was: Open) Improvement to single node cluster setup documentation for 0.23 --- Key: MAPREDUCE-3331 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3331 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Anupam Seth Assignee: Anupam Seth Priority: Minor Attachments: MRV2doc_review.patch This JIRA is to track some minor corrections and suggestions for improvement for the documentation for the setup of a single node cluster using 0.23 currently available at http://people.apache.org/~acmurthy/hadoop-0.23/hadoop-yarn/hadoop-yarn-site/SingleCluster.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3291) App fail to launch due to delegation token not found in cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3291: - Priority: Critical (was: Major) App fail to launch due to delegation token not found in cache - Key: MAPREDUCE-3291 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3291 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Critical Fix For: 0.23.1 Attachments: Log-MAPREDUCE-3291.rtf In secure mode, saw an app failure due to org.apache.hadoop.security.token.SecretManager$InvalidToken: token (HDFS_DELEGATION_TOKEN token id for user) can't be found in cache Exception in the next comment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3331) Improvement to single node cluster setup documentation for 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141682#comment-13141682 ] Anupam Seth commented on MAPREDUCE-3331: Is there a plan to have a document to set up a single node cluster with security as that might be useful? Improvement to single node cluster setup documentation for 0.23 --- Key: MAPREDUCE-3331 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3331 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Anupam Seth Assignee: Anupam Seth Priority: Minor Attachments: MRV2doc_review.patch This JIRA is to track some minor corrections and suggestions for improvement for the documentation for the setup of a single node cluster using 0.23 currently available at http://people.apache.org/~acmurthy/hadoop-0.23/hadoop-yarn/hadoop-yarn-site/SingleCluster.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3309) Report the AM of an application in the UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3309: --- Attachment: MAPREDUCE-3309.patch Report the AM of an application in the UI - Key: MAPREDUCE-3309 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3309 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Jonathan Eagles Fix For: 0.23.1 Attachments: MAPREDUCE-3309.patch Make provision to report the AM hostname of an application in the RM/JHS UI. It is difficult to trace back the AM on which an app ran when there are 100+ jobs in history. Digging through the logs is an option but since there is no consistency maintained in naming of apps in UI/logs/local dirs (MAPREDUCE-2793), debugging is all the more harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3309) Report the AM of an application in the UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141717#comment-13141717 ] Jonathan Eagles commented on MAPREDUCE-3309: Attaching a patch that has a working version of the RM portion of the fix. Report the AM of an application in the UI - Key: MAPREDUCE-3309 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3309 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Jonathan Eagles Fix For: 0.23.1 Attachments: MAPREDUCE-3309.patch Make provision to report the AM hostname of an application in the RM/JHS UI. It is difficult to trace back the AM on which an app ran when there are 100+ jobs in history. Digging through the logs is an option but since there is no consistency maintained in naming of apps in UI/logs/local dirs (MAPREDUCE-2793), debugging is all the more harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3331) Improvement to single node cluster setup documentation for 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141716#comment-13141716 ] Hadoop QA commented on MAPREDUCE-3331: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501851/MRV2doc_review.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1235//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1235//console This message is automatically generated. Improvement to single node cluster setup documentation for 0.23 --- Key: MAPREDUCE-3331 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3331 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Anupam Seth Assignee: Anupam Seth Priority: Minor Attachments: MRV2doc_review.patch This JIRA is to track some minor corrections and suggestions for improvement for the documentation for the setup of a single node cluster using 0.23 currently available at http://people.apache.org/~acmurthy/hadoop-0.23/hadoop-yarn/hadoop-yarn-site/SingleCluster.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3332) contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling
contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling Key: MAPREDUCE-3332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3332 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/raid Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Trivial Fix For: 0.23.1 [javac] /Users/Hitesh/dev/hadoop-common/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java:783: writeBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],org.apache.hadoop.hdfs.protocol.DatanodeInfo,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long,org.apache.hadoop.util.DataChecksum) in org.apache.hadoop.hdfs.protocol.datatransfer.Sender cannot be applied to (org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],nulltype,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long) [javac] new Sender(out).writeBlock(block.getBlock(), block.getBlockToken(), , [javac]^ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3332) contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling
[ https://issues.apache.org/jira/browse/MAPREDUCE-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3332: --- Status: Patch Available (was: Open) contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling Key: MAPREDUCE-3332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3332 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/raid Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Trivial Fix For: 0.23.1 Attachments: MR-3332.1.patch [javac] /Users/Hitesh/dev/hadoop-common/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java:783: writeBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],org.apache.hadoop.hdfs.protocol.DatanodeInfo,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long,org.apache.hadoop.util.DataChecksum) in org.apache.hadoop.hdfs.protocol.datatransfer.Sender cannot be applied to (org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],nulltype,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long) [javac] new Sender(out).writeBlock(block.getBlock(), block.getBlockToken(), , [javac]^ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3332) contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling
[ https://issues.apache.org/jira/browse/MAPREDUCE-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3332: - Fix Version/s: (was: 0.23.1) 0.23.0 contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling Key: MAPREDUCE-3332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3332 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/raid Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Trivial Fix For: 0.23.0 Attachments: MR-3332.1.patch [javac] /Users/Hitesh/dev/hadoop-common/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java:783: writeBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],org.apache.hadoop.hdfs.protocol.DatanodeInfo,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long,org.apache.hadoop.util.DataChecksum) in org.apache.hadoop.hdfs.protocol.datatransfer.Sender cannot be applied to (org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],nulltype,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long) [javac] new Sender(out).writeBlock(block.getBlock(), block.getBlockToken(), , [javac]^ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3332) contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling
[ https://issues.apache.org/jira/browse/MAPREDUCE-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141735#comment-13141735 ] Arun C Murthy commented on MAPREDUCE-3332: -- +1 on putting this in 0.23. I'm spinning a new rc1 anyway. contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling Key: MAPREDUCE-3332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3332 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/raid Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Trivial Fix For: 0.23.0 Attachments: MR-3332.1.patch [javac] /Users/Hitesh/dev/hadoop-common/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java:783: writeBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],org.apache.hadoop.hdfs.protocol.DatanodeInfo,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long,org.apache.hadoop.util.DataChecksum) in org.apache.hadoop.hdfs.protocol.datatransfer.Sender cannot be applied to (org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],nulltype,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long) [javac] new Sender(out).writeBlock(block.getBlock(), block.getBlockToken(), , [javac]^ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3332) contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling
[ https://issues.apache.org/jira/browse/MAPREDUCE-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3332: - Resolution: Fixed Status: Resolved (was: Patch Available) Just committed. Thanks Hitesh. contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling Key: MAPREDUCE-3332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3332 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/raid Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Trivial Fix For: 0.23.0 Attachments: MR-3332.1.patch [javac] /Users/Hitesh/dev/hadoop-common/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java:783: writeBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],org.apache.hadoop.hdfs.protocol.DatanodeInfo,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long,org.apache.hadoop.util.DataChecksum) in org.apache.hadoop.hdfs.protocol.datatransfer.Sender cannot be applied to (org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],nulltype,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long) [javac] new Sender(out).writeBlock(block.getBlock(), block.getBlockToken(), , [javac]^ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3332) contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling
[ https://issues.apache.org/jira/browse/MAPREDUCE-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141742#comment-13141742 ] Hudson commented on MAPREDUCE-3332: --- Integrated in Hadoop-Hdfs-0.23-Commit #134 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/134/]) MAPREDUCE-3332. contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling (Hitesh Shah via mahadev) -Merging r1196356 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1196366 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling Key: MAPREDUCE-3332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3332 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/raid Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Trivial Fix For: 0.23.0 Attachments: MR-3332.1.patch [javac] /Users/Hitesh/dev/hadoop-common/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java:783: writeBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],org.apache.hadoop.hdfs.protocol.DatanodeInfo,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long,org.apache.hadoop.util.DataChecksum) in org.apache.hadoop.hdfs.protocol.datatransfer.Sender cannot be applied to (org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],nulltype,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long) [javac] new Sender(out).writeBlock(block.getBlock(), block.getBlockToken(), , [javac]^ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3332) contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling
[ https://issues.apache.org/jira/browse/MAPREDUCE-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141744#comment-13141744 ] Hudson commented on MAPREDUCE-3332: --- Integrated in Hadoop-Common-trunk-Commit #1230 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1230/]) MAPREDUCE-3332. contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling (Hitesh Shah via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1196356 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling Key: MAPREDUCE-3332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3332 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/raid Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Trivial Fix For: 0.23.0 Attachments: MR-3332.1.patch [javac] /Users/Hitesh/dev/hadoop-common/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java:783: writeBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],org.apache.hadoop.hdfs.protocol.DatanodeInfo,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long,org.apache.hadoop.util.DataChecksum) in org.apache.hadoop.hdfs.protocol.datatransfer.Sender cannot be applied to (org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],nulltype,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long) [javac] new Sender(out).writeBlock(block.getBlock(), block.getBlockToken(), , [javac]^ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3332) contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling
[ https://issues.apache.org/jira/browse/MAPREDUCE-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141745#comment-13141745 ] Hudson commented on MAPREDUCE-3332: --- Integrated in Hadoop-Common-0.23-Commit #133 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/133/]) MAPREDUCE-3332. contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling (Hitesh Shah via mahadev) -Merging r1196356 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1196366 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling Key: MAPREDUCE-3332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3332 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/raid Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Trivial Fix For: 0.23.0 Attachments: MR-3332.1.patch [javac] /Users/Hitesh/dev/hadoop-common/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java:783: writeBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],org.apache.hadoop.hdfs.protocol.DatanodeInfo,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long,org.apache.hadoop.util.DataChecksum) in org.apache.hadoop.hdfs.protocol.datatransfer.Sender cannot be applied to (org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],nulltype,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long) [javac] new Sender(out).writeBlock(block.getBlock(), block.getBlockToken(), , [javac]^ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3332) contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling
[ https://issues.apache.org/jira/browse/MAPREDUCE-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141758#comment-13141758 ] Hadoop QA commented on MAPREDUCE-3332: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501866/MR-3332.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1236//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1236//console This message is automatically generated. contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling Key: MAPREDUCE-3332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3332 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/raid Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Trivial Fix For: 0.23.0 Attachments: MR-3332.1.patch [javac] /Users/Hitesh/dev/hadoop-common/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java:783: writeBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],org.apache.hadoop.hdfs.protocol.DatanodeInfo,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long,org.apache.hadoop.util.DataChecksum) in org.apache.hadoop.hdfs.protocol.datatransfer.Sender cannot be applied to (org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],nulltype,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long) [javac] new Sender(out).writeBlock(block.getBlock(), block.getBlockToken(), , [javac]^ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3332) contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling
[ https://issues.apache.org/jira/browse/MAPREDUCE-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141765#comment-13141765 ] Hudson commented on MAPREDUCE-3332: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1253 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1253/]) MAPREDUCE-3332. contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling (Hitesh Shah via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1196356 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling Key: MAPREDUCE-3332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3332 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/raid Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Trivial Fix For: 0.23.0 Attachments: MR-3332.1.patch [javac] /Users/Hitesh/dev/hadoop-common/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java:783: writeBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],org.apache.hadoop.hdfs.protocol.DatanodeInfo,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long,org.apache.hadoop.util.DataChecksum) in org.apache.hadoop.hdfs.protocol.datatransfer.Sender cannot be applied to (org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],nulltype,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long) [javac] new Sender(out).writeBlock(block.getBlock(), block.getBlockToken(), , [javac]^ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3332) contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling
[ https://issues.apache.org/jira/browse/MAPREDUCE-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141764#comment-13141764 ] Hudson commented on MAPREDUCE-3332: --- Integrated in Hadoop-Mapreduce-0.23-Commit #144 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/144/]) MAPREDUCE-3332. contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling (Hitesh Shah via mahadev) -Merging r1196356 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1196366 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/Sender#writeBlock related to checksum handling Key: MAPREDUCE-3332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3332 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/raid Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Trivial Fix For: 0.23.0 Attachments: MR-3332.1.patch [javac] /Users/Hitesh/dev/hadoop-common/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java:783: writeBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],org.apache.hadoop.hdfs.protocol.DatanodeInfo,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long,org.apache.hadoop.util.DataChecksum) in org.apache.hadoop.hdfs.protocol.datatransfer.Sender cannot be applied to (org.apache.hadoop.hdfs.protocol.ExtendedBlock,org.apache.hadoop.security.token.Tokenorg.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier,java.lang.String,org.apache.hadoop.hdfs.protocol.DatanodeInfo[],nulltype,org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage,int,long,long,long) [javac] new Sender(out).writeBlock(block.getBlock(), block.getBlockToken(), , [javac]^ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3139) SlivePartitioner generates negative partitions
[
https://issues.apache.org/jira/browse/MAPREDUCE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated MAPREDUCE-3139:
---
Attachment: MR-3139-0.patch
Quick patch to guarantee hash value is always positive. No tests as not easily
testable, but easily verifiable with math. After review will commit to 22, 23
and trunk.
SlivePartitioner generates negative partitions
--
Key: MAPREDUCE-3139
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3139
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 0.22.0
Reporter: Konstantin Shvachko
Assignee: Jakob Homan
Fix For: 0.24.0
Attachments: MR-3139-0.patch
{{SlivePartitioner.getPartition()}} returns negative partition numbers on
some occasions, which is illegal.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3139) SlivePartitioner generates negative partitions
[
https://issues.apache.org/jira/browse/MAPREDUCE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated MAPREDUCE-3139:
---
Status: Patch Available (was: Open)
SlivePartitioner generates negative partitions
--
Key: MAPREDUCE-3139
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3139
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 0.22.0
Reporter: Konstantin Shvachko
Assignee: Jakob Homan
Fix For: 0.24.0
Attachments: MR-3139-0.patch
{{SlivePartitioner.getPartition()}} returns negative partition numbers on
some occasions, which is illegal.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3139) SlivePartitioner generates negative partitions
[
https://issues.apache.org/jira/browse/MAPREDUCE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141848#comment-13141848
]
Konstantin Shvachko commented on MAPREDUCE-3139:
+1 This solves the problem.
SlivePartitioner generates negative partitions
--
Key: MAPREDUCE-3139
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3139
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 0.22.0
Reporter: Konstantin Shvachko
Assignee: Jakob Homan
Fix For: 0.24.0
Attachments: MR-3139-0.patch
{{SlivePartitioner.getPartition()}} returns negative partition numbers on
some occasions, which is illegal.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3323) Add new interface for Distributed Cache, which special for Map or Reduce,but not Both.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Azuryy(Chijiong) updated MAPREDUCE-3323: Summary: Add new interface for Distributed Cache, which special for Map or Reduce,but not Both. (was: Distributed Cache for Map or Reduce or Both) Add new interface for Distributed Cache, which special for Map or Reduce,but not Both. --- Key: MAPREDUCE-3323 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3323 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distributed-cache, tasktracker Affects Versions: 0.20.203.0 Reporter: Azuryy(Chijiong) Attachments: DistributedCache.patch, TaskTracker.patch We put some file into Distributed Cache, but sometimes, only Map or Reduce use thses cached files, not useful for both. but TaskTracker always download cached files from HDFS, if there are some little bit big files in cache, it's time expensive. so, this patch add some new API in the DistributedCache.java as follow: addArchiveToClassPathForMap addArchiveToClassPathForReduce addFileToClassPathForMap addFileToClassPathForReduce addCacheFileForMap addCacheFileForReduce addCacheArchiveForMap addCacheArchiveForReduce New API doesn't affect original interface. but they are specified for only map or reduce, not both of them. But if you do need cache file during both map and reduce, then use original interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3139) SlivePartitioner generates negative partitions
[
https://issues.apache.org/jira/browse/MAPREDUCE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141870#comment-13141870
]
Hadoop QA commented on MAPREDUCE-3139:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12501896/MR-3139-0.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1237//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1237//console
This message is automatically generated.
SlivePartitioner generates negative partitions
--
Key: MAPREDUCE-3139
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3139
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 0.22.0
Reporter: Konstantin Shvachko
Assignee: Jakob Homan
Fix For: 0.24.0
Attachments: MR-3139-0.patch
{{SlivePartitioner.getPartition()}} returns negative partition numbers on
some occasions, which is illegal.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
