[jira] [Created] (MAPREDUCE-3454) [Gridmix] TestDistCacheEmulation is broken
[Gridmix] TestDistCacheEmulation is broken -- Key: MAPREDUCE-3454 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3454 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Fix For: 0.23.1 TestDistCacheEmulation is broken as 'MapReduceTestUtil' no longer exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3454) [Gridmix] TestDistCacheEmulation is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154968#comment-13154968 ] Hitesh Shah commented on MAPREDUCE-3454: I believe its now in hadoop-mapreduce-client-jobclient-tests.jar. [Gridmix] TestDistCacheEmulation is broken -- Key: MAPREDUCE-3454 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3454 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: gridmix Fix For: 0.23.1 TestDistCacheEmulation is broken as 'MapReduceTestUtil' no longer exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3454) [Gridmix] TestDistCacheEmulation is broken
[
https://issues.apache.org/jira/browse/MAPREDUCE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154971#comment-13154971
]
Amar Kamat commented on MAPREDUCE-3454:
---
I see lot of test with such issues. A simple search for the
'createDummyMapTaskAttemptContext(' gives the following result.
{noformat}
hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java
hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/input/TestMRKeyValueTextInputFormat.java
hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/input/TestMRKeyValueTextInputFormat.java
hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/input/TestMRKeyValueTextInputFormat.java
hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/input/TestMRKeyValueTextInputFormat.java
hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/input/TestMRSequenceFileAsBinaryInputFormat.java
hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/input/TestMRSequenceFileAsTextInputFormat.java
hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/input/TestMRSequenceFileInputFilter.java
hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/input/TestNLineInputFormat.java
hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/join/TestJoinProperties.java
hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestMRSequenceFileAsBinaryOutputFormat.java
{noformat}
[Gridmix] TestDistCacheEmulation is broken
--
Key: MAPREDUCE-3454
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3454
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix
Affects Versions: 0.24.0
Reporter: Amar Kamat
Assignee: Amar Kamat
Labels: gridmix
Fix For: 0.23.1
TestDistCacheEmulation is broken as 'MapReduceTestUtil' no longer exists.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3454) [Gridmix] TestDistCacheEmulation is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154976#comment-13154976 ] Amar Kamat commented on MAPREDUCE-3454: --- Hitesh, I assume that you are aware of the code changes required to fix this JIRA. Can you kindly take this up? [Gridmix] TestDistCacheEmulation is broken -- Key: MAPREDUCE-3454 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3454 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: gridmix Fix For: 0.23.1 TestDistCacheEmulation is broken as 'MapReduceTestUtil' no longer exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3412) 'ant docs' is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3412: --- Hadoop Flags: Reviewed Status: Patch Available (was: Open) All of this will go away after the mavenisation goes the full distance. Looks innocuous. +1 for this patch in the interim. 'ant docs' is broken Key: MAPREDUCE-3412 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3412 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: docs Fix For: 0.24.0 Attachments: mapreduce-3412-v1.0.patch 'ant docs' no longer work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3412) 'ant docs' is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-3412: -- Resolution: Fixed Release Note: Fixes 'ant docs' by removing stale references to capacity-scheduler docs. Status: Resolved (was: Patch Available) I just committed this to trunk. Thanks Vinod! 'ant docs' is broken Key: MAPREDUCE-3412 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3412 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: docs Fix For: 0.24.0 Attachments: mapreduce-3412-v1.0.patch 'ant docs' no longer work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3412) 'ant docs' is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154995#comment-13154995 ] Hudson commented on MAPREDUCE-3412: --- Integrated in Hadoop-Hdfs-trunk-Commit #1385 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1385/]) MAPREDUCE-3412. Fix 'ant docs'. (amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1204885 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/build.xml * /hadoop/common/trunk/hadoop-mapreduce-project/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml 'ant docs' is broken Key: MAPREDUCE-3412 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3412 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: docs Fix For: 0.24.0 Attachments: mapreduce-3412-v1.0.patch 'ant docs' no longer work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3412) 'ant docs' is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154993#comment-13154993 ] Amar Kamat commented on MAPREDUCE-3412: --- I committed this to branch-0.23 too. 'ant docs' is broken Key: MAPREDUCE-3412 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3412 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: docs Fix For: 0.24.0 Attachments: mapreduce-3412-v1.0.patch 'ant docs' no longer work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3412) 'ant docs' is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154998#comment-13154998 ] Hadoop QA commented on MAPREDUCE-3412: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12503919/mapreduce-3412-v1.0.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 12 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1337//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1337//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-examples.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1337//console This message is automatically generated. 'ant docs' is broken Key: MAPREDUCE-3412 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3412 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: docs Fix For: 0.24.0 Attachments: mapreduce-3412-v1.0.patch 'ant docs' no longer work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2765) DistCp Rewrite
[ https://issues.apache.org/jira/browse/MAPREDUCE-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155000#comment-13155000 ] Alejandro Abdelnur commented on MAPREDUCE-2765: --- Any update on this JIRA? The module hadoop-tools is now available. DistCp Rewrite -- Key: MAPREDUCE-2765 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2765 Project: Hadoop Map/Reduce Issue Type: New Feature Components: distcp Affects Versions: 0.20.203.0 Reporter: Mithun Radhakrishnan Assignee: Mithun Radhakrishnan Attachments: distcpv2.20.203.patch, distcpv2_trunk.patch, distcpv2_trunk_post_review_1.patch This is a slightly modified version of the DistCp rewrite that Yahoo uses in production today. The rewrite was ground-up, with specific focus on: 1. improved startup time (postponing as much work as possible to the MR job) 2. support for multiple copy-strategies 3. new features (e.g. -atomic, -async, -bandwidth.) 4. improved programmatic use Some effort has gone into refactoring what used to be achieved by a single large (1.7 KLOC) source file, into a design that (hopefully) reads better too. The proposed DistCpV2 preserves command-line-compatibility with the old version, and should be a drop-in replacement. New to v2: 1. Copy-strategies and the DynamicInputFormat: A copy-strategy determines the policy by which source-file-paths are distributed between map-tasks. (These boil down to the choice of the input-format.) If no strategy is explicitly specified on the command-line, the policy chosen is uniform size, where v2 behaves identically to old-DistCp. (The number of bytes transferred by each map-task is roughly equal, at a per-file granularity.) Alternatively, v2 ships with a dynamic copy-strategy (in the DynamicInputFormat). This policy acknowledges that (a) dividing files based only on file-size might not be an even distribution (E.g. if some datanodes are slower than others, or if some files are skipped.) (b) a static association of a source-path to a map increases the likelihood of long-tails during copy. The dynamic strategy divides the list-of-source-paths into a number ( nMaps) of smaller parts. When each map completes its current list of paths, it picks up a new list to process, if available. So if a map-task is stuck on a slow (and not necessarily large) file, other maps can pick up the slack. The thinner the file-list is sliced, the greater the parallelism (and the lower the chances of long-tails). Within reason, of course: the number of these short-lived list-files is capped at an overridable maximum. Internal benchmarks against source/target clusters with some slow(ish) datanodes have indicated significant performance gains when using the dynamic-strategy. Gains are most pronounced when nFiles greatly exceeds nMaps. Please note that the DynamicInputFormat might prove useful outside of DistCp. It is hence available as a mapred/lib, unfettered to DistCpV2. Also note that the copy-strategies have no bearing on the CopyMapper.map() implementation. 2. Improved startup-time and programmatic use: When the old-DistCp runs with -update, and creates the list-of-source-paths, it attempts to filter out files that might be skipped (by comparing file-sizes, checksums, etc.) This significantly increases the startup time (or the time spent in serial processing till the MR job is launched), blocking the calling-thread. This becomes pronounced as nFiles increases. (Internal benchmarks have seen situations where more time is spent setting up the job than on the actual transfer.) DistCpV2 postpones as much work as possible to the MR job. The file-listing isn't filtered until the map-task runs (at which time, identical files are skipped). DistCpV2 can now be run asynchronously. The program quits at job-launch, logging the job-id for tracking. Programmatically, the DistCp.execute() returns a Job instance for progress-tracking. 3. New features: (a) -async: As described in #2. (b) -atomic: Data is copied to a (user-specifiable) tmp-location, and then moved atomically to destination. (c) -bandwidth: Enforces a limit on the bandwidth consumed per map. (d) -strategy: As above. A more comprehensive description the newer features, how the dynamic-strategy works, etc. is available in src/site/xdoc/, and in the pdf that's generated therefrom, during the build. High on the list of things to do is support to parallelize copies on a per-block level. (i.e. Incorporation of HDFS-222.) I look forward to comments, suggestions and discussion that will hopefully ensue. I have
[jira] [Commented] (MAPREDUCE-3412) 'ant docs' is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155003#comment-13155003 ] Hudson commented on MAPREDUCE-3412: --- Integrated in Hadoop-Common-0.23-Commit #203 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/203/]) MAPREDUCE-3412. Fix 'ant docs'. (amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1204892 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml 'ant docs' is broken Key: MAPREDUCE-3412 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3412 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: docs Fix For: 0.24.0 Attachments: mapreduce-3412-v1.0.patch 'ant docs' no longer work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3412) 'ant docs' is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155005#comment-13155005 ] Hudson commented on MAPREDUCE-3412: --- Integrated in Hadoop-Hdfs-0.23-Commit #200 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/200/]) MAPREDUCE-3412. Fix 'ant docs'. (amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1204892 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml 'ant docs' is broken Key: MAPREDUCE-3412 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3412 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: docs Fix For: 0.24.0 Attachments: mapreduce-3412-v1.0.patch 'ant docs' no longer work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3412) 'ant docs' is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155006#comment-13155006 ] Hudson commented on MAPREDUCE-3412: --- Integrated in Hadoop-Mapreduce-0.23-Commit #214 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/214/]) MAPREDUCE-3412. Fix 'ant docs'. (amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1204892 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml 'ant docs' is broken Key: MAPREDUCE-3412 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3412 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: docs Fix For: 0.24.0 Attachments: mapreduce-3412-v1.0.patch 'ant docs' no longer work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3412) 'ant docs' is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155008#comment-13155008 ] Hudson commented on MAPREDUCE-3412: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1335 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1335/]) MAPREDUCE-3412. Fix 'ant docs'. (amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1204885 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/build.xml * /hadoop/common/trunk/hadoop-mapreduce-project/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml 'ant docs' is broken Key: MAPREDUCE-3412 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3412 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: docs Fix For: 0.24.0 Attachments: mapreduce-3412-v1.0.patch 'ant docs' no longer work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3008) [Gridmix] Improve cumulative CPU usage emulation for short running tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-3008: -- Attachment: mapreduce-2591-v1.7-0.20-security.patch Backported patch for the branch-0.20-security branch. System tests passed. Our analysis shows improvements in emulation. I couldn't run test-patch and JUnit tests as the code compilation fails because of task-controller. [Gridmix] Improve cumulative CPU usage emulation for short running tasks Key: MAPREDUCE-3008 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3008 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: cpu-emulation, gridmix Fix For: 0.24.0 Attachments: mapreduce-2591-v1.4.2.patch, mapreduce-2591-v1.7-0.20-security.patch, mapreduce-2591-v1.7.patch CPU emulation in Gridmix fails to meet the expected target if the map has no data to sort/spill/merge. There are 2 major reasons for this: 1. The map task end immediately ends soon after the map task. The map progress is 67% while the map phase ends. 2. Currently, the sort (comparator) doesnt emulate CPU. If the map is short lived, the CPU emulation thread (spawned from the map task in cleanup) doesn't get a chance to emulate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3008) [Gridmix] Improve cumulative CPU usage emulation for short running tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155027#comment-13155027 ] Vinay Kumar Thota commented on MAPREDUCE-3008: -- +1 for the patch. [Gridmix] Improve cumulative CPU usage emulation for short running tasks Key: MAPREDUCE-3008 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3008 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: cpu-emulation, gridmix Fix For: 0.24.0 Attachments: mapreduce-2591-v1.4.2.patch, mapreduce-2591-v1.7-0.20-security.patch, mapreduce-2591-v1.7.patch CPU emulation in Gridmix fails to meet the expected target if the map has no data to sort/spill/merge. There are 2 major reasons for this: 1. The map task end immediately ends soon after the map task. The map progress is 67% while the map phase ends. 2. Currently, the sort (comparator) doesnt emulate CPU. If the map is short lived, the CPU emulation thread (spawned from the map task in cleanup) doesn't get a chance to emulate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3008) [Gridmix] Improve cumulative CPU usage emulation for short running tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155029#comment-13155029 ] Amar Kamat commented on MAPREDUCE-3008: --- I just committed the backported patch to branch-0.20-security. Thanks Vinay! [Gridmix] Improve cumulative CPU usage emulation for short running tasks Key: MAPREDUCE-3008 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3008 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: cpu-emulation, gridmix Fix For: 0.24.0 Attachments: mapreduce-2591-v1.4.2.patch, mapreduce-2591-v1.7-0.20-security.patch, mapreduce-2591-v1.7.patch CPU emulation in Gridmix fails to meet the expected target if the map has no data to sort/spill/merge. There are 2 major reasons for this: 1. The map task end immediately ends soon after the map task. The map progress is 67% while the map phase ends. 2. Currently, the sort (comparator) doesnt emulate CPU. If the map is short lived, the CPU emulation thread (spawned from the map task in cleanup) doesn't get a chance to emulate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3447) mapreduce examples not working
[ https://issues.apache.org/jira/browse/MAPREDUCE-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155059#comment-13155059 ] Hudson commented on MAPREDUCE-3447: --- Integrated in Hadoop-Hdfs-0.23-Build #84 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/84/]) MAPREDUCE-3447. mapreduce examples not working (mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1204696 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml mapreduce examples not working -- Key: MAPREDUCE-3447 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3447 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Mahadev konar Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3447.patch Since the mavenization went in the mapreduce examples jar no longer works. $ hadoop jar ./hadoop-0.23.0-SNAPSHOT/modules/hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar wordcount input output Exception in thread main java.lang.ClassNotFoundException: wordcount at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:252) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.util.RunJar.main(RunJar.java:193) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3434) Nightly build broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155058#comment-13155058 ] Hudson commented on MAPREDUCE-3434: --- Integrated in Hadoop-Hdfs-0.23-Build #84 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/84/]) MAPREDUCE-3434. Nightly build broken (Hitesh Shah via mahadev) - merging r1204638 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1204641 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml Nightly build broken - Key: MAPREDUCE-3434 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3434 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.1 Attachments: MR-3434.1.patch https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk/901/ Results : Failed tests: testSleepJob(org.apache.hadoop.mapreduce.v2.TestMRJobs) testRandomWriter(org.apache.hadoop.mapreduce.v2.TestMRJobs) testDistributedCache(org.apache.hadoop.mapreduce.v2.TestMRJobs) Tests in error: org.apache.hadoop.mapreduce.v2.TestMROldApiJobs: Failed to Start org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapreduce.v2.TestUberAM: Failed to Start org.apache.hadoop.mapreduce.v2.TestMRJobs Likely due to either of: MAPREDUCE-3415. improve MiniMRYarnCluster DistributedShell JAR resolution (tucu) MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2. (Ahmed via tucu) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3444) trunk/0.23 builds broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155062#comment-13155062 ] Hudson commented on MAPREDUCE-3444: --- Integrated in Hadoop-Hdfs-0.23-Build #84 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/84/]) MAPREDUCE-3444. trunk/0.23 builds broken (Hitesh Shah via mahadev) - Merging r1204698 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1204700 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/ivy.xml trunk/0.23 builds broken - Key: MAPREDUCE-3444 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3444 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.1 Attachments: MR-3444.1.patch https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/208/ https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1310/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3412) 'ant docs' is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155063#comment-13155063 ] Hudson commented on MAPREDUCE-3412: --- Integrated in Hadoop-Hdfs-0.23-Build #84 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/84/]) MAPREDUCE-3412. Fix 'ant docs'. (amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1204892 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml 'ant docs' is broken Key: MAPREDUCE-3412 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3412 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: docs Fix For: 0.24.0 Attachments: mapreduce-3412-v1.0.patch 'ant docs' no longer work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3444) trunk/0.23 builds broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155080#comment-13155080 ] Hudson commented on MAPREDUCE-3444: --- Integrated in Hadoop-Hdfs-trunk #871 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/871/]) MAPREDUCE-3444. trunk/0.23 builds broken (Hitesh Shah via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1204698 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/build.xml * /hadoop/common/trunk/hadoop-mapreduce-project/ivy.xml trunk/0.23 builds broken - Key: MAPREDUCE-3444 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3444 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.1 Attachments: MR-3444.1.patch https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/208/ https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1310/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3447) mapreduce examples not working
[ https://issues.apache.org/jira/browse/MAPREDUCE-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155078#comment-13155078 ] Hudson commented on MAPREDUCE-3447: --- Integrated in Hadoop-Hdfs-trunk #871 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/871/]) MAPREDUCE-3447. mapreduce examples not working (mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1204695 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml mapreduce examples not working -- Key: MAPREDUCE-3447 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3447 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Mahadev konar Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3447.patch Since the mavenization went in the mapreduce examples jar no longer works. $ hadoop jar ./hadoop-0.23.0-SNAPSHOT/modules/hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar wordcount input output Exception in thread main java.lang.ClassNotFoundException: wordcount at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:252) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.util.RunJar.main(RunJar.java:193) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3412) 'ant docs' is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155081#comment-13155081 ] Hudson commented on MAPREDUCE-3412: --- Integrated in Hadoop-Hdfs-trunk #871 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/871/]) MAPREDUCE-3412. Fix 'ant docs'. (amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1204885 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/build.xml * /hadoop/common/trunk/hadoop-mapreduce-project/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml 'ant docs' is broken Key: MAPREDUCE-3412 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3412 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: docs Fix For: 0.24.0 Attachments: mapreduce-3412-v1.0.patch 'ant docs' no longer work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3434) Nightly build broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155098#comment-13155098 ] Hudson commented on MAPREDUCE-3434: --- Integrated in Hadoop-Mapreduce-0.23-Build #101 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/101/]) MAPREDUCE-3434. Nightly build broken (Hitesh Shah via mahadev) - merging r1204638 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1204641 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml Nightly build broken - Key: MAPREDUCE-3434 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3434 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.1 Attachments: MR-3434.1.patch https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk/901/ Results : Failed tests: testSleepJob(org.apache.hadoop.mapreduce.v2.TestMRJobs) testRandomWriter(org.apache.hadoop.mapreduce.v2.TestMRJobs) testDistributedCache(org.apache.hadoop.mapreduce.v2.TestMRJobs) Tests in error: org.apache.hadoop.mapreduce.v2.TestMROldApiJobs: Failed to Start org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapreduce.v2.TestUberAM: Failed to Start org.apache.hadoop.mapreduce.v2.TestMRJobs Likely due to either of: MAPREDUCE-3415. improve MiniMRYarnCluster DistributedShell JAR resolution (tucu) MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2. (Ahmed via tucu) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3447) mapreduce examples not working
[ https://issues.apache.org/jira/browse/MAPREDUCE-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155099#comment-13155099 ] Hudson commented on MAPREDUCE-3447: --- Integrated in Hadoop-Mapreduce-0.23-Build #101 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/101/]) MAPREDUCE-3447. mapreduce examples not working (mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1204696 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml mapreduce examples not working -- Key: MAPREDUCE-3447 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3447 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Mahadev konar Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3447.patch Since the mavenization went in the mapreduce examples jar no longer works. $ hadoop jar ./hadoop-0.23.0-SNAPSHOT/modules/hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar wordcount input output Exception in thread main java.lang.ClassNotFoundException: wordcount at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:252) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.util.RunJar.main(RunJar.java:193) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3444) trunk/0.23 builds broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155102#comment-13155102 ] Hudson commented on MAPREDUCE-3444: --- Integrated in Hadoop-Mapreduce-0.23-Build #101 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/101/]) MAPREDUCE-3444. trunk/0.23 builds broken (Hitesh Shah via mahadev) - Merging r1204698 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1204700 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/ivy.xml trunk/0.23 builds broken - Key: MAPREDUCE-3444 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3444 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.1 Attachments: MR-3444.1.patch https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/208/ https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1310/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3412) 'ant docs' is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155103#comment-13155103 ] Hudson commented on MAPREDUCE-3412: --- Integrated in Hadoop-Mapreduce-0.23-Build #101 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/101/]) MAPREDUCE-3412. Fix 'ant docs'. (amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1204892 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml 'ant docs' is broken Key: MAPREDUCE-3412 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3412 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: docs Fix For: 0.24.0 Attachments: mapreduce-3412-v1.0.patch 'ant docs' no longer work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3412) 'ant docs' is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155125#comment-13155125 ] Hudson commented on MAPREDUCE-3412: --- Integrated in Hadoop-Mapreduce-trunk #905 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/905/]) MAPREDUCE-3412. Fix 'ant docs'. (amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1204885 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/build.xml * /hadoop/common/trunk/hadoop-mapreduce-project/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml 'ant docs' is broken Key: MAPREDUCE-3412 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3412 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: docs Fix For: 0.24.0 Attachments: mapreduce-3412-v1.0.patch 'ant docs' no longer work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3447) mapreduce examples not working
[ https://issues.apache.org/jira/browse/MAPREDUCE-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155122#comment-13155122 ] Hudson commented on MAPREDUCE-3447: --- Integrated in Hadoop-Mapreduce-trunk #905 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/905/]) MAPREDUCE-3447. mapreduce examples not working (mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1204695 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml mapreduce examples not working -- Key: MAPREDUCE-3447 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3447 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Mahadev konar Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3447.patch Since the mavenization went in the mapreduce examples jar no longer works. $ hadoop jar ./hadoop-0.23.0-SNAPSHOT/modules/hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar wordcount input output Exception in thread main java.lang.ClassNotFoundException: wordcount at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:252) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.util.RunJar.main(RunJar.java:193) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3444) trunk/0.23 builds broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155124#comment-13155124 ] Hudson commented on MAPREDUCE-3444: --- Integrated in Hadoop-Mapreduce-trunk #905 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/905/]) MAPREDUCE-3444. trunk/0.23 builds broken (Hitesh Shah via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1204698 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/build.xml * /hadoop/common/trunk/hadoop-mapreduce-project/ivy.xml trunk/0.23 builds broken - Key: MAPREDUCE-3444 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3444 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.1 Attachments: MR-3444.1.patch https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/208/ https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1310/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3434) Nightly build broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155123#comment-13155123 ] Hudson commented on MAPREDUCE-3434: --- Integrated in Hadoop-Mapreduce-trunk #905 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/905/]) MAPREDUCE-3434. Nightly build broken (Hitesh Shah via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1204638 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml Nightly build broken - Key: MAPREDUCE-3434 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3434 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.1 Attachments: MR-3434.1.patch https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk/901/ Results : Failed tests: testSleepJob(org.apache.hadoop.mapreduce.v2.TestMRJobs) testRandomWriter(org.apache.hadoop.mapreduce.v2.TestMRJobs) testDistributedCache(org.apache.hadoop.mapreduce.v2.TestMRJobs) Tests in error: org.apache.hadoop.mapreduce.v2.TestMROldApiJobs: Failed to Start org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapreduce.v2.TestUberAM: Failed to Start org.apache.hadoop.mapreduce.v2.TestMRJobs Likely due to either of: MAPREDUCE-3415. improve MiniMRYarnCluster DistributedShell JAR resolution (tucu) MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2. (Ahmed via tucu) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155304#comment-13155304 ] Arun C Murthy commented on MAPREDUCE-3451: -- Thanks for taking this up Patrick. Let me know if you need help. Port Fair Scheduler to MR2 -- Key: MAPREDUCE-3451 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451 Project: Hadoop Map/Reduce Issue Type: New Feature Components: scheduler Reporter: Patrick Wendell Assignee: Patrick Wendell The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3424) Some LinuxTaskController cleanup
[ https://issues.apache.org/jira/browse/MAPREDUCE-3424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155342#comment-13155342 ] Todd Lipcon commented on MAPREDUCE-3424: +1. Looks good. Some LinuxTaskController cleanup Key: MAPREDUCE-3424 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3424 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: tasktracker Affects Versions: 0.20.205.0 Reporter: Eli Collins Assignee: Eli Collins Priority: Minor Attachments: mapreduce-3424-1.patch, mapreduce-3424-2.patch, mapreduce-3424-3.patch MR-2415 had some tabs and weird indenting and spacing. Also would be more clear if LTC explicitly overrides createLogDir. Let's clean this up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3454) [Gridmix] TestDistCacheEmulation is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah reassigned MAPREDUCE-3454: -- Assignee: Hitesh Shah (was: Amar Kamat) [Gridmix] TestDistCacheEmulation is broken -- Key: MAPREDUCE-3454 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3454 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Hitesh Shah Labels: gridmix Fix For: 0.23.1 Attachments: MR-3454.1.patch TestDistCacheEmulation is broken as 'MapReduceTestUtil' no longer exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3454) [Gridmix] TestDistCacheEmulation is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3454: --- Attachment: MR-3454.1.patch [Gridmix] TestDistCacheEmulation is broken -- Key: MAPREDUCE-3454 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3454 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Hitesh Shah Labels: gridmix Fix For: 0.23.1 Attachments: MR-3454.1.patch TestDistCacheEmulation is broken as 'MapReduceTestUtil' no longer exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3454) [Gridmix] TestDistCacheEmulation is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3454: --- Status: Patch Available (was: Open) [Gridmix] TestDistCacheEmulation is broken -- Key: MAPREDUCE-3454 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3454 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Hitesh Shah Labels: gridmix Fix For: 0.23.1 Attachments: MR-3454.1.patch TestDistCacheEmulation is broken as 'MapReduceTestUtil' no longer exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3454) [Gridmix] TestDistCacheEmulation is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155355#comment-13155355 ] Mahadev konar commented on MAPREDUCE-3454: -- Looks good. We should probably delete the streaming dir. [Gridmix] TestDistCacheEmulation is broken -- Key: MAPREDUCE-3454 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3454 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Hitesh Shah Labels: gridmix Fix For: 0.23.1 Attachments: MR-3454.1.patch TestDistCacheEmulation is broken as 'MapReduceTestUtil' no longer exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[
https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155358#comment-13155358
]
Matei Zaharia commented on MAPREDUCE-2905:
--
The patch looks good, except that there's an extra space on a line in
TestFairScheduler:
{code:java}
ListTask tasks = scheduler.assignTasks(tracker(taskTrackerName))
{code}
Other than that, I think it's good to go.
CapBasedLoadManager incorrectly allows assignment when assignMultiple is true
(was: assignmultiple per job)
---
Key: MAPREDUCE-2905
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/fair-share
Affects Versions: 0.20.2
Reporter: Jeff Bean
Assignee: Jeff Bean
Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2,
mr-2905.txt, mr-2905.txt, screenshot-1.jpg
We encountered a situation where in the same cluster, large jobs benefit from
mapred.fairscheduler.assignmultiple, but small jobs with small numbers of
mappers do not: the mappers all clump to fully occupy just a few nodes, which
causes those nodes to saturate and bottleneck. The desired behavior is to
spread the job across more nodes so that a relatively small job doesn't
saturate any node in the cluster.
Testing has shown that setting mapred.fairscheduler.assignmultiple to false
gives the desired behavior for small jobs, but is unnecessary for large jobs.
However, since this is a cluster-wide setting, we can't properly tune.
It'd be nice if jobs can set a param similar to
mapred.fairscheduler.assignmultiple on submission to better control the task
distribution of a particular job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved MAPREDUCE-2905. Resolution: Fixed Fix Version/s: 0.20.206.0 Hadoop Flags: Reviewed Committed to 0.20-security. CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job) --- Key: MAPREDUCE-2905 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/fair-share Affects Versions: 0.20.2 Reporter: Jeff Bean Assignee: Jeff Bean Fix For: 0.20.206.0 Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, mr-2905.txt, mr-2905.txt, screenshot-1.jpg We encountered a situation where in the same cluster, large jobs benefit from mapred.fairscheduler.assignmultiple, but small jobs with small numbers of mappers do not: the mappers all clump to fully occupy just a few nodes, which causes those nodes to saturate and bottleneck. The desired behavior is to spread the job across more nodes so that a relatively small job doesn't saturate any node in the cluster. Testing has shown that setting mapred.fairscheduler.assignmultiple to false gives the desired behavior for small jobs, but is unnecessary for large jobs. However, since this is a cluster-wide setting, we can't properly tune. It'd be nice if jobs can set a param similar to mapred.fairscheduler.assignmultiple on submission to better control the task distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3454) [Gridmix] TestDistCacheEmulation is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155378#comment-13155378 ] Hudson commented on MAPREDUCE-3454: --- Integrated in Hadoop-Hdfs-trunk-Commit #1386 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1386/]) MAPREDUCE-3454. [Gridmix] TestDistCacheEmulation is broken (Hitesh Shah MAPREDUCE-3454. via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205143 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/ivy.xml * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/streaming [Gridmix] TestDistCacheEmulation is broken -- Key: MAPREDUCE-3454 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3454 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Hitesh Shah Labels: gridmix Fix For: 0.23.1 Attachments: MR-3454.1.patch TestDistCacheEmulation is broken as 'MapReduceTestUtil' no longer exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3454) [Gridmix] TestDistCacheEmulation is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155384#comment-13155384 ] Hudson commented on MAPREDUCE-3454: --- Integrated in Hadoop-Common-trunk-Commit #1313 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1313/]) MAPREDUCE-3454. [Gridmix] TestDistCacheEmulation is broken (Hitesh Shah MAPREDUCE-3454. via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205143 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/ivy.xml * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/streaming [Gridmix] TestDistCacheEmulation is broken -- Key: MAPREDUCE-3454 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3454 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Hitesh Shah Labels: gridmix Fix For: 0.23.1 Attachments: MR-3454.1.patch TestDistCacheEmulation is broken as 'MapReduceTestUtil' no longer exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3454) [Gridmix] TestDistCacheEmulation is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155387#comment-13155387 ] Hudson commented on MAPREDUCE-3454: --- Integrated in Hadoop-Mapreduce-0.23-Commit #215 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/215/]) MAPREDUCE-3454. [Gridmix] TestDistCacheEmulation is broken (Hitesh Shah via mahadev) - Merging r1205143 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205144 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/ivy.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/streaming [Gridmix] TestDistCacheEmulation is broken -- Key: MAPREDUCE-3454 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3454 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Hitesh Shah Labels: gridmix Fix For: 0.23.1 Attachments: MR-3454.1.patch TestDistCacheEmulation is broken as 'MapReduceTestUtil' no longer exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3454) [Gridmix] TestDistCacheEmulation is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155388#comment-13155388 ] Hudson commented on MAPREDUCE-3454: --- Integrated in Hadoop-Common-0.23-Commit #204 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/204/]) MAPREDUCE-3454. [Gridmix] TestDistCacheEmulation is broken (Hitesh Shah via mahadev) - Merging r1205143 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205144 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/ivy.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/streaming [Gridmix] TestDistCacheEmulation is broken -- Key: MAPREDUCE-3454 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3454 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Hitesh Shah Labels: gridmix Fix For: 0.23.1 Attachments: MR-3454.1.patch TestDistCacheEmulation is broken as 'MapReduceTestUtil' no longer exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3454) [Gridmix] TestDistCacheEmulation is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155392#comment-13155392 ] Hudson commented on MAPREDUCE-3454: --- Integrated in Hadoop-Hdfs-0.23-Commit #201 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/201/]) MAPREDUCE-3454. [Gridmix] TestDistCacheEmulation is broken (Hitesh Shah via mahadev) - Merging r1205143 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205144 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/ivy.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/streaming [Gridmix] TestDistCacheEmulation is broken -- Key: MAPREDUCE-3454 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3454 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Hitesh Shah Labels: gridmix Fix For: 0.23.1 Attachments: MR-3454.1.patch TestDistCacheEmulation is broken as 'MapReduceTestUtil' no longer exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3454) [Gridmix] TestDistCacheEmulation is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155393#comment-13155393 ] Hadoop QA commented on MAPREDUCE-3454: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504784/MR-3454.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 12 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1338//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1338//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-examples.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1338//console This message is automatically generated. [Gridmix] TestDistCacheEmulation is broken -- Key: MAPREDUCE-3454 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3454 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Hitesh Shah Labels: gridmix Fix For: 0.23.1 Attachments: MR-3454.1.patch TestDistCacheEmulation is broken as 'MapReduceTestUtil' no longer exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3454) [Gridmix] TestDistCacheEmulation is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155399#comment-13155399 ] Hudson commented on MAPREDUCE-3454: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1336 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1336/]) MAPREDUCE-3454. [Gridmix] TestDistCacheEmulation is broken (Hitesh Shah MAPREDUCE-3454. via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205143 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/ivy.xml * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/streaming [Gridmix] TestDistCacheEmulation is broken -- Key: MAPREDUCE-3454 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3454 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Hitesh Shah Labels: gridmix Fix For: 0.23.1 Attachments: MR-3454.1.patch TestDistCacheEmulation is broken as 'MapReduceTestUtil' no longer exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2103) task-controller shouldn't require o-r permissions
[
https://issues.apache.org/jira/browse/MAPREDUCE-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155419#comment-13155419
]
Todd Lipcon commented on MAPREDUCE-2103:
+1 on the 20s patch
task-controller shouldn't require o-r permissions
-
Key: MAPREDUCE-2103
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2103
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: task-controller
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Trivial
Attachments: mapreduce-2103-20x.patch, mapreduce-2103.txt,
mapreduce-2103.txt
The task-controller currently checks that other users don't have read
permissions. This is unnecessary - we just need to make it's not executable.
The debian policy manual explains it well:
{quote}
Setuid and setgid executables should be mode 4755 or 2755 respectively, and
owned by the appropriate user or group. They should not be made unreadable
(modes like 4711 or 2711 or even 4111); doing so achieves no extra security,
because anyone can find the binary in the freely available Debian package; it
is merely inconvenient. For the same reason you should not restrict read or
execute permissions on non-set-id executables.
Some setuid programs need to be restricted to particular sets of users, using
file permissions. In this case they should be owned by the uid to which they
are set-id, and by the group which should be allowed to execute them. They
should have mode 4754; again there is no point in making them unreadable to
those users who must not be allowed to execute them.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3455) TokenCache should only call getDelegationToken if getDelegationTokens does not return valid tokens
TokenCache should only call getDelegationToken if getDelegationTokens does not return valid tokens -- Key: MAPREDUCE-3455 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3455 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Currently getDelegationToken is called irrespective of whether getDelegationTokens gives back valid tokens. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3433) Finding counters by legacy group name returns empty counters
[ https://issues.apache.org/jira/browse/MAPREDUCE-3433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3433: - Status: Open (was: Patch Available) Tom, looks like the tests are failing. Mind taking a look? Finding counters by legacy group name returns empty counters Key: MAPREDUCE-3433 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3433 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client, mrv2 Reporter: Tom White Assignee: Tom White Fix For: 0.24.0, 0.23.1 Attachments: MAPREDUCE-3433-jenkins.patch.patch, MAPREDUCE-3433.patch Attempting to find counters with a legacy group name (e.g. org.apache.hadoop.mapred.Task$Counter rather than the new org.apache.hadoop.mapreduce.TaskCounter) returns empty counters. This causes TestStreamingCombiner to fail when run with YARN. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3446) Move hadoop-examples to hadoop-mapreduce-client/hadoop-examples?
[ https://issues.apache.org/jira/browse/MAPREDUCE-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155509#comment-13155509 ] Alejandro Abdelnur commented on MAPREDUCE-3446: --- sure, and it should be hadoop-mapreduce-examples, right? Move hadoop-examples to hadoop-mapreduce-client/hadoop-examples? Key: MAPREDUCE-3446 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3446 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Arun C Murthy Should we consider moving hadoop-examples to hadoop-mapreduce-client/hadoop-examples? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3408) yarn-daemon.sh unconditionnaly sets yarn.root.logger
[
https://issues.apache.org/jira/browse/MAPREDUCE-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar updated MAPREDUCE-3408:
-
Resolution: Fixed
Fix Version/s: 0.23.1
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Looks good. Just committed this to trunk and 0.23. Thanks Bruno.
yarn-daemon.sh unconditionnaly sets yarn.root.logger
Key: MAPREDUCE-3408
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3408
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager, resourcemanager
Affects Versions: 0.23.0, 0.23.1
Reporter: Bruno Mahé
Assignee: Bruno Mahé
Labels: bigtop
Fix For: 0.23.1
Attachments: MAPREDUCE-3408.patch
yarn-daemon.sh unconditionnaly sets yarn.root.logger which then prevent any
override from happening.
From ./hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh:
export YARN_ROOT_LOGGER=INFO,DRFA
export YARN_JHS_LOGGER=INFO,JSA
and then yarn-daemon.sh will call $YARN_HOME/bin/yarn which does the
following:
YARN_OPTS=$YARN_OPTS
-Dhadoop.root.logger=${YARN_ROOT_LOGGER:-INFO,console}
YARN_OPTS=$YARN_OPTS -Dyarn.root.logger=${YARN_ROOT_LOGGER:-INFO,console}
This has at least 2 issues:
* I cannot override hadoop.root.logger when using the yarn-daemon.sh script
* I cannot have different values for hadoop.root.logger and yarn.root.logger
I currently see two different ways to proceed forward:
1/ Make the script yarn-daemon.sh only sets a default value for
YARN_ROOT_LOGGER if this variable is not defined
2/ Remove the quoted code from yarn-daemon.sh since yarn already does
something similar
3/ Entirely remove that chunk and let people define their logging however
they want through some properties files (see log4j.properties in the conf
directories for instance)
I would also use the variable HADOOP_ROOT_LOGGER for hadoop.root.logger if
either option 1/ or 2/ would be taken.
I don't really have any preference toward any of these solutions. What would
you recommend? What is the Apache Hadoop way for this matter?
Note: This is probably happening as well for the other daemons, and I will
take a look at it once this issue is resolved.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3369) Migrate MR1 tests to run on MR2 using the new interfaces introduced in MAPREDUCE-3169
[ https://issues.apache.org/jira/browse/MAPREDUCE-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3369: - Status: Open (was: Patch Available) cancelling while tom's concerns are addressed. Migrate MR1 tests to run on MR2 using the new interfaces introduced in MAPREDUCE-3169 - Key: MAPREDUCE-3369 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3369 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, mrv2, test Affects Versions: 0.23.0 Reporter: Ahmed Radwan Assignee: Ahmed Radwan Attachments: MAPREDUCE-3369.patch, MAPREDUCE-3369_rev2_part1.patch, MAPREDUCE-3369_rev2_part2.patch, mv-cp.sh This ticket tracks the migration of MR1 tests (currently residing in hadoop-mapreduce-project/src/test/) to run on MR2. The migration is using the new interfaces introduced in MAPREDUCE-3169. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3408) yarn-daemon.sh unconditionnaly sets yarn.root.logger
[
https://issues.apache.org/jira/browse/MAPREDUCE-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155525#comment-13155525
]
Hudson commented on MAPREDUCE-3408:
---
Integrated in Hadoop-Hdfs-0.23-Commit #203 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/203/])
MAPREDUCE-3408. yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno
Mahe via mahadev) - Merging r1205200 from trunk
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205201
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
yarn-daemon.sh unconditionnaly sets yarn.root.logger
Key: MAPREDUCE-3408
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3408
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager, resourcemanager
Affects Versions: 0.23.0, 0.23.1
Reporter: Bruno Mahé
Assignee: Bruno Mahé
Labels: bigtop
Fix For: 0.23.1
Attachments: MAPREDUCE-3408.patch
yarn-daemon.sh unconditionnaly sets yarn.root.logger which then prevent any
override from happening.
From ./hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh:
export YARN_ROOT_LOGGER=INFO,DRFA
export YARN_JHS_LOGGER=INFO,JSA
and then yarn-daemon.sh will call $YARN_HOME/bin/yarn which does the
following:
YARN_OPTS=$YARN_OPTS
-Dhadoop.root.logger=${YARN_ROOT_LOGGER:-INFO,console}
YARN_OPTS=$YARN_OPTS -Dyarn.root.logger=${YARN_ROOT_LOGGER:-INFO,console}
This has at least 2 issues:
* I cannot override hadoop.root.logger when using the yarn-daemon.sh script
* I cannot have different values for hadoop.root.logger and yarn.root.logger
I currently see two different ways to proceed forward:
1/ Make the script yarn-daemon.sh only sets a default value for
YARN_ROOT_LOGGER if this variable is not defined
2/ Remove the quoted code from yarn-daemon.sh since yarn already does
something similar
3/ Entirely remove that chunk and let people define their logging however
they want through some properties files (see log4j.properties in the conf
directories for instance)
I would also use the variable HADOOP_ROOT_LOGGER for hadoop.root.logger if
either option 1/ or 2/ would be taken.
I don't really have any preference toward any of these solutions. What would
you recommend? What is the Apache Hadoop way for this matter?
Note: This is probably happening as well for the other daemons, and I will
take a look at it once this issue is resolved.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3408) yarn-daemon.sh unconditionnaly sets yarn.root.logger
[
https://issues.apache.org/jira/browse/MAPREDUCE-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155526#comment-13155526
]
Hudson commented on MAPREDUCE-3408:
---
Integrated in Hadoop-Common-0.23-Commit #206 (See
[https://builds.apache.org/job/Hadoop-Common-0.23-Commit/206/])
MAPREDUCE-3408. yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno
Mahe via mahadev) - Merging r1205200 from trunk
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205201
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
yarn-daemon.sh unconditionnaly sets yarn.root.logger
Key: MAPREDUCE-3408
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3408
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager, resourcemanager
Affects Versions: 0.23.0, 0.23.1
Reporter: Bruno Mahé
Assignee: Bruno Mahé
Labels: bigtop
Fix For: 0.23.1
Attachments: MAPREDUCE-3408.patch
yarn-daemon.sh unconditionnaly sets yarn.root.logger which then prevent any
override from happening.
From ./hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh:
export YARN_ROOT_LOGGER=INFO,DRFA
export YARN_JHS_LOGGER=INFO,JSA
and then yarn-daemon.sh will call $YARN_HOME/bin/yarn which does the
following:
YARN_OPTS=$YARN_OPTS
-Dhadoop.root.logger=${YARN_ROOT_LOGGER:-INFO,console}
YARN_OPTS=$YARN_OPTS -Dyarn.root.logger=${YARN_ROOT_LOGGER:-INFO,console}
This has at least 2 issues:
* I cannot override hadoop.root.logger when using the yarn-daemon.sh script
* I cannot have different values for hadoop.root.logger and yarn.root.logger
I currently see two different ways to proceed forward:
1/ Make the script yarn-daemon.sh only sets a default value for
YARN_ROOT_LOGGER if this variable is not defined
2/ Remove the quoted code from yarn-daemon.sh since yarn already does
something similar
3/ Entirely remove that chunk and let people define their logging however
they want through some properties files (see log4j.properties in the conf
directories for instance)
I would also use the variable HADOOP_ROOT_LOGGER for hadoop.root.logger if
either option 1/ or 2/ would be taken.
I don't really have any preference toward any of these solutions. What would
you recommend? What is the Apache Hadoop way for this matter?
Note: This is probably happening as well for the other daemons, and I will
take a look at it once this issue is resolved.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3411) Performance Upgrade for jQuery
[ https://issues.apache.org/jira/browse/MAPREDUCE-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3411: - Resolution: Fixed Fix Version/s: 0.23.1 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Just committed this. Thanks Jonathan. Performance Upgrade for jQuery -- Key: MAPREDUCE-3411 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3411 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.24.0, 0.23.1 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor Fix For: 0.23.1 Attachments: MAPREDUCE-3411.patch jQuery 1.6.4 is almost twice as fast as current version 1.4.4 on modern browsers on some operations. There are also many modern browser compatibility fixes http://jsperf.com/jquery-15-unique-traversal/15 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3408) yarn-daemon.sh unconditionnaly sets yarn.root.logger
[
https://issues.apache.org/jira/browse/MAPREDUCE-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155533#comment-13155533
]
Hudson commented on MAPREDUCE-3408:
---
Integrated in Hadoop-Hdfs-trunk-Commit #1388 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1388/])
MAPREDUCE-3408. yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno
Mahe via mahadev)
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205200
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
yarn-daemon.sh unconditionnaly sets yarn.root.logger
Key: MAPREDUCE-3408
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3408
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager, resourcemanager
Affects Versions: 0.23.0, 0.23.1
Reporter: Bruno Mahé
Assignee: Bruno Mahé
Labels: bigtop
Fix For: 0.23.1
Attachments: MAPREDUCE-3408.patch
yarn-daemon.sh unconditionnaly sets yarn.root.logger which then prevent any
override from happening.
From ./hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh:
export YARN_ROOT_LOGGER=INFO,DRFA
export YARN_JHS_LOGGER=INFO,JSA
and then yarn-daemon.sh will call $YARN_HOME/bin/yarn which does the
following:
YARN_OPTS=$YARN_OPTS
-Dhadoop.root.logger=${YARN_ROOT_LOGGER:-INFO,console}
YARN_OPTS=$YARN_OPTS -Dyarn.root.logger=${YARN_ROOT_LOGGER:-INFO,console}
This has at least 2 issues:
* I cannot override hadoop.root.logger when using the yarn-daemon.sh script
* I cannot have different values for hadoop.root.logger and yarn.root.logger
I currently see two different ways to proceed forward:
1/ Make the script yarn-daemon.sh only sets a default value for
YARN_ROOT_LOGGER if this variable is not defined
2/ Remove the quoted code from yarn-daemon.sh since yarn already does
something similar
3/ Entirely remove that chunk and let people define their logging however
they want through some properties files (see log4j.properties in the conf
directories for instance)
I would also use the variable HADOOP_ROOT_LOGGER for hadoop.root.logger if
either option 1/ or 2/ would be taken.
I don't really have any preference toward any of these solutions. What would
you recommend? What is the Apache Hadoop way for this matter?
Note: This is probably happening as well for the other daemons, and I will
take a look at it once this issue is resolved.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3408) yarn-daemon.sh unconditionnaly sets yarn.root.logger
[
https://issues.apache.org/jira/browse/MAPREDUCE-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155534#comment-13155534
]
Hudson commented on MAPREDUCE-3408:
---
Integrated in Hadoop-Common-trunk-Commit #1315 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1315/])
MAPREDUCE-3408. yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno
Mahe via mahadev)
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205200
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
yarn-daemon.sh unconditionnaly sets yarn.root.logger
Key: MAPREDUCE-3408
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3408
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager, resourcemanager
Affects Versions: 0.23.0, 0.23.1
Reporter: Bruno Mahé
Assignee: Bruno Mahé
Labels: bigtop
Fix For: 0.23.1
Attachments: MAPREDUCE-3408.patch
yarn-daemon.sh unconditionnaly sets yarn.root.logger which then prevent any
override from happening.
From ./hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh:
export YARN_ROOT_LOGGER=INFO,DRFA
export YARN_JHS_LOGGER=INFO,JSA
and then yarn-daemon.sh will call $YARN_HOME/bin/yarn which does the
following:
YARN_OPTS=$YARN_OPTS
-Dhadoop.root.logger=${YARN_ROOT_LOGGER:-INFO,console}
YARN_OPTS=$YARN_OPTS -Dyarn.root.logger=${YARN_ROOT_LOGGER:-INFO,console}
This has at least 2 issues:
* I cannot override hadoop.root.logger when using the yarn-daemon.sh script
* I cannot have different values for hadoop.root.logger and yarn.root.logger
I currently see two different ways to proceed forward:
1/ Make the script yarn-daemon.sh only sets a default value for
YARN_ROOT_LOGGER if this variable is not defined
2/ Remove the quoted code from yarn-daemon.sh since yarn already does
something similar
3/ Entirely remove that chunk and let people define their logging however
they want through some properties files (see log4j.properties in the conf
directories for instance)
I would also use the variable HADOOP_ROOT_LOGGER for hadoop.root.logger if
either option 1/ or 2/ would be taken.
I don't really have any preference toward any of these solutions. What would
you recommend? What is the Apache Hadoop way for this matter?
Note: This is probably happening as well for the other daemons, and I will
take a look at it once this issue is resolved.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3408) yarn-daemon.sh unconditionnaly sets yarn.root.logger
[
https://issues.apache.org/jira/browse/MAPREDUCE-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155540#comment-13155540
]
Hudson commented on MAPREDUCE-3408:
---
Integrated in Hadoop-Mapreduce-0.23-Commit #217 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/217/])
MAPREDUCE-3408. yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno
Mahe via mahadev) - Merging r1205200 from trunk
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205201
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
yarn-daemon.sh unconditionnaly sets yarn.root.logger
Key: MAPREDUCE-3408
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3408
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager, resourcemanager
Affects Versions: 0.23.0, 0.23.1
Reporter: Bruno Mahé
Assignee: Bruno Mahé
Labels: bigtop
Fix For: 0.23.1
Attachments: MAPREDUCE-3408.patch
yarn-daemon.sh unconditionnaly sets yarn.root.logger which then prevent any
override from happening.
From ./hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh:
export YARN_ROOT_LOGGER=INFO,DRFA
export YARN_JHS_LOGGER=INFO,JSA
and then yarn-daemon.sh will call $YARN_HOME/bin/yarn which does the
following:
YARN_OPTS=$YARN_OPTS
-Dhadoop.root.logger=${YARN_ROOT_LOGGER:-INFO,console}
YARN_OPTS=$YARN_OPTS -Dyarn.root.logger=${YARN_ROOT_LOGGER:-INFO,console}
This has at least 2 issues:
* I cannot override hadoop.root.logger when using the yarn-daemon.sh script
* I cannot have different values for hadoop.root.logger and yarn.root.logger
I currently see two different ways to proceed forward:
1/ Make the script yarn-daemon.sh only sets a default value for
YARN_ROOT_LOGGER if this variable is not defined
2/ Remove the quoted code from yarn-daemon.sh since yarn already does
something similar
3/ Entirely remove that chunk and let people define their logging however
they want through some properties files (see log4j.properties in the conf
directories for instance)
I would also use the variable HADOOP_ROOT_LOGGER for hadoop.root.logger if
either option 1/ or 2/ would be taken.
I don't really have any preference toward any of these solutions. What would
you recommend? What is the Apache Hadoop way for this matter?
Note: This is probably happening as well for the other daemons, and I will
take a look at it once this issue is resolved.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3411) Performance Upgrade for jQuery
[ https://issues.apache.org/jira/browse/MAPREDUCE-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155541#comment-13155541 ] Hudson commented on MAPREDUCE-3411: --- Integrated in Hadoop-Common-0.23-Commit #207 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/207/]) MAPREDUCE-3411. Performance Upgrade for jQuery (Jonathan Eagles via mahadev) - Merging r1205210 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205213 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/JQueryUI.java Performance Upgrade for jQuery -- Key: MAPREDUCE-3411 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3411 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.24.0, 0.23.1 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor Fix For: 0.23.1 Attachments: MAPREDUCE-3411.patch jQuery 1.6.4 is almost twice as fast as current version 1.4.4 on modern browsers on some operations. There are also many modern browser compatibility fixes http://jsperf.com/jquery-15-unique-traversal/15 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3411) Performance Upgrade for jQuery
[ https://issues.apache.org/jira/browse/MAPREDUCE-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155542#comment-13155542 ] Hudson commented on MAPREDUCE-3411: --- Integrated in Hadoop-Hdfs-trunk-Commit #1389 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1389/]) MAPREDUCE-3411. Performance Upgrade for jQuery (Jonathan Eagles via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205210 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/JQueryUI.java Performance Upgrade for jQuery -- Key: MAPREDUCE-3411 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3411 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.24.0, 0.23.1 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor Fix For: 0.23.1 Attachments: MAPREDUCE-3411.patch jQuery 1.6.4 is almost twice as fast as current version 1.4.4 on modern browsers on some operations. There are also many modern browser compatibility fixes http://jsperf.com/jquery-15-unique-traversal/15 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3411) Performance Upgrade for jQuery
[ https://issues.apache.org/jira/browse/MAPREDUCE-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155549#comment-13155549 ] Hudson commented on MAPREDUCE-3411: --- Integrated in Hadoop-Common-trunk-Commit #1316 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1316/]) MAPREDUCE-3411. Performance Upgrade for jQuery (Jonathan Eagles via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205210 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/JQueryUI.java Performance Upgrade for jQuery -- Key: MAPREDUCE-3411 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3411 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.24.0, 0.23.1 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor Fix For: 0.23.1 Attachments: MAPREDUCE-3411.patch jQuery 1.6.4 is almost twice as fast as current version 1.4.4 on modern browsers on some operations. There are also many modern browser compatibility fixes http://jsperf.com/jquery-15-unique-traversal/15 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3408) yarn-daemon.sh unconditionnaly sets yarn.root.logger
[
https://issues.apache.org/jira/browse/MAPREDUCE-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1312#comment-1312
]
Hudson commented on MAPREDUCE-3408:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #1338 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1338/])
MAPREDUCE-3408. yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno
Mahe via mahadev)
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205200
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
yarn-daemon.sh unconditionnaly sets yarn.root.logger
Key: MAPREDUCE-3408
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3408
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager, resourcemanager
Affects Versions: 0.23.0, 0.23.1
Reporter: Bruno Mahé
Assignee: Bruno Mahé
Labels: bigtop
Fix For: 0.23.1
Attachments: MAPREDUCE-3408.patch
yarn-daemon.sh unconditionnaly sets yarn.root.logger which then prevent any
override from happening.
From ./hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh:
export YARN_ROOT_LOGGER=INFO,DRFA
export YARN_JHS_LOGGER=INFO,JSA
and then yarn-daemon.sh will call $YARN_HOME/bin/yarn which does the
following:
YARN_OPTS=$YARN_OPTS
-Dhadoop.root.logger=${YARN_ROOT_LOGGER:-INFO,console}
YARN_OPTS=$YARN_OPTS -Dyarn.root.logger=${YARN_ROOT_LOGGER:-INFO,console}
This has at least 2 issues:
* I cannot override hadoop.root.logger when using the yarn-daemon.sh script
* I cannot have different values for hadoop.root.logger and yarn.root.logger
I currently see two different ways to proceed forward:
1/ Make the script yarn-daemon.sh only sets a default value for
YARN_ROOT_LOGGER if this variable is not defined
2/ Remove the quoted code from yarn-daemon.sh since yarn already does
something similar
3/ Entirely remove that chunk and let people define their logging however
they want through some properties files (see log4j.properties in the conf
directories for instance)
I would also use the variable HADOOP_ROOT_LOGGER for hadoop.root.logger if
either option 1/ or 2/ would be taken.
I don't really have any preference toward any of these solutions. What would
you recommend? What is the Apache Hadoop way for this matter?
Note: This is probably happening as well for the other daemons, and I will
take a look at it once this issue is resolved.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3411) Performance Upgrade for jQuery
[ https://issues.apache.org/jira/browse/MAPREDUCE-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1313#comment-1313 ] Hudson commented on MAPREDUCE-3411: --- Integrated in Hadoop-Mapreduce-0.23-Commit #218 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/218/]) MAPREDUCE-3411. Performance Upgrade for jQuery (Jonathan Eagles via mahadev) - Merging r1205210 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205213 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/JQueryUI.java Performance Upgrade for jQuery -- Key: MAPREDUCE-3411 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3411 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.24.0, 0.23.1 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor Fix For: 0.23.1 Attachments: MAPREDUCE-3411.patch jQuery 1.6.4 is almost twice as fast as current version 1.4.4 on modern browsers on some operations. There are also many modern browser compatibility fixes http://jsperf.com/jquery-15-unique-traversal/15 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3456) $HADOOP_PREFIX/bin/yarn should set defaults for $HADOOP_*_HOME
$HADOOP_PREFIX/bin/yarn should set defaults for $HADOOP_*_HOME -- Key: MAPREDUCE-3456 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3456 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Eric Payne If the $HADOOP_PREFIX/hadoop-dist/target/hadoop-0.23.0-SNAPSHOT.tar.gz tarball is used to distribute hadoop, all of the HADOOP components (HDFS, MAPRED, COMMON) are all under one directory. In this use case, HADOOP_PREFIX should be set and should point to the root directory for all components, and it should not be necessary to set HADOOP_HDFS_HOME, HADOOP_COMMON_HOME, and HADOOP_MAPRED_HOME. However, the $HADOOP_PREFIX/bin/yarn script requires these 3 to be set explicitly in the calling environment or it won't run. $HADOOP_PREFIX/bin/yarn should check if $HADOOP_PREFIX is set and, if it is, use that value for the 3 other HADOOP_*_HOME variables. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3411) Performance Upgrade for jQuery
[ https://issues.apache.org/jira/browse/MAPREDUCE-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155569#comment-13155569 ] Hudson commented on MAPREDUCE-3411: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1339 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1339/]) MAPREDUCE-3411. Performance Upgrade for jQuery (Jonathan Eagles via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205210 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/JQueryUI.java Performance Upgrade for jQuery -- Key: MAPREDUCE-3411 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3411 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.24.0, 0.23.1 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor Fix For: 0.23.1 Attachments: MAPREDUCE-3411.patch jQuery 1.6.4 is almost twice as fast as current version 1.4.4 on modern browsers on some operations. There are also many modern browser compatibility fixes http://jsperf.com/jquery-15-unique-traversal/15 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3456) $HADOOP_PREFIX/bin/yarn should set defaults for $HADOOP_*_HOME
[ https://issues.apache.org/jira/browse/MAPREDUCE-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3456: - Priority: Critical (was: Major) $HADOOP_PREFIX/bin/yarn should set defaults for $HADOOP_*_HOME -- Key: MAPREDUCE-3456 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3456 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Eric Payne Priority: Critical If the $HADOOP_PREFIX/hadoop-dist/target/hadoop-0.23.0-SNAPSHOT.tar.gz tarball is used to distribute hadoop, all of the HADOOP components (HDFS, MAPRED, COMMON) are all under one directory. In this use case, HADOOP_PREFIX should be set and should point to the root directory for all components, and it should not be necessary to set HADOOP_HDFS_HOME, HADOOP_COMMON_HOME, and HADOOP_MAPRED_HOME. However, the $HADOOP_PREFIX/bin/yarn script requires these 3 to be set explicitly in the calling environment or it won't run. $HADOOP_PREFIX/bin/yarn should check if $HADOOP_PREFIX is set and, if it is, use that value for the 3 other HADOOP_*_HOME variables. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3443) Oozie jobs are running as oozie user even though they create the jobclient as doAs.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar updated MAPREDUCE-3443:
-
Attachment: MAPREDUCE-3443.patch
Updated patch with some bug fixes.
Oozie jobs are running as oozie user even though they create the jobclient as
doAs.
---
Key: MAPREDUCE-3443
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3443
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Mahadev konar
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.1
Attachments: MAPREDUCE-3443.patch, MAPREDUCE-3443.patch
Oozie is having issues with job submission, since it does the following:
{code}
doAs(userwhosubmittedjob) {
jobclient = new JobClient(jobconf);
}
jobclient.submitjob()
{code}
In 0.20.2** this works because the JT proxy is created as soon as we call new
JobClient(). But in 0.23 this is no longer true since the client has to talk
to multiple servers (AM/RM/JHS). To keep this behavior we will have to store
the ugi in new JobClient() and make sure all the calls are run with a doAs()
inside the jobclient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3457) MR examples can not be built from source tarball
MR examples can not be built from source tarball
Key: MAPREDUCE-3457
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3457
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build, examples
Affects Versions: 0.23.0
Reporter: Andrew Bayer
When attempting to build the mapreduce examples in hadoop-mapreduce-project,
from the contents of the released source tarball hadoop-0.23.0-src.tar.gz, you
hit a failure due to a missing file:
{quote}
andrew@ubuntu-slave01:/tmp/hadoop-0.23.0-src/hadoop-mapreduce-project$ ant
examples -Dresolvers=internal
Buildfile: /tmp/hadoop-0.23.0-src/hadoop-mapreduce-project/build.xml
BUILD FAILED
/tmp/hadoop-0.23.0-src/hadoop-mapreduce-project/build.xml:2011: Cannot find
/tmp/hadoop-0.23.0-src/hadoop-mapreduce-project/src/test/aop/build/aop.xml
imported from /tmp/hadoop-0.23.0-src/hadoop-mapreduce-project/build.xml
Total time: 1 second
andrew@ubuntu-slave01:/tmp/hadoop-0.23.0-src/hadoop-mapreduce-project$
{quote}
Note that this is after having run mvn -Pdist -Pnative -Psrc -Dtar -DskipTests
-DskipTest -DskipITs install in the /tmp/hadoop-0.23.0-src directory. I can't
find any aop.xml files in the source tarball, though they're there in svn.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3446) Move hadoop-examples to hadoop-mapreduce-client/hadoop-examples?
[ https://issues.apache.org/jira/browse/MAPREDUCE-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155612#comment-13155612 ] Arun C Murthy commented on MAPREDUCE-3446: -- +1 Move hadoop-examples to hadoop-mapreduce-client/hadoop-examples? Key: MAPREDUCE-3446 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3446 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Arun C Murthy Should we consider moving hadoop-examples to hadoop-mapreduce-client/hadoop-examples? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3458) Fix findbugs warnings in hadoop-examples
Fix findbugs warnings in hadoop-examples Key: MAPREDUCE-3458 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3458 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.1 Reporter: Arun C Murthy I see 12 findbugs warnings in hadoop-examples: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1336//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-examples.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3329) capacity schedule maximum-capacity allowed to be less then capacity
[ https://issues.apache.org/jira/browse/MAPREDUCE-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3329: - Resolution: Fixed Fix Version/s: 0.23.1 Status: Resolved (was: Patch Available) I just committed the trivial fix. I've opened MAPREDUCE-3458 to track the findbugs warnings in hadoop-examples. capacity schedule maximum-capacity allowed to be less then capacity --- Key: MAPREDUCE-3329 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3329 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3329.patch When configuring the capacity scheduler capacity and maximum-capacity, it allows the maximum-capacity to be less then the capacity. I did not test to see what true limit is, I assume maximum capacity. output from mapred queue -list where capacity = 10%, max capacity = 5%. Queue Name : test2 Queue State : running Scheduling Info : queueName: test2, capacity: 0.1, maximumCapacity: 5.0, currentCapacity: 0.0, state: Q_RUNNING, -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3329) capacity schedule maximum-capacity allowed to be less then capacity
[ https://issues.apache.org/jira/browse/MAPREDUCE-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155619#comment-13155619 ] Hudson commented on MAPREDUCE-3329: --- Integrated in Hadoop-Mapreduce-0.23-Commit #220 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/220/]) Merge -c 1205260 from trunk to branch-0.23 to fix MAPREDUCE-3329. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205261 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc *
[jira] [Commented] (MAPREDUCE-3329) capacity schedule maximum-capacity allowed to be less then capacity
[ https://issues.apache.org/jira/browse/MAPREDUCE-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155627#comment-13155627 ] Hudson commented on MAPREDUCE-3329: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1340 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1340/]) MAPREDUCE-3329. Fixed CapacityScheduler to ensure maximum-capacity cannot be lesser than capacity for any queue. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205260 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java capacity schedule maximum-capacity allowed to be less then capacity --- Key: MAPREDUCE-3329 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3329 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3329.patch When configuring the capacity scheduler capacity and maximum-capacity, it allows the maximum-capacity to be less then the capacity. I did not test to see what true limit is, I assume maximum capacity. output from mapred queue -list where capacity = 10%, max capacity = 5%. Queue Name : test2 Queue State : running Scheduling Info : queueName: test2, capacity: 0.1, maximumCapacity: 5.0, currentCapacity: 0.0, state: Q_RUNNING, -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3459) ant build is broken in branch-23
ant build is broken in branch-23
Key: MAPREDUCE-3459
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3459
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: John George
ant build is broken in 0.23. A small snippet:
[javac] Compiling 330 source files to
/home//hadoop/branch-0.23/hadoop-mapreduce-project/build/test/mapred/classes
[javac]
/home//hadoop/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/NotificationTestCase.java:28:
cannot find symbol
[javac] symbol : class MapReduceTestUtil
[javac] location: package org.apache.hadoop.mapreduce
[javac] import org.apache.hadoop.mapreduce.MapReduceTestUtil;
[javac] ^
[javac]
/home//hadoop/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestBadRecords.java:43:
cannot find symbol
[javac] symbol: class ClusterMapReduceTestCase
[javac] public class TestBadRecords extends ClusterMapReduceTestCase {
[javac] ^
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3459) ant build is broken in branch-23
[
https://issues.apache.org/jira/browse/MAPREDUCE-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155637#comment-13155637
]
Hitesh Shah commented on MAPREDUCE-3459:
Seems a dup of MAPREDUCE-3444. Could you purge your ivy cache and retry?
ant build is broken in branch-23
Key: MAPREDUCE-3459
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3459
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: John George
ant build is broken in 0.23. A small snippet:
[javac] Compiling 330 source files to
/home//hadoop/branch-0.23/hadoop-mapreduce-project/build/test/mapred/classes
[javac]
/home//hadoop/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/NotificationTestCase.java:28:
cannot find symbol
[javac] symbol : class MapReduceTestUtil
[javac] location: package org.apache.hadoop.mapreduce
[javac] import org.apache.hadoop.mapreduce.MapReduceTestUtil;
[javac] ^
[javac]
/home//hadoop/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestBadRecords.java:43:
cannot find symbol
[javac] symbol: class ClusterMapReduceTestCase
[javac] public class TestBadRecords extends ClusterMapReduceTestCase {
[javac] ^
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3329) capacity schedule maximum-capacity allowed to be less then capacity
[ https://issues.apache.org/jira/browse/MAPREDUCE-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155647#comment-13155647 ] Hudson commented on MAPREDUCE-3329: --- Integrated in Hadoop-Common-trunk-Commit #1317 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1317/]) MAPREDUCE-3329. Fixed CapacityScheduler to ensure maximum-capacity cannot be lesser than capacity for any queue. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205260 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java capacity schedule maximum-capacity allowed to be less then capacity --- Key: MAPREDUCE-3329 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3329 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3329.patch When configuring the capacity scheduler capacity and maximum-capacity, it allows the maximum-capacity to be less then the capacity. I did not test to see what true limit is, I assume maximum capacity. output from mapred queue -list where capacity = 10%, max capacity = 5%. Queue Name : test2 Queue State : running Scheduling Info : queueName: test2, capacity: 0.1, maximumCapacity: 5.0, currentCapacity: 0.0, state: Q_RUNNING, -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3329) capacity schedule maximum-capacity allowed to be less then capacity
[ https://issues.apache.org/jira/browse/MAPREDUCE-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155648#comment-13155648 ] Hudson commented on MAPREDUCE-3329: --- Integrated in Hadoop-Hdfs-0.23-Commit #206 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/206/]) Merge -c 1205260 from trunk to branch-0.23 to fix MAPREDUCE-3329. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205261 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc *
[jira] [Commented] (MAPREDUCE-3459) ant build is broken in branch-23
[
https://issues.apache.org/jira/browse/MAPREDUCE-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155649#comment-13155649
]
Mahadev konar commented on MAPREDUCE-3459:
--
I just tried it out, I dont see the issue either.
ant build is broken in branch-23
Key: MAPREDUCE-3459
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3459
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: John George
ant build is broken in 0.23. A small snippet:
[javac] Compiling 330 source files to
/home//hadoop/branch-0.23/hadoop-mapreduce-project/build/test/mapred/classes
[javac]
/home//hadoop/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/NotificationTestCase.java:28:
cannot find symbol
[javac] symbol : class MapReduceTestUtil
[javac] location: package org.apache.hadoop.mapreduce
[javac] import org.apache.hadoop.mapreduce.MapReduceTestUtil;
[javac] ^
[javac]
/home//hadoop/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestBadRecords.java:43:
cannot find symbol
[javac] symbol: class ClusterMapReduceTestCase
[javac] public class TestBadRecords extends ClusterMapReduceTestCase {
[javac] ^
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3265) Reduce log level on MR2 IPC construction, etc
[ https://issues.apache.org/jira/browse/MAPREDUCE-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3265: - Attachment: MAPREDUCE-3265.patch Changed a bunch of LOG.info statements. No tests since it's only logging changes. Reduce log level on MR2 IPC construction, etc - Key: MAPREDUCE-3265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3265 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Arun C Murthy Priority: Blocker Attachments: MAPREDUCE-3265.patch Currently MR's IPC logging is very verbose. For example, I see a lot of: 11/10/25 12:14:06 INFO ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC 11/10/25 12:14:06 INFO mapred.ResourceMgrDelegate: Connecting to ResourceManager at c0309.hal.cloudera.com/172.29.81.91:40012 11/10/25 12:14:06 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol 11/10/25 12:14:07 INFO mapred.ResourceMgrDelegate: Connected to ResourceManager at c0309.hal.cloudera.com/172.29.81.91:40012 11/10/25 12:14:08 INFO mapred.ClientCache: Connecting to HistoryServer at: c0309.hal.cloudera.com:10020 11/10/25 12:14:08 INFO ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC 11/10/25 12:14:08 INFO mapred.ClientCache: Connected to HistoryServer at: c0309.hal.cloudera.com:10020 11/10/25 12:14:08 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocol ... when submitting a job. This should be DEBUG level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3329) capacity schedule maximum-capacity allowed to be less then capacity
[ https://issues.apache.org/jira/browse/MAPREDUCE-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155650#comment-13155650 ] Hudson commented on MAPREDUCE-3329: --- Integrated in Hadoop-Common-0.23-Commit #209 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/209/]) Merge -c 1205260 from trunk to branch-0.23 to fix MAPREDUCE-3329. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205261 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc *
[jira] [Updated] (MAPREDUCE-3265) Reduce log level on MR2 IPC construction, etc
[ https://issues.apache.org/jira/browse/MAPREDUCE-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3265: - Fix Version/s: 0.23.1 Status: Patch Available (was: Open) Reduce log level on MR2 IPC construction, etc - Key: MAPREDUCE-3265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3265 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3265.patch Currently MR's IPC logging is very verbose. For example, I see a lot of: 11/10/25 12:14:06 INFO ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC 11/10/25 12:14:06 INFO mapred.ResourceMgrDelegate: Connecting to ResourceManager at c0309.hal.cloudera.com/172.29.81.91:40012 11/10/25 12:14:06 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol 11/10/25 12:14:07 INFO mapred.ResourceMgrDelegate: Connected to ResourceManager at c0309.hal.cloudera.com/172.29.81.91:40012 11/10/25 12:14:08 INFO mapred.ClientCache: Connecting to HistoryServer at: c0309.hal.cloudera.com:10020 11/10/25 12:14:08 INFO ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC 11/10/25 12:14:08 INFO mapred.ClientCache: Connected to HistoryServer at: c0309.hal.cloudera.com:10020 11/10/25 12:14:08 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocol ... when submitting a job. This should be DEBUG level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3329) capacity schedule maximum-capacity allowed to be less then capacity
[ https://issues.apache.org/jira/browse/MAPREDUCE-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155652#comment-13155652 ] Hudson commented on MAPREDUCE-3329: --- Integrated in Hadoop-Hdfs-trunk-Commit #1390 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1390/]) MAPREDUCE-3329. Fixed CapacityScheduler to ensure maximum-capacity cannot be lesser than capacity for any queue. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205260 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java capacity schedule maximum-capacity allowed to be less then capacity --- Key: MAPREDUCE-3329 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3329 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3329.patch When configuring the capacity scheduler capacity and maximum-capacity, it allows the maximum-capacity to be less then the capacity. I did not test to see what true limit is, I assume maximum capacity. output from mapred queue -list where capacity = 10%, max capacity = 5%. Queue Name : test2 Queue State : running Scheduling Info : queueName: test2, capacity: 0.1, maximumCapacity: 5.0, currentCapacity: 0.0, state: Q_RUNNING, -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3265) Reduce log level on MR2 IPC construction, etc
[ https://issues.apache.org/jira/browse/MAPREDUCE-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155654#comment-13155654 ] Mahadev konar commented on MAPREDUCE-3265: -- +1 looks good to me. Reduce log level on MR2 IPC construction, etc - Key: MAPREDUCE-3265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3265 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3265.patch Currently MR's IPC logging is very verbose. For example, I see a lot of: 11/10/25 12:14:06 INFO ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC 11/10/25 12:14:06 INFO mapred.ResourceMgrDelegate: Connecting to ResourceManager at c0309.hal.cloudera.com/172.29.81.91:40012 11/10/25 12:14:06 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol 11/10/25 12:14:07 INFO mapred.ResourceMgrDelegate: Connected to ResourceManager at c0309.hal.cloudera.com/172.29.81.91:40012 11/10/25 12:14:08 INFO mapred.ClientCache: Connecting to HistoryServer at: c0309.hal.cloudera.com:10020 11/10/25 12:14:08 INFO ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC 11/10/25 12:14:08 INFO mapred.ClientCache: Connected to HistoryServer at: c0309.hal.cloudera.com:10020 11/10/25 12:14:08 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocol ... when submitting a job. This should be DEBUG level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3265) Reduce log level on MR2 IPC construction, etc
[
https://issues.apache.org/jira/browse/MAPREDUCE-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155656#comment-13155656
]
Arun C Murthy commented on MAPREDUCE-3265:
--
With the patch, the output looks so:
{noformat}
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
11/11/23 03:05:12 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
Running on 5 nodes to sort from
hdfs://hrt12n01.cc1.ygridcore.net:8020/user/hrt_1/in into
hdfs://nn.foobar.com:8020/user/hrt_1/out with 9 reduces.
Job started: Wed Nov 23 03:05:12 UTC 2011
11/11/23 03:05:12 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 379
for hrt_1 on xx.xx.xx.xx:8020
11/11/23 03:05:12 INFO security.TokenCache: Got dt for
hdfs://nn.foobar.com:8020:8020;uri=xx.xx.xx.xx:8020;t.service=xx.xx.xx.xx:8020
11/11/23 03:05:12 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 380
for hrt_1 on xx.xx.xx.xx:8020
11/11/23 03:05:12 INFO security.TokenCache: Got dt for
hdfs://nn.foobar.com:8020:8020;uri=xx.xx.xx.xx:8020;t.service=xx.xx.xx.xx:8020
11/11/23 03:05:12 WARN conf.Configuration: fs.default.name is deprecated.
Instead, use fs.defaultFS
11/11/23 03:05:15 INFO input.FileInputFormat: Total input paths to process : 5
11/11/23 03:05:17 INFO mapreduce.JobSubmitter: number of splits:5
11/11/23 03:05:18 INFO mapred.ResourceMgrDelegate: Submitted application
application_1322016908667_0002 to ResourceManager at
rm.foobar.com/yy.yy.yy.yy:8040
11/11/23 03:05:18 INFO mapreduce.Job: Running job: job_1322016908667_0002
11/11/23 03:05:19 INFO mapreduce.Job: map 0% reduce 0%
11/11/23 03:05:29 INFO mapred.ClientServiceDelegate: Job tracking-url:
rm.foobar.com:8088/proxy/application_1322016908667_0002/
11/11/23 03:05:40 INFO mapreduce.Job: map 100% reduce 0%
{noformat}
Reduce log level on MR2 IPC construction, etc
-
Key: MAPREDUCE-3265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Arun C Murthy
Priority: Blocker
Fix For: 0.23.1
Attachments: MAPREDUCE-3265.patch
Currently MR's IPC logging is very verbose. For example, I see a lot of:
11/10/25 12:14:06 INFO ipc.YarnRPC: Creating YarnRPC for
org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
11/10/25 12:14:06 INFO mapred.ResourceMgrDelegate: Connecting to
ResourceManager at c0309.hal.cloudera.com/172.29.81.91:40012
11/10/25 12:14:06 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy
for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol
11/10/25 12:14:07 INFO mapred.ResourceMgrDelegate: Connected to
ResourceManager at c0309.hal.cloudera.com/172.29.81.91:40012
11/10/25 12:14:08 INFO mapred.ClientCache: Connecting to HistoryServer at:
c0309.hal.cloudera.com:10020
11/10/25 12:14:08 INFO ipc.YarnRPC: Creating YarnRPC for
org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
11/10/25 12:14:08 INFO mapred.ClientCache: Connected to HistoryServer at:
c0309.hal.cloudera.com:10020
11/10/25 12:14:08 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy
for protocol interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocol
... when submitting a job. This should be DEBUG level.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3459) ant build is broken in branch-23
[
https://issues.apache.org/jira/browse/MAPREDUCE-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155657#comment-13155657
]
John George commented on MAPREDUCE-3459:
Yep - cleaned up my caches and it worked! Thanks for trying Mahadev and Hitesh.
ant build is broken in branch-23
Key: MAPREDUCE-3459
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3459
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: John George
ant build is broken in 0.23. A small snippet:
[javac] Compiling 330 source files to
/home//hadoop/branch-0.23/hadoop-mapreduce-project/build/test/mapred/classes
[javac]
/home//hadoop/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/NotificationTestCase.java:28:
cannot find symbol
[javac] symbol : class MapReduceTestUtil
[javac] location: package org.apache.hadoop.mapreduce
[javac] import org.apache.hadoop.mapreduce.MapReduceTestUtil;
[javac] ^
[javac]
/home//hadoop/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestBadRecords.java:43:
cannot find symbol
[javac] symbol: class ClusterMapReduceTestCase
[javac] public class TestBadRecords extends ClusterMapReduceTestCase {
[javac] ^
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3459) ant build is broken in branch-23
[
https://issues.apache.org/jira/browse/MAPREDUCE-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John George resolved MAPREDUCE-3459.
Resolution: Invalid
ant build is broken in branch-23
Key: MAPREDUCE-3459
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3459
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: John George
ant build is broken in 0.23. A small snippet:
[javac] Compiling 330 source files to
/home//hadoop/branch-0.23/hadoop-mapreduce-project/build/test/mapred/classes
[javac]
/home//hadoop/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/NotificationTestCase.java:28:
cannot find symbol
[javac] symbol : class MapReduceTestUtil
[javac] location: package org.apache.hadoop.mapreduce
[javac] import org.apache.hadoop.mapreduce.MapReduceTestUtil;
[javac] ^
[javac]
/home//hadoop/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestBadRecords.java:43:
cannot find symbol
[javac] symbol: class ClusterMapReduceTestCase
[javac] public class TestBadRecords extends ClusterMapReduceTestCase {
[javac] ^
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3265) Reduce log level on MR2 IPC construction, etc
[ https://issues.apache.org/jira/browse/MAPREDUCE-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155667#comment-13155667 ] Hadoop QA commented on MAPREDUCE-3265: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504844/MAPREDUCE-3265.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 12 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1339//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1339//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-examples.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1339//console This message is automatically generated. Reduce log level on MR2 IPC construction, etc - Key: MAPREDUCE-3265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3265 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3265.patch Currently MR's IPC logging is very verbose. For example, I see a lot of: 11/10/25 12:14:06 INFO ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC 11/10/25 12:14:06 INFO mapred.ResourceMgrDelegate: Connecting to ResourceManager at c0309.hal.cloudera.com/172.29.81.91:40012 11/10/25 12:14:06 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol 11/10/25 12:14:07 INFO mapred.ResourceMgrDelegate: Connected to ResourceManager at c0309.hal.cloudera.com/172.29.81.91:40012 11/10/25 12:14:08 INFO mapred.ClientCache: Connecting to HistoryServer at: c0309.hal.cloudera.com:10020 11/10/25 12:14:08 INFO ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC 11/10/25 12:14:08 INFO mapred.ClientCache: Connected to HistoryServer at: c0309.hal.cloudera.com:10020 11/10/25 12:14:08 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocol ... when submitting a job. This should be DEBUG level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3424) Some LinuxTaskController cleanup
[ https://issues.apache.org/jira/browse/MAPREDUCE-3424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins resolved MAPREDUCE-3424. Resolution: Fixed Hadoop Flags: Reviewed Thanks Todd. I've committed this. Some LinuxTaskController cleanup Key: MAPREDUCE-3424 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3424 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: tasktracker Affects Versions: 0.20.205.0 Reporter: Eli Collins Assignee: Eli Collins Priority: Minor Attachments: mapreduce-3424-1.patch, mapreduce-3424-2.patch, mapreduce-3424-3.patch MR-2415 had some tabs and weird indenting and spacing. Also would be more clear if LTC explicitly overrides createLogDir. Let's clean this up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3460) MR AM can hang if containers are allocated on a node blacklisted by the AM
MR AM can hang if containers are allocated on a node blacklisted by the AM -- Key: MAPREDUCE-3460 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3460 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Priority: Blocker When an AM is assigned a FAILED_MAP (priority = 5) container on a nodemanager which it has blacklisted - it tries to find a corresponding container request. This uses the hostname to find the matching container request - and can end up returning any of the ContainerRequests which may have requested a container on this node. This container request is cleaned to remove the bad node - and then added back to the RM 'ask' list. The AM cleans the 'ask' list after each heartbeat - The RM Allocator is still aware of the priority=5 container (in 'remoteRequestsTable') - but this never gets added back to the 'ask' set - which is what is sent to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2103) task-controller shouldn't require o-r permissions
[
https://issues.apache.org/jira/browse/MAPREDUCE-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155678#comment-13155678
]
Eli Collins commented on MAPREDUCE-2103:
Thanks Todd. I've committed this.
task-controller shouldn't require o-r permissions
-
Key: MAPREDUCE-2103
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2103
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: task-controller
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Trivial
Attachments: mapreduce-2103-20x.patch, mapreduce-2103.txt,
mapreduce-2103.txt
The task-controller currently checks that other users don't have read
permissions. This is unnecessary - we just need to make it's not executable.
The debian policy manual explains it well:
{quote}
Setuid and setgid executables should be mode 4755 or 2755 respectively, and
owned by the appropriate user or group. They should not be made unreadable
(modes like 4711 or 2711 or even 4111); doing so achieves no extra security,
because anyone can find the binary in the freely available Debian package; it
is merely inconvenient. For the same reason you should not restrict read or
execute permissions on non-set-id executables.
Some setuid programs need to be restricted to particular sets of users, using
file permissions. In this case they should be owned by the uid to which they
are set-id, and by the group which should be allowed to execute them. They
should have mode 4754; again there is no point in making them unreadable to
those users who must not be allowed to execute them.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3026) When user adds hierarchical queues to the cluster, mapred queue -list returns NULL Pointer Exception
[ https://issues.apache.org/jira/browse/MAPREDUCE-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated MAPREDUCE-3026: --- Description: When User adds the hierarchical queues, and try to see them from the command line using mapred queue -list It returns Null Pointer Exception. was: When User adds the hierarchical queues, and try to see them from the command line using mapred queue -list It returns Null Pointer Exception. Assignee: Mayank Bansal When user adds hierarchical queues to the cluster, mapred queue -list returns NULL Pointer Exception Key: MAPREDUCE-3026 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3026 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.22.0 Reporter: Mayank Bansal Assignee: Mayank Bansal Labels: patch Fix For: 0.22.0 Attachments: patch-22, patch-22.patch When User adds the hierarchical queues, and try to see them from the command line using mapred queue -list It returns Null Pointer Exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2939) Ant setup on hadoop7 jenkins host
[ https://issues.apache.org/jira/browse/MAPREDUCE-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated MAPREDUCE-2939: --- Assignee: Joep Rottinghuis Issue Type: Task (was: Bug) Ant setup on hadoop7 jenkins host - Key: MAPREDUCE-2939 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2939 Project: Hadoop Map/Reduce Issue Type: Task Components: build Affects Versions: 0.22.0 Environment: Jenkins: https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-22-branch/65/console Reporter: Joep Rottinghuis Assignee: Joep Rottinghuis Fix For: 0.22.0 From the build error it looks like a) ant is not set up on the machine b) $ANT_HOME point to the wrong spot -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1853) MultipleOutputs does not cache TaskAttemptContext
[
https://issues.apache.org/jira/browse/MAPREDUCE-1853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated MAPREDUCE-1853:
---
Component/s: task
Description:
In MultipleOutputs there is
{code}
private TaskAttemptContext getContext(String nameOutput) throws IOException {
// The following trick leverages the instantiation of a record writer via
// the job thus supporting arbitrary output formats.
Job job = new Job(context.getConfiguration());
job.setOutputFormatClass(getNamedOutputFormatClass(context, nameOutput));
job.setOutputKeyClass(getNamedOutputKeyClass(context, nameOutput));
job.setOutputValueClass(getNamedOutputValueClass(context, nameOutput));
TaskAttemptContext taskContext =
new TaskAttemptContextImpl(job.getConfiguration(),
context.getTaskAttemptID());
return taskContext;
}
{code}
so for every reduce call it creates a new Job instance ...which creates a new
LocalJobRunner.
That does not sound like a good idea.
You end up with a flood of jvm.JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
This should probably also be added to 0.22.
was:
In MultipleOutputs there is
[code]
private TaskAttemptContext getContext(String nameOutput) throws IOException {
// The following trick leverages the instantiation of a record writer via
// the job thus supporting arbitrary output formats.
Job job = new Job(context.getConfiguration());
job.setOutputFormatClass(getNamedOutputFormatClass(context, nameOutput));
job.setOutputKeyClass(getNamedOutputKeyClass(context, nameOutput));
job.setOutputValueClass(getNamedOutputValueClass(context, nameOutput));
TaskAttemptContext taskContext =
new TaskAttemptContextImpl(job.getConfiguration(),
context.getTaskAttemptID());
return taskContext;
}
[code]
so for every reduce call it creates a new Job instance ...which creates a new
LocalJobRunner.
That does not sound like a good idea.
You end up with a flood of jvm.JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
This should probably also be added to 0.22.
Assignee: Torsten Curdt
MultipleOutputs does not cache TaskAttemptContext
-
Key: MAPREDUCE-1853
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1853
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task
Affects Versions: 0.21.0, 0.22.0
Environment: OSX 10.6
java6
Reporter: Torsten Curdt
Assignee: Torsten Curdt
Priority: Critical
Fix For: 0.21.0, 0.22.0
Attachments: cache-task-attempts.diff
In MultipleOutputs there is
{code}
private TaskAttemptContext getContext(String nameOutput) throws IOException {
// The following trick leverages the instantiation of a record writer via
// the job thus supporting arbitrary output formats.
Job job = new Job(context.getConfiguration());
job.setOutputFormatClass(getNamedOutputFormatClass(context, nameOutput));
job.setOutputKeyClass(getNamedOutputKeyClass(context, nameOutput));
job.setOutputValueClass(getNamedOutputValueClass(context, nameOutput));
TaskAttemptContext taskContext =
new TaskAttemptContextImpl(job.getConfiguration(),
context.getTaskAttemptID());
return taskContext;
}
{code}
so for every reduce call it creates a new Job instance ...which creates a new
LocalJobRunner.
That does not sound like a good idea.
You end up with a flood of jvm.JvmMetrics: Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized
This should probably also be added to 0.22.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1517) streaming should support running on background
[ https://issues.apache.org/jira/browse/MAPREDUCE-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated MAPREDUCE-1517: --- Affects Version/s: 0.22.0 Assignee: Bochun Bai streaming should support running on background -- Key: MAPREDUCE-1517 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1517 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/streaming Affects Versions: 0.22.0 Reporter: Bochun Bai Assignee: Bochun Bai Fix For: 0.22.0 Attachments: contrib-streaming-background-with-test-with-license.patch, contrib-streaming-background-with-test-with-license2.patch, contrib-streaming-background-with-test.patch, contrib-streaming-background.patch, contrib-streaming-background.patch, hadoop-mapred-patch-logs.tar.gz StreamJob submit the job and use a while loop monitor the progress. I prefer it running on background. Just add at the end of command is a alternative solution, but it keeps a java process on client machine. When submit hundreds jobs at the same time, the client machine is overloaded. Adding a -background option to StreamJob, tell it only submit and don't monitor the progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1248) Redundant memory copying in StreamKeyValUtil
[ https://issues.apache.org/jira/browse/MAPREDUCE-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated MAPREDUCE-1248: --- Affects Version/s: 0.22.0 Assignee: Ruibang He Redundant memory copying in StreamKeyValUtil Key: MAPREDUCE-1248 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1248 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/streaming Affects Versions: 0.22.0 Reporter: Ruibang He Assignee: Ruibang He Priority: Minor Fix For: 0.22.0 Attachments: MAPREDUCE-1248-v1.0.patch I found that when MROutputThread collecting the output of Reducer, it calls StreamKeyValUtil.splitKeyVal() and two local byte-arrays are allocated there for each line of output. Later these two byte-arrays are passed to variable key and val. There are twice memory copying here, one is the System.arraycopy() method, the other is inside key.set() / val.set(). This causes double times of memory copying for the whole output (may lead to higher CPU consumption), and frequent temporay object allocation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1118) Capacity Scheduler scheduling information is hard to read / should be tabular format
[ https://issues.apache.org/jira/browse/MAPREDUCE-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated MAPREDUCE-1118: --- Component/s: contrib/capacity-sched Assignee: Milind Bhandarkar Capacity Scheduler scheduling information is hard to read / should be tabular format Key: MAPREDUCE-1118 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1118 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/capacity-sched Affects Versions: 0.20.2 Reporter: Allen Wittenauer Assignee: Milind Bhandarkar Fix For: 0.20.203.0, 0.22.0 Attachments: MR-1118-22.patch, mapred-1118-1.patch, mapred-1118-2.patch, mapred-1118-3.patch, mapred-1118.20S.patch, mapred-1118.patch The scheduling information provided by the capacity scheduler is extremely hard to read on the job tracker web page. Instead of just flat text, it should be presenting the information in a tabular format, similar to what the fair share scheduler provides. This makes it much easier to compare what different queues are doing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-1118) Capacity Scheduler scheduling information is hard to read / should be tabular format
[ https://issues.apache.org/jira/browse/MAPREDUCE-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-1118: Assignee: Krishna Ramachandran (was: Milind Bhandarkar) Capacity Scheduler scheduling information is hard to read / should be tabular format Key: MAPREDUCE-1118 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1118 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/capacity-sched Affects Versions: 0.20.2 Reporter: Allen Wittenauer Assignee: Krishna Ramachandran Fix For: 0.20.203.0, 0.22.0 Attachments: MR-1118-22.patch, mapred-1118-1.patch, mapred-1118-2.patch, mapred-1118-3.patch, mapred-1118.20S.patch, mapred-1118.patch The scheduling information provided by the capacity scheduler is extremely hard to read on the job tracker web page. Instead of just flat text, it should be presenting the information in a tabular format, similar to what the fair share scheduler provides. This makes it much easier to compare what different queues are doing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3461) Move hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site to hadoop-site
Move hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site to hadoop-site - Key: MAPREDUCE-3461 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3461 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 0.23.0 Reporter: Arun C Murthy Currently hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site has both HDFS and MR docs, we should move it to top-level hadoop-site. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
