[jira] [Commented] (MAPREDUCE-2798) Make the ResourceManager, NodeManager and HistoryServer run from Eclipse.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165117#comment-13165117 ] Harsh J commented on MAPREDUCE-2798: We could either provide README-sorta instructions on setting up the run configuration, or like has been done before, we can ship run configuration scripts that launch these java programs with the right environment. I'm not sure on how to do the latter, but running daemons from Eclipse is as simple as running any other java program - so long as you get the classpaths and env. vars. right. Make the ResourceManager, NodeManager and HistoryServer run from Eclipse. - Key: MAPREDUCE-2798 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2798 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Affects Versions: 0.23.0 Reporter: praveen sripati Priority: Minor Fix For: 0.24.0 Make the ResourceManager, NodeManager and HistoryServer run from Eclipse, so that it would be easy for development. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3327) RM web ui scheduler link doesn't show correct max value for queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165227#comment-13165227 ] Thomas Graves commented on MAPREDUCE-3327: -- Anupam, I think you need to handle the case where maximumCapacity isn't initialized (ie is = to CapacitySchedulerConfiguration.UNDEFINED). The line you added to divide maximumCapacity by 100 will cause issues if its UNDEFINED. RM web ui scheduler link doesn't show correct max value for queues -- Key: MAPREDUCE-3327 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3327 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Anupam Seth Priority: Critical Attachments: MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch Configure a cluster to use the capacity scheduler and then specifying a maximum-capacity 100% for a queue. If you go to the RM Web UI and hover over the queue, it always shows the max at 100%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3519) Deadlock in LocalDirsHandlerService and ShuffleHandler
[ https://issues.apache.org/jira/browse/MAPREDUCE-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165258#comment-13165258 ] Robert Joseph Evans commented on MAPREDUCE-3519: We have seen one other deadlock in the past when trying to lock the Configuration object. The other one I am thinking of was specifically withing Configuration itself when trying to write out the configuration to a stream and the reader of the stream in the same process was trying to read in a configuration value. That was also fixed by making sure that the configuration objects were separate instances. In yarn with the message passing framework and how initialization happens automatically we are moving much more towards having a single global Configuration object shared by all parts of the system. I think it is time for us to look at moving to read/write locks on the Configuration class. This will not solve all cases of deadlocks, because we could still have a writer holding the lock and someone else trying to read or write holding a different lock. But both of the situations I have seen so far are two readers deadlocked on reads. No writes at all. With read write locks the reads would both be able to finish successfully without blocking. This would also have the advantage of a potential performance boost for anyone using Configuration. It would break backwards compatibility for anyone who is doing what we see here and synchronizing on configuration so it would have to be part of the release notes and we would have to inform downstream projects. But I think that the benefits outweigh the drawbacks. Especially for YARN where we are already changing a lot. Deadlock in LocalDirsHandlerService and ShuffleHandler -- Key: MAPREDUCE-3519 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3519 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.1, 0.24.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.23.1 Attachments: 3519.patch, deadlock.txt MAPREDUCE-3121 cloned Configuration object in LocalDirsHandlerService.init() to avoid others to access that configuration object. But since it is set in the base class of LocalDirsHandlerService using super.init(conf), it is reflected and is accessible to some other services. This is causing a deadlock when accessing this configuration object from LocalDirsHandlerService and ShuffleHandler along with AllocatorPerContext object. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3519) Deadlock in LocalDirsHandlerService and ShuffleHandler
[ https://issues.apache.org/jira/browse/MAPREDUCE-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165293#comment-13165293 ] Ravi Gummadi commented on MAPREDUCE-3519: - As reads of Configuration object happen often and writes to Configuration object don't happen often, having separate locks for read and write should improve things a lot in terms of performance. Deadlock in LocalDirsHandlerService and ShuffleHandler -- Key: MAPREDUCE-3519 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3519 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.1, 0.24.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.23.1 Attachments: 3519.patch, deadlock.txt MAPREDUCE-3121 cloned Configuration object in LocalDirsHandlerService.init() to avoid others to access that configuration object. But since it is set in the base class of LocalDirsHandlerService using super.init(conf), it is reflected and is accessible to some other services. This is causing a deadlock when accessing this configuration object from LocalDirsHandlerService and ShuffleHandler along with AllocatorPerContext object. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3520) Allow for metrics to be exchanged between mapps and reduces
Allow for metrics to be exchanged between mapps and reduces --- Key: MAPREDUCE-3520 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3520 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Affects Versions: 0.24.0 Reporter: Robert Joseph Evans Some people on the mailing lists have requested having read access in reducers to counters that were set by mappers. I propose that we provide a new interface for doing this. The counter data in addition to being set periodically to the Application Master would be put into the intermediate output of the mapper for each reducer and be specially marked as such. When the reducers fetch that data they can combine it together and provide a read only interface to it. This allows users to potentially optimize their code on the reducer side if they know some special metadata before hand. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165307#comment-13165307 ] Jonathan Eagles commented on MAPREDUCE-3518: Small change to avoid NPE. No test required. mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Attachments: MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3518: --- Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1) Status: Open (was: Patch Available) canceling patch to address ant builds breaking. mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Attachments: MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3327) RM web ui scheduler link doesn't show correct max value for queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anupam Seth updated MAPREDUCE-3327: --- Attachment: MAPREDUCE-3327-branch_0_23.patch Thanks Tom and Mahadev. I have fixed the test failures, am re-submitting patch for review. RM web ui scheduler link doesn't show correct max value for queues -- Key: MAPREDUCE-3327 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3327 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Anupam Seth Priority: Critical Attachments: MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch Configure a cluster to use the capacity scheduler and then specifying a maximum-capacity 100% for a queue. If you go to the RM Web UI and hover over the queue, it always shows the max at 100%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3327) RM web ui scheduler link doesn't show correct max value for queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anupam Seth updated MAPREDUCE-3327: --- Tags: mrv2 Status: Patch Available (was: Open) RM web ui scheduler link doesn't show correct max value for queues -- Key: MAPREDUCE-3327 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3327 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Anupam Seth Priority: Critical Attachments: MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch Configure a cluster to use the capacity scheduler and then specifying a maximum-capacity 100% for a queue. If you go to the RM Web UI and hover over the queue, it always shows the max at 100%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3518: --- Attachment: MAPREDUCE-3518.patch mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Attachments: MAPREDUCE-3518.patch, MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165402#comment-13165402 ] Jonathan Eagles commented on MAPREDUCE-3518: Latest test fixes unit tests mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Attachments: MAPREDUCE-3518.patch, MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3518: --- Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1) Status: Patch Available (was: Open) mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Attachments: MAPREDUCE-3518.patch, MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3366) Mapreduce component should use consistent directory structure layout as HDFS/common
[
https://issues.apache.org/jira/browse/MAPREDUCE-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165415#comment-13165415
]
Tom White commented on MAPREDUCE-3366:
--
Does this subsume MAPREDUCE-3416?
Mapreduce component should use consistent directory structure layout as
HDFS/common
---
Key: MAPREDUCE-3366
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3366
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Environment: Java, Maven
Reporter: Eric Yang
Assignee: Eric Yang
Attachments: MAPREDUCE-3366-1-trunk.patch,
MAPREDUCE-3366-2-trunk.patch, MAPREDUCE-3366-3-trunk.patch,
MAPREDUCE-3366.patch, svn-rename-MAPREDUCE-3366-3.sh
Directory structure for MRv2 layout looks like:
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/conf
/lib
/modules
{noformat}
The directory structure layout should be updated to reflect changes
implemented in HADOOP-6255.
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/etc/hadoop
/lib
/libexec
/sbin
/share/hadoop
/share/hadoop/lib
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3327) RM web ui scheduler link doesn't show correct max value for queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165426#comment-13165426 ] Hadoop QA commented on MAPREDUCE-3327: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506633/MAPREDUCE-3327-branch_0_23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1411//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1411//console This message is automatically generated. RM web ui scheduler link doesn't show correct max value for queues -- Key: MAPREDUCE-3327 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3327 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Anupam Seth Priority: Critical Attachments: MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch Configure a cluster to use the capacity scheduler and then specifying a maximum-capacity 100% for a queue. If you go to the RM Web UI and hover over the queue, it always shows the max at 100%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165437#comment-13165437 ] Hadoop QA commented on MAPREDUCE-3518: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506635/MAPREDUCE-3518.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1412//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1412//console This message is automatically generated. mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Attachments: MAPREDUCE-3518.patch, MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3511) Counters occupy a good part of AM heap
[ https://issues.apache.org/jira/browse/MAPREDUCE-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165485#comment-13165485 ] Siddharth Seth commented on MAPREDUCE-3511: --- Good point. Calls to getTaskReports (all maps / all reduces) would cause AM heap usage to go up significantly. Don't think there's any call which asks for all task counters. Changing to mrv1 would also impact the UI components. Interning strings is still worth exploring though - not sure how well it will work with protocol buffers. Alternately try optimizing mrv2 counters or somehow send mrv1 counters over rpc. Counters occupy a good part of AM heap -- Key: MAPREDUCE-3511 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3511 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Devaraj K Per task counters seem to be occupying a good part of an AMs heap. Looks like more than 50% of what's used by a TaskAttemptImpl object. This could be optimized by interning strings or possibly using mrv1 counters which are optimized. Currently counters are converted from mrv1 to mrv2 format for in memory storage. The conversion could be delayed till it's actually required for RPC transfers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3339) Job is getting hanged indefinitely,if the child processes are killed on the NM. KILL_CONTAINER eventtype is continuosly sent to the containers that are not existing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3339: -- Status: Open (was: Patch Available) Job is getting hanged indefinitely,if the child processes are killed on the NM. KILL_CONTAINER eventtype is continuosly sent to the containers that are not existing - Key: MAPREDUCE-3339 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3339 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramgopal N Assignee: Siddharth Seth Priority: Blocker Attachments: MR3339_v1.txt I have only one NM running. I have submitted a job and all the child processes on the NM got killed continuosly.This made the Job to hang indefinitely. In the NM logs it is logging WARN message :org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Event EventType: KILL_CONTAINER sent to absent container container_1320301910500_0004_01_001359 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3417) job access controls not working app master and job history UI's
[ https://issues.apache.org/jira/browse/MAPREDUCE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anupam Seth reassigned MAPREDUCE-3417: -- Assignee: Anupam Seth job access controls not working app master and job history UI's --- Key: MAPREDUCE-3417 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3417 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Anupam Seth Priority: Critical tested with security on, no filters defined for httpserver, job acls set so that only I could view/modify the job. Then went to the web ui to app master and job history server and both allowed me to view the job details. The webui shows the user webuser. The RM properly rejected my request although it was using user Dr.Who. The exception shown in the log is: 11/11/16 18:58:53 INFO mapred.JobACLsManager: job checkAccess user is: webuser 11/11/16 18:58:53 WARN security.ShellBasedUnixGroupsMapping: got exception trying to get groups for user webuser org.apache.hadoop.util.Shell$ExitCodeException: id: webuser: No such user at org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at org.apache.hadoop.util.Shell.run(Shell.java:188) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at org.apache.hadoop.util.Shell.execCommand(Shell.java:467) at org.apache.hadoop.util.Shell.execCommand(Shell.java:450) at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:86) at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:55) at org.apache.hadoop.security.Groups.getGroups(Groups.java:88) at org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1043) at org.apache.hadoop.security.authorize.AccessControlList.isUserAllowed(AccessControlList.java:221) at org.apache.hadoop.mapred.JobACLsManager.checkAccess(JobACLsManager.java:103) at org.apache.hadoop.mapreduce.v2.hs.CompletedJob.checkAccess(CompletedJob.java:325) at org.apache.hadoop.mapreduce.v2.app.webapp.AppController.checkAccess(AppController.java:292) at org.apache.hadoop.mapreduce.v2.app.webapp.AppController.requireJob(AppController.java:313) at org.apache.hadoop.mapreduce.v2.app.webapp.AppController.job(AppController.java:97) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3521) Hadoop Streaming ignores unknown commands
Hadoop Streaming ignores unknown commands
-
Key: MAPREDUCE-3521
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3521
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Priority: Minor
The hadoop streaming command will ignore any command line arguments to it.
{code}
hadoop jar streaming.jar -input input -output output -mapper cat -reducer cat
ThisIsABadArgument
{code}
Works just fine. This can mask issues where quotes were mistakenly missed like
{code}
hadoop jar streaming.jar -input input -output output -mapper xargs cat -reducer
cat -archive someArchive.tgz
{code}
Streaming should fail if it encounters an unexpected command line parameter
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3388) Streaming task with special char gets wrong output
[
https://issues.apache.org/jira/browse/MAPREDUCE-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans resolved MAPREDUCE-3388.
Resolution: Invalid
As it turns out this was an issue with our tests which was masked by a small
bug in streaming. I have filed MAPREDUCE-3521 to address this issue in
streaming.
Streaming task with special char gets wrong output
---
Key: MAPREDUCE-3388
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3388
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Priority: Critical
In 0.20.204:
{noformat}
hadoop jar $HADOOP_HOME/hadoop-streaming.jar '-Dmapreduce.job.acl-view-job=*'
-files 'file:///tmp/InputDir#testlink!@$*()-_+=' -input input.txt -mapper
'xargs cat' -reducer cat -output output -jobconf
mapred.job.name=streamingTest-1000 -jobconf 'mapreduce.job.acl-view-job=*'
{noformat}
Output:
{noformat}
hadoop fs -cat output/*
and-so-is-the-c#
c++-also-supports-object-oriented-concepts
hadoop-apache-org-core
hadoop-streaming
hod-is-the-part-of-hadoop
it-is-residing-on-apache-under-repos-asf
java-an object-oriented-language
smalltalk-is-also-object-oriented-language
streaming-is-also-part-of-it
{noformat}
For 0.23:
{noformat}
hadoop jar $HADOOP_MAPRED_HOME/hadoop-streaming.jar
-Dmapreduce.job.acl-view-job=* -files
'file:///tmp/InputDir#testlink!@$*()-_+=' -input input.txt -mapper 'xargs
cat' -reducer cat -output output -jobconf mapred.job.name=streamingTest-1000
-jobconf 'mapreduce.job.acl-view-job=*'
{noformat}
Output:
{noformat}
testlink!@$*()-_+=/input1.txt testlink!@$*()-_+=/input2.txt
{noformat}
The contents of input.txt are as follows:
{noformat}
hadoop fs -cat Streaming/streaming-1000/input.txt
testlink!@$*()-_+=/input1.txt
testlink!@$*()-_+=/input2.txt
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3521) Hadoop Streaming ignores unknown parameters
[
https://issues.apache.org/jira/browse/MAPREDUCE-3521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-3521:
---
Summary: Hadoop Streaming ignores unknown parameters (was: Hadoop
Streaming ignores unknown commands)
Hadoop Streaming ignores unknown parameters
---
Key: MAPREDUCE-3521
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3521
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Priority: Minor
The hadoop streaming command will ignore any command line arguments to it.
{code}
hadoop jar streaming.jar -input input -output output -mapper cat -reducer cat
ThisIsABadArgument
{code}
Works just fine. This can mask issues where quotes were mistakenly missed
like
{code}
hadoop jar streaming.jar -input input -output output -mapper xargs cat
-reducer cat -archive someArchive.tgz
{code}
Streaming should fail if it encounters an unexpected command line parameter
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3522) Capacity Scheduler ACLs not inherited by default
Capacity Scheduler ACLs not inherited by default Key: MAPREDUCE-3522 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3522 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Hierarchical Queues do not inherit parent ACLs correctly by default. Instead, if no value is specified for submit or administer acls, then all access is granted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3523) Exclude mrapp-generated-classpath from published jars
Exclude mrapp-generated-classpath from published jars - Key: MAPREDUCE-3523 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3523 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Priority: Critical Fix For: 0.23.1, 0.24.0 Downstream projects will be generating their own classpath file to allow yarn to launch containers correctly in a unit-test env which may potentially create a clash if 2 versions are present. The current code in MRApps assumes that the generated classpath is always present which may need to be addressed if this file is not present when deployed on a cluster. An option for a fix could be to pass the location of the classpath file via the MiniYarnCluster constructor and propagate that to MRApps via some form of configuration value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3522) Capacity Scheduler ACLs not inherited by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3522: --- Attachment: (was: MAPREDUCE-3522-v2.patch) Capacity Scheduler ACLs not inherited by default Key: MAPREDUCE-3522 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3522 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Hierarchical Queues do not inherit parent ACLs correctly by default. Instead, if no value is specified for submit or administer acls, then all access is granted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3522) Capacity Scheduler ACLs not inherited by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3522: --- Attachment: MAPREDUCE-3522-v2.patch Patch is based off of MAPREDUCE-3510 and MAPREDUCE-3518 Capacity Scheduler ACLs not inherited by default Key: MAPREDUCE-3522 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3522 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Hierarchical Queues do not inherit parent ACLs correctly by default. Instead, if no value is specified for submit or administer acls, then all access is granted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3522) Capacity Scheduler ACLs not inherited by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3522: --- Attachment: MAPREDUCE-3522.patch Capacity Scheduler ACLs not inherited by default Key: MAPREDUCE-3522 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3522 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3522.patch Hierarchical Queues do not inherit parent ACLs correctly by default. Instead, if no value is specified for submit or administer acls, then all access is granted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3522) Capacity Scheduler ACLs not inherited by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3522: --- Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1) Status: Patch Available (was: Open) Capacity Scheduler ACLs not inherited by default Key: MAPREDUCE-3522 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3522 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3522.patch Hierarchical Queues do not inherit parent ACLs correctly by default. Instead, if no value is specified for submit or administer acls, then all access is granted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3522) Capacity Scheduler ACLs not inherited by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165712#comment-13165712 ] Hadoop QA commented on MAPREDUCE-3522: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506684/MAPREDUCE-3522.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1413//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1413//console This message is automatically generated. Capacity Scheduler ACLs not inherited by default Key: MAPREDUCE-3522 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3522 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3522.patch Hierarchical Queues do not inherit parent ACLs correctly by default. Instead, if no value is specified for submit or administer acls, then all access is granted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165725#comment-13165725 ] Mahadev konar commented on MAPREDUCE-3518: -- Jon, Do you think we can add a small test case with this? I think itll be easy to write one and we should probably add one testcase for this. What do you think? mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Attachments: MAPREDUCE-3518.patch, MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3327) RM web ui scheduler link doesn't show correct max value for queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3327: - Resolution: Fixed Fix Version/s: 0.23.1 Status: Resolved (was: Patch Available) Just committed this. Thanks Anupam! RM web ui scheduler link doesn't show correct max value for queues -- Key: MAPREDUCE-3327 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3327 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Anupam Seth Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch Configure a cluster to use the capacity scheduler and then specifying a maximum-capacity 100% for a queue. If you go to the RM Web UI and hover over the queue, it always shows the max at 100%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3327) RM web ui scheduler link doesn't show correct max value for queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165745#comment-13165745 ] Hudson commented on MAPREDUCE-3327: --- Integrated in Hadoop-Common-trunk-Commit #1385 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1385/]) MAPREDUCE-3327. RM web ui scheduler link doesn't show correct max value for queues (Anupam Seth via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1212212 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java RM web ui scheduler link doesn't show correct max value for queues -- Key: MAPREDUCE-3327 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3327 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Anupam Seth Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch Configure a cluster to use the capacity scheduler and then specifying a maximum-capacity 100% for a queue. If you go to the RM Web UI and hover over the queue, it always shows the max at 100%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3327) RM web ui scheduler link doesn't show correct max value for queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165746#comment-13165746 ] Hudson commented on MAPREDUCE-3327: --- Integrated in Hadoop-Hdfs-trunk-Commit #1459 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1459/]) MAPREDUCE-3327. RM web ui scheduler link doesn't show correct max value for queues (Anupam Seth via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1212212 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java RM web ui scheduler link doesn't show correct max value for queues -- Key: MAPREDUCE-3327 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3327 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Anupam Seth Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch Configure a cluster to use the capacity scheduler and then specifying a maximum-capacity 100% for a queue. If you go to the RM Web UI and hover over the queue, it always shows the max at 100%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3327) RM web ui scheduler link doesn't show correct max value for queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165752#comment-13165752 ] Hudson commented on MAPREDUCE-3327: --- Integrated in Hadoop-Hdfs-0.23-Commit #251 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/251/]) MAPREDUCE-3327. RM web ui scheduler link doesn't show correct max value for queues (Anupam Seth via mahadev) - Merging r1212212 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1212213 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java RM web ui scheduler link doesn't show correct max value for queues -- Key: MAPREDUCE-3327 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3327 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Anupam Seth Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch Configure a cluster to use the capacity scheduler and then specifying a maximum-capacity 100% for a queue. If you go to the RM Web UI and hover over the queue, it always shows the max at 100%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3478) Cannot build against ZooKeeper 3.4.0
[
https://issues.apache.org/jira/browse/MAPREDUCE-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165754#comment-13165754
]
Mahadev konar commented on MAPREDUCE-3478:
--
@Tom,
Looks like 3.4.0 has data consistency issues. We'll be doing a 3.4.1 soon (as
in should be available sometime next week). Want to wait for that?
Cannot build against ZooKeeper 3.4.0
Key: MAPREDUCE-3478
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3478
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Andrew Bayer
Priority: Minor
Labels: bigtop
Attachments: MAPREDUCE-3478.patch
I tried to see if one could build Hadoop 0.23.0 against ZooKeeper 3.4.0,
rather than 3.3.1 (3.3.3 does work, fwiw) and hit compilation errors:
{quote}
[INFO]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.3.2:testCompile
(default-testCompile) on project hadoop-yarn-server-common: Compilation
failure: Compilation failure:
[ERROR]
/Volumes/EssEssDee/abayer/src/asf-git/hadoop-common/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/lib/TestZKClient.java:[48,25]
cannot find symbol
[ERROR] symbol : class Factory
[ERROR] location: class org.apache.zookeeper.server.NIOServerCnxn
[ERROR]
/Volumes/EssEssDee/abayer/src/asf-git/hadoop-common/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/lib/TestZKClient.java:[150,33]
cannot find symbol
[ERROR] symbol : class Factory
[ERROR] location: class org.apache.zookeeper.server.NIOServerCnxn
[ERROR] - [Help 1]
{quote}
Presumably, Yarn needs to build against newer ZK releases eventually, hence
this bug. =)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3327) RM web ui scheduler link doesn't show correct max value for queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165756#comment-13165756 ] Hudson commented on MAPREDUCE-3327: --- Integrated in Hadoop-Common-0.23-Commit #261 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/261/]) MAPREDUCE-3327. RM web ui scheduler link doesn't show correct max value for queues (Anupam Seth via mahadev) - Merging r1212212 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1212213 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java RM web ui scheduler link doesn't show correct max value for queues -- Key: MAPREDUCE-3327 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3327 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Anupam Seth Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch Configure a cluster to use the capacity scheduler and then specifying a maximum-capacity 100% for a queue. If you go to the RM Web UI and hover over the queue, it always shows the max at 100%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3327) RM web ui scheduler link doesn't show correct max value for queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165757#comment-13165757 ] Hudson commented on MAPREDUCE-3327: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1409 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1409/]) MAPREDUCE-3327. RM web ui scheduler link doesn't show correct max value for queues (Anupam Seth via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1212212 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java RM web ui scheduler link doesn't show correct max value for queues -- Key: MAPREDUCE-3327 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3327 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Anupam Seth Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch Configure a cluster to use the capacity scheduler and then specifying a maximum-capacity 100% for a queue. If you go to the RM Web UI and hover over the queue, it always shows the max at 100%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3513) Capacity Scheduler web UI has a spelling mistake for Memory.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3513: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this. Thanks chackaravarthy! Capacity Scheduler web UI has a spelling mistake for Memory. Key: MAPREDUCE-3513 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3513 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: chackaravarthy Priority: Trivial Fix For: 0.23.1 Attachments: mapreduce-3513.patch, mapreduce-3513.patch The web page for capacity scheduler has a column named Memopry Total, a spelling mistake which needs to be fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3513) Capacity Scheduler web UI has a spelling mistake for Memory.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165761#comment-13165761 ] Hudson commented on MAPREDUCE-3513: --- Integrated in Hadoop-Hdfs-trunk-Commit #1460 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1460/]) MAPREDUCE-3513. Capacity Scheduler web UI has a spelling mistake for Memory. (chackaravarthy via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1212215 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NodePage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java Capacity Scheduler web UI has a spelling mistake for Memory. Key: MAPREDUCE-3513 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3513 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: chackaravarthy Priority: Trivial Fix For: 0.23.1 Attachments: mapreduce-3513.patch, mapreduce-3513.patch The web page for capacity scheduler has a column named Memopry Total, a spelling mistake which needs to be fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3513) Capacity Scheduler web UI has a spelling mistake for Memory.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165764#comment-13165764 ] Hudson commented on MAPREDUCE-3513: --- Integrated in Hadoop-Common-trunk-Commit #1386 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1386/]) MAPREDUCE-3513. Capacity Scheduler web UI has a spelling mistake for Memory. (chackaravarthy via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1212215 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NodePage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java Capacity Scheduler web UI has a spelling mistake for Memory. Key: MAPREDUCE-3513 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3513 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: chackaravarthy Priority: Trivial Fix For: 0.23.1 Attachments: mapreduce-3513.patch, mapreduce-3513.patch The web page for capacity scheduler has a column named Memopry Total, a spelling mistake which needs to be fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2863) Support web-services for RM NM
[
https://issues.apache.org/jira/browse/MAPREDUCE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165768#comment-13165768
]
Hitesh Shah commented on MAPREDUCE-2863:
For the most part looks good. Have not yet had a chance to run this on a secure
cluster. Had run the previous patch against one so do not expect any issues
with this. Will update jira again once I have done that.
Minor comments:
- inconsistency in use of MediaType.APPLICATION_JSON v/s application/json in
unit tests. Could switch all tests to use the defined constant.
- missing test for output when media type is not specified ( or both
specified ) to test default json behavior.
- no tests for xml. Could be done more thoroughly in an additional jira but
would help to have atleast one or 2 tests that accept xml only.
For further ( later ) investigation:
There might be some xss issues which may need to be looked at some point. For
example:
{code}throw new NotFoundException(app with id: + appId + not
found);{code}
Printing back user input as is ( without escaping ) could effectively execute
malicious javascript on a browser. This however depends on what form of input
parsing is done by the jaxb framework before the parameter gets to the actual
ws handling code.
Support web-services for RM NM
Key: MAPREDUCE-2863
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2863
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2, nodemanager, resourcemanager
Affects Versions: 0.23.0
Reporter: Arun C Murthy
Assignee: Thomas Graves
Priority: Blocker
Attachments: MAPREDUCE-2863.patch, MAPREDUCE-2863.patch,
MAPREDUCE-2863.patch, MAPREDUCE-2863.patch, MAPREDUCE-2863.patch,
MAPREDUCE-2863.patch, MAPREDUCE-2863.patch, MAPREDUCE-2863.patch,
amoutput.txt, appoutput.txt, hsoutput.txt, nmoutput.txt, nmoutput.txt,
nmoutput.txt, rmoutput.txt, rmoutput.txt, rmoutput.txt
It will be very useful for RM and NM to support web-services to export
json/xml.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3513) Capacity Scheduler web UI has a spelling mistake for Memory.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165776#comment-13165776 ] Hudson commented on MAPREDUCE-3513: --- Integrated in Hadoop-Hdfs-0.23-Commit #252 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/252/]) MAPREDUCE-3513. Capacity Scheduler web UI has a spelling mistake for Memory. (chackaravarthy via mahadev) - Merging r1212215 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1212216 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NodePage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java Capacity Scheduler web UI has a spelling mistake for Memory. Key: MAPREDUCE-3513 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3513 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: chackaravarthy Priority: Trivial Fix For: 0.23.1 Attachments: mapreduce-3513.patch, mapreduce-3513.patch The web page for capacity scheduler has a column named Memopry Total, a spelling mistake which needs to be fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3510) Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls
[ https://issues.apache.org/jira/browse/MAPREDUCE-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3510: - Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1) Status: Open (was: Patch Available) Jon, The patch no longer applies. Can you please update the patch? Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls - Key: MAPREDUCE-3510 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3510 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/capacity-sched, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3510.patch, MAPREDUCE-3510.patch mapred queue -showacls does not show inherited acls -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3513) Capacity Scheduler web UI has a spelling mistake for Memory.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165781#comment-13165781 ] Hudson commented on MAPREDUCE-3513: --- Integrated in Hadoop-Common-0.23-Commit #262 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/262/]) MAPREDUCE-3513. Capacity Scheduler web UI has a spelling mistake for Memory. (chackaravarthy via mahadev) - Merging r1212215 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1212216 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NodePage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java Capacity Scheduler web UI has a spelling mistake for Memory. Key: MAPREDUCE-3513 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3513 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: chackaravarthy Priority: Trivial Fix For: 0.23.1 Attachments: mapreduce-3513.patch, mapreduce-3513.patch The web page for capacity scheduler has a column named Memopry Total, a spelling mistake which needs to be fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3327) RM web ui scheduler link doesn't show correct max value for queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165784#comment-13165784 ] Hudson commented on MAPREDUCE-3327: --- Integrated in Hadoop-Mapreduce-0.23-Commit #271 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/271/]) MAPREDUCE-3327. RM web ui scheduler link doesn't show correct max value for queues (Anupam Seth via mahadev) - Merging r1212212 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1212213 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java RM web ui scheduler link doesn't show correct max value for queues -- Key: MAPREDUCE-3327 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3327 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Anupam Seth Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch, MAPREDUCE-3327-branch_0_23.patch Configure a cluster to use the capacity scheduler and then specifying a maximum-capacity 100% for a queue. If you go to the RM Web UI and hover over the queue, it always shows the max at 100%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3251) Network ACLs can prevent some clients to talk to MR ApplicationMaster
[
https://issues.apache.org/jira/browse/MAPREDUCE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165787#comment-13165787
]
Mahadev konar commented on MAPREDUCE-3251:
--
@Anupam,
I am not sure I understand the patch correctly:
{noformat}
logApplicationReportInfo(application);
+ LOG.info(Network ACL closed to AM for job + jobId
+ + . Redirecting to job history server.);
+ return checkAndGetHSProxy(null, JobState.RUNNING);
{noformat}
YOu are returning a jobhistory proxy when the switch is turned on? Why is that?
Also, one more thing, we should document the property in yarn-default.xml with
default as false.
Network ACLs can prevent some clients to talk to MR ApplicationMaster
-
Key: MAPREDUCE-3251
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3251
Project: Hadoop Map/Reduce
Issue Type: Task
Components: mrv2
Affects Versions: 0.23.0
Reporter: Anupam Seth
Assignee: Anupam Seth
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3251-branch_0_23.patch,
MAPREDUCE-3251-branch_0_23.patch, MAPREDUCE-3251_branch-0_23_preliminary.txt
In 0.20.xxx, the JobClient while polling goes to JT to get the job status.
With YARN, AM can be launched on any port and the client will have to have
ACL open to that port to talk to AM and get the job status. When the client
is within the same grid network access to AM is not a problem. But some
applications may have one installation per set of clusters and may launch
jobs even across such sets (on job trackers in another set of clusters). For
that to work only the JT port needs to be open currently. In case of YARN,
all ports will have to be opened up for things to work. That would be a
security no-no.
There are two possible solutions:
1) Make the job client only talk to RM (as an option) to get the job
status.
2) Limit the range of ports AM can listen on.
Option 2) may not be favorable as there is no direct OS API to find a free
port.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3251) Network ACLs can prevent some clients to talk to MR ApplicationMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3251: - Status: Open (was: Patch Available) Cancelling the patch to update the config and description. Network ACLs can prevent some clients to talk to MR ApplicationMaster - Key: MAPREDUCE-3251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3251 Project: Hadoop Map/Reduce Issue Type: Task Components: mrv2 Affects Versions: 0.23.0 Reporter: Anupam Seth Assignee: Anupam Seth Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3251-branch_0_23.patch, MAPREDUCE-3251-branch_0_23.patch, MAPREDUCE-3251_branch-0_23_preliminary.txt In 0.20.xxx, the JobClient while polling goes to JT to get the job status. With YARN, AM can be launched on any port and the client will have to have ACL open to that port to talk to AM and get the job status. When the client is within the same grid network access to AM is not a problem. But some applications may have one installation per set of clusters and may launch jobs even across such sets (on job trackers in another set of clusters). For that to work only the JT port needs to be open currently. In case of YARN, all ports will have to be opened up for things to work. That would be a security no-no. There are two possible solutions: 1) Make the job client only talk to RM (as an option) to get the job status. 2) Limit the range of ports AM can listen on. Option 2) may not be favorable as there is no direct OS API to find a free port. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3513) Capacity Scheduler web UI has a spelling mistake for Memory.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165790#comment-13165790 ] Hudson commented on MAPREDUCE-3513: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1410 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1410/]) MAPREDUCE-3513. Capacity Scheduler web UI has a spelling mistake for Memory. (chackaravarthy via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1212215 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NodePage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java Capacity Scheduler web UI has a spelling mistake for Memory. Key: MAPREDUCE-3513 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3513 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: chackaravarthy Priority: Trivial Fix For: 0.23.1 Attachments: mapreduce-3513.patch, mapreduce-3513.patch The web page for capacity scheduler has a column named Memopry Total, a spelling mistake which needs to be fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3513) Capacity Scheduler web UI has a spelling mistake for Memory.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165806#comment-13165806 ] Hudson commented on MAPREDUCE-3513: --- Integrated in Hadoop-Mapreduce-0.23-Commit #272 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/272/]) MAPREDUCE-3513. Capacity Scheduler web UI has a spelling mistake for Memory. (chackaravarthy via mahadev) - Merging r1212215 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1212216 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NodePage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java Capacity Scheduler web UI has a spelling mistake for Memory. Key: MAPREDUCE-3513 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3513 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: chackaravarthy Priority: Trivial Fix For: 0.23.1 Attachments: mapreduce-3513.patch, mapreduce-3513.patch The web page for capacity scheduler has a column named Memopry Total, a spelling mistake which needs to be fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3478) Cannot build against ZooKeeper 3.4.0
[
https://issues.apache.org/jira/browse/MAPREDUCE-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165840#comment-13165840
]
Tom White commented on MAPREDUCE-3478:
--
Certainly - I was thinking the same thing.
Cannot build against ZooKeeper 3.4.0
Key: MAPREDUCE-3478
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3478
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Andrew Bayer
Priority: Minor
Labels: bigtop
Attachments: MAPREDUCE-3478.patch
I tried to see if one could build Hadoop 0.23.0 against ZooKeeper 3.4.0,
rather than 3.3.1 (3.3.3 does work, fwiw) and hit compilation errors:
{quote}
[INFO]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.3.2:testCompile
(default-testCompile) on project hadoop-yarn-server-common: Compilation
failure: Compilation failure:
[ERROR]
/Volumes/EssEssDee/abayer/src/asf-git/hadoop-common/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/lib/TestZKClient.java:[48,25]
cannot find symbol
[ERROR] symbol : class Factory
[ERROR] location: class org.apache.zookeeper.server.NIOServerCnxn
[ERROR]
/Volumes/EssEssDee/abayer/src/asf-git/hadoop-common/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/lib/TestZKClient.java:[150,33]
cannot find symbol
[ERROR] symbol : class Factory
[ERROR] location: class org.apache.zookeeper.server.NIOServerCnxn
[ERROR] - [Help 1]
{quote}
Presumably, Yarn needs to build against newer ZK releases eventually, hence
this bug. =)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3349) No rack-name logged in JobHistory for unsuccessful tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-3349: -- Attachment: MAPREDUCE-3349-v1.6.patch Attaching a patch incorporating Sid's comments. I was unable to test this patch. I will upload the test results soon. No rack-name logged in JobHistory for unsuccessful tasks Key: MAPREDUCE-3349 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3349 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3349-v1.4.patch, MAPREDUCE-3349-v1.6.patch, MAPREDUCE-3349.patch Found this while running jobs on a cluster with [~Karams]. This is because TaskAttemptUnsuccessfulCompletionEvent history record doesn't have a rack field. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
