[jira] [Commented] (MAPREDUCE-3366) Mapreduce component should use consistent directory structure layout as HDFS/common
[
https://issues.apache.org/jira/browse/MAPREDUCE-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167402#comment-13167402
]
Eric Yang commented on MAPREDUCE-3366:
--
Mahadev, yes I will update the document for completeness. Thanks for the
pointer.
Mapreduce component should use consistent directory structure layout as
HDFS/common
---
Key: MAPREDUCE-3366
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3366
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Environment: Java, Maven
Reporter: Eric Yang
Assignee: Eric Yang
Labels: bigtop
Attachments: MAPREDUCE-3366-1-trunk.patch,
MAPREDUCE-3366-2-trunk.patch, MAPREDUCE-3366-3-trunk.patch,
MAPREDUCE-3366.patch, svn-rename-MAPREDUCE-3366-3.sh
Directory structure for MRv2 layout looks like:
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/conf
/lib
/modules
{noformat}
The directory structure layout should be updated to reflect changes
implemented in HADOOP-6255.
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/etc/hadoop
/lib
/libexec
/sbin
/share/hadoop
/share/hadoop/lib
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH needs to be overridable
[
https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Radwan updated MAPREDUCE-3505:
Attachment: MAPREDUCE-3505.patch
A possible way of addressing that is to push this classpath as part of
configuration, where the default value will be the current value in
ApplicationConstants.APPLICATION_CLASSPATH. This value can be then overridden
(if needed) in the conf files.
The attached patch does that.
yarn APPLICATION_CLASSPATH needs to be overridable
--
Key: MAPREDUCE-3505
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.23.1
Reporter: Bruno Mahé
Labels: bigtop
Attachments: MAPREDUCE-3505.patch
Right now MRApps sets the classpath to just being mrapp-generated-classpath,
its content and a hardcoded list of directories.
If I understand correctly mrapp-generated-classpath is only there for testing
and may change or disappear at any time
The list of hardcoded directories is defined in
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
at line 92.
For convenience, here is its current content:
{noformat}
/**
* Classpath for typical applications.
*/
public static final String[] APPLICATION_CLASSPATH =
new String[] {
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/share/hadoop/common/*,
$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,
$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
$YARN_HOME/modules/*,
$YARN_HOME/lib/*
};
{noformat}
Not all deployment scenarii fit in this layout and therefore we need a
standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH needs to be overridable
[
https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Radwan reassigned MAPREDUCE-3505:
---
Assignee: Ahmed Radwan
yarn APPLICATION_CLASSPATH needs to be overridable
--
Key: MAPREDUCE-3505
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.23.1
Reporter: Bruno Mahé
Assignee: Ahmed Radwan
Labels: bigtop
Attachments: MAPREDUCE-3505.patch
Right now MRApps sets the classpath to just being mrapp-generated-classpath,
its content and a hardcoded list of directories.
If I understand correctly mrapp-generated-classpath is only there for testing
and may change or disappear at any time
The list of hardcoded directories is defined in
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
at line 92.
For convenience, here is its current content:
{noformat}
/**
* Classpath for typical applications.
*/
public static final String[] APPLICATION_CLASSPATH =
new String[] {
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/share/hadoop/common/*,
$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,
$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
$YARN_HOME/modules/*,
$YARN_HOME/lib/*
};
{noformat}
Not all deployment scenarii fit in this layout and therefore we need a
standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3533) have the service interface extend Closeable and use close() as its shutdown operation
have the service interface extend Closeable and use close() as its shutdown operation - Key: MAPREDUCE-3533 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3533 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0, 0.23.1, 0.24.0 Reporter: Steve Loughran Priority: Minor This is probably going be marked down as a wontfix on the basis that it is (almost) encoded in the interfaces, but it may not be too late. If the service interface interface extended Closeable and used close() instead of stop(), it would work automatically in the java7 automatic resource management blocks... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH needs to be overridable
[
https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167458#comment-13167458
]
Hadoop QA commented on MAPREDUCE-3505:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12506990/MAPREDUCE-3505.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1420//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1420//console
This message is automatically generated.
yarn APPLICATION_CLASSPATH needs to be overridable
--
Key: MAPREDUCE-3505
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.23.1
Reporter: Bruno Mahé
Assignee: Ahmed Radwan
Labels: bigtop
Attachments: MAPREDUCE-3505.patch
Right now MRApps sets the classpath to just being mrapp-generated-classpath,
its content and a hardcoded list of directories.
If I understand correctly mrapp-generated-classpath is only there for testing
and may change or disappear at any time
The list of hardcoded directories is defined in
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
at line 92.
For convenience, here is its current content:
{noformat}
/**
* Classpath for typical applications.
*/
public static final String[] APPLICATION_CLASSPATH =
new String[] {
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/share/hadoop/common/*,
$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,
$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
$YARN_HOME/modules/*,
$YARN_HOME/lib/*
};
{noformat}
Not all deployment scenarii fit in this layout and therefore we need a
standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3534) Compression runtime is increased by 13% as well as throughput is decreased by 24% in 0.23
Compression runtime is increased by 13% as well as throughput is decreased by 24% in 0.23 -- Key: MAPREDUCE-3534 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3534 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Vinay Kumar Thota Compression runtime is increased by 13% as well as throughput is decreased by 24% in 0.23 when compared to 0.20.204 on a 350 node size cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3535) Yarn service subclasses don't check for service state before executing their state change operations
Yarn service subclasses don't check for service state before executing their
state change operations
Key: MAPREDUCE-3535
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3535
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Steve Loughran
Although there are checks in the lifecycle state change methods (start, stop,
...), they only get checked in the superclass. The subclasses don't check it;
they don't exit the start() method if they are already started, and they don't
bail out early on a stop if they are already stopped -even when the superclass
returns without doing anything.
This means that calling {{Service.start()}} twice may leak resources,
{{Service.start()}} twice try to shut down twice, etc. And that's on the same
thread...
My preferred action here would be for the ave the operations return true if a
state change took place, the implementation would be synchronised and return
the correct value
The subclasses look for this and only execute their state changes took place
e.g
{code}
boolean start() {
if (!super.start()) return false;
//do my own startup
return true;
}
{code}
The {{Service.stop()}} operation is trickier as the subclasses tend to call the
superclass last for a better unwinding. I think it may be safer to work the
other way around, but code reviews would be need to ensure that this doesn't
break assumptions in the subclass about termination order. It may be possible
to do more complex designs, but it is hard to chain this down a hierarchy of
classes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3533) have the service interface extend Closeable and use close() as its shutdown operation
[ https://issues.apache.org/jira/browse/MAPREDUCE-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167513#comment-13167513 ] Robert Joseph Evans commented on MAPREDUCE-3533: I too hope it is not too late. have the service interface extend Closeable and use close() as its shutdown operation - Key: MAPREDUCE-3533 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3533 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0, 0.23.1, 0.24.0 Reporter: Steve Loughran Priority: Minor This is probably going be marked down as a wontfix on the basis that it is (almost) encoded in the interfaces, but it may not be too late. If the service interface interface extended Closeable and used close() instead of stop(), it would work automatically in the java7 automatic resource management blocks... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3536) consider whether the same instance of a ServiceStateChangeListener should be allowed to listen to events
consider whether the same instance of a ServiceStateChangeListener should be
allowed to listen to events
Key: MAPREDUCE-3536
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3536
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Steve Loughran
Priority: Trivial
Currently there is no limit on the number of times a listener can register for
events; it's a simple list. A service must unregister the same number of times
that it registers.
Is this the desired behaviour? If so it should be documented in the {{Service}}
interface rather than just implicitly in the {{AbstractService}}
implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3416) Rename {start,stop}-all.sh to {start,stop}-yarn.sh for consistency with HDFS
[
https://issues.apache.org/jira/browse/MAPREDUCE-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles resolved MAPREDUCE-3416.
Resolution: Duplicate
Rename {start,stop}-all.sh to {start,stop}-yarn.sh for consistency with HDFS
Key: MAPREDUCE-3416
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3416
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.23.0
Reporter: Tom White
Assignee: Jonathan Eagles
Attachments: MAPREDUCE-3416.patch
There are already {start,stop}-all.sh scripts for starting and stopping all
Hadoop daemons which conflict with the YARN {start,stop}-all.sh scripts. The
latter should be renamed {start,stop}-yarn.sh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3416) Rename {start,stop}-all.sh to {start,stop}-yarn.sh for consistency with HDFS
[
https://issues.apache.org/jira/browse/MAPREDUCE-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167611#comment-13167611
]
Jonathan Eagles commented on MAPREDUCE-3416:
Thanks for pointing out MAPREDUCE-3366, Mahadev.
Rename {start,stop}-all.sh to {start,stop}-yarn.sh for consistency with HDFS
Key: MAPREDUCE-3416
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3416
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.23.0
Reporter: Tom White
Assignee: Jonathan Eagles
Attachments: MAPREDUCE-3416.patch
There are already {start,stop}-all.sh scripts for starting and stopping all
Hadoop daemons which conflict with the YARN {start,stop}-all.sh scripts. The
latter should be renamed {start,stop}-yarn.sh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3527) Fix minor API incompatibilities between 1.0 and 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167614#comment-13167614 ] Tom White commented on MAPREDUCE-3527: -- I found these using SigTest (see HADOOP-7738 and https://issues.apache.org/jira/secure/attachment/12498825/apicheck-hadoop-0.20.204.0-0.24.0-SNAPSHOT.txt), which looks for API incompatibilities. It would be great to run this tool as a part of test-patch at some point. Fix minor API incompatibilities between 1.0 and 0.23 Key: MAPREDUCE-3527 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3527 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Attachments: MAPREDUCE-3527.patch There are a few minor incompatibilities that were found in HADOOP-7738 and are straightforward to fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3534) Compression runtime is increased by 13% as well as throughput is decreased by 24% in 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3534: - Priority: Critical (was: Major) Compression runtime is increased by 13% as well as throughput is decreased by 24% in 0.23 -- Key: MAPREDUCE-3534 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3534 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Vinay Kumar Thota Priority: Critical Compression runtime is increased by 13% as well as throughput is decreased by 24% in 0.23 when compared to 0.20.204 on a 350 node size cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3328) mapred queue -list output inconsistent and missing child queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3328: Status: Patch Available (was: Open) mapred queue -list output inconsistent and missing child queues --- Key: MAPREDUCE-3328 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3328 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Ravi Prakash Priority: Critical Attachments: MAPREDUCE-3328.branch-0.23.patch, MAPREDUCE-3328.trunk.patch, MAPREDUCE-3328.trunk.patch, MAPREDUCE-3328.trunk.patch When running mapred queue -list on a 0.23.0 cluster with capacity scheduler configured with child queues. In my case I have queues default, test1, and test2. test1 has subqueues of a1, a2. test2 has subqueues of a3 and a4. - the child queues do not show up - The output of maximum capacity doesn't match the format of the current capacity and capacity. the latter two use float while the maximum is specified as int: Queue Name : default Queue State : running Scheduling Info : queueName: default, capacity: 0.7, maximumCapacity: 90.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test Queue State : running Scheduling Info : queueName: test, capacity: 0.2, maximumCapacity: -1.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test2 Queue State : running Scheduling Info : queueName: test2, capacity: 0.1, maximumCapacity: 5.0, currentCapacity: 0.0, state: Q_RUNNING, == here default is configured to have capacity=70% and maximum capacity = 90% -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-778) [Rumen] Need a standalone JobHistory log anonymizer
[ https://issues.apache.org/jira/browse/MAPREDUCE-778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-778: - Attachment: mapreduce-778-v1.14-12.patch Attaching a patch that adds the remaining features to the Anonymizer. Some newly added features: 1. Note that Gridmix runs with the anonymized trace. 2. Classname now have a filter which allows the user to specify which packages to pass through. 3. Job config parsing and filtering is done. Only MR (framework-level) configs are parsed and allowed. There is a config to extend this functionality and allow users to handle other keys. E.g. Pig etc. Read the Rumen manual for details on the Anonymizer and its configuration parameters. Testing: test-patch and ant tests passed. Also tested on 2 days worth of job history data. Todos: 1. Currently, the job properties only consideres MR (framework-level) properties. Add job config parsers for Input/Output file formats, pig, etc. 2. Chunking of data (esp. job names) to preserve some useful stats like daily/weekly/monthly etc. [Rumen] Need a standalone JobHistory log anonymizer --- Key: MAPREDUCE-778 URL: https://issues.apache.org/jira/browse/MAPREDUCE-778 Project: Hadoop Map/Reduce Issue Type: New Feature Components: tools/rumen Reporter: Hong Tang Assignee: Amar Kamat Labels: anonymization, rumen Attachments: anonymizer.patch, anonymizer.py, mapreduce-778-v1.14-12.patch, mapreduce-778-v1.2-2.patch, same.py Job history logs contain a rich set of information that can help understand and characterize cluster workload and individual job execution. Examples of work that parses or utilizes job history include HADOOP-3585, MAPREDUCE-534, HDFS-459, MAPREDUCE-728, and MAPREDUCE-776. Some of the parsing tools developed in previous work already contains a component to anonymize the logs. It would be nice to combine these effort and have a common standalone tool that can anonymizes job history logs and preserve much of the structure of the files so that existing tools on top of job history logs continue work with no modification. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3527) Fix minor API incompatibilities between 1.0 and 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-3527: - Resolution: Fixed Fix Version/s: 0.24.0 0.23.1 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this. The two extra javac warnings are due to deprecations on the reinstated methods. There are no new tests included since the change just reinstated some methods that were previously removed. Fix minor API incompatibilities between 1.0 and 0.23 Key: MAPREDUCE-3527 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3527 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Fix For: 0.23.1, 0.24.0 Attachments: MAPREDUCE-3527.patch There are a few minor incompatibilities that were found in HADOOP-7738 and are straightforward to fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3534) Compression runtime is increased by 13% as well as throughput is decreased by 24% in 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167674#comment-13167674 ] Todd Lipcon commented on MAPREDUCE-3534: What's the test/benchmark used for this result? Compression runtime is increased by 13% as well as throughput is decreased by 24% in 0.23 -- Key: MAPREDUCE-3534 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3534 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Vinay Kumar Thota Priority: Critical Compression runtime is increased by 13% as well as throughput is decreased by 24% in 0.23 when compared to 0.20.204 on a 350 node size cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3527) Fix minor API incompatibilities between 1.0 and 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167685#comment-13167685 ] Hudson commented on MAPREDUCE-3527: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1423 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1423/]) MAPREDUCE-3527. Fix minor API incompatibilities between 1.0 and 0.23. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213350 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/InvalidJobConfException.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/RunningJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/CombineFileInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/TaskAttemptID.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/TaskID.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/KeyValueLineRecordReader.java Fix minor API incompatibilities between 1.0 and 0.23 Key: MAPREDUCE-3527 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3527 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Fix For: 0.23.1, 0.24.0 Attachments: MAPREDUCE-3527.patch There are a few minor incompatibilities that were found in HADOOP-7738 and are straightforward to fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3502) Review all Service.stop() operations and make sure that they work before a service is started
[ https://issues.apache.org/jira/browse/MAPREDUCE-3502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-3502: -- Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1) Status: Open (was: Patch Available) Review all Service.stop() operations and make sure that they work before a service is started - Key: MAPREDUCE-3502 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3502 Project: Hadoop Map/Reduce Issue Type: Task Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Steve Loughran Assignee: Steve Loughran Attachments: MAPREDUCE-3502.patch Original Estimate: 24h Time Spent: 0.5h Remaining Estimate: 23.5h MAPREDUCE-3431 has shown that some of the key services's shutdown operations are not robust against being invoked before the service is started. They need to be by # not calling other things if the other things are null # not being re-entrant (i.e. make synchronized if possible), Maybe # have a StopService operation that only stops a service if it is live # factor out the is-running test from the base service class and make it a pre-check for all the child services, so they bail out sooner rather than later. This would be the best as it would be the one guaranteed to work consistently across all instances, so only one or two would need testing my first iteration will skip the sync though it's something to consider. Testing: try to create each instance; call stop() straight after construction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3527) Fix minor API incompatibilities between 1.0 and 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167689#comment-13167689 ] Hudson commented on MAPREDUCE-3527: --- Integrated in Hadoop-Mapreduce-0.23-Commit #281 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/281/]) Merge -r 1213349:1213350 from trunk to branch-0.23. Fixes: MAPREDUCE-3527 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213355 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/InvalidJobConfException.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/RunningJob.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/CombineFileInputFormat.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/TaskAttemptID.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/TaskID.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/KeyValueLineRecordReader.java Fix minor API incompatibilities between 1.0 and 0.23 Key: MAPREDUCE-3527 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3527 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Fix For: 0.23.1, 0.24.0 Attachments: MAPREDUCE-3527.patch There are a few minor incompatibilities that were found in HADOOP-7738 and are straightforward to fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3502) Review all Service.stop() operations and make sure that they work before a service is started
[ https://issues.apache.org/jira/browse/MAPREDUCE-3502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-3502: -- Attachment: MAPREDUCE-3502.patch Review all Service.stop() operations and make sure that they work before a service is started - Key: MAPREDUCE-3502 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3502 Project: Hadoop Map/Reduce Issue Type: Task Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Steve Loughran Assignee: Steve Loughran Attachments: MAPREDUCE-3502.patch, MAPREDUCE-3502.patch Original Estimate: 24h Time Spent: 0.5h Remaining Estimate: 23.5h MAPREDUCE-3431 has shown that some of the key services's shutdown operations are not robust against being invoked before the service is started. They need to be by # not calling other things if the other things are null # not being re-entrant (i.e. make synchronized if possible), Maybe # have a StopService operation that only stops a service if it is live # factor out the is-running test from the base service class and make it a pre-check for all the child services, so they bail out sooner rather than later. This would be the best as it would be the one guaranteed to work consistently across all instances, so only one or two would need testing my first iteration will skip the sync though it's something to consider. Testing: try to create each instance; call stop() straight after construction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3502) Review all Service.stop() operations and make sure that they work before a service is started
[ https://issues.apache.org/jira/browse/MAPREDUCE-3502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-3502: -- Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1) Status: Patch Available (was: Open) Review all Service.stop() operations and make sure that they work before a service is started - Key: MAPREDUCE-3502 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3502 Project: Hadoop Map/Reduce Issue Type: Task Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Steve Loughran Assignee: Steve Loughran Attachments: MAPREDUCE-3502.patch, MAPREDUCE-3502.patch Original Estimate: 24h Time Spent: 2.5h Remaining Estimate: 21.5h MAPREDUCE-3431 has shown that some of the key services's shutdown operations are not robust against being invoked before the service is started. They need to be by # not calling other things if the other things are null # not being re-entrant (i.e. make synchronized if possible), Maybe # have a StopService operation that only stops a service if it is live # factor out the is-running test from the base service class and make it a pre-check for all the child services, so they bail out sooner rather than later. This would be the best as it would be the one guaranteed to work consistently across all instances, so only one or two would need testing my first iteration will skip the sync though it's something to consider. Testing: try to create each instance; call stop() straight after construction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3535) Yarn service subclasses don't check for service state before executing their state change operations
[
https://issues.apache.org/jira/browse/MAPREDUCE-3535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167708#comment-13167708
]
Steve Loughran commented on MAPREDUCE-3535:
---
Having a look at what I'd done w.r.t service lifecycle
[http://svn.eu.apache.org/viewvc/hadoop/common/branches/HADOOP-3628-2/src/core/org/apache/hadoop/util/Service.java?view=markup]
I avoided this by having the base class do all the checks in final methods and
have overridable {{innerStart()}}, {{innerStop()}} etc. methods that subclasses
would use to perform their custom operations, along with a static class to
actually walk a service into its started state.
There is no reason why the {{AbstractService}} class cannot use the same
technique, without changing the {{Service}} interface. It would declare it's
init/start/stop methods final, do the state change, then delegate to the
{{protected}} methods {{innerInit()}} {{innerStart()}}, {{innerStop()}}. These
would not be externally visible, and not get invoked until the validity of the
operation had been tested.
Effort:
# time to rework the older service lifecycle methods into the new framework, 1
h
# time to go through all the subclasses and rename their init/start/stop
methods to be the inner ones.
We could use a better prefix than inner if anyone can think of it.
Yarn service subclasses don't check for service state before executing their
state change operations
Key: MAPREDUCE-3535
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3535
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Steve Loughran
Although there are checks in the lifecycle state change methods (start, stop,
...), they only get checked in the superclass. The subclasses don't check it;
they don't exit the start() method if they are already started, and they
don't bail out early on a stop if they are already stopped -even when the
superclass returns without doing anything.
This means that calling {{Service.start()}} twice may leak resources,
{{Service.start()}} twice try to shut down twice, etc. And that's on the same
thread...
My preferred action here would be for the ave the operations return true if a
state change took place, the implementation would be synchronised and return
the correct value
The subclasses look for this and only execute their state changes took place
e.g
{code}
boolean start() {
if (!super.start()) return false;
//do my own startup
return true;
}
{code}
The {{Service.stop()}} operation is trickier as the subclasses tend to call
the superclass last for a better unwinding. I think it may be safer to work
the other way around, but code reviews would be need to ensure that this
doesn't break assumptions in the subclass about termination order. It may be
possible to do more complex designs, but it is hard to chain this down a
hierarchy of classes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3527) Fix minor API incompatibilities between 1.0 and 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167726#comment-13167726 ] Hudson commented on MAPREDUCE-3527: --- Integrated in Hadoop-Hdfs-trunk-Commit #1475 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1475/]) MAPREDUCE-3527. Fix minor API incompatibilities between 1.0 and 0.23. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213350 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/InvalidJobConfException.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/RunningJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/CombineFileInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/TaskAttemptID.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/TaskID.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/KeyValueLineRecordReader.java Fix minor API incompatibilities between 1.0 and 0.23 Key: MAPREDUCE-3527 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3527 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Fix For: 0.23.1, 0.24.0 Attachments: MAPREDUCE-3527.patch There are a few minor incompatibilities that were found in HADOOP-7738 and are straightforward to fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3527) Fix minor API incompatibilities between 1.0 and 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167731#comment-13167731 ] Hudson commented on MAPREDUCE-3527: --- Integrated in Hadoop-Common-trunk-Commit #1401 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1401/]) MAPREDUCE-3527. Fix minor API incompatibilities between 1.0 and 0.23. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213350 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/InvalidJobConfException.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/RunningJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/CombineFileInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/TaskAttemptID.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/TaskID.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/KeyValueLineRecordReader.java Fix minor API incompatibilities between 1.0 and 0.23 Key: MAPREDUCE-3527 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3527 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Fix For: 0.23.1, 0.24.0 Attachments: MAPREDUCE-3527.patch There are a few minor incompatibilities that were found in HADOOP-7738 and are straightforward to fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3527) Fix minor API incompatibilities between 1.0 and 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167736#comment-13167736 ] Hudson commented on MAPREDUCE-3527: --- Integrated in Hadoop-Hdfs-0.23-Commit #261 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/261/]) Merge -r 1213349:1213350 from trunk to branch-0.23. Fixes: MAPREDUCE-3527 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213355 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/InvalidJobConfException.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/RunningJob.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/CombineFileInputFormat.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/TaskAttemptID.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/TaskID.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/KeyValueLineRecordReader.java Fix minor API incompatibilities between 1.0 and 0.23 Key: MAPREDUCE-3527 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3527 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Fix For: 0.23.1, 0.24.0 Attachments: MAPREDUCE-3527.patch There are a few minor incompatibilities that were found in HADOOP-7738 and are straightforward to fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3527) Fix minor API incompatibilities between 1.0 and 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167746#comment-13167746 ] Hudson commented on MAPREDUCE-3527: --- Integrated in Hadoop-Common-0.23-Commit #271 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/271/]) Merge -r 1213349:1213350 from trunk to branch-0.23. Fixes: MAPREDUCE-3527 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213355 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/InvalidJobConfException.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/RunningJob.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/CombineFileInputFormat.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/TaskAttemptID.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/TaskID.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/KeyValueLineRecordReader.java Fix minor API incompatibilities between 1.0 and 0.23 Key: MAPREDUCE-3527 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3527 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Fix For: 0.23.1, 0.24.0 Attachments: MAPREDUCE-3527.patch There are a few minor incompatibilities that were found in HADOOP-7738 and are straightforward to fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3502) Review all Service.stop() operations and make sure that they work before a service is started
[ https://issues.apache.org/jira/browse/MAPREDUCE-3502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167767#comment-13167767 ] Hadoop QA commented on MAPREDUCE-3502: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507037/MAPREDUCE-3502.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1422//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1422//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1422//console This message is automatically generated. Review all Service.stop() operations and make sure that they work before a service is started - Key: MAPREDUCE-3502 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3502 Project: Hadoop Map/Reduce Issue Type: Task Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Steve Loughran Assignee: Steve Loughran Attachments: MAPREDUCE-3502.patch, MAPREDUCE-3502.patch Original Estimate: 24h Time Spent: 2.5h Remaining Estimate: 21.5h MAPREDUCE-3431 has shown that some of the key services's shutdown operations are not robust against being invoked before the service is started. They need to be by # not calling other things if the other things are null # not being re-entrant (i.e. make synchronized if possible), Maybe # have a StopService operation that only stops a service if it is live # factor out the is-running test from the base service class and make it a pre-check for all the child services, so they bail out sooner rather than later. This would be the best as it would be the one guaranteed to work consistently across all instances, so only one or two would need testing my first iteration will skip the sync though it's something to consider. Testing: try to create each instance; call stop() straight after construction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3328) mapred queue -list output inconsistent and missing child queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167769#comment-13167769 ] Hadoop QA commented on MAPREDUCE-3328: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505380/MAPREDUCE-3328.trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1423//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1423//console This message is automatically generated. mapred queue -list output inconsistent and missing child queues --- Key: MAPREDUCE-3328 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3328 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Ravi Prakash Priority: Critical Attachments: MAPREDUCE-3328.branch-0.23.patch, MAPREDUCE-3328.trunk.patch, MAPREDUCE-3328.trunk.patch, MAPREDUCE-3328.trunk.patch When running mapred queue -list on a 0.23.0 cluster with capacity scheduler configured with child queues. In my case I have queues default, test1, and test2. test1 has subqueues of a1, a2. test2 has subqueues of a3 and a4. - the child queues do not show up - The output of maximum capacity doesn't match the format of the current capacity and capacity. the latter two use float while the maximum is specified as int: Queue Name : default Queue State : running Scheduling Info : queueName: default, capacity: 0.7, maximumCapacity: 90.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test Queue State : running Scheduling Info : queueName: test, capacity: 0.2, maximumCapacity: -1.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test2 Queue State : running Scheduling Info : queueName: test2, capacity: 0.1, maximumCapacity: 5.0, currentCapacity: 0.0, state: Q_RUNNING, == here default is configured to have capacity=70% and maximum capacity = 90% -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3518: --- Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1) Status: Patch Available (was: Open) mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Attachments: MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3518: --- Attachment: MAPREDUCE-3518.patch mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Attachments: MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167781#comment-13167781 ] Jonathan Eagles commented on MAPREDUCE-3518: I appreciate your comment on public APIs. Something definitely needs to be addressed. The difficulty lies in that the testing class needs the ability to mock out the cluster (protected no good), but exists in a different package (package-private no good). I have tried to address all concerns with the latest patch. In this patch I merely add no public API or vars in CLI or JobClient. Instead, I have created a small derived class for testing that is able to set the protected cluster. mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Attachments: MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3518: --- Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1) Status: Open (was: Patch Available) mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Attachments: MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167816#comment-13167816 ] Hadoop QA commented on MAPREDUCE-3518: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507054/MAPREDUCE-3518.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1424//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1424//console This message is automatically generated. mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Attachments: MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2593) Random read benchmark for DFS
[ https://issues.apache.org/jira/browse/MAPREDUCE-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson updated MAPREDUCE-2593: - Attachment: RndRead-TestDFSIO-MR2593-trunk121211.patch Updated path names for trunk 0.23 (12/12/11). Otherwise, it's the same patch. Random read benchmark for DFS - Key: MAPREDUCE-2593 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2593 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Raghu Angadi Assignee: Dave Thompson Attachments: HDFS-236.patch, RndRead-TestDFSIO-061011.patch, RndRead-TestDFSIO-MR2593-trunk121211.patch, RndRead-TestDFSIO.patch We should have at least one random read benchmark that can be run with rest of Hadoop benchmarks regularly. Please provide benchmark ideas or requirements. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3522) Capacity Scheduler ACLs not inherited by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3522: --- Attachment: MAPREDUCE-3522.patch Capacity Scheduler ACLs not inherited by default Key: MAPREDUCE-3522 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3522 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3522.patch, MAPREDUCE-3522.patch Hierarchical Queues do not inherit parent ACLs correctly by default. Instead, if no value is specified for submit or administer acls, then all access is granted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3421) Error in MR AM shutdown when running an uber-job
[ https://issues.apache.org/jira/browse/MAPREDUCE-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah resolved MAPREDUCE-3421. Resolution: Duplicate Assignee: Hitesh Shah Error in MR AM shutdown when running an uber-job Key: MAPREDUCE-3421 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3421 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Fix For: 0.23.1 Exception thrown during the un-registration process in the ContainerAllocator#stop. Exception trace in next comment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167852#comment-13167852 ] Mahadev konar commented on MAPREDUCE-3518: -- @Jon, +1 the patch looks good. Thanks for your patience. Committing now.. mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Attachments: MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2593) Random read benchmark for DFS
[ https://issues.apache.org/jira/browse/MAPREDUCE-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167874#comment-13167874 ] Hadoop QA commented on MAPREDUCE-2593: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507064/RndRead-TestDFSIO-MR2593-trunk121211.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1425//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1425//console This message is automatically generated. Random read benchmark for DFS - Key: MAPREDUCE-2593 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2593 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Raghu Angadi Assignee: Dave Thompson Attachments: HDFS-236.patch, RndRead-TestDFSIO-061011.patch, RndRead-TestDFSIO-MR2593-trunk121211.patch, RndRead-TestDFSIO.patch We should have at least one random read benchmark that can be run with rest of Hadoop benchmarks regularly. Please provide benchmark ideas or requirements. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167881#comment-13167881 ] Jonathan Eagles commented on MAPREDUCE-3518: @Mahadev, Thanks for taking the time. I have learned a lot from this JIRA. mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Attachments: MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3426) uber-jobs tried to write outputs into wrong dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167892#comment-13167892 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3426: I am looking at the patch. In the mean while can you please provide a brief summary of the patch? Thanks! uber-jobs tried to write outputs into wrong dir --- Key: MAPREDUCE-3426 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3426 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3426.1.patch, MR-3426.2.patch Incorrect setup of the uber tasks causes tasks to try to write intermidiate outputs into dirs that the user does not have permissions to write to on a secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167890#comment-13167890 ] Hudson commented on MAPREDUCE-3518: --- Integrated in Hadoop-Common-trunk-Commit #1403 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1403/]) MAPREDUCE-3518. mapred queue -info queue -showJobs throws NPE. (Jonathan Eagles via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213464 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/JobClientUnitTest.java mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Attachments: MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3426) uber-jobs tried to write outputs into wrong dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167894#comment-13167894 ] Hitesh Shah commented on MAPREDUCE-3426: Summary of changes: LocalContainerLauncher changes: - in runSubTask(), set the correct config values for task specific information as well as the local dirs setting so that the output is written to the correct dir ( which the process has write permissions to ) - cleaned up commented out code. Added flags to be able to profile map and reduce tasks with separate profiling params. MRAppMaster changes: - fixed disabling of speculative execution if job is uberized. - Moved uber decision into new function - hardcoded max reduce count supported to 1 as multiple reduces are not supported. - using MR AM vm size as memory bound check for uber decision RMCommunicator change: - for MR-3421 JobStatus/JobReport changes: - Print out uber choice on client side to inform user whether running job in uber or non-uber mode Comments changes: - Removed comments on race condition in local launcher for MapTask can't send TA_UPDATE before TA_CONTAINER_LAUNCHED - the state machine has a write lock, also the launched event is sent first followed by any updates if any from the task itself. - renameMapOutputForReduce is still required. Output files are generated into top level dir and not a task specific dir. uber-jobs tried to write outputs into wrong dir --- Key: MAPREDUCE-3426 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3426 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3426.1.patch, MR-3426.2.patch Incorrect setup of the uber tasks causes tasks to try to write intermidiate outputs into dirs that the user does not have permissions to write to on a secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167896#comment-13167896 ] Hudson commented on MAPREDUCE-3518: --- Integrated in Hadoop-Hdfs-trunk-Commit #1477 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1477/]) MAPREDUCE-3518. mapred queue -info queue -showJobs throws NPE. (Jonathan Eagles via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213464 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/JobClientUnitTest.java mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Attachments: MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167913#comment-13167913 ] Hudson commented on MAPREDUCE-3518: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1425 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1425/]) MAPREDUCE-3518. mapred queue -info queue -showJobs throws NPE. (Jonathan Eagles via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213464 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/JobClientUnitTest.java mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Attachments: MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2863) Support web-services for RM NM
[ https://issues.apache.org/jira/browse/MAPREDUCE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-2863: - Attachment: MAPREDUCE-2863.patch rebased patch off latest trunk. Support web-services for RM NM Key: MAPREDUCE-2863 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2863 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2, nodemanager, resourcemanager Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Thomas Graves Priority: Blocker Attachments: MAPREDUCE-2863.patch, MAPREDUCE-2863.patch, MAPREDUCE-2863.patch, MAPREDUCE-2863.patch, MAPREDUCE-2863.patch, MAPREDUCE-2863.patch, MAPREDUCE-2863.patch, MAPREDUCE-2863.patch, MAPREDUCE-2863.patch, amoutput.txt, appoutput.txt, hsoutput.txt, nmoutput.txt, nmoutput.txt, nmoutput.txt, rmoutput.txt, rmoutput.txt, rmoutput.txt It will be very useful for RM and NM to support web-services to export json/xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3518: - Resolution: Fixed Fix Version/s: 0.23.1 Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1) Status: Resolved (was: Patch Available) I just committed this. Thanks Jon. Nice follow ups on the comments! mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3522) Capacity Scheduler ACLs not inherited by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3522: --- Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1) Status: Patch Available (was: Open) Capacity Scheduler ACLs not inherited by default Key: MAPREDUCE-3522 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3522 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3522.patch, MAPREDUCE-3522.patch Hierarchical Queues do not inherit parent ACLs correctly by default. Instead, if no value is specified for submit or administer acls, then all access is granted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3426) uber-jobs tried to write outputs into wrong dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167942#comment-13167942 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3426: Mostly looks good. Minor comments on the patch. - Jobimpl changes: Not this patch's fault, duplicate comment starting User has overall veto power over uberization - RMCommunicator.stop() is still not thread-safe/idempotent - the shutdown hook and the stop-event handling may both find the volatile boolean be true and proceed. This method needs synchronization. - JobStatus constructors - in both mapred and map reduce classes - need to backwards compatible. uber-jobs tried to write outputs into wrong dir --- Key: MAPREDUCE-3426 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3426 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3426.1.patch, MR-3426.2.patch Incorrect setup of the uber tasks causes tasks to try to write intermidiate outputs into dirs that the user does not have permissions to write to on a secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2863) Support web-services for RM NM
[ https://issues.apache.org/jira/browse/MAPREDUCE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167952#comment-13167952 ] Hadoop QA commented on MAPREDUCE-2863: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507086/MAPREDUCE-2863.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1426//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1426//console This message is automatically generated. Support web-services for RM NM Key: MAPREDUCE-2863 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2863 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2, nodemanager, resourcemanager Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Thomas Graves Priority: Blocker Attachments: MAPREDUCE-2863.patch, MAPREDUCE-2863.patch, MAPREDUCE-2863.patch, MAPREDUCE-2863.patch, MAPREDUCE-2863.patch, MAPREDUCE-2863.patch, MAPREDUCE-2863.patch, MAPREDUCE-2863.patch, MAPREDUCE-2863.patch, amoutput.txt, appoutput.txt, hsoutput.txt, nmoutput.txt, nmoutput.txt, nmoutput.txt, rmoutput.txt, rmoutput.txt, rmoutput.txt It will be very useful for RM and NM to support web-services to export json/xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3530) Sometimes NODE_UPDATE to the scheduler throws NPE causes scheduling to stop
[
https://issues.apache.org/jira/browse/MAPREDUCE-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy reassigned MAPREDUCE-3530:
Assignee: Vinod Kumar Vavilapalli
Sometimes NODE_UPDATE to the scheduler throws NPE causes scheduling to stop
---
Key: MAPREDUCE-3530
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3530
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, resourcemanager, scheduler
Affects Versions: 0.23.1
Reporter: Karam Singh
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Sometimes NODE_UPDATE to the scheduler throws NPE causes scheduling to stop
but ResourceManager keeps on running.
I have been observing intermitently for last 3 weeks.
But with latest svn code. I tried to run sort twice and both times Job got
stuck due to NPE.
{code}
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.containerLaunchedOnNode(SchedulerApp.java:181)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.containerLaunchedOnNode(CapacityScheduler.java:596)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:539)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:617)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
at java.lang.Thread.run(Thread.java:619)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3510) Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls
[ https://issues.apache.org/jira/browse/MAPREDUCE-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167977#comment-13167977 ] Mahadev konar commented on MAPREDUCE-3510: -- +1, patch looks good to me. Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls - Key: MAPREDUCE-3510 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3510 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/capacity-sched, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3510.patch, MAPREDUCE-3510.patch, MAPREDUCE-3510.patch, MAPREDUCE-3510.patch mapred queue -showacls does not show inherited acls -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167994#comment-13167994 ] Hudson commented on MAPREDUCE-3518: --- Integrated in Hadoop-Mapreduce-0.23-Commit #284 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/284/]) mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3510) Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls
[ https://issues.apache.org/jira/browse/MAPREDUCE-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168018#comment-13168018 ] Hudson commented on MAPREDUCE-3510: --- Integrated in Hadoop-Mapreduce-0.23-Commit #285 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/285/]) MAPREDUCE-3510. Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls (Jonathan Eagles via mahadev) - Merging r1213511 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213516 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls - Key: MAPREDUCE-3510 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3510 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/capacity-sched, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.1 Attachments: MAPREDUCE-3510.patch, MAPREDUCE-3510.patch, MAPREDUCE-3510.patch, MAPREDUCE-3510.patch mapred queue -showacls does not show inherited acls -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3510) Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls
[ https://issues.apache.org/jira/browse/MAPREDUCE-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168020#comment-13168020 ] Hudson commented on MAPREDUCE-3510: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1427 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1427/]) MAPREDUCE-3510. Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls (Jonathan Eagles via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213511 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls - Key: MAPREDUCE-3510 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3510 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/capacity-sched, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.1 Attachments: MAPREDUCE-3510.patch, MAPREDUCE-3510.patch, MAPREDUCE-3510.patch, MAPREDUCE-3510.patch mapred queue -showacls does not show inherited acls -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3366) Mapreduce component should use consistent directory structure layout as HDFS/common
[
https://issues.apache.org/jira/browse/MAPREDUCE-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated MAPREDUCE-3366:
-
Attachment: MAPREDUCE-3366-4-trunk.patch
Include updated documentation.
Mapreduce component should use consistent directory structure layout as
HDFS/common
---
Key: MAPREDUCE-3366
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3366
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Environment: Java, Maven
Reporter: Eric Yang
Assignee: Eric Yang
Labels: bigtop
Attachments: MAPREDUCE-3366-1-trunk.patch,
MAPREDUCE-3366-2-trunk.patch, MAPREDUCE-3366-3-trunk.patch,
MAPREDUCE-3366-4-trunk.patch, MAPREDUCE-3366.patch,
svn-rename-MAPREDUCE-3366-3.sh
Directory structure for MRv2 layout looks like:
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/conf
/lib
/modules
{noformat}
The directory structure layout should be updated to reflect changes
implemented in HADOOP-6255.
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/etc/hadoop
/lib
/libexec
/sbin
/share/hadoop
/share/hadoop/lib
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3366) Mapreduce component should use consistent directory structure layout as HDFS/common
[
https://issues.apache.org/jira/browse/MAPREDUCE-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168030#comment-13168030
]
Hadoop QA commented on MAPREDUCE-3366:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12507108/MAPREDUCE-3366-4-trunk.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1428//console
This message is automatically generated.
Mapreduce component should use consistent directory structure layout as
HDFS/common
---
Key: MAPREDUCE-3366
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3366
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Environment: Java, Maven
Reporter: Eric Yang
Assignee: Eric Yang
Labels: bigtop
Attachments: MAPREDUCE-3366-1-trunk.patch,
MAPREDUCE-3366-2-trunk.patch, MAPREDUCE-3366-3-trunk.patch,
MAPREDUCE-3366-4-trunk.patch, MAPREDUCE-3366.patch,
svn-rename-MAPREDUCE-3366-3.sh
Directory structure for MRv2 layout looks like:
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/conf
/lib
/modules
{noformat}
The directory structure layout should be updated to reflect changes
implemented in HADOOP-6255.
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/etc/hadoop
/lib
/libexec
/sbin
/share/hadoop
/share/hadoop/lib
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168027#comment-13168027 ] Hudson commented on MAPREDUCE-3518: --- Integrated in Hadoop-Hdfs-0.23-Commit #263 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/263/]) MAPREDUCE-3518. mapred queue -info queue -showJobs throws NPE. (Jonathan Eagles via mahadev) - Merging r1213464 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213493 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/JobClientUnitTest.java mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3510) Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls
[ https://issues.apache.org/jira/browse/MAPREDUCE-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168028#comment-13168028 ] Hudson commented on MAPREDUCE-3510: --- Integrated in Hadoop-Hdfs-0.23-Commit #263 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/263/]) MAPREDUCE-3510. Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls (Jonathan Eagles via mahadev) - Merging r1213511 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213516 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls - Key: MAPREDUCE-3510 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3510 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/capacity-sched, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.1 Attachments: MAPREDUCE-3510.patch, MAPREDUCE-3510.patch, MAPREDUCE-3510.patch, MAPREDUCE-3510.patch mapred queue -showacls does not show inherited acls -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3328) mapred queue -list output inconsistent and missing child queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168029#comment-13168029 ] Hudson commented on MAPREDUCE-3328: --- Integrated in Hadoop-Hdfs-0.23-Commit #263 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/263/]) MAPREDUCE-3328. mapred queue -list output inconsistent and missing child queues. (Ravi Prakash via mahadev) - Merging r1213504 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213505 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobQueueClient.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java mapred queue -list output inconsistent and missing child queues --- Key: MAPREDUCE-3328 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3328 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Ravi Prakash Priority: Critical Attachments: MAPREDUCE-3328.branch-0.23.patch, MAPREDUCE-3328.trunk.patch, MAPREDUCE-3328.trunk.patch, MAPREDUCE-3328.trunk.patch When running mapred queue -list on a 0.23.0 cluster with capacity scheduler configured with child queues. In my case I have queues default, test1, and test2. test1 has subqueues of a1, a2. test2 has subqueues of a3 and a4. - the child queues do not show up - The output of maximum capacity doesn't match the format of the current capacity and capacity. the latter two use float while the maximum is specified as int: Queue Name : default Queue State : running Scheduling Info : queueName: default, capacity: 0.7, maximumCapacity: 90.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test Queue State : running Scheduling Info : queueName: test, capacity: 0.2, maximumCapacity: -1.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test2 Queue State : running Scheduling Info : queueName: test2, capacity: 0.1, maximumCapacity: 5.0, currentCapacity: 0.0, state: Q_RUNNING, == here default is configured to have capacity=70% and maximum capacity = 90% -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3518) mapred queue -info queue -showJobs throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168031#comment-13168031 ] Hudson commented on MAPREDUCE-3518: --- Integrated in Hadoop-Common-0.23-Commit #273 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/273/]) MAPREDUCE-3518. mapred queue -info queue -showJobs throws NPE. (Jonathan Eagles via mahadev) - Merging r1213464 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213493 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/JobClientUnitTest.java mapred queue -info queue -showJobs throws NPE --- Key: MAPREDUCE-3518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3518 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch, MAPREDUCE-3518.patch mapred queue -info default -showJobs Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572) at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190) at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:234) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3510) Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls
[ https://issues.apache.org/jira/browse/MAPREDUCE-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168032#comment-13168032 ] Hudson commented on MAPREDUCE-3510: --- Integrated in Hadoop-Common-0.23-Commit #273 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/273/]) MAPREDUCE-3510. Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls (Jonathan Eagles via mahadev) - Merging r1213511 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213516 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls - Key: MAPREDUCE-3510 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3510 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/capacity-sched, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.1 Attachments: MAPREDUCE-3510.patch, MAPREDUCE-3510.patch, MAPREDUCE-3510.patch, MAPREDUCE-3510.patch mapred queue -showacls does not show inherited acls -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3328) mapred queue -list output inconsistent and missing child queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168033#comment-13168033 ] Hudson commented on MAPREDUCE-3328: --- Integrated in Hadoop-Common-0.23-Commit #273 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/273/]) MAPREDUCE-3328. mapred queue -list output inconsistent and missing child queues. (Ravi Prakash via mahadev) - Merging r1213504 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213505 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobQueueClient.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java mapred queue -list output inconsistent and missing child queues --- Key: MAPREDUCE-3328 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3328 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Ravi Prakash Priority: Critical Attachments: MAPREDUCE-3328.branch-0.23.patch, MAPREDUCE-3328.trunk.patch, MAPREDUCE-3328.trunk.patch, MAPREDUCE-3328.trunk.patch When running mapred queue -list on a 0.23.0 cluster with capacity scheduler configured with child queues. In my case I have queues default, test1, and test2. test1 has subqueues of a1, a2. test2 has subqueues of a3 and a4. - the child queues do not show up - The output of maximum capacity doesn't match the format of the current capacity and capacity. the latter two use float while the maximum is specified as int: Queue Name : default Queue State : running Scheduling Info : queueName: default, capacity: 0.7, maximumCapacity: 90.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test Queue State : running Scheduling Info : queueName: test, capacity: 0.2, maximumCapacity: -1.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test2 Queue State : running Scheduling Info : queueName: test2, capacity: 0.1, maximumCapacity: 5.0, currentCapacity: 0.0, state: Q_RUNNING, == here default is configured to have capacity=70% and maximum capacity = 90% -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3537) DefaultContainerExecutor has a race condn. with multiple concurrent containers
DefaultContainerExecutor has a race condn. with multiple concurrent containers -- Key: MAPREDUCE-3537 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3537 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Arun C Murthy Priority: Blocker -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3510) Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls
[ https://issues.apache.org/jira/browse/MAPREDUCE-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168036#comment-13168036 ] Hudson commented on MAPREDUCE-3510: --- Integrated in Hadoop-Hdfs-trunk-Commit #1478 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1478/]) MAPREDUCE-3510. Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls (Jonathan Eagles via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213511 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls - Key: MAPREDUCE-3510 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3510 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/capacity-sched, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.1 Attachments: MAPREDUCE-3510.patch, MAPREDUCE-3510.patch, MAPREDUCE-3510.patch, MAPREDUCE-3510.patch mapred queue -showacls does not show inherited acls -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3537) DefaultContainerExecutor has a race condn. with multiple concurrent containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3537: - Description: DCE relies cwd before calling ContainerLocalizer.runLocalization. However, with multiple containers setting cwd on same localFS reference leads to race. Assignee: Arun C Murthy DefaultContainerExecutor has a race condn. with multiple concurrent containers -- Key: MAPREDUCE-3537 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3537 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker DCE relies cwd before calling ContainerLocalizer.runLocalization. However, with multiple containers setting cwd on same localFS reference leads to race. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3328) mapred queue -list output inconsistent and missing child queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168037#comment-13168037 ] Hudson commented on MAPREDUCE-3328: --- Integrated in Hadoop-Hdfs-trunk-Commit #1478 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1478/]) MAPREDUCE-3328. mapred queue -list output inconsistent and missing child queues. (Ravi Prakash via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213504 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobQueueClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java mapred queue -list output inconsistent and missing child queues --- Key: MAPREDUCE-3328 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3328 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Ravi Prakash Priority: Critical Attachments: MAPREDUCE-3328.branch-0.23.patch, MAPREDUCE-3328.trunk.patch, MAPREDUCE-3328.trunk.patch, MAPREDUCE-3328.trunk.patch When running mapred queue -list on a 0.23.0 cluster with capacity scheduler configured with child queues. In my case I have queues default, test1, and test2. test1 has subqueues of a1, a2. test2 has subqueues of a3 and a4. - the child queues do not show up - The output of maximum capacity doesn't match the format of the current capacity and capacity. the latter two use float while the maximum is specified as int: Queue Name : default Queue State : running Scheduling Info : queueName: default, capacity: 0.7, maximumCapacity: 90.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test Queue State : running Scheduling Info : queueName: test, capacity: 0.2, maximumCapacity: -1.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test2 Queue State : running Scheduling Info : queueName: test2, capacity: 0.1, maximumCapacity: 5.0, currentCapacity: 0.0, state: Q_RUNNING, == here default is configured to have capacity=70% and maximum capacity = 90% -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3537) DefaultContainerExecutor has a race condn. with multiple concurrent containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3537: - Attachment: MAPREDUCE-3537.patch Quick fix to add synchronization to unblock Pig, Oozie etc. We should redo the DCE more throughly. DefaultContainerExecutor has a race condn. with multiple concurrent containers -- Key: MAPREDUCE-3537 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3537 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Attachments: MAPREDUCE-3537.patch DCE relies cwd before calling ContainerLocalizer.runLocalization. However, with multiple containers setting cwd on same localFS reference leads to race. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3537) DefaultContainerExecutor has a race condn. with multiple concurrent containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3537: - Target Version/s: 0.23.1 Status: Patch Available (was: Open) DefaultContainerExecutor has a race condn. with multiple concurrent containers -- Key: MAPREDUCE-3537 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3537 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Attachments: MAPREDUCE-3537.patch DCE relies cwd before calling ContainerLocalizer.runLocalization. However, with multiple containers setting cwd on same localFS reference leads to race. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3538) DCE shouldn't directly call ContainerLocalizer.runLocalization
DCE shouldn't directly call ContainerLocalizer.runLocalization -- Key: MAPREDUCE-3538 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3538 Project: Hadoop Map/Reduce Issue Type: Bug Components: nodemanager Affects Versions: 0.23.0 Reporter: Arun C Murthy Leads to races like MAPREDUCE-3537. We should probably fork the localizer similar to LCE. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3510) Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls
[ https://issues.apache.org/jira/browse/MAPREDUCE-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168040#comment-13168040 ] Hudson commented on MAPREDUCE-3510: --- Integrated in Hadoop-Common-trunk-Commit #1404 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1404/]) MAPREDUCE-3510. Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls (Jonathan Eagles via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213511 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls - Key: MAPREDUCE-3510 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3510 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/capacity-sched, mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.1 Attachments: MAPREDUCE-3510.patch, MAPREDUCE-3510.patch, MAPREDUCE-3510.patch, MAPREDUCE-3510.patch mapred queue -showacls does not show inherited acls -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3328) mapred queue -list output inconsistent and missing child queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168041#comment-13168041 ] Hudson commented on MAPREDUCE-3328: --- Integrated in Hadoop-Common-trunk-Commit #1404 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1404/]) MAPREDUCE-3328. mapred queue -list output inconsistent and missing child queues. (Ravi Prakash via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213504 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobQueueClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java mapred queue -list output inconsistent and missing child queues --- Key: MAPREDUCE-3328 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3328 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Ravi Prakash Priority: Critical Attachments: MAPREDUCE-3328.branch-0.23.patch, MAPREDUCE-3328.trunk.patch, MAPREDUCE-3328.trunk.patch, MAPREDUCE-3328.trunk.patch When running mapred queue -list on a 0.23.0 cluster with capacity scheduler configured with child queues. In my case I have queues default, test1, and test2. test1 has subqueues of a1, a2. test2 has subqueues of a3 and a4. - the child queues do not show up - The output of maximum capacity doesn't match the format of the current capacity and capacity. the latter two use float while the maximum is specified as int: Queue Name : default Queue State : running Scheduling Info : queueName: default, capacity: 0.7, maximumCapacity: 90.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test Queue State : running Scheduling Info : queueName: test, capacity: 0.2, maximumCapacity: -1.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test2 Queue State : running Scheduling Info : queueName: test2, capacity: 0.1, maximumCapacity: 5.0, currentCapacity: 0.0, state: Q_RUNNING, == here default is configured to have capacity=70% and maximum capacity = 90% -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3522) Capacity Scheduler ACLs not inherited by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168057#comment-13168057 ] Hadoop QA commented on MAPREDUCE-3522: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507068/MAPREDUCE-3522.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1427//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1427//console This message is automatically generated. Capacity Scheduler ACLs not inherited by default Key: MAPREDUCE-3522 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3522 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3522.patch, MAPREDUCE-3522.patch Hierarchical Queues do not inherit parent ACLs correctly by default. Instead, if no value is specified for submit or administer acls, then all access is granted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3522) Capacity Scheduler ACLs not inherited by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168070#comment-13168070 ] Jonathan Eagles commented on MAPREDUCE-3522: bq. @jon, the patch looks good. Is there anywhere we can document this? Would hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm be a good place? Good catch. I'll go ahead and add this as part of this JIRA so the documentation keeps up with the behavior. Capacity Scheduler ACLs not inherited by default Key: MAPREDUCE-3522 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3522 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3522.patch, MAPREDUCE-3522.patch Hierarchical Queues do not inherit parent ACLs correctly by default. Instead, if no value is specified for submit or administer acls, then all access is granted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3426) uber-jobs tried to write outputs into wrong dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3426: --- Attachment: MR-3426.3.patch addressed comments. uber-jobs tried to write outputs into wrong dir --- Key: MAPREDUCE-3426 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3426 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3426.1.patch, MR-3426.2.patch, MR-3426.3.patch Incorrect setup of the uber tasks causes tasks to try to write intermidiate outputs into dirs that the user does not have permissions to write to on a secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3426) uber-jobs tried to write outputs into wrong dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3426: --- Status: Patch Available (was: Open) uber-jobs tried to write outputs into wrong dir --- Key: MAPREDUCE-3426 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3426 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3426.1.patch, MR-3426.2.patch, MR-3426.3.patch Incorrect setup of the uber tasks causes tasks to try to write intermidiate outputs into dirs that the user does not have permissions to write to on a secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3522) Capacity Scheduler ACLs not inherited by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3522: --- Attachment: MAPREDUCE-3522.patch Capacity Scheduler ACLs not inherited by default Key: MAPREDUCE-3522 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3522 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3522.patch, MAPREDUCE-3522.patch, MAPREDUCE-3522.patch Hierarchical Queues do not inherit parent ACLs correctly by default. Instead, if no value is specified for submit or administer acls, then all access is granted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3522) Capacity Scheduler ACLs not inherited by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3522: --- Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1) Status: Open (was: Patch Available) Capacity Scheduler ACLs not inherited by default Key: MAPREDUCE-3522 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3522 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3522.patch, MAPREDUCE-3522.patch, MAPREDUCE-3522.patch Hierarchical Queues do not inherit parent ACLs correctly by default. Instead, if no value is specified for submit or administer acls, then all access is granted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3522) Capacity Scheduler ACLs not inherited by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3522: --- Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1) Status: Patch Available (was: Open) Capacity Scheduler ACLs not inherited by default Key: MAPREDUCE-3522 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3522 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3522.patch, MAPREDUCE-3522.patch, MAPREDUCE-3522.patch Hierarchical Queues do not inherit parent ACLs correctly by default. Instead, if no value is specified for submit or administer acls, then all access is granted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3426) uber-jobs tried to write outputs into wrong dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168123#comment-13168123 ] Hadoop QA commented on MAPREDUCE-3426: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507118/MR-3426.3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1431//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1431//console This message is automatically generated. uber-jobs tried to write outputs into wrong dir --- Key: MAPREDUCE-3426 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3426 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3426.1.patch, MR-3426.2.patch, MR-3426.3.patch Incorrect setup of the uber tasks causes tasks to try to write intermidiate outputs into dirs that the user does not have permissions to write to on a secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3537) DefaultContainerExecutor has a race condn. with multiple concurrent containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168121#comment-13168121 ] Hadoop QA commented on MAPREDUCE-3537: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507110/MAPREDUCE-3537.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1429//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1429//console This message is automatically generated. DefaultContainerExecutor has a race condn. with multiple concurrent containers -- Key: MAPREDUCE-3537 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3537 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Attachments: MAPREDUCE-3537.patch DCE relies cwd before calling ContainerLocalizer.runLocalization. However, with multiple containers setting cwd on same localFS reference leads to race. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3522) Capacity Scheduler ACLs not inherited by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168122#comment-13168122 ] Hadoop QA commented on MAPREDUCE-3522: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507127/MAPREDUCE-3522.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1430//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1430//console This message is automatically generated. Capacity Scheduler ACLs not inherited by default Key: MAPREDUCE-3522 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3522 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3522.patch, MAPREDUCE-3522.patch, MAPREDUCE-3522.patch Hierarchical Queues do not inherit parent ACLs correctly by default. Instead, if no value is specified for submit or administer acls, then all access is granted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3522) Capacity Scheduler ACLs not inherited by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168126#comment-13168126 ] Jonathan Eagles commented on MAPREDUCE-3522: The documentation has been update to reflect the current behavior. I have also verified that mvn site:site still produces valid html documentation and that my new changes are present and correct. Capacity Scheduler ACLs not inherited by default Key: MAPREDUCE-3522 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3522 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-3522.patch, MAPREDUCE-3522.patch, MAPREDUCE-3522.patch Hierarchical Queues do not inherit parent ACLs correctly by default. Instead, if no value is specified for submit or administer acls, then all access is granted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-778) [Rumen] Need a standalone JobHistory log anonymizer
[ https://issues.apache.org/jira/browse/MAPREDUCE-778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-778: - Fix Version/s: 0.24.0 Affects Version/s: 0.24.0 Status: Patch Available (was: Open) Running the latest patch (v1.14-12) through Hudson. [Rumen] Need a standalone JobHistory log anonymizer --- Key: MAPREDUCE-778 URL: https://issues.apache.org/jira/browse/MAPREDUCE-778 Project: Hadoop Map/Reduce Issue Type: New Feature Components: tools/rumen Affects Versions: 0.24.0 Reporter: Hong Tang Assignee: Amar Kamat Labels: anonymization, rumen Fix For: 0.24.0 Attachments: anonymizer.patch, anonymizer.py, mapreduce-778-v1.14-12.patch, mapreduce-778-v1.2-2.patch, same.py Job history logs contain a rich set of information that can help understand and characterize cluster workload and individual job execution. Examples of work that parses or utilizes job history include HADOOP-3585, MAPREDUCE-534, HDFS-459, MAPREDUCE-728, and MAPREDUCE-776. Some of the parsing tools developed in previous work already contains a component to anonymize the logs. It would be nice to combine these effort and have a common standalone tool that can anonymizes job history logs and preserve much of the structure of the files so that existing tools on top of job history logs continue work with no modification. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3398) Log Aggregation broken in Secure Mode
[ https://issues.apache.org/jira/browse/MAPREDUCE-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168132#comment-13168132 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3398: Few comments on the patch. NodeStatusPBImpl.setKeepAliveApplications(): clears containerStatuses? (!) I guess we can knock off YarnConfiguration.RM_DELEGATION_TOKEN_KEEPALIVE_TIME_MS altogether to instead use the NM-expiry interval itself. This configuration anyways needs to capped by the expiry interval. We may need it only if we ever want to lessen the keep-alive, but with the token-expiry usually in the range of 24 hours, we will hardly ever need to lessen it. After doing the above, we can drop the documentation changes too. removal-interval-ms doesn't need to be documented. NodeStatusUpdaterImpl: - createKeepAliveApplicationList(): The two loops can be rolled into one, a single loop iterating over appTokenKeepAliveMap? DelegationTokenRenewer: - May I suggesting renaming DelegationTokenRenewer.removeApplication() to applicationFinished() and removeApplicationInternal() to removeApplicationFromRenewal()? Log Aggregation broken in Secure Mode - Key: MAPREDUCE-3398 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3398 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Critical Fix For: 0.23.1 Attachments: MR3398_v1.txt, MR3398_v2.txt, MR3398_v3.txt Log aggregation in secure mode does not work with MAPREDUCE-2977. The nodemanager relies on the users credentials to write out logs to HDFS. These credentials are currently cancelled once a job completes, before the NM can write out the logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-778) [Rumen] Need a standalone JobHistory log anonymizer
[ https://issues.apache.org/jira/browse/MAPREDUCE-778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168140#comment-13168140 ] Hadoop QA commented on MAPREDUCE-778: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507028/mapreduce-778-v1.14-12.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 44 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1432//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1432//console This message is automatically generated. [Rumen] Need a standalone JobHistory log anonymizer --- Key: MAPREDUCE-778 URL: https://issues.apache.org/jira/browse/MAPREDUCE-778 Project: Hadoop Map/Reduce Issue Type: New Feature Components: tools/rumen Affects Versions: 0.24.0 Reporter: Hong Tang Assignee: Amar Kamat Labels: anonymization, rumen Fix For: 0.24.0 Attachments: anonymizer.patch, anonymizer.py, mapreduce-778-v1.14-12.patch, mapreduce-778-v1.2-2.patch, same.py Job history logs contain a rich set of information that can help understand and characterize cluster workload and individual job execution. Examples of work that parses or utilizes job history include HADOOP-3585, MAPREDUCE-534, HDFS-459, MAPREDUCE-728, and MAPREDUCE-776. Some of the parsing tools developed in previous work already contains a component to anonymize the logs. It would be nice to combine these effort and have a common standalone tool that can anonymizes job history logs and preserve much of the structure of the files so that existing tools on top of job history logs continue work with no modification. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3534) Compression runtime is increased by 13% as well as throughput is decreased by 24% in 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168142#comment-13168142 ] Vinay Kumar Thota commented on MAPREDUCE-3534: -- Ran compression benchmark with following environments. Codec type : BLOCK Codec : com.hadoop.compression.lzo.LzoCodec Compression runtime is increased by 13% as well as throughput is decreased by 24% in 0.23 -- Key: MAPREDUCE-3534 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3534 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Vinay Kumar Thota Priority: Critical Compression runtime is increased by 13% as well as throughput is decreased by 24% in 0.23 when compared to 0.20.204 on a 350 node size cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2637) Providing options to debug the mapreduce user code (Mapper, Reducer, Combiner, Sort implementations)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168157#comment-13168157 ] Ravi Teja Ch N V commented on MAPREDUCE-2637: - With IsolationRunner no longer functional(MAPREDUCE-2606),we need to find a alternative for it, as it makes the task debug a bit difficult. Providing options to debug the mapreduce user code (Mapper, Reducer, Combiner, Sort implementations) Key: MAPREDUCE-2637 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2637 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tasktracker Reporter: Devaraj K Assignee: Devaraj K Presently Hadoop provides mapred.child.java.opts configuration which can be used to set JVM options for Child JVM running Map or Reduce Task. If we need to remote debug the Child JVM, we can add remote debugging options to this configuration value. But this will work only for single Child JVM. Other children will fail as the remote debugging port is already used. We cannot specify the remote debugging port dynamically. As a result, it's not possible to remote debug multiple Child JVMs. As a solution to this problem, we can provide a configuration to debug Task JVMs in this scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3537) DefaultContainerExecutor has a race condn. with multiple concurrent containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168161#comment-13168161 ] Mahadev konar commented on MAPREDUCE-3537: -- +1 the patch looks good. DefaultContainerExecutor has a race condn. with multiple concurrent containers -- Key: MAPREDUCE-3537 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3537 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Attachments: MAPREDUCE-3537.patch DCE relies cwd before calling ContainerLocalizer.runLocalization. However, with multiple containers setting cwd on same localFS reference leads to race. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3537) DefaultContainerExecutor has a race condn. with multiple concurrent containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3537: - Fix Version/s: 0.23.1 DefaultContainerExecutor has a race condn. with multiple concurrent containers -- Key: MAPREDUCE-3537 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3537 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3537.patch DCE relies cwd before calling ContainerLocalizer.runLocalization. However, with multiple containers setting cwd on same localFS reference leads to race. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3537) DefaultContainerExecutor has a race condn. with multiple concurrent containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3537: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. DefaultContainerExecutor has a race condn. with multiple concurrent containers -- Key: MAPREDUCE-3537 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3537 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Attachments: MAPREDUCE-3537.patch DCE relies cwd before calling ContainerLocalizer.runLocalization. However, with multiple containers setting cwd on same localFS reference leads to race. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3537) DefaultContainerExecutor has a race condn. with multiple concurrent containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168172#comment-13168172 ] Hudson commented on MAPREDUCE-3537: --- Integrated in Hadoop-Hdfs-trunk-Commit #1480 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1480/]) MAPREDUCE-3537. Fix race condition in DefaultContainerExecutor which led to container localization occuring in wrong directories. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213575 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java DefaultContainerExecutor has a race condn. with multiple concurrent containers -- Key: MAPREDUCE-3537 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3537 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3537.patch DCE relies cwd before calling ContainerLocalizer.runLocalization. However, with multiple containers setting cwd on same localFS reference leads to race. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3537) DefaultContainerExecutor has a race condn. with multiple concurrent containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168171#comment-13168171 ] Hudson commented on MAPREDUCE-3537: --- Integrated in Hadoop-Hdfs-0.23-Commit #264 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/264/]) Merge -c 1213575 from trunk to branch-0.23 to fix MAPREDUCE-3537. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213576 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java DefaultContainerExecutor has a race condn. with multiple concurrent containers -- Key: MAPREDUCE-3537 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3537 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3537.patch DCE relies cwd before calling ContainerLocalizer.runLocalization. However, with multiple containers setting cwd on same localFS reference leads to race. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3534) Compression benchmark run-time increased by 13% in 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Kumar Thota updated MAPREDUCE-3534: - Summary: Compression benchmark run-time increased by 13% in 0.23 (was: Compression runtime is increased by 13% as well as throughput is decreased by 24% in 0.23 ) Compression benchmark run-time increased by 13% in 0.23 --- Key: MAPREDUCE-3534 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3534 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Vinay Kumar Thota Priority: Critical Compression runtime is increased by 13% as well as throughput is decreased by 24% in 0.23 when compared to 0.20.204 on a 350 node size cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3537) DefaultContainerExecutor has a race condn. with multiple concurrent containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168176#comment-13168176 ] Hudson commented on MAPREDUCE-3537: --- Integrated in Hadoop-Common-0.23-Commit #274 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/274/]) Merge -c 1213575 from trunk to branch-0.23 to fix MAPREDUCE-3537. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213576 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java DefaultContainerExecutor has a race condn. with multiple concurrent containers -- Key: MAPREDUCE-3537 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3537 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3537.patch DCE relies cwd before calling ContainerLocalizer.runLocalization. However, with multiple containers setting cwd on same localFS reference leads to race. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3537) DefaultContainerExecutor has a race condn. with multiple concurrent containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168175#comment-13168175 ] Hudson commented on MAPREDUCE-3537: --- Integrated in Hadoop-Common-trunk-Commit #1406 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1406/]) MAPREDUCE-3537. Fix race condition in DefaultContainerExecutor which led to container localization occuring in wrong directories. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213575 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java DefaultContainerExecutor has a race condn. with multiple concurrent containers -- Key: MAPREDUCE-3537 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3537 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3537.patch DCE relies cwd before calling ContainerLocalizer.runLocalization. However, with multiple containers setting cwd on same localFS reference leads to race. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3537) DefaultContainerExecutor has a race condn. with multiple concurrent containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168178#comment-13168178 ] Hudson commented on MAPREDUCE-3537: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1430 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1430/]) MAPREDUCE-3537. Fix race condition in DefaultContainerExecutor which led to container localization occuring in wrong directories. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213575 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java DefaultContainerExecutor has a race condn. with multiple concurrent containers -- Key: MAPREDUCE-3537 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3537 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3537.patch DCE relies cwd before calling ContainerLocalizer.runLocalization. However, with multiple containers setting cwd on same localFS reference leads to race. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3537) DefaultContainerExecutor has a race condn. with multiple concurrent containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168181#comment-13168181 ] Hudson commented on MAPREDUCE-3537: --- Integrated in Hadoop-Mapreduce-0.23-Commit #286 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/286/]) Merge -c 1213575 from trunk to branch-0.23 to fix MAPREDUCE-3537. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213576 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java DefaultContainerExecutor has a race condn. with multiple concurrent containers -- Key: MAPREDUCE-3537 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3537 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3537.patch DCE relies cwd before calling ContainerLocalizer.runLocalization. However, with multiple containers setting cwd on same localFS reference leads to race. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
