[jira] [Commented] (MAPREDUCE-4807) Allow MapOutputBuffer to be pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13533344#comment-13533344 ] Hudson commented on MAPREDUCE-4807: --- Integrated in Hadoop-Yarn-trunk #67 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/67/]) MAPREDUCE-4807. Allow MapOutputBuffer to be pluggable. (masokan via tucu) (Revision 1422345) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1422345 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IndexRecord.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapOutputCollector.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SpillRecord.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMerge.java Allow MapOutputBuffer to be pluggable - Key: MAPREDUCE-4807 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4807 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.0.2-alpha Reporter: Arun C Murthy Assignee: Mariappan Asokan Fix For: trunk Attachments: COMBO-mapreduce-4809-4807.patch, COMBO-mapreduce-4809-4807.patch, COMBO-mapreduce-4809-4807.patch, mapreduce-4807.patch, mapreduce-4807.patch, mapreduce-4807.patch, mapreduce-4807.patch, mapreduce-4807.patch, mapreduce-4807.patch Allow MapOutputBuffer to be pluggable -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4809) Change visibility of classes for pluggable sort changes
[ https://issues.apache.org/jira/browse/MAPREDUCE-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13533348#comment-13533348 ] Hudson commented on MAPREDUCE-4809: --- Integrated in Hadoop-Yarn-trunk #67 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/67/]) MAPREDUCE-4809. Change visibility of classes for pluggable sort changes. (masokan via tucu) (Revision 1422341) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1422341 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ExceptionReporter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MapHost.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MapOutput.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Shuffle.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleClientMetrics.java Change visibility of classes for pluggable sort changes --- Key: MAPREDUCE-4809 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4809 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.0.2-alpha Reporter: Arun C Murthy Assignee: Mariappan Asokan Fix For: trunk Attachments: MAPREDUCE-4809-1.patch, mapreduce-4809.patch, mapreduce-4809.patch, mapreduce-4809.patch Make classes required for MAPREDUCE-2454 to be java public (with LimitedPrivate) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4809) Change visibility of classes for pluggable sort changes
[ https://issues.apache.org/jira/browse/MAPREDUCE-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13533386#comment-13533386 ] Hudson commented on MAPREDUCE-4809: --- Integrated in Hadoop-Hdfs-trunk #1256 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1256/]) MAPREDUCE-4809. Change visibility of classes for pluggable sort changes. (masokan via tucu) (Revision 1422341) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1422341 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ExceptionReporter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MapHost.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MapOutput.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Shuffle.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleClientMetrics.java Change visibility of classes for pluggable sort changes --- Key: MAPREDUCE-4809 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4809 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.0.2-alpha Reporter: Arun C Murthy Assignee: Mariappan Asokan Fix For: trunk Attachments: MAPREDUCE-4809-1.patch, mapreduce-4809.patch, mapreduce-4809.patch, mapreduce-4809.patch Make classes required for MAPREDUCE-2454 to be java public (with LimitedPrivate) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4807) Allow MapOutputBuffer to be pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13533382#comment-13533382 ] Hudson commented on MAPREDUCE-4807: --- Integrated in Hadoop-Hdfs-trunk #1256 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1256/]) MAPREDUCE-4807. Allow MapOutputBuffer to be pluggable. (masokan via tucu) (Revision 1422345) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1422345 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IndexRecord.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapOutputCollector.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SpillRecord.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMerge.java Allow MapOutputBuffer to be pluggable - Key: MAPREDUCE-4807 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4807 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.0.2-alpha Reporter: Arun C Murthy Assignee: Mariappan Asokan Fix For: trunk Attachments: COMBO-mapreduce-4809-4807.patch, COMBO-mapreduce-4809-4807.patch, COMBO-mapreduce-4809-4807.patch, mapreduce-4807.patch, mapreduce-4807.patch, mapreduce-4807.patch, mapreduce-4807.patch, mapreduce-4807.patch, mapreduce-4807.patch Allow MapOutputBuffer to be pluggable -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4807) Allow MapOutputBuffer to be pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13533392#comment-13533392 ] Hudson commented on MAPREDUCE-4807: --- Integrated in Hadoop-Mapreduce-trunk #1287 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1287/]) MAPREDUCE-4807. Allow MapOutputBuffer to be pluggable. (masokan via tucu) (Revision 1422345) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1422345 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IndexRecord.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapOutputCollector.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SpillRecord.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMerge.java Allow MapOutputBuffer to be pluggable - Key: MAPREDUCE-4807 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4807 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.0.2-alpha Reporter: Arun C Murthy Assignee: Mariappan Asokan Fix For: trunk Attachments: COMBO-mapreduce-4809-4807.patch, COMBO-mapreduce-4809-4807.patch, COMBO-mapreduce-4809-4807.patch, mapreduce-4807.patch, mapreduce-4807.patch, mapreduce-4807.patch, mapreduce-4807.patch, mapreduce-4807.patch, mapreduce-4807.patch Allow MapOutputBuffer to be pluggable -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4809) Change visibility of classes for pluggable sort changes
[ https://issues.apache.org/jira/browse/MAPREDUCE-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13533396#comment-13533396 ] Hudson commented on MAPREDUCE-4809: --- Integrated in Hadoop-Mapreduce-trunk #1287 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1287/]) MAPREDUCE-4809. Change visibility of classes for pluggable sort changes. (masokan via tucu) (Revision 1422341) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1422341 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ExceptionReporter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MapHost.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MapOutput.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Shuffle.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleClientMetrics.java Change visibility of classes for pluggable sort changes --- Key: MAPREDUCE-4809 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4809 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.0.2-alpha Reporter: Arun C Murthy Assignee: Mariappan Asokan Fix For: trunk Attachments: MAPREDUCE-4809-1.patch, mapreduce-4809.patch, mapreduce-4809.patch, mapreduce-4809.patch Make classes required for MAPREDUCE-2454 to be java public (with LimitedPrivate) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4859) TestRecoveryManager fails on branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13533421#comment-13533421 ] Amir Sanjar commented on MAPREDUCE-4859: is there a patch for release 1.1.1? .. thanks TestRecoveryManager fails on branch-1 - Key: MAPREDUCE-4859 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4859 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.1 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 1.1.2 Attachments: MAPREDUCE-4859.patch, MAPREDUCE-4859.patch, MAPREDUCE-4859.patch Looks like the tests are extremely flaky and just hang. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4882) Error in estimating the length of the output file in Spill Phase
Lijie Xu created MAPREDUCE-4882:
---
Summary: Error in estimating the length of the output file in
Spill Phase
Key: MAPREDUCE-4882
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4882
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.0.3, 0.20.2
Environment: Any Environment
Reporter: Lijie Xu
The sortAndSpill() method in MapTask.java has an error in estimating the length
of the output file.
The long size should be (bufvoid - bufstart) + bufend not (bufvoid -
bufend) + bufstart when bufend bufstart.
Here is the original code in MapTask.java.
private void sortAndSpill() throws IOException, ClassNotFoundException,
InterruptedException {
//approximate the length of the output file to be the length of the
//buffer + header lengths for the partitions
long size = (bufend = bufstart
? bufend - bufstart
: (bufvoid - bufend) + bufstart) +
partitions * APPROX_HEADER_LENGTH;
FSDataOutputStream out = null;
--
I had a test on TeraSort. A snippet from mapper's log is as follows:
MapTask: Spilling map output: record full = true
MapTask: bufstart = 157286200; bufend = 10485460; bufvoid = 199229440
MapTask: kvstart = 262142; kvend = 131069; length = 655360
MapTask: Finished spill 3
In this occasioin, Spill Bytes should be (199229440 - 157286200) + 10485460 =
52428700 (52 MB) because the number of spilled records is 524287 and each
record costs 100B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4883) Reducer's Maximum Shuffle Buffer Size should be enlarged for 64bit JVM
Lijie Xu created MAPREDUCE-4883:
---
Summary: Reducer's Maximum Shuffle Buffer Size should be enlarged
for 64bit JVM
Key: MAPREDUCE-4883
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4883
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 1.0.3, 0.20.2
Environment: Especially for 64bit JVM
Reporter: Lijie Xu
In hadoop-0.20.2, hadoop-1.0.3 or other versions, reducer's shuffle buffer size
cannot exceed 2048MB (i.e., Integer.MAX_VALUE). This is reasonable for 32bit
JVM.
But for 64bit JVM, although reducer's JVM size can be set more than 2048MB
(e.g., mapred.child.java.opts=-Xmx4000m), the heap size used for shuffle buffer
is at most 2048MB * maxInMemCopyUse (default 0.7) not 4000MB *
maxInMemCopyUse.
So the pointed piece of code in ReduceTask.java needs modification for 64bit
JVM.
---
private final long maxSize;
private final long maxSingleShuffleLimit;
private long size = 0;
private Object dataAvailable = new Object();
private long fullSize = 0;
private int numPendingRequests = 0;
private int numRequiredMapOutputs = 0;
private int numClosed = 0;
private boolean closed = false;
public ShuffleRamManager(Configuration conf) throws IOException {
final float maxInMemCopyUse =
conf.getFloat(mapred.job.shuffle.input.buffer.percent, 0.70f);
if (maxInMemCopyUse 1.0 || maxInMemCopyUse 0.0) {
throw new IOException(mapred.job.shuffle.input.buffer.percent +
maxInMemCopyUse);
}
// Allow unit tests to fix Runtime memory
-- maxSize = (int)(conf.getInt(mapred.job.reduce.total.mem.bytes,
--(int)Math.min(Runtime.getRuntime().maxMemory(), Integer.MAX_VALUE))
-- * maxInMemCopyUse);
maxSingleShuffleLimit = (long)(maxSize *
MAX_SINGLE_SHUFFLE_SEGMENT_FRACTION);
LOG.info(ShuffleRamManager: MemoryLimit= + maxSize +
, MaxSingleShuffleLimit= + maxSingleShuffleLimit);
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4884) streaming tests fail to start MiniMRCluster due to Queue configuration missing child queue names for root
Chris Nauroth created MAPREDUCE-4884:
Summary: streaming tests fail to start MiniMRCluster due to Queue
configuration missing child queue names for root
Key: MAPREDUCE-4884
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4884
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, test
Affects Versions: 3.0.0, trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Multiple tests in hadoop-streaming, such as {{TestFileArgs}}, fail to
initialize {{MiniMRCluster}} due to a {{YarnException}} with reason Queue
configuration missing child queue names for root.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4884) streaming tests fail to start MiniMRCluster due to Queue configuration missing child queue names for root
[
https://issues.apache.org/jira/browse/MAPREDUCE-4884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13533681#comment-13533681
]
Chris Nauroth commented on MAPREDUCE-4884:
--
{noformat}
org.apache.hadoop.yarn.YarnException: Failed to Start
org.apache.hadoop.mapred.MiniMRCluster
at
org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78)
at
org.apache.hadoop.mapred.MiniMRClientClusterFactory.create(MiniMRClientClusterFactory.java:75)
at org.apache.hadoop.mapred.MiniMRCluster.init(MiniMRCluster.java:180)
at org.apache.hadoop.mapred.MiniMRCluster.init(MiniMRCluster.java:170)
at org.apache.hadoop.mapred.MiniMRCluster.init(MiniMRCluster.java:162)
at org.apache.hadoop.mapred.MiniMRCluster.init(MiniMRCluster.java:154)
at org.apache.hadoop.mapred.MiniMRCluster.init(MiniMRCluster.java:147)
at org.apache.hadoop.mapred.MiniMRCluster.init(MiniMRCluster.java:140)
at org.apache.hadoop.mapred.MiniMRCluster.init(MiniMRCluster.java:133)
at org.apache.hadoop.mapred.MiniMRCluster.init(MiniMRCluster.java:128)
at org.apache.hadoop.streaming.TestFileArgs.init(TestFileArgs.java:57)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:209)
at
org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:258)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:255)
at
org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: org.apache.hadoop.yarn.YarnException:
java.lang.IllegalStateException: Queue configuration missing child queue names
for root
at
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:184)
at
org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
... 39 more
Caused by: java.lang.IllegalStateException: Queue configuration missing child
queue names for root
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:303)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:228)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:198)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.init(ResourceManager.java:201)
at
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:164)
... 40 more
[jira] [Updated] (MAPREDUCE-4884) streaming tests fail to start MiniMRCluster due to Queue configuration missing child queue names for root
[
https://issues.apache.org/jira/browse/MAPREDUCE-4884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated MAPREDUCE-4884:
-
Attachment: MAPREDUCE-4884.1.patch
The attached patch fixes the problem by adding a testResources stanza to the
hadoop-streaming pom.xml, which references the capacity-scheduler.xml
configuration file already present in the YARN project. This guarantees that a
copy of capacity-scheduler.xml is on the classpath when the streaming tests run.
streaming tests fail to start MiniMRCluster due to Queue configuration
missing child queue names for root
---
Key: MAPREDUCE-4884
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4884
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, test
Affects Versions: 3.0.0, trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4884.1.patch
Multiple tests in hadoop-streaming, such as {{TestFileArgs}}, fail to
initialize {{MiniMRCluster}} due to a {{YarnException}} with reason Queue
configuration missing child queue names for root.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4884) streaming tests fail to start MiniMRCluster due to Queue configuration missing child queue names for root
[
https://issues.apache.org/jira/browse/MAPREDUCE-4884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated MAPREDUCE-4884:
-
Status: Patch Available (was: Open)
streaming tests fail to start MiniMRCluster due to Queue configuration
missing child queue names for root
---
Key: MAPREDUCE-4884
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4884
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, test
Affects Versions: 3.0.0, trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4884.1.patch
Multiple tests in hadoop-streaming, such as {{TestFileArgs}}, fail to
initialize {{MiniMRCluster}} due to a {{YarnException}} with reason Queue
configuration missing child queue names for root.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (MAPREDUCE-4885) YARN tests have multiple failures on Windows due to Queue configuration missing
[ https://issues.apache.org/jira/browse/MAPREDUCE-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth moved YARN-260 to MAPREDUCE-4885: --- Component/s: (was: applications) test contrib/streaming Target Version/s: trunk-win (was: trunk-win) Affects Version/s: (was: trunk-win) trunk-win Key: MAPREDUCE-4885 (was: YARN-260) Project: Hadoop Map/Reduce (was: Hadoop YARN) YARN tests have multiple failures on Windows due to Queue configuration missing - Key: MAPREDUCE-4885 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4885 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, test Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth There are multiple test failures due to Queue configuration missing child queue names for root. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4885) streaming tests have multiple failures on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated MAPREDUCE-4885: - Summary: streaming tests have multiple failures on Windows (was: YARN tests have multiple failures on Windows due to Queue configuration missing) The patch attached to MAPREDUCE-4884 will resolve the original issue mentioned of Queue configuration missing child queue names for root. I am moving this jira from YARN to MAPREDUCE to record the current scope more accurately: fully working streaming tests on Windows. streaming tests have multiple failures on Windows - Key: MAPREDUCE-4885 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4885 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, test Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth There are multiple test failures due to Queue configuration missing child queue names for root. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4884) streaming tests fail to start MiniMRCluster due to Queue configuration missing child queue names for root
[
https://issues.apache.org/jira/browse/MAPREDUCE-4884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13533693#comment-13533693
]
Hadoop QA commented on MAPREDUCE-4884:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12561251/MAPREDUCE-4884.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-streaming.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3127//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3127//console
This message is automatically generated.
streaming tests fail to start MiniMRCluster due to Queue configuration
missing child queue names for root
---
Key: MAPREDUCE-4884
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4884
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, test
Affects Versions: 3.0.0, trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4884.1.patch
Multiple tests in hadoop-streaming, such as {{TestFileArgs}}, fail to
initialize {{MiniMRCluster}} due to a {{YarnException}} with reason Queue
configuration missing child queue names for root.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4884) streaming tests fail to start MiniMRCluster due to Queue configuration missing child queue names for root
[
https://issues.apache.org/jira/browse/MAPREDUCE-4884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13533696#comment-13533696
]
Chris Nauroth commented on MAPREDUCE-4884:
--
Jenkins gave -1 for no new tests, but this patch fixes multiple existing tests.
streaming tests fail to start MiniMRCluster due to Queue configuration
missing child queue names for root
---
Key: MAPREDUCE-4884
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4884
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, test
Affects Versions: 3.0.0, trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4884.1.patch
Multiple tests in hadoop-streaming, such as {{TestFileArgs}}, fail to
initialize {{MiniMRCluster}} due to a {{YarnException}} with reason Queue
configuration missing child queue names for root.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
