[jira] [Updated] (MAPREDUCE-5004) Somebody working on Genetic Algorithm library on Map Reduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Singh Rajput updated MAPREDUCE-5004: Assignee: Vaibhav Singh Rajput Somebody working on Genetic Algorithm library on Map Reduce --- Key: MAPREDUCE-5004 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5004 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Abhishek Bajpai Assignee: Vaibhav Singh Rajput -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3801) org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator fails intermittently
[ https://issues.apache.org/jira/browse/MAPREDUCE-3801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13634881#comment-13634881 ] Tsuyoshi OZAWA commented on MAPREDUCE-3801: --- I got similar results. This is the result of testing: $ mvn test -Dtest=TestRuntimeEstimators --- T E S T S --- Running org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 38.515 sec FAILURE! testLegacyEstimator(org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators) Time elapsed: 20276 sec FAILURE! java.lang.AssertionError: We got the wrong number of successful speculations. expected:3 but was:5 at org.junit.Assert.fail(Assert.java:91) at org.junit.Assert.failNotEquals(Assert.java:645) at org.junit.Assert.assertEquals(Assert.java:126) at org.junit.Assert.assertEquals(Assert.java:470) at org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.coreTestEstimator(TestRuntimeEstimators.java:222) at org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testLegacyEstimator(TestRuntimeEstimators.java:229) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) Results : Failed tests: testLegacyEstimator(org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators): We got the wrong number of successful speculations. expected:3 but was:5 org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator fails intermittently -- Key: MAPREDUCE-3801 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3801 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0 Reporter: Robert Joseph Evans Attachments: org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators-output.txt, org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.txt, TEST-org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.xml org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator fails intermittently -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA
[jira] [Updated] (MAPREDUCE-3801) org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator fails intermittently
[ https://issues.apache.org/jira/browse/MAPREDUCE-3801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-3801: -- Assignee: Tsuyoshi OZAWA org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator fails intermittently -- Key: MAPREDUCE-3801 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3801 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0 Reporter: Robert Joseph Evans Assignee: Tsuyoshi OZAWA Attachments: org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators-output.txt, org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.txt, TEST-org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.xml org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator fails intermittently -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5128) mapred-default.xml is missing a bunch of history server configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635077#comment-13635077 ] Hudson commented on MAPREDUCE-5128: --- Integrated in Hadoop-Yarn-trunk #187 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/187/]) MAPREDUCE-5128. mapred-default.xml is missing a bunch of history server configs. (sandyr via tucu) (Revision 1469095) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1469095 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml mapred-default.xml is missing a bunch of history server configs --- Key: MAPREDUCE-5128 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5128 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation, jobhistoryserver Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.0.5-beta Attachments: MAPREDUCE-5128.patch mapred-default.xml is missing many configs that work for the job history server. mapreduce.jobhistory.cleaner.enable, mapreduce.jobhistory.done-dir, and mapreduce.jobhistory.datestring.cache.size are a few examples. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5140) MR part of YARN-514
[ https://issues.apache.org/jira/browse/MAPREDUCE-5140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635082#comment-13635082 ] Hudson commented on MAPREDUCE-5140: --- Integrated in Hadoop-Yarn-trunk #187 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/187/]) MAPREDUCE-5140. MR part of YARN-514 (Zhijie Shen via bikas) (Revision 1469060) Result = FAILURE bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1469060 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java MR part of YARN-514 --- Key: MAPREDUCE-5140 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5140 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5140.1.patch In YARN-514, application store needs to be delayed to unblock application submission, such that a new state of MRApp needs to be created. On mapreduce side, there's some function to map yarn states to mapreduce ones. This mapping needs to be updated due to the newly added state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3801) org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator fails intermittently
[ https://issues.apache.org/jira/browse/MAPREDUCE-3801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-3801: -- Assignee: (was: Tsuyoshi OZAWA) org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator fails intermittently -- Key: MAPREDUCE-3801 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3801 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0 Reporter: Robert Joseph Evans Attachments: org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators-output.txt, org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.txt, TEST-org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.xml org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator fails intermittently -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5128) mapred-default.xml is missing a bunch of history server configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635133#comment-13635133 ] Hudson commented on MAPREDUCE-5128: --- Integrated in Hadoop-Hdfs-trunk #1376 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1376/]) MAPREDUCE-5128. mapred-default.xml is missing a bunch of history server configs. (sandyr via tucu) (Revision 1469095) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1469095 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml mapred-default.xml is missing a bunch of history server configs --- Key: MAPREDUCE-5128 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5128 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation, jobhistoryserver Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.0.5-beta Attachments: MAPREDUCE-5128.patch mapred-default.xml is missing many configs that work for the job history server. mapreduce.jobhistory.cleaner.enable, mapreduce.jobhistory.done-dir, and mapreduce.jobhistory.datestring.cache.size are a few examples. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5140) MR part of YARN-514
[ https://issues.apache.org/jira/browse/MAPREDUCE-5140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635138#comment-13635138 ] Hudson commented on MAPREDUCE-5140: --- Integrated in Hadoop-Hdfs-trunk #1376 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1376/]) MAPREDUCE-5140. MR part of YARN-514 (Zhijie Shen via bikas) (Revision 1469060) Result = FAILURE bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1469060 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java MR part of YARN-514 --- Key: MAPREDUCE-5140 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5140 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5140.1.patch In YARN-514, application store needs to be delayed to unblock application submission, such that a new state of MRApp needs to be created. On mapreduce side, there's some function to map yarn states to mapreduce ones. This mapping needs to be updated due to the newly added state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5164) command mapred job and mapred queue omit HADOOP_CLIENT_OPTS
nemon lou created MAPREDUCE-5164: Summary: command mapred job and mapred queue omit HADOOP_CLIENT_OPTS Key: MAPREDUCE-5164 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5164 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: nemon lou Assignee: nemon lou HADOOP_CLIENT_OPTS does not take effect when type mapred job -list and mapred queue -list. The mapred script omit it -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4383) HadoopPipes.cc needs to include unistd.h
[
https://issues.apache.org/jira/browse/MAPREDUCE-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635166#comment-13635166
]
Ravi Prakash commented on MAPREDUCE-4383:
-
Hi,
Could this please be committed to 0.23 too?
HadoopPipes.cc needs to include unistd.h
Key: MAPREDUCE-4383
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4383
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: pipes
Affects Versions: 2.0.0-alpha
Reporter: Andy Isaacson
Assignee: Andy Isaacson
Priority: Minor
Fix For: 2.0.2-alpha
Attachments: mapreduce-4383.txt
Since MAPREDUCE-4267 I've seen mvn -Pnative compile failing with:
{quote}
[exec] /usr/bin/c++-g -Wall -O2 -D_REENTRANT -D_FILE_OFFSET_BITS=64
-I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/utils/api
-I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/api
-I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src-o
CMakeFiles/hadooppipes.dir/main/native/pipes/impl/HadoopPipes.cc.o -c
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
In member function ‘std::string
HadoopPipes::BinaryProtocol::createDigest(std::string, std::string)’:
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:439:21:
warning: value computed is not used [-Wunused-value]
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
In function ‘void* HadoopPipes::ping(void*)’:
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1049:16:
error: ‘sleep’ was not declared in this scope
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1067:30:
error: ‘close’ was not declared in this scope
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
In function ‘bool HadoopPipes::runTask(const HadoopPipes::Factory)’:
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1162:28:
error: ‘close’ was not declared in this scope
[exec] make[2]: ***
[CMakeFiles/hadooppipes.dir/main/native/pipes/impl/HadoopPipes.cc.o] Error 1
{quote}
I believe the failure is new simply because I wasn't compiling pipes before.
The fix is pretty simple, just include unistd.h in HadoopPipes.cc.
My environment is debian unstable, amd64, g++ 4.7.0-6, openjdk-6-jdk
6b24-1.11.1-6.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5164) command mapred job and mapred queue omit HADOOP_CLIENT_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nemon lou updated MAPREDUCE-5164: - Attachment: MAPREDUCE-5164.patch command mapred job and mapred queue omit HADOOP_CLIENT_OPTS - Key: MAPREDUCE-5164 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5164 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: nemon lou Assignee: nemon lou Attachments: MAPREDUCE-5164.patch HADOOP_CLIENT_OPTS does not take effect when type mapred job -list and mapred queue -list. The mapred script omit it -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5128) mapred-default.xml is missing a bunch of history server configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635176#comment-13635176 ] Hudson commented on MAPREDUCE-5128: --- Integrated in Hadoop-Mapreduce-trunk #1403 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1403/]) MAPREDUCE-5128. mapred-default.xml is missing a bunch of history server configs. (sandyr via tucu) (Revision 1469095) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1469095 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml mapred-default.xml is missing a bunch of history server configs --- Key: MAPREDUCE-5128 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5128 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation, jobhistoryserver Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.0.5-beta Attachments: MAPREDUCE-5128.patch mapred-default.xml is missing many configs that work for the job history server. mapreduce.jobhistory.cleaner.enable, mapreduce.jobhistory.done-dir, and mapreduce.jobhistory.datestring.cache.size are a few examples. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5140) MR part of YARN-514
[ https://issues.apache.org/jira/browse/MAPREDUCE-5140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635181#comment-13635181 ] Hudson commented on MAPREDUCE-5140: --- Integrated in Hadoop-Mapreduce-trunk #1403 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1403/]) MAPREDUCE-5140. MR part of YARN-514 (Zhijie Shen via bikas) (Revision 1469060) Result = SUCCESS bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1469060 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java MR part of YARN-514 --- Key: MAPREDUCE-5140 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5140 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5140.1.patch In YARN-514, application store needs to be delayed to unblock application submission, such that a new state of MRApp needs to be created. On mapreduce side, there's some function to map yarn states to mapreduce ones. This mapping needs to be updated due to the newly added state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4383) HadoopPipes.cc needs to include unistd.h
[
https://issues.apache.org/jira/browse/MAPREDUCE-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves updated MAPREDUCE-4383:
-
Fix Version/s: 0.23.8
I merged into branch-0.23
HadoopPipes.cc needs to include unistd.h
Key: MAPREDUCE-4383
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4383
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: pipes
Affects Versions: 2.0.0-alpha
Reporter: Andy Isaacson
Assignee: Andy Isaacson
Priority: Minor
Fix For: 2.0.2-alpha, 0.23.8
Attachments: mapreduce-4383.txt
Since MAPREDUCE-4267 I've seen mvn -Pnative compile failing with:
{quote}
[exec] /usr/bin/c++-g -Wall -O2 -D_REENTRANT -D_FILE_OFFSET_BITS=64
-I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/utils/api
-I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/api
-I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src-o
CMakeFiles/hadooppipes.dir/main/native/pipes/impl/HadoopPipes.cc.o -c
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
In member function ‘std::string
HadoopPipes::BinaryProtocol::createDigest(std::string, std::string)’:
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:439:21:
warning: value computed is not used [-Wunused-value]
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
In function ‘void* HadoopPipes::ping(void*)’:
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1049:16:
error: ‘sleep’ was not declared in this scope
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1067:30:
error: ‘close’ was not declared in this scope
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
In function ‘bool HadoopPipes::runTask(const HadoopPipes::Factory)’:
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1162:28:
error: ‘close’ was not declared in this scope
[exec] make[2]: ***
[CMakeFiles/hadooppipes.dir/main/native/pipes/impl/HadoopPipes.cc.o] Error 1
{quote}
I believe the failure is new simply because I wasn't compiling pipes before.
The fix is pretty simple, just include unistd.h in HadoopPipes.cc.
My environment is debian unstable, amd64, g++ 4.7.0-6, openjdk-6-jdk
6b24-1.11.1-6.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5147) Maven build should create hadoop-mapreduce-client-app-VERSION.jar directly
[ https://issues.apache.org/jira/browse/MAPREDUCE-5147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated MAPREDUCE-5147: - Attachment: MAPREDUCE-5147-branch2.patch MAPREDUCE-5147-branch0.23.patch Maven build should create hadoop-mapreduce-client-app-VERSION.jar directly -- Key: MAPREDUCE-5147 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5147 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: trunk Reporter: Robert Parker Assignee: Robert Parker Attachments: MAPREDUCE-5147-branch0.23.patch, MAPREDUCE-5147-branch2.patch, MAPREDUCE-5147.patch Currently the build creates mr-app.jar and links it to the proper name. All hard links to mr-app.jar appear to have been removed. The maven build should be simplified to directly build the jar. Related -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5147) Maven build should create hadoop-mapreduce-client-app-VERSION.jar directly
[
https://issues.apache.org/jira/browse/MAPREDUCE-5147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635232#comment-13635232
]
Hadoop QA commented on MAPREDUCE-5147:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12579334/MAPREDUCE-5147-branch2.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3536//console
This message is automatically generated.
Maven build should create hadoop-mapreduce-client-app-VERSION.jar directly
--
Key: MAPREDUCE-5147
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5147
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: trunk
Reporter: Robert Parker
Assignee: Robert Parker
Attachments: MAPREDUCE-5147-branch0.23.patch,
MAPREDUCE-5147-branch2.patch, MAPREDUCE-5147.patch
Currently the build creates mr-app.jar and links it to the proper name. All
hard links to mr-app.jar appear to have been removed. The maven build should
be simplified to directly build the jar.
Related
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5165) Create MiniMRCluster version which uses the mapreduce package.
jay vyas created MAPREDUCE-5165: --- Summary: Create MiniMRCluster version which uses the mapreduce package. Key: MAPREDUCE-5165 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5165 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: jay vyas Priority: Minor The MiniMapRedCluster class references some older mapred.* classes. It could be recreated in the mapreduce package to use the Configuration class instead of JobConf, which would make it simpler to use and integrate with new FS implementations and test harnesses that use new Configuration (not JobConf) objects to drive tests. This could be done many ways: 1) using inheritance or else 2) by copying the code directly The appropriate implementation depends on wether or not 1) Is it okay for mapreduce.* classes to depend on mapred.* classes ? 2) Is the mapred MiniMRCluster implementation going to be deprecated or eliminated anytime? 3) What is the future of the JobConf class - which has been deprecated and then undeprecated ? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4898) FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635338#comment-13635338
]
Robert Kanter commented on MAPREDUCE-4898:
--
I created a job like you said and ran it on Hadoop 1 and Hadoop 2 with and
without setting the working dir. The behavior is the same in both Hadoop 1 and
Hadoop 2: without setting the working dir, it resolves a relative path to the
homedir; with setting the working dir, it resolves a relative path to that dir
(even in Hadoop 1). So no behavioral change here :)
FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath
incompatible with MR1
--
Key: MAPREDUCE-4898
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4898
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Robert Kanter
Assignee: Robert Kanter
Fix For: 2.0.5-beta
Attachments: MAPREDUCE-4898.patch
In MR1,
{{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs}}
throws {{org.apache.hadoop.mapred.FileAlreadyExistsException}} but now it
throws {{org.apache.hadoop.fs.FileAlreadyExistsException}} instead, making
them incompatible.
In MR1,
{{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath}}
doesn't throw any exceptions but now it throws an {{IOException}}, making
them incompatible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5165) Create MiniMRCluster version which uses the mapreduce package.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635378#comment-13635378 ] jay vyas commented on MAPREDUCE-5165: - I have just found that rehashes contents from another JIRA https://issues.apache.org/jira/browse/MAPREDUCE-3169... and there is a MiniMRYarnCluster in fact in the mapreduce package. So.. then... Is MiniMRCluster an artifact of the MR1 days that will be less used once MR2 takes over ? Is MiniMrYarnCluster a generic version of MiniMRCluster which will one day obviate the implementation specific MiniMRCluster altogether for the Hadoop-MapReduce jobs that are implemented in the MR2 YARN framework? Create MiniMRCluster version which uses the mapreduce package. -- Key: MAPREDUCE-5165 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5165 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: jay vyas Priority: Minor The MiniMapRedCluster class references some older mapred.* classes. It could be recreated in the mapreduce package to use the Configuration class instead of JobConf, which would make it simpler to use and integrate with new FS implementations and test harnesses that use new Configuration (not JobConf) objects to drive tests. This could be done many ways: 1) using inheritance or else 2) by copying the code directly The appropriate implementation depends on wether or not 1) Is it okay for mapreduce.* classes to depend on mapred.* classes ? 2) Is the mapred MiniMRCluster implementation going to be deprecated or eliminated anytime? 3) What is the future of the JobConf class - which has been deprecated and then undeprecated ? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5165) Create MiniMRCluster version which uses the mapreduce package.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jay vyas updated MAPREDUCE-5165: Description: The MiniMapRedCluster class references some older mapred.* classes (as per comments below however, there is the MiniMRYarnCluster, which may aim to replace it). It could be recreated in the mapreduce package to use the Configuration class instead of JobConf, which would make it simpler to use and integrate with new FS implementations and test harnesses that use new Configuration (not JobConf) objects to drive tests. This could be done many ways: 1) using inheritance or else 2) by copying the code directly The appropriate implementation depends on wether or not 1) Is it okay for mapreduce.* classes to depend on mapred.* classes ? 2) Is the mapred MiniMRCluster implementation going to be deprecated or eliminated anytime? 3) What is the future of the JobConf class - which has been deprecated and then undeprecated ? Note that This is all intimately linked to the role that MiniMRYarnCluster will play. Relevant classes: .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java was: The MiniMapRedCluster class references some older mapred.* classes. It could be recreated in the mapreduce package to use the Configuration class instead of JobConf, which would make it simpler to use and integrate with new FS implementations and test harnesses that use new Configuration (not JobConf) objects to drive tests. This could be done many ways: 1) using inheritance or else 2) by copying the code directly The appropriate implementation depends on wether or not 1) Is it okay for mapreduce.* classes to depend on mapred.* classes ? 2) Is the mapred MiniMRCluster implementation going to be deprecated or eliminated anytime? 3) What is the future of the JobConf class - which has been deprecated and then undeprecated ? Create MiniMRCluster version which uses the mapreduce package. -- Key: MAPREDUCE-5165 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5165 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: jay vyas Priority: Minor The MiniMapRedCluster class references some older mapred.* classes (as per comments below however, there is the MiniMRYarnCluster, which may aim to replace it). It could be recreated in the mapreduce package to use the Configuration class instead of JobConf, which would make it simpler to use and integrate with new FS implementations and test harnesses that use new Configuration (not JobConf) objects to drive tests. This could be done many ways: 1) using inheritance or else 2) by copying the code directly The appropriate implementation depends on wether or not 1) Is it okay for mapreduce.* classes to depend on mapred.* classes ? 2) Is the mapred MiniMRCluster implementation going to be deprecated or eliminated anytime? 3) What is the future of the JobConf class - which has been deprecated and then undeprecated ? Note that This is all intimately linked to the role that MiniMRYarnCluster will play. Relevant classes: .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5165) Create MiniMRCluster version which uses the mapreduce package.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635418#comment-13635418 ] Alejandro Abdelnur commented on MAPREDUCE-5165: --- MiniMRCluster has been added to Hadoop2 to enabled backward compatibility for downstream projects that used Hadoop1 MiniMRCluster in their testcases. In other words, to support legacy stuff. If you are starting something from scratch you should use the new MiniMRYarnCluster directly. IMO, I'd close this JIRA as not-a-problem. Create MiniMRCluster version which uses the mapreduce package. -- Key: MAPREDUCE-5165 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5165 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: jay vyas Priority: Minor The MiniMapRedCluster class references some older mapred.* classes (as per comments below however, there is the MiniMRYarnCluster, which may aim to replace it). It could be recreated in the mapreduce package to use the Configuration class instead of JobConf, which would make it simpler to use and integrate with new FS implementations and test harnesses that use new Configuration (not JobConf) objects to drive tests. This could be done many ways: 1) using inheritance or else 2) by copying the code directly The appropriate implementation depends on wether or not 1) Is it okay for mapreduce.* classes to depend on mapred.* classes ? 2) Is the mapred MiniMRCluster implementation going to be deprecated or eliminated anytime? 3) What is the future of the JobConf class - which has been deprecated and then undeprecated ? Note that This is all intimately linked to the role that MiniMRYarnCluster will play. Relevant classes: .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5165) Create MiniMRCluster version which uses the mapreduce package.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635448#comment-13635448 ] jay vyas commented on MAPREDUCE-5165: - +1 to close, but the deprecation story is somewhat tricky not sure how to improve it. Maybe just a wiki page update to http://wiki.apache.org/hadoop/HowToContribute to explain the changes to MRMiniCluster would be in order here, or something. Create MiniMRCluster version which uses the mapreduce package. -- Key: MAPREDUCE-5165 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5165 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: jay vyas Priority: Minor The MiniMapRedCluster class references some older mapred.* classes (as per comments below however, there is the MiniMRYarnCluster, which may aim to replace it). It could be recreated in the mapreduce package to use the Configuration class instead of JobConf, which would make it simpler to use and integrate with new FS implementations and test harnesses that use new Configuration (not JobConf) objects to drive tests. This could be done many ways: 1) using inheritance or else 2) by copying the code directly The appropriate implementation depends on wether or not 1) Is it okay for mapreduce.* classes to depend on mapred.* classes ? 2) Is the mapred MiniMRCluster implementation going to be deprecated or eliminated anytime? 3) What is the future of the JobConf class - which has been deprecated and then undeprecated ? Note that This is all intimately linked to the role that MiniMRYarnCluster will play. Relevant classes: .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5140) MR part of YARN-514
[ https://issues.apache.org/jira/browse/MAPREDUCE-5140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5140: --- Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed MR part of YARN-514 --- Key: MAPREDUCE-5140 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5140 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.0.5-beta Attachments: MAPREDUCE-5140.1.patch In YARN-514, application store needs to be delayed to unblock application submission, such that a new state of MRApp needs to be created. On mapreduce side, there's some function to map yarn states to mapreduce ones. This mapping needs to be updated due to the newly added state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5166) ConcurrentModificationException in LocalJobRunner
Gunther Hagleitner created MAPREDUCE-5166:
-
Summary: ConcurrentModificationException in LocalJobRunner
Key: MAPREDUCE-5166
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5166
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Gunther Hagleitner
Priority: Critical
With the latest version hive unit tests fail in various places with the
following stack trace. The problem seems related to: MAPREDUCE-2931
{noformat}
[junit] java.util.ConcurrentModificationException
[junit] at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
[junit] at java.util.HashMap$ValueIterator.next(HashMap.java:822)
[junit] at
org.apache.hadoop.mapred.Counters.incrAllCounters(Counters.java:505)
[junit] at org.apache.hadoop.mapred.Counters.sum(Counters.java:528)
[junit] at
org.apache.hadoop.mapred.LocalJobRunner$Job.getCurrentCounters(LocalJobRunner.java:490)
[junit] at
org.apache.hadoop.mapred.LocalJobRunner.getJobCounters(LocalJobRunner.java:634)
[junit] at
org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:418)
[junit] at
org.apache.hadoop.hive.ql.exec.HadoopJobExecHelper$ExecDriverTaskHandle.getCounters(HadoopJobExecHelper.java:465)
[junit] at
org.apache.hadoop.hive.ql.exec.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:300)
[junit] at
org.apache.hadoop.hive.ql.exec.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:532)
[junit] at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:453)
[junit] at
org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:681)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[junit] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit] at java.lang.reflect.Method.invoke(Method.java:597)
[junit] at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5148) Syslog missing from Map/Reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635498#comment-13635498 ] Sandy Ryza commented on MAPREDUCE-5148: --- I ran a tar build and the task-log4j.properties showed up (and syslog appeared correctly). Is there a packaging/upgrade process that updates code but not config files? Syslog missing from Map/Reduce tasks Key: MAPREDUCE-5148 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5148 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 1.2.0 Reporter: yeshavora Attachments: Screenshot_MAPREDUCE-5148.png MAPREDUCE-4970 introduced incompatible change and causes syslog to be missing from tasktracker on old clusters which just have log4j.properties configured -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5166) ConcurrentModificationException in LocalJobRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-5166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635502#comment-13635502
]
Sandy Ryza commented on MAPREDUCE-5166:
---
Gunther, which version of Hadoop does this appear in?
ConcurrentModificationException in LocalJobRunner
-
Key: MAPREDUCE-5166
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5166
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Gunther Hagleitner
Priority: Critical
With the latest version hive unit tests fail in various places with the
following stack trace. The problem seems related to: MAPREDUCE-2931
{noformat}
[junit] java.util.ConcurrentModificationException
[junit] at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
[junit] at java.util.HashMap$ValueIterator.next(HashMap.java:822)
[junit] at
org.apache.hadoop.mapred.Counters.incrAllCounters(Counters.java:505)
[junit] at org.apache.hadoop.mapred.Counters.sum(Counters.java:528)
[junit] at
org.apache.hadoop.mapred.LocalJobRunner$Job.getCurrentCounters(LocalJobRunner.java:490)
[junit] at
org.apache.hadoop.mapred.LocalJobRunner.getJobCounters(LocalJobRunner.java:634)
[junit] at
org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:418)
[junit] at
org.apache.hadoop.hive.ql.exec.HadoopJobExecHelper$ExecDriverTaskHandle.getCounters(HadoopJobExecHelper.java:465)
[junit] at
org.apache.hadoop.hive.ql.exec.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:300)
[junit] at
org.apache.hadoop.hive.ql.exec.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:532)
[junit] at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:453)
[junit] at
org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:681)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[junit] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit] at java.lang.reflect.Method.invoke(Method.java:597)
[junit] at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4898) FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635517#comment-13635517
]
Alejandro Abdelnur commented on MAPREDUCE-4898:
---
Thanks for verifying this Robert.
+1
FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath
incompatible with MR1
--
Key: MAPREDUCE-4898
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4898
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Robert Kanter
Assignee: Robert Kanter
Fix For: 2.0.5-beta
Attachments: MAPREDUCE-4898.patch
In MR1,
{{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs}}
throws {{org.apache.hadoop.mapred.FileAlreadyExistsException}} but now it
throws {{org.apache.hadoop.fs.FileAlreadyExistsException}} instead, making
them incompatible.
In MR1,
{{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath}}
doesn't throw any exceptions but now it throws an {{IOException}}, making
them incompatible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4898) FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated MAPREDUCE-4898:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Robert. Committed to trunk and branch-2.
FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath
incompatible with MR1
--
Key: MAPREDUCE-4898
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4898
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Robert Kanter
Assignee: Robert Kanter
Fix For: 2.0.5-beta
Attachments: MAPREDUCE-4898.patch
In MR1,
{{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs}}
throws {{org.apache.hadoop.mapred.FileAlreadyExistsException}} but now it
throws {{org.apache.hadoop.fs.FileAlreadyExistsException}} instead, making
them incompatible.
In MR1,
{{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath}}
doesn't throw any exceptions but now it throws an {{IOException}}, making
them incompatible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4898) FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635540#comment-13635540
]
Hudson commented on MAPREDUCE-4898:
---
Integrated in Hadoop-trunk-Commit #3629 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3629/])
MAPREDUCE-4898. FileOutputFormat.checkOutputSpecs and
FileOutputFormat.setOutputPath incompatible with MR1. (rkanter via tucu)
(Revision 1469523)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1469523
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputFormat.java
FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath
incompatible with MR1
--
Key: MAPREDUCE-4898
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4898
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Robert Kanter
Assignee: Robert Kanter
Fix For: 2.0.5-beta
Attachments: MAPREDUCE-4898.patch
In MR1,
{{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs}}
throws {{org.apache.hadoop.mapred.FileAlreadyExistsException}} but now it
throws {{org.apache.hadoop.fs.FileAlreadyExistsException}} instead, making
them incompatible.
In MR1,
{{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath}}
doesn't throw any exceptions but now it throws an {{IOException}}, making
them incompatible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5152) MR App is not using Container from RM
[
https://issues.apache.org/jira/browse/MAPREDUCE-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635546#comment-13635546
]
Bikas Saha commented on MAPREDUCE-5152:
---
+1 overall. Minor nits
Reuse TA.getNodeId() instead of duplicating code?
{code}
-+ ] using containerId: [ + containerID + on NM: [
-+ containerMgrAddress + ]);
++ ] using containerId: [ + container.getId() + on NM: [
++ StringInterner.weakIntern(container.getNodeId().toString()) + ]);
{code}
{code}
- taskAttempt.containerID, taskAttempt.containerMgrAddress,
- taskAttempt.containerToken,
+ taskAttempt.container.getId(), StringInterner
+ .weakIntern(taskAttempt.container.getNodeId().toString()),
{code}
Unnecessary variable?
{code}
+ Container container = cEvent.getContainer();
+ taskAttempt.container = container;
{code}
Looks like there is still some avoidable copying happening in
ContainerRemoteLauchEvent and its base and in ContainerLaunchImpl.Container but
we may leave it for a later date.
MR App is not using Container from RM
-
Key: MAPREDUCE-5152
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5152
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.5-beta
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli
Attachments: MAPREDUCE-5152-20130415.1.txt,
MAPREDUCE-5152-20130415.txt, MAPREDUCE-5152-20130417.txt
The goal of YARN-486 was to make AMs just pass information encapsulated in
Container along to NM instead of doing it themselves by duplicating
information. We still do not do this pass-through as intended as YARN-486
avoided the individual field duplication but failed to avoid the duplication
of container itself.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5152) MR App is not using Container from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635614#comment-13635614 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5152: TA.getNodeId() takes a readLock again, it's not a big deal but is unnecessary and so I avoided it. The container field is also needed once again in the same method, not a big deal again, I'll let it be. I'm checking this in to unblock YARN-571. MR App is not using Container from RM - Key: MAPREDUCE-5152 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5152 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.5-beta Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Attachments: MAPREDUCE-5152-20130415.1.txt, MAPREDUCE-5152-20130415.txt, MAPREDUCE-5152-20130417.txt The goal of YARN-486 was to make AMs just pass information encapsulated in Container along to NM instead of doing it themselves by duplicating information. We still do not do this pass-through as intended as YARN-486 avoided the individual field duplication but failed to avoid the duplication of container itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5152) MR App is not using Container from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5152: --- Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I committed this to trunk and branch-2. Thanks for the reviews Bikas. MR App is not using Container from RM - Key: MAPREDUCE-5152 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5152 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.5-beta Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 2.0.5-beta Attachments: MAPREDUCE-5152-20130415.1.txt, MAPREDUCE-5152-20130415.txt, MAPREDUCE-5152-20130417.txt The goal of YARN-486 was to make AMs just pass information encapsulated in Container along to NM instead of doing it themselves by duplicating information. We still do not do this pass-through as intended as YARN-486 avoided the individual field duplication but failed to avoid the duplication of container itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5165) Create MiniMRCluster version which uses the mapreduce package.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635631#comment-13635631 ] Siddharth Seth commented on MAPREDUCE-5165: --- MiniMRClientClusterFactory should be used instead of using MiniMRCluster or MiniMRYarnCluster. Create MiniMRCluster version which uses the mapreduce package. -- Key: MAPREDUCE-5165 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5165 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: jay vyas Priority: Minor The MiniMapRedCluster class references some older mapred.* classes (as per comments below however, there is the MiniMRYarnCluster, which may aim to replace it). It could be recreated in the mapreduce package to use the Configuration class instead of JobConf, which would make it simpler to use and integrate with new FS implementations and test harnesses that use new Configuration (not JobConf) objects to drive tests. This could be done many ways: 1) using inheritance or else 2) by copying the code directly The appropriate implementation depends on wether or not 1) Is it okay for mapreduce.* classes to depend on mapred.* classes ? 2) Is the mapred MiniMRCluster implementation going to be deprecated or eliminated anytime? 3) What is the future of the JobConf class - which has been deprecated and then undeprecated ? Note that This is all intimately linked to the role that MiniMRYarnCluster will play. Relevant classes: .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java .//hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5161) CombineFileInputFormat fix for paths not on default FS merge from branch-1 to branch-1-win
[
https://issues.apache.org/jira/browse/MAPREDUCE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635634#comment-13635634
]
Bikas Saha commented on MAPREDUCE-5161:
---
Patch looks like a clean merge of MAPREDUCE-1806. Its not clear whether it
reverts the independent fix in branch-1-win that is mentioned in the
description?
CombineFileInputFormat fix for paths not on default FS merge from branch-1 to
branch-1-win
--
Key: MAPREDUCE-5161
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5161
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv1
Affects Versions: 1-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-5161-branch-1-win.1.patch
MAPREDUCE-1806 fixed a bug related to use of {{CombineFileInputFormat}} with
paths that are not on the default file system. This same bug was fixed
independently on branch-1-win. The code was slightly different, but
equivalent to the branch-1 fix. This jira will apply the branch-1 fix to
branch-1-win to keep the 2 code lines in agreement.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5152) MR App is not using Container from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635641#comment-13635641 ] Hudson commented on MAPREDUCE-5152: --- Integrated in Hadoop-trunk-Commit #3630 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3630/]) MAPREDUCE-5152. Make MR App to simply pass through the container from RM instead of extracting and populating information itself to start any container. Contributed by Vinod Kumar Vavilapalli. (Revision 1469544) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1469544 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerRemoteLaunchEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestMapReduceChildJVM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java MR App is not using Container from RM - Key: MAPREDUCE-5152 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5152 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.5-beta Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 2.0.5-beta Attachments: MAPREDUCE-5152-20130415.1.txt, MAPREDUCE-5152-20130415.txt, MAPREDUCE-5152-20130417.txt The goal of YARN-486 was to make AMs just pass information encapsulated in Container along to NM instead of doing it themselves by duplicating information. We still do not do this pass-through as intended as YARN-486 avoided the individual field duplication but failed to avoid the duplication of container itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5161) CombineFileInputFormat fix for paths not on default FS merge from branch-1 to branch-1-win
[
https://issues.apache.org/jira/browse/MAPREDUCE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635668#comment-13635668
]
Chris Nauroth commented on MAPREDUCE-5161:
--
{quote}
Its not clear whether it reverts the independent fix in branch-1-win that is
mentioned in the description?
{quote}
Yes, this patch reverts that fix so that branch-1 and branch-1-win are
identical for this logic. For reference, I've included a diff below showing
the earlier fix that was made straight to branch-1-win, so you can compare.
The branch-1 version is preferable and includes more tests.
{code}
diff --git src/mapred/org/apache/hadoop/mapred/lib/CombineFileInputFormat.java
src/mapred/org/apache/hadoop/mapred/lib/CombineFileInputFormat.java
index c55df11..c439bad 100644
--- src/mapred/org/apache/hadoop/mapred/lib/CombineFileInputFormat.java
+++ src/mapred/org/apache/hadoop/mapred/lib/CombineFileInputFormat.java
@@ -194,7 +194,7 @@ public abstract class CombineFileInputFormatK, V
continue;
}
FileSystem fs = paths[i].getFileSystem(job);
-Path p = new Path(paths[i].toUri().getPath());
+Path p = new Path(paths[i].toString());
if (onepool.accept(p)) {
myPaths.add(paths[i]); // add it to my output set
paths[i] = null; // already processed
diff --git
src/mapred/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
src/mapred/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
index c9fa549..c7929e4 100644
--- src/mapred/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
+++ src/mapred/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
@@ -211,7 +211,7 @@ public abstract class CombineFileInputFormatK, V
// times, one time each for each pool in the next loop.
ListPath newpaths = new LinkedListPath();
for (int i = 0; i paths.length; i++) {
- Path p = new Path(paths[i].toUri().getPath());
+ Path p = new Path(paths[i].toString());
newpaths.add(p);
}
paths = null;
diff --git
src/test/org/apache/hadoop/mapred/lib/TestCombineFileInputFormat.java
src/test/org/apache/hadoop/mapred/lib/TestCombineFileInputFormat.java
index 8f7c4be..f013bb8 100644
--- src/test/org/apache/hadoop/mapred/lib/TestCombineFileInputFormat.java
+++ src/test/org/apache/hadoop/mapred/lib/TestCombineFileInputFormat.java
@@ -462,7 +462,8 @@ public class TestCombineFileInputFormat extends TestCase{
// returns true if the specified path matches the prefix stored
// in this TestFilter.
public boolean accept(Path path) {
- if (path.toString().indexOf(p.toString()) == 0) {
+ Path uriPath = new Path(path.toUri().getPath());
+ if (uriPath.toString().indexOf(p.toString()) == 0) {
return true;
}
return false;
diff --git
src/test/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
src/test/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
index c80c70d..16345bd 100644
---
src/test/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
+++
src/test/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
@@ -1122,7 +1122,8 @@ public class TestCombineFileInputFormat extends TestCase {
// returns true if the specified path matches the prefix stored
// in this TestFilter.
public boolean accept(Path path) {
- if (path.toString().indexOf(p.toString()) == 0) {
+ Path uriPath = new Path(path.toUri().getPath());
+ if (uriPath.toString().indexOf(p.toString()) == 0) {
return true;
}
return false;
{code}
CombineFileInputFormat fix for paths not on default FS merge from branch-1 to
branch-1-win
--
Key: MAPREDUCE-5161
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5161
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv1
Affects Versions: 1-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-5161-branch-1-win.1.patch
MAPREDUCE-1806 fixed a bug related to use of {{CombineFileInputFormat}} with
paths that are not on the default file system. This same bug was fixed
independently on branch-1-win. The code was slightly different, but
equivalent to the branch-1 fix. This jira will apply the branch-1 fix to
branch-1-win to keep the 2 code lines in agreement.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4932) mapreduce.job#getTaskCompletionEvents incompatible with Hadoop 1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated MAPREDUCE-4932:
--
Summary: mapreduce.job#getTaskCompletionEvents incompatible with Hadoop 1
(was: mapreduce.job#getTaskCompletionEvents incompatible with MR1)
mapreduce.job#getTaskCompletionEvents incompatible with Hadoop 1
Key: MAPREDUCE-4932
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4932
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.2-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
Attachments: MAPREDUCE-4932.patch, MAPREDUCE-4932.patch,
MAPREDUCE-4932.patch
In MR1, {{org.apache.hadoop.mapreduce.Job#getTaskCompletionEvents}} takes one
argument: {{int startFrom}}. In MR2, it now takes an additional argument:
{{int numEvents}} (which is the max number of events to get). This makes
them incompatible.
I propose we add a second {{getTaskCompletionEvents}} method that simply
calls the other one with {{numEvents}} set to {{Integer.MAX_VALUE}} to
replicate the behavior of the MR1 version.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4932) mapreduce.job#getTaskCompletionEvents incompatible with Hadoop 1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated MAPREDUCE-4932:
--
Resolution: Fixed
Fix Version/s: 2.0.5-beta
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Robert. Committed to trunk and branch-2.
mapreduce.job#getTaskCompletionEvents incompatible with Hadoop 1
Key: MAPREDUCE-4932
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4932
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.2-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
Fix For: 2.0.5-beta
Attachments: MAPREDUCE-4932.patch, MAPREDUCE-4932.patch,
MAPREDUCE-4932.patch
In MR1, {{org.apache.hadoop.mapreduce.Job#getTaskCompletionEvents}} takes one
argument: {{int startFrom}}. In MR2, it now takes an additional argument:
{{int numEvents}} (which is the max number of events to get). This makes
them incompatible.
I propose we add a second {{getTaskCompletionEvents}} method that simply
calls the other one with {{numEvents}} set to {{Integer.MAX_VALUE}} to
replicate the behavior of the MR1 version.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4932) mapreduce.job#getTaskCompletionEvents incompatible with Hadoop 1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635743#comment-13635743
]
Hudson commented on MAPREDUCE-4932:
---
Integrated in Hadoop-trunk-Commit #3631 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3631/])
MAPREDUCE-4932. mapreduce.job#getTaskCompletionEvents incompatible with
Hadoop 1. (rkanter via tucu) (Revision 1469587)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1469587
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
mapreduce.job#getTaskCompletionEvents incompatible with Hadoop 1
Key: MAPREDUCE-4932
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4932
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.2-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
Fix For: 2.0.5-beta
Attachments: MAPREDUCE-4932.patch, MAPREDUCE-4932.patch,
MAPREDUCE-4932.patch
In MR1, {{org.apache.hadoop.mapreduce.Job#getTaskCompletionEvents}} takes one
argument: {{int startFrom}}. In MR2, it now takes an additional argument:
{{int numEvents}} (which is the max number of events to get). This makes
them incompatible.
I propose we add a second {{getTaskCompletionEvents}} method that simply
calls the other one with {{numEvents}} set to {{Integer.MAX_VALUE}} to
replicate the behavior of the MR1 version.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635825#comment-13635825
]
Bikas Saha commented on MAPREDUCE-4987:
---
I thought that static blocks are executed once per class loader. So I am not
sure why this one would be executed per inner class object creation. In any
case, moving the code to @BeforeClass is a right thing to do in general.
Are the test times similar in Linux and Windows or they are close to timing out
only on Windows. I would be wary of simply doubling the test timeouts.
this probably can be done once instead of multiple times right? I am assuming
this is a slow filesystem operation.
{code}
+if (new File(entry.getKey().toUri().getPath()).isDirectory()) {
THIS
+ newClassPath.append(Path.SEPARATOR);
+}
{code}
btw, there doesnt seem to be a test about explicitly adding local resources to
the classpath in this patch, right?
Finally, this will have to be split into common, mr and yarn jiras+patches,
though we will need a combined patch to get a successful jenkins run. we can
attach the combined patch to the common jira because that will be committed
first.
TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior
of symlinks
---
Key: MAPREDUCE-4987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, nodemanager
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4987.1.patch, MAPREDUCE-4987.2.patch,
MAPREDUCE-4987.3.patch, MAPREDUCE-4987.4.patch, MAPREDUCE-4987.5.patch
On Windows, {{TestMRJobs#testDistributedCache}} fails on an assertion while
checking the length of a symlink. It expects to see the length of the target
of the symlink, but Java 6 on Windows always reports that a symlink has
length 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5161) CombineFileInputFormat fix for paths not on default FS merge from branch-1 to branch-1-win
[
https://issues.apache.org/jira/browse/MAPREDUCE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635841#comment-13635841
]
Bikas Saha commented on MAPREDUCE-5161:
---
Then I dont understand why some diff show up? e.g.
From the diff in your comment above
{code}
FileSystem fs = paths[i].getFileSystem(job);
-Path p = new Path(paths[i].toUri().getPath());
+Path p = new Path(paths[i].toString());
if (onepool.accept(p)) {
{code}
From the attached patch.
{code}
FileSystem fs = paths[i].getFileSystem(job);
-Path p = new Path(paths[i].toUri().getPath());
+Path p = fs.makeQualified(paths[i]);
if (onepool.accept(p)) {
{code}
Shouldnt I see?
{code}
FileSystem fs = paths[i].getFileSystem(job);
-Path p = new Path(paths[i].toString());
+Path p = fs.makeQualified(paths[i]);
if (onepool.accept(p)) {
{code}
CombineFileInputFormat fix for paths not on default FS merge from branch-1 to
branch-1-win
--
Key: MAPREDUCE-5161
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5161
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv1
Affects Versions: 1-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-5161-branch-1-win.1.patch
MAPREDUCE-1806 fixed a bug related to use of {{CombineFileInputFormat}} with
paths that are not on the default file system. This same bug was fixed
independently on branch-1-win. The code was slightly different, but
equivalent to the branch-1 fix. This jira will apply the branch-1 fix to
branch-1-win to keep the 2 code lines in agreement.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5161) CombineFileInputFormat fix for paths not on default FS merge from branch-1 to branch-1-win
[
https://issues.apache.org/jira/browse/MAPREDUCE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635868#comment-13635868
]
Chris Nauroth commented on MAPREDUCE-5161:
--
Sorry for the confusion. It turns out that the diff in my comment above came
from a personal branch. The independent fix I described never actually got
committed to branch-1-win, so really, this is just a simple merge of the
branch-1 fix to branch-1-win.
I've updated the ticket description to state that this is a merge (and not a
revert of a prior independent fix).
Thanks!
CombineFileInputFormat fix for paths not on default FS merge from branch-1 to
branch-1-win
--
Key: MAPREDUCE-5161
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5161
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv1
Affects Versions: 1-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-5161-branch-1-win.1.patch
MAPREDUCE-1806 fixed a bug related to use of {{CombineFileInputFormat}} with
paths that are not on the default file system. This jira will merge the
branch-1 fix to branch-1-win.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5163) Update MR App after YARN-441
[ https://issues.apache.org/jira/browse/MAPREDUCE-5163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-5163. Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed Committed the latest patch to trunk and branch-2 along with YARN-441. Closing this as resolved. Update MR App after YARN-441 Key: MAPREDUCE-5163 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5163 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Vinod Kumar Vavilapalli Assignee: Xuan Gong Fix For: 2.0.5-beta Attachments: MAPREDUCE-5163.patch YARN-441 is removing some collection APIs that are utility methods on top the base APIs. MR App needs to be updated to not use those anymore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5167) Update MR App after YARN-562
Vinod Kumar Vavilapalli created MAPREDUCE-5167: -- Summary: Update MR App after YARN-562 Key: MAPREDUCE-5167 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5167 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Vinod Kumar Vavilapalli Assignee: Jian He Tracking JIRA for MR changes at YARN-562. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5163) Update MR App after YARN-441
[ https://issues.apache.org/jira/browse/MAPREDUCE-5163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635938#comment-13635938 ] Hudson commented on MAPREDUCE-5163: --- Integrated in Hadoop-trunk-Commit #3633 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3633/]) YARN-441. Removed unused utility methods for collections from two API records. Contributed by Xuan Gong. MAPREDUCE-5163. Update MR App to not use API utility methods for collections after YARN-441. Contributed by Xuan Gong. (Revision 1469657) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1469657 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/StartContainerResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateRequestPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/StartContainerResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestContainerManagerSecurity.java Update MR App after YARN-441 Key: MAPREDUCE-5163 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5163 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Vinod Kumar Vavilapalli Assignee: Xuan Gong Fix For: 2.0.5-beta Attachments: MAPREDUCE-5163.patch YARN-441 is removing some collection APIs that are utility methods on top the base APIs. MR App needs to be updated to not use those anymore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5168) Reducer can OOM during shuffle because on-disk output stream not released
Jason Lowe created MAPREDUCE-5168: - Summary: Reducer can OOM during shuffle because on-disk output stream not released Key: MAPREDUCE-5168 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5168 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical If a reducer needs to shuffle a map output to disk, it opens an output stream and writes the data to disk. However it does not release the reference to the output stream within the MapOutput, and the output stream can have a 128K buffer attached to it. If enough of these on-disk outputs are queued up waiting to be merged, it can cause the reducer to OOM during the shuffle phase. In one case I saw there were 1200 on-disk outputs queued up to be merged, leading to an extra 150MB of pressure on the heap due to the output stream buffers that were no longer necessary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5168) Reducer can OOM during shuffle because on-disk output stream not released
[ https://issues.apache.org/jira/browse/MAPREDUCE-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5168: -- Attachment: MAPREDUCE-5168.patch Patch to release reference to output stream when disk map output is committed or aborted. Reducer can OOM during shuffle because on-disk output stream not released - Key: MAPREDUCE-5168 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5168 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Attachments: MAPREDUCE-5168.patch If a reducer needs to shuffle a map output to disk, it opens an output stream and writes the data to disk. However it does not release the reference to the output stream within the MapOutput, and the output stream can have a 128K buffer attached to it. If enough of these on-disk outputs are queued up waiting to be merged, it can cause the reducer to OOM during the shuffle phase. In one case I saw there were 1200 on-disk outputs queued up to be merged, leading to an extra 150MB of pressure on the heap due to the output stream buffers that were no longer necessary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5168) Reducer can OOM during shuffle because on-disk output stream not released
[ https://issues.apache.org/jira/browse/MAPREDUCE-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5168: -- Target Version/s: 2.0.5-beta, 0.23.8 Affects Version/s: 2.0.5-beta Status: Patch Available (was: Open) Reducer can OOM during shuffle because on-disk output stream not released - Key: MAPREDUCE-5168 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5168 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7, 2.0.5-beta Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Attachments: MAPREDUCE-5168.patch If a reducer needs to shuffle a map output to disk, it opens an output stream and writes the data to disk. However it does not release the reference to the output stream within the MapOutput, and the output stream can have a 128K buffer attached to it. If enough of these on-disk outputs are queued up waiting to be merged, it can cause the reducer to OOM during the shuffle phase. In one case I saw there were 1200 on-disk outputs queued up to be merged, leading to an extra 150MB of pressure on the heap due to the output stream buffers that were no longer necessary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5168) Reducer can OOM during shuffle because on-disk output stream not released
[
https://issues.apache.org/jira/browse/MAPREDUCE-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13636105#comment-13636105
]
Hadoop QA commented on MAPREDUCE-5168:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12579479/MAPREDUCE-5168.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3537//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3537//console
This message is automatically generated.
Reducer can OOM during shuffle because on-disk output stream not released
-
Key: MAPREDUCE-5168
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5168
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.7, 2.0.5-beta
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Critical
Attachments: MAPREDUCE-5168.patch
If a reducer needs to shuffle a map output to disk, it opens an output stream
and writes the data to disk. However it does not release the reference to
the output stream within the MapOutput, and the output stream can have a 128K
buffer attached to it. If enough of these on-disk outputs are queued up
waiting to be merged, it can cause the reducer to OOM during the shuffle
phase. In one case I saw there were 1200 on-disk outputs queued up to be
merged, leading to an extra 150MB of pressure on the heap due to the output
stream buffers that were no longer necessary.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
