[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549752#comment-14549752 ] sam liu commented on MAPREDUCE-6204: Hi Tsuyoshi, In fact, there is no hadoop cluster and hadoop related configuration in my dev env on power linux, so I also did not sure why the 'MAPRED_MAP_TASK_JAVA_OPTS' and 'MAPRED_REDUCE_TASK_JAVA_OPTS' has above default and unexpected value there. At the same time, we mentioned that the root causes of the issue is described in MAPREDUCE-6205, however its patch is still not reviewed/accepted yet. Therefore, current fix will greatly make sense to the ut TestJobCounters: it explicitly replaces the deprecated property with the legal ones, and really could set correct value to MAP/REDUCE OPTS properties and fix the unexpected env/configurations issue and make the ut more robust. Thanks! TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Labels: BB2015-05-RFC Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204-2.patch, MAPREDUCE-6204-3.patch, MAPREDUCE-6204-4.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14543278#comment-14543278 ] sam liu commented on MAPREDUCE-6204: It's really weird that there always be some ut failures, however none of them was brought by current fix and all the ut could pass in my env. I guess there might be some env issues. I suggest to rerun the ut or deliver this fix directly after reviewing. Thanks! TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Labels: BB2015-05-RFC Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204-2.patch, MAPREDUCE-6204-3.patch, MAPREDUCE-6204-4.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6204: --- Attachment: MAPREDUCE-6204-3.patch TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Labels: BB2015-05-RFC Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204-2.patch, MAPREDUCE-6204-3.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6204: --- Attachment: MAPREDUCE-6204-4.patch TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Labels: BB2015-05-RFC Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204-2.patch, MAPREDUCE-6204-3.patch, MAPREDUCE-6204-4.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537761#comment-14537761 ] sam liu commented on MAPREDUCE-6204: These ut should not fail, as they could pass in my env. I studied the outputs of ut execution and seems the failures were caused by env issue, like 'java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/ApplicationIdNotProvidedException'. Could please help rerun the ut? Thanks! TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Labels: BB2015-05-RFC Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204-2.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6204: --- Attachment: MAPREDUCE-6204-2.patch Created a new patch basing on latest trunk code(2015/05/06) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Labels: BB2015-05-TBR Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204-2.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5804) TestMRJobsWithProfiler#testProfiler timesout
[
https://issues.apache.org/jira/browse/MAPREDUCE-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14282193#comment-14282193
]
sam liu commented on MAPREDUCE-5804:
I applied this fix, but still encounter timeout issue. After changing '15'
to '25', this test could pass on my env. So could we enlarge the timeout
value, like to '25'?
TestMRJobsWithProfiler#testProfiler timesout
Key: MAPREDUCE-5804
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5804
Project: Hadoop Map/Reduce
Issue Type: Test
Affects Versions: 2.4.0
Reporter: Mit Desai
Assignee: Mit Desai
Fix For: 2.5.0
Attachments: LOG.txt, MAPREDUCE-5804.patch
{noformat}
testProfiler(org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler) Time
elapsed: 154.972 sec ERROR!
java.lang.Exception: test timed out after 12 milliseconds
at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)
at java.io.File.exists(File.java:813)
at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1080)
at sun.misc.URLClassPath.getResource(URLClassPath.java:199)
at java.net.URLClassLoader$1.run(URLClassLoader.java:358)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.apache.log4j.spi.LoggingEvent.init(LoggingEvent.java:165)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
at
org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:208)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:338)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:532)
at org.apache.hadoop.mapreduce.Job$1.run(Job.java:314)
at org.apache.hadoop.mapreduce.Job$1.run(Job.java:311)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1570)
at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:311)
at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:599)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1344)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1306)
at
org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testProfiler(TestMRJobsWithProfiler.java:138)
Results :
Tests in error:
TestMRJobsWithProfiler.testProfiler:138 ยป test timed out after 12
millise...
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property mapred.child.java.opts
[
https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267371#comment-14267371
]
sam liu commented on MAPREDUCE-6205:
Gera,
The root cause of above failure is that after deserializing both the deprecated
and the non-deprecated versions of a config are set. So, before deserializing,
mapreduce.reduce.java.opts is equals to $ {mapred.child.java.opts}; after
deserializing, mapreduce.reduce.java.opts is equals to '-Xmx200m'.
To resolve this issue, I have a proposal that in mapred-default.xml:
- Comment out the property 'mapred.child.java.opts'
- Uncomment both mapreduce.map|reduce.java.opts and set their value as
'-Xmx200m'.
I think this makes sense, because 'mapred.child.java.opts' is deprecated, but
'mapreduce.map|reduce.java.opts' are recommended by hadoop. With such
modifications, test TestWritableJobConf could pass. Also I ran some other
regarding tests, and I found only test TestJobConf should be updated till now.
What's your comments on the proposal?
Thanks!
Update the value of the new version properties of the deprecated property
mapred.child.java.opts
--
Key: MAPREDUCE-6205
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: trunk
Reporter: sam liu
Assignee: sam liu
Priority: Minor
Attachments: MAPREDUCE-6205.patch, MAPREDUCE-6205.patch
In current hadoop code, the old property mapred.child.java.opts is
deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and
MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the
deprecated property mapred.child.java.opts, hadoop won't automatically
update its new versions properties
MRJobConfig.MAP_JAVA_OPTS(mapreduce.map.java.opts) and
MRJobConfig.REDUCE_JAVA_OPTS(mapreduce.reduce.java.opts). As hadoop will
update the new version properties for many other deprecated properties, we
also should support such feature on the old property
mapred.child.java.opts, otherwise it might bring some imcompatible issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property mapred.child.java.opts
[
https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265742#comment-14265742
]
sam liu commented on MAPREDUCE-6205:
Gera,
Thanks for your reminder!
After implementing suggestion 3(update mapred-default.xml), the unit test
failed on 'assertEquals(conf, deser)', but with different issue:
- the expected is: 'mapreduce.reduce.java.opts=${mapred.child.java.opts},
mapreduce.map.java.opts=${mapred.child.java.opts}'
- but is: 'mapreduce.reduce.java.opts=-Xmx200m,
mapreduce.map.java.opts=-Xmx200m'
However, I could get following output by 'conf.get(key)' or 'deser.get(key)'
during running this test:
1. The properties of 'conf':
mapred.child.java.opts=-Xmx200m
mapred.map.child.java.opts=-Xmx200m
mapred.reduce.child.java.opts=-Xmx200m
mapreduce.map.java.opts=-Xmx200m
mapreduce.reduce.java.opts=-Xmx200m
2. The properties of 'deser':
mapred.child.java.opts=-Xmx200m
mapred.map.child.java.opts=-Xmx200m
mapred.reduce.child.java.opts=-Xmx200m
mapreduce.map.java.opts=-Xmx200m
mapreduce.reduce.java.opts=-Xmx200m
Any comments?
Update the value of the new version properties of the deprecated property
mapred.child.java.opts
--
Key: MAPREDUCE-6205
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: trunk
Reporter: sam liu
Assignee: sam liu
Priority: Minor
Attachments: MAPREDUCE-6205.patch, MAPREDUCE-6205.patch
In current hadoop code, the old property mapred.child.java.opts is
deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and
MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the
deprecated property mapred.child.java.opts, hadoop won't automatically
update its new versions properties
MRJobConfig.MAP_JAVA_OPTS(mapreduce.map.java.opts) and
MRJobConfig.REDUCE_JAVA_OPTS(mapreduce.reduce.java.opts). As hadoop will
update the new version properties for many other deprecated properties, we
also should support such feature on the old property
mapred.child.java.opts, otherwise it might bring some imcompatible issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property mapred.child.java.opts
[ https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262063#comment-14262063 ] sam liu commented on MAPREDUCE-6205: Gera, Yes, there is an existing deprecation deltas mapred.map|reduce.child.java.opts - mapreduce.map|reduce.java.opts. However, after I add the deprecation delta as mapred.child.java.opts - mapred.map|reduce.child.java.opts, the test TestWritableJobConf#testEmptyConfiguration() failed. In this test, it compares two configurations(conf and deser): -conf: JobConf conf = new JobConf(); -deser: Configuration deser = serDeser(conf); [A] Before I added the new deprecation delta, I got following output during running this test and the test passed. [A.1] The properties of 'conf': mapred.child.java.opts=-Xmx200m 2014-12-31 01:13:12,092 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1013)) - mapred.map.child.java.opts is deprecated. Instead, use mapreduce.map.java.opts mapred.map.child.java.opts=null 2014-12-31 01:13:12,107 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1013)) - mapred.reduce.child.java.opts is deprecated. Instead, use mapreduce.reduce.java.opts mapred.reduce.child.java.opts=null mapreduce.map.java.opts=null mapreduce.reduce.java.opts=null [A.2] The properties of 'deser': mapred.child.java.opts=-Xmx200m mapred.map.child.java.opts=null mapred.reduce.child.java.opts=null mapreduce.map.java.opts=null mapreduce.reduce.java.opts=null [B] After I added the new deprecation delta, I got following output during running this test and the test failed. [B.1] The properties of 'conf': 2014-12-31 01:06:22,313 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1013)) - mapred.child.java.opts is deprecated. Instead, use mapred.map.child.java.opts, mapred.reduce.child.java.opts mapred.child.java.opts=-Xmx200m mapred.map.child.java.opts=null mapred.reduce.child.java.opts=null mapreduce.map.java.opts=null mapreduce.reduce.java.opts=null [B.2] The properties of 'deser': mapred.child.java.opts=-Xmx200m mapred.map.child.java.opts=-Xmx200m mapred.reduce.child.java.opts=-Xmx200m mapreduce.map.java.opts=-Xmx200m mapreduce.reduce.java.opts=-Xmx200m Furthermore, if I did not add the deprecation delta as mapred.child.java.opts - mapred.map|reduce.child.java.opts, but add the deprecation delta as mapred.child.java.opts - mapreduce.map|reduce.java.opts, the test TestWritableJobConf#testEmptyConfiguration() could pass. What's your opinion? Thanks! Update the value of the new version properties of the deprecated property mapred.child.java.opts -- Key: MAPREDUCE-6205 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: trunk Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6205.patch, MAPREDUCE-6205.patch In current hadoop code, the old property mapred.child.java.opts is deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the deprecated property mapred.child.java.opts, hadoop won't automatically update its new versions properties MRJobConfig.MAP_JAVA_OPTS(mapreduce.map.java.opts) and MRJobConfig.REDUCE_JAVA_OPTS(mapreduce.reduce.java.opts). As hadoop will update the new version properties for many other deprecated properties, we also should support such feature on the old property mapred.child.java.opts, otherwise it might bring some imcompatible issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260694#comment-14260694 ] sam liu commented on MAPREDUCE-6204: Gera, thanks for your comments and I will update the patch of MAPREDUCE-6205 according to your suggestion later. TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259797#comment-14259797 ] sam liu commented on MAPREDUCE-6204: For many deprecated properties, hadoop code always set its value to its new version of property. Absolutely, this solution is clear and won't bring any misunderstanding on the conception of 'deprecation'. However, for the deprecated property mapred.child.java.opts, hadoop uses a different method: it won't update the value of its new versions MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS. Yes, finally hadoop will set a correct value on the classpath, however the program like MapReduce could not get correct value of MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS: it definately bring confusing to the users. Furthermore, this ut will fail without this patch on ppc64 platform, but could pass on the same env with this patch. Actually the root cause is the insuffcient completion of the deprecation of property mapred.child.java.opts. We could not say this patch is useless or simply pass the buck to 'env issue': if this is an env issue, this ut should fail on the ppc64 platform either with the deprecated property mapred.child.java.opts or with the new version of properties MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS. Therefore this patch is necessary and could remedy the insuffcient completion of the deprecation of property mapred.child.java.opts. TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property mapred.child.java.opts
[ https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259140#comment-14259140 ] sam liu commented on MAPREDUCE-6205: Yes, the MapReduceChildJVM#getChildJavaOpts makes the old property as fallback. However, this solution has two disadvantages: 1. During map/reduce task execution, the program might get incorrect value of new property MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS, if only the value of mapred.child.java.opts is defined before. 2. Inconsistent handling style. For most of other deprecated properties, Hadoop will automatically update the value of their new version with the value of old version. However, hadoop does not use same style to handle the deprecation of property mapred.child.java.opts Update the value of the new version properties of the deprecated property mapred.child.java.opts -- Key: MAPREDUCE-6205 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: trunk Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6205.patch, MAPREDUCE-6205.patch In current hadoop code, the old property mapred.child.java.opts is deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the deprecated property mapred.child.java.opts, hadoop won't automatically update its new versions properties MRJobConfig.MAP_JAVA_OPTS(mapreduce.map.java.opts) and MRJobConfig.REDUCE_JAVA_OPTS(mapreduce.reduce.java.opts). As hadoop will update the new version properties for many other deprecated properties, we also should support such feature on the old property mapred.child.java.opts, otherwise it might bring some imcompatible issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property mapred.child.java.opts
[ https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259141#comment-14259141 ] sam liu commented on MAPREDUCE-6205: Yes, the MapReduceChildJVM#getChildJavaOpts makes the old property as fallback. However, this solution has two disadvantages: 1. During map/reduce task execution, the program might get incorrect value of new property MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS, if only the value of mapred.child.java.opts is defined before. 2. Inconsistent handling style. For most of other deprecated properties, Hadoop will automatically update the value of their new version with the value of old version. However, hadoop does not use same style to handle the deprecation of property mapred.child.java.opts Update the value of the new version properties of the deprecated property mapred.child.java.opts -- Key: MAPREDUCE-6205 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: trunk Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6205.patch, MAPREDUCE-6205.patch In current hadoop code, the old property mapred.child.java.opts is deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the deprecated property mapred.child.java.opts, hadoop won't automatically update its new versions properties MRJobConfig.MAP_JAVA_OPTS(mapreduce.map.java.opts) and MRJobConfig.REDUCE_JAVA_OPTS(mapreduce.reduce.java.opts). As hadoop will update the new version properties for many other deprecated properties, we also should support such feature on the old property mapred.child.java.opts, otherwise it might bring some imcompatible issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property mapred.child.java.opts
[ https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6205: --- Attachment: MAPREDUCE-6205.patch Updated TestJobConf according the code changes Update the value of the new version properties of the deprecated property mapred.child.java.opts -- Key: MAPREDUCE-6205 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: trunk Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6205.patch, MAPREDUCE-6205.patch In current hadoop code, the old property mapred.child.java.opts is deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the deprecated property mapred.child.java.opts, hadoop won't automatically update its new versions properties MRJobConfig.MAP_JAVA_OPTS(mapreduce.map.java.opts) and MRJobConfig.REDUCE_JAVA_OPTS(mapreduce.reduce.java.opts). As hadoop will update the new version properties for many other deprecated properties, we also should support such feature on the old property mapred.child.java.opts, otherwise it might bring some imcompatible issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257775#comment-14257775 ] sam liu commented on MAPREDUCE-6204: I did many tests and it does not work if only making MAPRED_TASK_JAVA_OPTS larger heap size. Actually, MapReduceChildJVM#getChildJavaOpts is only be invoked by method MapReduceChildJVM#getVMCommand which is not be invoked by any other existing hadoop code: that's why in my tests the value of MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS is still null and not be the same as MAPRED_TASK_JAVA_OPTS. Only I set correct value to MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS, the map/reduce tasks could get correct java opts as below and this unit test could pass on PPC64 platform: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_REDUCE_TASK_JAVA_OPTS=-Xms32m -Xmx1G TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257845#comment-14257845 ] sam liu commented on MAPREDUCE-6204: Gera, I see, but could you please help explain why the map/reduce tasks could not get expected value of MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS, but only return 'null' in the test? I added following code to get the property value during map/reduce task execution in TestJobCounters#MemoryLoaderMapper#configure() and TestJobCounters#MemoryLoaderReducer#configure(): System.out.println(MAPRED_TASK_JAVA_OPTS= + conf.get(JobConf.MAPRED_TASK_JAVA_OPTS)); System.out.println(MAPRED_MAP_TASK_JAVA_OPTS= + conf.get(JobConf.MAPRED_MAP_TASK_JAVA_OPTS)); System.out.println(MAPRED_REDUCE_TASK_JAVA_OPTS= + conf.get(JobConf.MAPRED_REDUCE_TASK_JAVA_OPTS)); TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257920#comment-14257920 ] sam liu commented on MAPREDUCE-6204: Tsuyoshi, Yes, you are correct and the two properties won't be updated. I did a test: 1. In TestTaskAttemptContainerRequest.java, I added 'jobConf.set(JobConf.MAPRED_TASK_JAVA_OPTS, -Xms32m -Xmx1G)' 2. Then I found the command args created by 'ContainerLaunchContext launchCtx = TaskAttemptImpl.createContainerLaunchContext...' likes: =$JAVA_HOME/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xms32m -Xmx1G -Djava.io.tmpdir=$PWD/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=LOG_DIR -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 127.0.0.1 0 attempt_1_0001_m_01_1 0 1LOG_DIR/stdout 2LOG_DIR/stderr Please mention that only a commn heap setting -Xms32m -Xmx1G is added into the command args, but the two properties(MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS) were not updated as expected. So I think it's still necessary to update MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS values in TestJobCounters, as that test could pass on PPC64 only with such updates. TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6205) Updated the value of the new version properties of the deprecated property mapred.child.java.opts
sam liu created MAPREDUCE-6205: -- Summary: Updated the value of the new version properties of the deprecated property mapred.child.java.opts Key: MAPREDUCE-6205 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: sam liu Assignee: sam liu Priority: Minor In current hadoop code, the old property mapred.child.java.opts is deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the deprecated property mapred.child.java.opts, hadoop won't automatically update its new versions properties MRJobConfig.MAP_JAVA_OPTS(mapreduce.map.java.opts) and MRJobConfig.REDUCE_JAVA_OPTS(mapreduce.reduce.java.opts). As hadoop will update the new version properties for many other deprecated properties, we also should support such feature on the old property mapred.child.java.opts, otherwise it might bring some imcompatible issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6205) Updated the value of the new version properties of the deprecated property mapred.child.java.opts
[ https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6205: --- Attachment: MAPREDUCE-6205.patch Unlike many other deprecated properties, the old property mapred.child.java.opts does not only has one new version property, but has two new version properties: MRJobConfig.MAP_JAVA_OPTS, MRJobConfig.REDUCE_JAVA_OPTS. So we should add a new function Configuration#DeprecationDelta(String key, String[] newKeys) to support it. Updated the value of the new version properties of the deprecated property mapred.child.java.opts --- Key: MAPREDUCE-6205 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6205.patch In current hadoop code, the old property mapred.child.java.opts is deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the deprecated property mapred.child.java.opts, hadoop won't automatically update its new versions properties MRJobConfig.MAP_JAVA_OPTS(mapreduce.map.java.opts) and MRJobConfig.REDUCE_JAVA_OPTS(mapreduce.reduce.java.opts). As hadoop will update the new version properties for many other deprecated properties, we also should support such feature on the old property mapred.child.java.opts, otherwise it might bring some imcompatible issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6205) Updated the value of the new version properties of the deprecated property mapred.child.java.opts
[ https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6205: --- Affects Version/s: trunk Status: Patch Available (was: Open) If we only set value -Xms32m -Xmx1G to the deprecated property mapred.child.java.opts before MR job execution: - Without this patch, during map/reduce tasks execution, we can NOT get correct value of MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS, but get null value: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MRJobConfig.MAP_JAVA_OPTS=null MRJobConfig.REDUCE_JAVA_OPTS=null - With this patch, during map/reduce tasks execution, we can get correct value of them: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MRJobConfig.MAP_JAVA_OPTS=-Xms32m -Xmx1G MRJobConfig.REDUCE_JAVA_OPTS=-Xms32m -Xmx1G Updated the value of the new version properties of the deprecated property mapred.child.java.opts --- Key: MAPREDUCE-6205 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: trunk Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6205.patch In current hadoop code, the old property mapred.child.java.opts is deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the deprecated property mapred.child.java.opts, hadoop won't automatically update its new versions properties MRJobConfig.MAP_JAVA_OPTS(mapreduce.map.java.opts) and MRJobConfig.REDUCE_JAVA_OPTS(mapreduce.reduce.java.opts). As hadoop will update the new version properties for many other deprecated properties, we also should support such feature on the old property mapred.child.java.opts, otherwise it might bring some imcompatible issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property mapred.child.java.opts
[ https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6205: --- Summary: Update the value of the new version properties of the deprecated property mapred.child.java.opts (was: Updated the value of the new version properties of the deprecated property mapred.child.java.opts) Update the value of the new version properties of the deprecated property mapred.child.java.opts -- Key: MAPREDUCE-6205 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: trunk Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6205.patch In current hadoop code, the old property mapred.child.java.opts is deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the deprecated property mapred.child.java.opts, hadoop won't automatically update its new versions properties MRJobConfig.MAP_JAVA_OPTS(mapreduce.map.java.opts) and MRJobConfig.REDUCE_JAVA_OPTS(mapreduce.reduce.java.opts). As hadoop will update the new version properties for many other deprecated properties, we also should support such feature on the old property mapred.child.java.opts, otherwise it might bring some imcompatible issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258086#comment-14258086 ] sam liu commented on MAPREDUCE-6204: Hi Tsuyoshi, I did more tests on PPC64, it has some difference from x86: 1. Without any change, the map/reduce tasks will print: 1.1 on x86: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS==null MAPRED_REDUCE_TASK_JAVA_OPTS=null 1.2 on PPC64: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS=-Xmx1000m -Xms1000m -Xmn100m -Xtune:virtualized -Xshareclasses:name=mrscc_%g,groupAccess,cacheDir=/var/hadoop/tmp,nonFatal -Xscmx20m -Xdump:java:file=/var/hadoop/tmp/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt -Xdump:heap:file=/var/hadoop/tmp/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd MAPRED_REDUCE_TASK_JAVA_OPTS=-Xmx1000m -Xms1000m -Xmn100m -Xtune:virtualized -Xshareclasses:name=mrscc_%g,groupAccess,cacheDir=/var/hadoop/tmp,nonFatal -Xscmx20m -Xdump:java:file=/var/hadoop/tmp/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt -Xdump:heap:file=/var/hadoop/tmp/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd 2. With this patch, both x86 and PPC64 will print same message: MAPRED_TASK_JAVA_OPTS=-Xmx200m MAPRED_MAP_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_REDUCE_TASK_JAVA_OPTS=-Xms32m -Xmx1G I will look into PPC64 part to see if there is any other issue there. I also opened another jira MAPREDUCE-6205 to update MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS with MAPRED_TASK_JAVA_OPTS together, but the patch brought another new ut failure. I also will look into that jira later. So this ut really need this patch on PPC64 now and as you said we should use new properties instead of deprecated one. Thanks! TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
sam liu created MAPREDUCE-6204: -- Summary: TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu reassigned MAPREDUCE-6204: -- Assignee: sam liu TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256575#comment-14256575 ] sam liu commented on MAPREDUCE-6204: In Hadoop 2.x, the property JobConf.MAPRED_TASK_JAVA_OPTS is deprecated, and hadoop will ignore this property directly without replacing its new values(MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS). This issue will cause test TestJobCounters fail on PPC64 platform sometimes, as PPC64 JVM sometimes comsumes more memory when executing map/reduce tasks and might encounter JVM crashing issue once the default heap size is exhausted. TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6204: --- Status: Patch Available (was: Open) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6204: --- Attachment: MAPREDUCE-6204.patch The solution is to replace the deprecated property MAPRED_TASK_JAVA_OPTS with its new values MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS in the test. After applying the patch, current test could pass on PPC64 platform all the time. TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256687#comment-14256687 ] sam liu commented on MAPREDUCE-6204: Gera, I did a test: 1. Without the patch, current test TestJobCounters#MemoryLoaderMapper and MemoryLoaderReducer will get value 'null' for properties MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS during MR job execution: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS=null MAPRED_REDUCE_TASK_JAVA_OPTS=null 2. With the patch, current test TestJobCounters#MemoryLoaderMapper and MemoryLoaderReducer will get correct value for all the properties: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_REDUCE_TASK_JAVA_OPTS=-Xms32m -Xmx1G TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6204: --- Attachment: MAPREDUCE-6204-1.patch Attached new patch. Thanks! TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6204: --- Status: Open (was: Patch Available) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6204: --- Status: Patch Available (was: Open) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256718#comment-14256718 ] sam liu commented on MAPREDUCE-6204: Hi Tsuyoshi, I attached new patch MAPREDUCE-6204-1.patch and could you please help take a review? Thanks! TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6191) TestJavaSerialization fails with getting incorrect MR job result
[ https://issues.apache.org/jira/browse/MAPREDUCE-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14243628#comment-14243628 ] sam liu commented on MAPREDUCE-6191: In the latest test result, this UT passed and the patch works well. Running org.apache.hadoop.mapred.TestJavaSerialization Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.204 sec - in org.apache.hadoop.mapred.TestJavaSerialization https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5072//consoleFull TestJavaSerialization fails with getting incorrect MR job result Key: MAPREDUCE-6191 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6191 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Fix For: trunk Attachments: MAPREDUCE-6191.patch TestJavaSerialization#testMapReduceJob() fails with getting incorrect MR job result: junit.framework.ComparisonFailure: expected:[a ]1 but was:[0 1]1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6191) TestJavaSerialization fails with getting incorrect MR job result
sam liu created MAPREDUCE-6191: -- Summary: TestJavaSerialization fails with getting incorrect MR job result Key: MAPREDUCE-6191 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6191 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor TestJavaSerialization#testMapReduceJob() fails with getting incorrect MR job result: junit.framework.ComparisonFailure: expected:[a ]1 but was:[0 1]1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6191) TestJavaSerialization fails with getting incorrect MR job result
[ https://issues.apache.org/jira/browse/MAPREDUCE-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14242068#comment-14242068 ] sam liu commented on MAPREDUCE-6191: [A] Reason of failure: Before executing the UT, there is already a file under the INPUT_DIR used as the MR input dir, such as 'hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/input/0'. So the MR result is incorrect, as the input dir includes extra file. [B] Solution: Update UT code to remove the whole INPUT_DIR before execution, not only removing INPUT_FILE. (I did such tests and they all passed) TestJavaSerialization fails with getting incorrect MR job result Key: MAPREDUCE-6191 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6191 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor TestJavaSerialization#testMapReduceJob() fails with getting incorrect MR job result: junit.framework.ComparisonFailure: expected:[a ]1 but was:[0 1]1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6191) TestJavaSerialization fails with getting incorrect MR job result
[ https://issues.apache.org/jira/browse/MAPREDUCE-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6191: --- Fix Version/s: trunk Status: Patch Available (was: Open) Remove the whole INPUT_DIR before runing MR job, not only remove the INPUT_FILE. This enhancement will also remove other unexpected files under INPUT_DIR. TestJavaSerialization fails with getting incorrect MR job result Key: MAPREDUCE-6191 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6191 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Fix For: trunk TestJavaSerialization#testMapReduceJob() fails with getting incorrect MR job result: junit.framework.ComparisonFailure: expected:[a ]1 but was:[0 1]1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6191) TestJavaSerialization fails with getting incorrect MR job result
[ https://issues.apache.org/jira/browse/MAPREDUCE-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6191: --- Status: Open (was: Patch Available) TestJavaSerialization fails with getting incorrect MR job result Key: MAPREDUCE-6191 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6191 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Fix For: trunk TestJavaSerialization#testMapReduceJob() fails with getting incorrect MR job result: junit.framework.ComparisonFailure: expected:[a ]1 but was:[0 1]1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6191) TestJavaSerialization fails with getting incorrect MR job result
[ https://issues.apache.org/jira/browse/MAPREDUCE-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6191: --- Attachment: MAPREDUCE-6191.patch Before running the UT, to remove the whole INPUT_DIR, not only the INPUT_FILE. This enhancement will clean other unexpected files under folder INPUT_DIR. TestJavaSerialization fails with getting incorrect MR job result Key: MAPREDUCE-6191 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6191 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Fix For: trunk Attachments: MAPREDUCE-6191.patch TestJavaSerialization#testMapReduceJob() fails with getting incorrect MR job result: junit.framework.ComparisonFailure: expected:[a ]1 but was:[0 1]1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6191) TestJavaSerialization fails with getting incorrect MR job result
[ https://issues.apache.org/jira/browse/MAPREDUCE-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6191: --- Status: Patch Available (was: Open) I ran some tests after applying this patch and they all passed. TestJavaSerialization fails with getting incorrect MR job result Key: MAPREDUCE-6191 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6191 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Fix For: trunk Attachments: MAPREDUCE-6191.patch TestJavaSerialization#testMapReduceJob() fails with getting incorrect MR job result: junit.framework.ComparisonFailure: expected:[a ]1 but was:[0 1]1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6185) YARN M/R may return NaN for reducer progress after Job completion
sam liu created MAPREDUCE-6185: -- Summary: YARN M/R may return NaN for reducer progress after Job completion Key: MAPREDUCE-6185 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6185 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.4.1 Reporter: sam liu Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6185) YARN M/R may return NaN for reducer progress after Job completion
[
https://issues.apache.org/jira/browse/MAPREDUCE-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14237635#comment-14237635
]
sam liu commented on MAPREDUCE-6185:
This issue is about method 'jobClient.getJob(jobid).reduceProgress()' and only
happens on the job which only has map tasks, but no reduce task, such as
Teragen.
[A] My Test Code:
// Using hadoop-mapreduce-client-core\org.apache.hadoop.mapred.JobClient.java
JobClient jobClient = new JobClient(new Configuration());
for (int i = 0; i 120; i++){
Thread.sleep(1000);
System.out.println(jobClient.getJob( + args[0] +
).reduceProgress()= + jobClient.getJob(args[0]).reduceProgress());
}
[B] WordCount: Got expected result
Can get reduceProgress during job running(0.0 to 1.0) and after job
completion(1.0).
$ hadoop jar yarn.hadoop.jar hadoop.yarn.proj.test.YarnClientTest
job_1417679271770_0169
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.2084
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.2084
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.2084
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.3334
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.3334
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.3334
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.6765683
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.6765683
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.6765683
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.76213264
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.76213264
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.76213264
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.8284452
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.8284452
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.8284452
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.9177592
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.9177592
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.9177592
jobClient.getJob(job_1417679271770_0169).reduceProgress()=1.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=1.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=1.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=1.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=1.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=1.0
14/12/04 22:54:21 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
jobClient.getJob(job_1417679271770_0169).reduceProgress()=1.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=1.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=1.0
[C] Teragen: Failed to get expected result
- Can get reduceProgress during job running(0.0 )
- But get NaN after job completion
As Teragen only has Mapper, but does not has Reducer, so I think it's correct.
$ hadoop jar yarn.hadoop.jar hadoop.yarn.proj.test.YarnClientTest
job_1417679271770_0168
14/12/04 22:51:53 INFO client.RMProxy: Connecting to ResourceManager at
cluster.nn1/9.31.111.201:8032
jobClient.getJob(job_1417679271770_0168).reduceProgress()=0.0
jobClient.getJob(job_1417679271770_0168).reduceProgress()=0.0
jobClient.getJob(job_1417679271770_0168).reduceProgress()=0.0
jobClient.getJob(job_1417679271770_0168).reduceProgress()=0.0
jobClient.getJob(job_1417679271770_0168).reduceProgress()=0.0
jobClient.getJob(job_1417679271770_0168).reduceProgress()=0.0
jobClient.getJob(job_1417679271770_0168).reduceProgress()=0.0
14/12/04 22:52:17 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
jobClient.getJob(job_1417679271770_0168).reduceProgress()=NaN
jobClient.getJob(job_1417679271770_0168).reduceProgress()=NaN
jobClient.getJob(job_1417679271770_0168).reduceProgress()=NaN
jobClient.getJob(job_1417679271770_0168).reduceProgress()=NaN
jobClient.getJob(job_1417679271770_0168).reduceProgress()=NaN
Expected result: '0.0'.
Actual result: 'NaN'.
Comment:
- Even after job completion, method reduceProgress() still should return same
result '0.0', but not 'NaN'.
- As 'Redirecting to job history server', marked the component of this jira as
'jobhistoryserver'
YARN M/R may return NaN for reducer progress after Job completion
-
[jira] [Updated] (MAPREDUCE-4490) JVM reuse is incompatible with LinuxTaskController (and therefore incompatible with Security)
[
https://issues.apache.org/jira/browse/MAPREDUCE-4490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sam liu updated MAPREDUCE-4490:
---
Status: Patch Available (was: Open)
JVM reuse is incompatible with LinuxTaskController (and therefore
incompatible with Security)
-
Key: MAPREDUCE-4490
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4490
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task-controller, tasktracker
Affects Versions: 1.2.1, 1.0.3, 0.20.205.0
Reporter: George Datskos
Assignee: sam liu
Priority: Critical
Labels: patch
Fix For: 1.2.1
Attachments: MAPREDUCE-4490.patch, MAPREDUCE-4490.patch,
MAPREDUCE-4490.patch
When using LinuxTaskController, JVM reuse (mapred.job.reuse.jvm.num.tasks
1) with more map tasks in a job than there are map slots in the cluster will
result in immediate task failures for the second task in each JVM (and then
the JVM exits). We have investigated this bug and the root cause is as
follows. When using LinuxTaskController, the userlog directory for a task
attempt (../userlogs/job/task-attempt) is created only on the first
invocation (when the JVM is launched) because userlogs directories are
created by the task-controller binary which only runs *once* per JVM.
Therefore, attempting to create log.index is guaranteed to fail with ENOENT
leading to immediate task failure and child JVM exit.
{quote}
2012-07-24 14:29:11,914 INFO org.apache.hadoop.mapred.TaskLog: Starting
logging for a new task attempt_201207241401_0013_m_27_0 in the same JVM
as that of the first task
/var/log/hadoop/mapred/userlogs/job_201207241401_0013/attempt_201207241401_0013_m_06_0
2012-07-24 14:29:11,915 WARN org.apache.hadoop.mapred.Child: Error running
child
ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
at
org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:161)
at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:296)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
at org.apache.hadoop.mapred.Child.main(Child.java:229)
{quote}
The above error occurs in a JVM which runs tasks 6 and 27. Task6 goes
smoothly. Then Task27 starts. The directory
/var/log/hadoop/mapred/userlogs/job_201207241401_0013/attempt_201207241401_0013_m_027_0
is never created so when mapred.Child tries to write the log.index file for
Task27, it fails with ENOENT because the
attempt_201207241401_0013_m_027_0 directory does not exist. Therefore,
the second task in each JVM is guaranteed to fail (and then the JVM exits)
every time when using LinuxTaskController. Note that this problem does not
occur when using the DefaultTaskController because the userlogs directories
are created for each task (not just for each JVM as with LinuxTaskController).
For each task, the TaskRunner calls the TaskController's createLogDir method
before attempting to write out an index file.
* DefaultTaskController#createLogDir: creates log directory for each task
* LinuxTaskController#createLogDir: does nothing
** task-controller binary creates log directory [create_attempt_directories]
(but only for the first task)
Possible Solution: add a new command to task-controller *initialize task* to
create attempt directories. Call that command, with ShellCommandExecutor, in
the LinuxTaskController#createLogDir method
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-4490) JVM reuse is incompatible with LinuxTaskController (and therefore incompatible with Security)
[
https://issues.apache.org/jira/browse/MAPREDUCE-4490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sam liu updated MAPREDUCE-4490:
---
Attachment: MAPREDUCE-4490.patch
New patch basing on latest branch origin/branch-1.2
JVM reuse is incompatible with LinuxTaskController (and therefore
incompatible with Security)
-
Key: MAPREDUCE-4490
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4490
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task-controller, tasktracker
Affects Versions: 0.20.205.0, 1.0.3, 1.2.1
Reporter: George Datskos
Assignee: sam liu
Priority: Critical
Labels: patch
Fix For: 1.2.1
Attachments: MAPREDUCE-4490.patch, MAPREDUCE-4490.patch,
MAPREDUCE-4490.patch
When using LinuxTaskController, JVM reuse (mapred.job.reuse.jvm.num.tasks
1) with more map tasks in a job than there are map slots in the cluster will
result in immediate task failures for the second task in each JVM (and then
the JVM exits). We have investigated this bug and the root cause is as
follows. When using LinuxTaskController, the userlog directory for a task
attempt (../userlogs/job/task-attempt) is created only on the first
invocation (when the JVM is launched) because userlogs directories are
created by the task-controller binary which only runs *once* per JVM.
Therefore, attempting to create log.index is guaranteed to fail with ENOENT
leading to immediate task failure and child JVM exit.
{quote}
2012-07-24 14:29:11,914 INFO org.apache.hadoop.mapred.TaskLog: Starting
logging for a new task attempt_201207241401_0013_m_27_0 in the same JVM
as that of the first task
/var/log/hadoop/mapred/userlogs/job_201207241401_0013/attempt_201207241401_0013_m_06_0
2012-07-24 14:29:11,915 WARN org.apache.hadoop.mapred.Child: Error running
child
ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
at
org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:161)
at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:296)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
at org.apache.hadoop.mapred.Child.main(Child.java:229)
{quote}
The above error occurs in a JVM which runs tasks 6 and 27. Task6 goes
smoothly. Then Task27 starts. The directory
/var/log/hadoop/mapred/userlogs/job_201207241401_0013/attempt_201207241401_0013_m_027_0
is never created so when mapred.Child tries to write the log.index file for
Task27, it fails with ENOENT because the
attempt_201207241401_0013_m_027_0 directory does not exist. Therefore,
the second task in each JVM is guaranteed to fail (and then the JVM exits)
every time when using LinuxTaskController. Note that this problem does not
occur when using the DefaultTaskController because the userlogs directories
are created for each task (not just for each JVM as with LinuxTaskController).
For each task, the TaskRunner calls the TaskController's createLogDir method
before attempting to write out an index file.
* DefaultTaskController#createLogDir: creates log directory for each task
* LinuxTaskController#createLogDir: does nothing
** task-controller binary creates log directory [create_attempt_directories]
(but only for the first task)
Possible Solution: add a new command to task-controller *initialize task* to
create attempt directories. Call that command, with ShellCommandExecutor, in
the LinuxTaskController#createLogDir method
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (MAPREDUCE-4490) JVM reuse is incompatible with LinuxTaskController (and therefore incompatible with Security)
[
https://issues.apache.org/jira/browse/MAPREDUCE-4490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sam liu updated MAPREDUCE-4490:
---
Status: Open (was: Patch Available)
Will upload new patch for latest code base of branch origin/branch-1.2
JVM reuse is incompatible with LinuxTaskController (and therefore
incompatible with Security)
-
Key: MAPREDUCE-4490
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4490
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task-controller, tasktracker
Affects Versions: 1.2.1, 1.0.3, 0.20.205.0
Reporter: George Datskos
Assignee: sam liu
Priority: Critical
Labels: patch
Fix For: 1.2.1
Attachments: MAPREDUCE-4490.patch, MAPREDUCE-4490.patch,
MAPREDUCE-4490.patch
When using LinuxTaskController, JVM reuse (mapred.job.reuse.jvm.num.tasks
1) with more map tasks in a job than there are map slots in the cluster will
result in immediate task failures for the second task in each JVM (and then
the JVM exits). We have investigated this bug and the root cause is as
follows. When using LinuxTaskController, the userlog directory for a task
attempt (../userlogs/job/task-attempt) is created only on the first
invocation (when the JVM is launched) because userlogs directories are
created by the task-controller binary which only runs *once* per JVM.
Therefore, attempting to create log.index is guaranteed to fail with ENOENT
leading to immediate task failure and child JVM exit.
{quote}
2012-07-24 14:29:11,914 INFO org.apache.hadoop.mapred.TaskLog: Starting
logging for a new task attempt_201207241401_0013_m_27_0 in the same JVM
as that of the first task
/var/log/hadoop/mapred/userlogs/job_201207241401_0013/attempt_201207241401_0013_m_06_0
2012-07-24 14:29:11,915 WARN org.apache.hadoop.mapred.Child: Error running
child
ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
at
org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:161)
at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:296)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
at org.apache.hadoop.mapred.Child.main(Child.java:229)
{quote}
The above error occurs in a JVM which runs tasks 6 and 27. Task6 goes
smoothly. Then Task27 starts. The directory
/var/log/hadoop/mapred/userlogs/job_201207241401_0013/attempt_201207241401_0013_m_027_0
is never created so when mapred.Child tries to write the log.index file for
Task27, it fails with ENOENT because the
attempt_201207241401_0013_m_027_0 directory does not exist. Therefore,
the second task in each JVM is guaranteed to fail (and then the JVM exits)
every time when using LinuxTaskController. Note that this problem does not
occur when using the DefaultTaskController because the userlogs directories
are created for each task (not just for each JVM as with LinuxTaskController).
For each task, the TaskRunner calls the TaskController's createLogDir method
before attempting to write out an index file.
* DefaultTaskController#createLogDir: creates log directory for each task
* LinuxTaskController#createLogDir: does nothing
** task-controller binary creates log directory [create_attempt_directories]
(but only for the first task)
Possible Solution: add a new command to task-controller *initialize task* to
create attempt directories. Call that command, with ShellCommandExecutor, in
the LinuxTaskController#createLogDir method
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Assigned] (MAPREDUCE-4798) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use'
[ https://issues.apache.org/jira/browse/MAPREDUCE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu reassigned MAPREDUCE-4798: -- Assignee: sam liu TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use' - Key: MAPREDUCE-4798 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4798 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, test Affects Versions: 1.0.3 Environment: Red Hat Ent Server 6.2 Reporter: sam liu Assignee: sam liu Priority: Minor Labels: patch Fix For: 1.1.2 Attachments: MAPREDUCE-4798.patch, MAPREDUCE-4798_branch-1.patch Original Estimate: 3h Remaining Estimate: 3h UT Failure in IHC 1.0.3: org.apache.hadoop.mapred.TestJobHistoryServer. This UT fails sometimes. The error message is: 'Testcase: testHistoryServerStandalone took 5.376 sec Caused an ERROR Address already in use java.lang.AssertionError: Address already in use at org.apache.hadoop.mapred.TestJobHistoryServer.testHistoryServerStandalone(TestJobHistoryServer.java:113)' -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (MAPREDUCE-4490) JVM reuse is incompatible with LinuxTaskController (and therefore incompatible with Security)
[
https://issues.apache.org/jira/browse/MAPREDUCE-4490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sam liu updated MAPREDUCE-4490:
---
Attachment: MAPREDUCE-4490.patch
Update patch to remove create_attempt_directories() invocation from
task-controller.c#run_task_as_user(). That invocation is unnecessary because
task-controller.c#initialize_task() always does same work.
JVM reuse is incompatible with LinuxTaskController (and therefore
incompatible with Security)
-
Key: MAPREDUCE-4490
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4490
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task-controller, tasktracker
Affects Versions: 0.20.205.0, 1.0.3, 1.2.1
Reporter: George Datskos
Assignee: sam liu
Priority: Critical
Labels: patch
Fix For: 1.2.1
Attachments: MAPREDUCE-4490.patch, MAPREDUCE-4490.patch
When using LinuxTaskController, JVM reuse (mapred.job.reuse.jvm.num.tasks
1) with more map tasks in a job than there are map slots in the cluster will
result in immediate task failures for the second task in each JVM (and then
the JVM exits). We have investigated this bug and the root cause is as
follows. When using LinuxTaskController, the userlog directory for a task
attempt (../userlogs/job/task-attempt) is created only on the first
invocation (when the JVM is launched) because userlogs directories are
created by the task-controller binary which only runs *once* per JVM.
Therefore, attempting to create log.index is guaranteed to fail with ENOENT
leading to immediate task failure and child JVM exit.
{quote}
2012-07-24 14:29:11,914 INFO org.apache.hadoop.mapred.TaskLog: Starting
logging for a new task attempt_201207241401_0013_m_27_0 in the same JVM
as that of the first task
/var/log/hadoop/mapred/userlogs/job_201207241401_0013/attempt_201207241401_0013_m_06_0
2012-07-24 14:29:11,915 WARN org.apache.hadoop.mapred.Child: Error running
child
ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
at
org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:161)
at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:296)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
at org.apache.hadoop.mapred.Child.main(Child.java:229)
{quote}
The above error occurs in a JVM which runs tasks 6 and 27. Task6 goes
smoothly. Then Task27 starts. The directory
/var/log/hadoop/mapred/userlogs/job_201207241401_0013/attempt_201207241401_0013_m_027_0
is never created so when mapred.Child tries to write the log.index file for
Task27, it fails with ENOENT because the
attempt_201207241401_0013_m_027_0 directory does not exist. Therefore,
the second task in each JVM is guaranteed to fail (and then the JVM exits)
every time when using LinuxTaskController. Note that this problem does not
occur when using the DefaultTaskController because the userlogs directories
are created for each task (not just for each JVM as with LinuxTaskController).
For each task, the TaskRunner calls the TaskController's createLogDir method
before attempting to write out an index file.
* DefaultTaskController#createLogDir: creates log directory for each task
* LinuxTaskController#createLogDir: does nothing
** task-controller binary creates log directory [create_attempt_directories]
(but only for the first task)
Possible Solution: add a new command to task-controller *initialize task* to
create attempt directories. Call that command, with ShellCommandExecutor, in
the LinuxTaskController#createLogDir method
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (MAPREDUCE-4490) JVM reuse is incompatible with LinuxTaskController (and therefore incompatible with Security)
[
https://issues.apache.org/jira/browse/MAPREDUCE-4490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sam liu reassigned MAPREDUCE-4490:
--
Assignee: sam liu
JVM reuse is incompatible with LinuxTaskController (and therefore
incompatible with Security)
-
Key: MAPREDUCE-4490
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4490
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task-controller, tasktracker
Affects Versions: 0.20.205.0, 1.0.3
Reporter: George Datskos
Assignee: sam liu
When using LinuxTaskController, JVM reuse (mapred.job.reuse.jvm.num.tasks
1) with more map tasks in a job than there are map slots in the cluster will
result in immediate task failures for the second task in each JVM (and then
the JVM exits). We have investigated this bug and the root cause is as
follows. When using LinuxTaskController, the userlog directory for a task
attempt (../userlogs/job/task-attempt) is created only on the first
invocation (when the JVM is launched) because userlogs directories are
created by the task-controller binary which only runs *once* per JVM.
Therefore, attempting to create log.index is guaranteed to fail with ENOENT
leading to immediate task failure and child JVM exit.
{quote}
2012-07-24 14:29:11,914 INFO org.apache.hadoop.mapred.TaskLog: Starting
logging for a new task attempt_201207241401_0013_m_27_0 in the same JVM
as that of the first task
/var/log/hadoop/mapred/userlogs/job_201207241401_0013/attempt_201207241401_0013_m_06_0
2012-07-24 14:29:11,915 WARN org.apache.hadoop.mapred.Child: Error running
child
ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
at
org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:161)
at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:296)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
at org.apache.hadoop.mapred.Child.main(Child.java:229)
{quote}
The above error occurs in a JVM which runs tasks 6 and 27. Task6 goes
smoothly. Then Task27 starts. The directory
/var/log/hadoop/mapred/userlogs/job_201207241401_0013/attempt_201207241401_0013_m_027_0
is never created so when mapred.Child tries to write the log.index file for
Task27, it fails with ENOENT because the
attempt_201207241401_0013_m_027_0 directory does not exist. Therefore,
the second task in each JVM is guaranteed to fail (and then the JVM exits)
every time when using LinuxTaskController. Note that this problem does not
occur when using the DefaultTaskController because the userlogs directories
are created for each task (not just for each JVM as with LinuxTaskController).
For each task, the TaskRunner calls the TaskController's createLogDir method
before attempting to write out an index file.
* DefaultTaskController#createLogDir: creates log directory for each task
* LinuxTaskController#createLogDir: does nothing
** task-controller binary creates log directory [create_attempt_directories]
(but only for the first task)
Possible Solution: add a new command to task-controller *initialize task* to
create attempt directories. Call that command, with ShellCommandExecutor, in
the LinuxTaskController#createLogDir method
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-4490) JVM reuse is incompatible with LinuxTaskController (and therefore incompatible with Security)
[
https://issues.apache.org/jira/browse/MAPREDUCE-4490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sam liu updated MAPREDUCE-4490:
---
Attachment: MAPREDUCE-4490.patch
Attached patch works well in my local environment and could resolve current
issue. Any feedback is welcome!
JVM reuse is incompatible with LinuxTaskController (and therefore
incompatible with Security)
-
Key: MAPREDUCE-4490
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4490
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task-controller, tasktracker
Affects Versions: 0.20.205.0, 1.0.3
Reporter: George Datskos
Assignee: sam liu
Attachments: MAPREDUCE-4490.patch
When using LinuxTaskController, JVM reuse (mapred.job.reuse.jvm.num.tasks
1) with more map tasks in a job than there are map slots in the cluster will
result in immediate task failures for the second task in each JVM (and then
the JVM exits). We have investigated this bug and the root cause is as
follows. When using LinuxTaskController, the userlog directory for a task
attempt (../userlogs/job/task-attempt) is created only on the first
invocation (when the JVM is launched) because userlogs directories are
created by the task-controller binary which only runs *once* per JVM.
Therefore, attempting to create log.index is guaranteed to fail with ENOENT
leading to immediate task failure and child JVM exit.
{quote}
2012-07-24 14:29:11,914 INFO org.apache.hadoop.mapred.TaskLog: Starting
logging for a new task attempt_201207241401_0013_m_27_0 in the same JVM
as that of the first task
/var/log/hadoop/mapred/userlogs/job_201207241401_0013/attempt_201207241401_0013_m_06_0
2012-07-24 14:29:11,915 WARN org.apache.hadoop.mapred.Child: Error running
child
ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
at
org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:161)
at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:296)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
at org.apache.hadoop.mapred.Child.main(Child.java:229)
{quote}
The above error occurs in a JVM which runs tasks 6 and 27. Task6 goes
smoothly. Then Task27 starts. The directory
/var/log/hadoop/mapred/userlogs/job_201207241401_0013/attempt_201207241401_0013_m_027_0
is never created so when mapred.Child tries to write the log.index file for
Task27, it fails with ENOENT because the
attempt_201207241401_0013_m_027_0 directory does not exist. Therefore,
the second task in each JVM is guaranteed to fail (and then the JVM exits)
every time when using LinuxTaskController. Note that this problem does not
occur when using the DefaultTaskController because the userlogs directories
are created for each task (not just for each JVM as with LinuxTaskController).
For each task, the TaskRunner calls the TaskController's createLogDir method
before attempting to write out an index file.
* DefaultTaskController#createLogDir: creates log directory for each task
* LinuxTaskController#createLogDir: does nothing
** task-controller binary creates log directory [create_attempt_directories]
(but only for the first task)
Possible Solution: add a new command to task-controller *initialize task* to
create attempt directories. Call that command, with ShellCommandExecutor, in
the LinuxTaskController#createLogDir method
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-4490) JVM reuse is incompatible with LinuxTaskController (and therefore incompatible with Security)
[
https://issues.apache.org/jira/browse/MAPREDUCE-4490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sam liu updated MAPREDUCE-4490:
---
Priority: Critical (was: Major)
JVM reuse is incompatible with LinuxTaskController (and therefore
incompatible with Security)
-
Key: MAPREDUCE-4490
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4490
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task-controller, tasktracker
Affects Versions: 0.20.205.0, 1.0.3
Reporter: George Datskos
Assignee: sam liu
Priority: Critical
Attachments: MAPREDUCE-4490.patch
When using LinuxTaskController, JVM reuse (mapred.job.reuse.jvm.num.tasks
1) with more map tasks in a job than there are map slots in the cluster will
result in immediate task failures for the second task in each JVM (and then
the JVM exits). We have investigated this bug and the root cause is as
follows. When using LinuxTaskController, the userlog directory for a task
attempt (../userlogs/job/task-attempt) is created only on the first
invocation (when the JVM is launched) because userlogs directories are
created by the task-controller binary which only runs *once* per JVM.
Therefore, attempting to create log.index is guaranteed to fail with ENOENT
leading to immediate task failure and child JVM exit.
{quote}
2012-07-24 14:29:11,914 INFO org.apache.hadoop.mapred.TaskLog: Starting
logging for a new task attempt_201207241401_0013_m_27_0 in the same JVM
as that of the first task
/var/log/hadoop/mapred/userlogs/job_201207241401_0013/attempt_201207241401_0013_m_06_0
2012-07-24 14:29:11,915 WARN org.apache.hadoop.mapred.Child: Error running
child
ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
at
org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:161)
at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:296)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
at org.apache.hadoop.mapred.Child.main(Child.java:229)
{quote}
The above error occurs in a JVM which runs tasks 6 and 27. Task6 goes
smoothly. Then Task27 starts. The directory
/var/log/hadoop/mapred/userlogs/job_201207241401_0013/attempt_201207241401_0013_m_027_0
is never created so when mapred.Child tries to write the log.index file for
Task27, it fails with ENOENT because the
attempt_201207241401_0013_m_027_0 directory does not exist. Therefore,
the second task in each JVM is guaranteed to fail (and then the JVM exits)
every time when using LinuxTaskController. Note that this problem does not
occur when using the DefaultTaskController because the userlogs directories
are created for each task (not just for each JVM as with LinuxTaskController).
For each task, the TaskRunner calls the TaskController's createLogDir method
before attempting to write out an index file.
* DefaultTaskController#createLogDir: creates log directory for each task
* LinuxTaskController#createLogDir: does nothing
** task-controller binary creates log directory [create_attempt_directories]
(but only for the first task)
Possible Solution: add a new command to task-controller *initialize task* to
create attempt directories. Call that command, with ShellCommandExecutor, in
the LinuxTaskController#createLogDir method
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-4490) JVM reuse is incompatible with LinuxTaskController (and therefore incompatible with Security)
[
https://issues.apache.org/jira/browse/MAPREDUCE-4490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sam liu updated MAPREDUCE-4490:
---
Fix Version/s: 1.2.1
Labels: patch (was: )
Target Version/s: 1.2.1
Affects Version/s: 1.2.1
Status: Patch Available (was: Open)
As above comments/description, the root cause of this issue is that userlogs
directories are created by the task-controller binary which only runs once per
JVM when using LinuxTaskController. So the major purpose of the patch is to add
a new command to task-controller initialize task to create attempt directories
and invoke it, with ShellCommandExecutor, in the
LinuxTaskController#createLogDir method. Below are the main details of the
modifications:
1. src/c++/task-controller/impl/task-controller.h:
Add declaration to new method initialize_task()
2. src/c++/task-controller/impl/task-controller.c:
Implement the new method initialize_task() which invokes existing method
create_attempt_directories()
3. src/c++/task-controller/impl/main.c:
To allow to invoke new method initialize_task() from ShellCommandExecutor
4. src/mapred/org/apache/hadoop/mapred/LinuxTaskController.java:
In method createLogDir() to invoke initialize_task() from ShellCommandExecutor
to create attempt directory before launching each task
JVM reuse is incompatible with LinuxTaskController (and therefore
incompatible with Security)
-
Key: MAPREDUCE-4490
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4490
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task-controller, tasktracker
Affects Versions: 1.2.1, 1.0.3, 0.20.205.0
Reporter: George Datskos
Assignee: sam liu
Priority: Critical
Labels: patch
Fix For: 1.2.1
Attachments: MAPREDUCE-4490.patch
When using LinuxTaskController, JVM reuse (mapred.job.reuse.jvm.num.tasks
1) with more map tasks in a job than there are map slots in the cluster will
result in immediate task failures for the second task in each JVM (and then
the JVM exits). We have investigated this bug and the root cause is as
follows. When using LinuxTaskController, the userlog directory for a task
attempt (../userlogs/job/task-attempt) is created only on the first
invocation (when the JVM is launched) because userlogs directories are
created by the task-controller binary which only runs *once* per JVM.
Therefore, attempting to create log.index is guaranteed to fail with ENOENT
leading to immediate task failure and child JVM exit.
{quote}
2012-07-24 14:29:11,914 INFO org.apache.hadoop.mapred.TaskLog: Starting
logging for a new task attempt_201207241401_0013_m_27_0 in the same JVM
as that of the first task
/var/log/hadoop/mapred/userlogs/job_201207241401_0013/attempt_201207241401_0013_m_06_0
2012-07-24 14:29:11,915 WARN org.apache.hadoop.mapred.Child: Error running
child
ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
at
org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:161)
at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:296)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
at org.apache.hadoop.mapred.Child.main(Child.java:229)
{quote}
The above error occurs in a JVM which runs tasks 6 and 27. Task6 goes
smoothly. Then Task27 starts. The directory
/var/log/hadoop/mapred/userlogs/job_201207241401_0013/attempt_201207241401_0013_m_027_0
is never created so when mapred.Child tries to write the log.index file for
Task27, it fails with ENOENT because the
attempt_201207241401_0013_m_027_0 directory does not exist. Therefore,
the second task in each JVM is guaranteed to fail (and then the JVM exits)
every time when using LinuxTaskController. Note that this problem does not
occur when using the DefaultTaskController because the userlogs directories
are created for each task (not just for each JVM as with LinuxTaskController).
For each task, the TaskRunner calls the TaskController's createLogDir method
before attempting to write out an index file.
* DefaultTaskController#createLogDir: creates log directory for each task
* LinuxTaskController#createLogDir: does nothing
** task-controller binary creates log directory [create_attempt_directories]
(but only for the first task)
Possible Solution: add a new command to task-controller *initialize task* to
create attempt directories. Call that command, with ShellCommandExecutor, in
the LinuxTaskController#createLogDir method
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5506) Hadoop-1.1.1 occurs ArrayIndexOutOfBoundsException with MultithreadedMapRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13768201#comment-13768201
]
sam liu commented on MAPREDUCE-5506:
MultithreadedMapRunner launches multiple threads to run map() method in
parallel, but seems all the threads share a singal object of OutputCollector to
write output of map() method into buffer. It should be the reason that the
ArrayIndexOutOfBoundsException occurs at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1331).
Therefore, I modified my WordCount MR application to move Mapper/Reducer
output.collect(...) statements into a 'synchronized (output)' block. And then,
no ArrayIndexOutOfBoundsException occurs any more, and the job could execute
successfully. Taking my updated code in map() method as example:
synchronized (output) {
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
output.collect(word, one);
}
}
And its original code is as below:
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
output.collect(word, one);
}
If that's the case, I would like to make a summay: When using
MultithreadedMapRunner, the ways to invoke OutputCollector object in Map/Reduce
methods should be thread safe. And a general way is to move Mapper/Reducer
output.collect(...) statements into a 'synchronized (output)' block.
Any comments will be welcomed!
Hadoop-1.1.1 occurs ArrayIndexOutOfBoundsException with MultithreadedMapRunner
--
Key: MAPREDUCE-5506
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5506
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Affects Versions: 1.1.1
Environment: RHEL 6.3 x86_64
Reporter: sam liu
Priority: Blocker
After I set:
- 'jobConf.setMapRunnerClass(MultithreadedMapRunner.class);' in MR app
- 'mapred.map.multithreadedrunner.threads = 2' in mapred-site.xml
A simple MR app failed as its Map task encountered
ArrayIndexOutOfBoundsException as below(please ignore the line numbers in the
exception as I added some log print codes):
java.lang.ArrayIndexOutOfBoundsException
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1331)
at java.io.DataOutputStream.write(DataOutputStream.java:101)
at org.apache.hadoop.io.Text.write(Text.java:282)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1060)
at
org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:591)
at study.hadoop.mapreduce.sample.WordCount$Map.map(WordCount.java:41)
at study.hadoop.mapreduce.sample.WordCount$Map.map(WordCount.java:1)
at
org.apache.hadoop.mapred.lib.MultithreadedMapRunner$MapperInvokeRunable.run(MultithreadedMapRunner.java:231)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
at java.lang.Thread.run(Thread.java:738)
And the exception happens on line 'System.arraycopy(b, off, kvbuffer,
bufindex, len)' in MapTask.java#MapOutputBuffer#Buffer#write(). When the
exception occurs, 'b.length=4' but 'len=9'.
Btw, if I set 'mapred.map.multithreadedrunner.threads = 1', no exception
happened. So it should be an issue caused by multiple threads.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5506) Hadoop-1.1.1 occurs ArrayIndexOutOfBoundsException with MultithreadedMapRunner
sam liu created MAPREDUCE-5506: -- Summary: Hadoop-1.1.1 occurs ArrayIndexOutOfBoundsException with MultithreadedMapRunner Key: MAPREDUCE-5506 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5506 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.1.1 Environment: RHEL 6.3 x86_64 Reporter: sam liu Priority: Blocker After I set: - 'jobConf.setMapRunnerClass(MultithreadedMapRunner.class);' in MR app - 'mapred.map.multithreadedrunner.threads = 2' in mapred-site.xml A simple MR app failed as its Map task encountered ArrayIndexOutOfBoundsException as below(please ignore the line numbers in the exception as I added some log print codes): java.lang.ArrayIndexOutOfBoundsException at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1331) at java.io.DataOutputStream.write(DataOutputStream.java:101) at org.apache.hadoop.io.Text.write(Text.java:282) at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90) at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1060) at org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:591) at study.hadoop.mapreduce.sample.WordCount$Map.map(WordCount.java:41) at study.hadoop.mapreduce.sample.WordCount$Map.map(WordCount.java:1) at org.apache.hadoop.mapred.lib.MultithreadedMapRunner$MapperInvokeRunable.run(MultithreadedMapRunner.java:231) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919) at java.lang.Thread.run(Thread.java:738) And the exception happens on line 'System.arraycopy(b, off, kvbuffer, bufindex, len)' in MapTask.java#MapOutputBuffer#Buffer#write(). When the exception occurs, 'b.length=4' but 'len=9'. Btw, if I set 'mapred.map.multithreadedrunner.threads = 1', no exception happened. So it should be an issue caused by multiple threads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4490) JVM reuse is incompatible with LinuxTaskController (and therefore incompatible with Security)
[
https://issues.apache.org/jira/browse/MAPREDUCE-4490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13744793#comment-13744793
]
sam liu commented on MAPREDUCE-4490:
Hi,
According to the description, I am trying to provide a patch, as we encountered
same issue in our Hadoop cluster.
First, I added a function in task-controller.c:
int initialize_task(const char* user,
const char * good_local_dirs, const char *job_id, const char *task_id) {
// Prepare the attempt directories for the task JVM.
int result = create_attempt_directories(user, good_local_dirs, job_id,
task_id);
return result;
}
Of cause, I also modified task-controller.h/task-controller.c/main.c. After
that, I try to call this feature through ShellCommandExecutor in
LinuxTaskController#createLogDir. However, I found the default
LinuxTaskController#createLogDir only has two input parameters (TaskAttemptID
taskID,boolean isCleanup), and does not satisfy the input parameters of
function initialize_task(const char* user,const char * good_local_dirs, const
char *job_id, const char *task_id): we can not get user, dir, jobid, taskid
from LinuxTaskController#createLogDir.
Any suggestions on the issue which is blocking my progress?
Thanks a lot!
JVM reuse is incompatible with LinuxTaskController (and therefore
incompatible with Security)
-
Key: MAPREDUCE-4490
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4490
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task-controller, tasktracker
Affects Versions: 0.20.205.0, 1.0.3
Reporter: George Datskos
When using LinuxTaskController, JVM reuse (mapred.job.reuse.jvm.num.tasks
1) with more map tasks in a job than there are map slots in the cluster will
result in immediate task failures for the second task in each JVM (and then
the JVM exits). We have investigated this bug and the root cause is as
follows. When using LinuxTaskController, the userlog directory for a task
attempt (../userlogs/job/task-attempt) is created only on the first
invocation (when the JVM is launched) because userlogs directories are
created by the task-controller binary which only runs *once* per JVM.
Therefore, attempting to create log.index is guaranteed to fail with ENOENT
leading to immediate task failure and child JVM exit.
{quote}
2012-07-24 14:29:11,914 INFO org.apache.hadoop.mapred.TaskLog: Starting
logging for a new task attempt_201207241401_0013_m_27_0 in the same JVM
as that of the first task
/var/log/hadoop/mapred/userlogs/job_201207241401_0013/attempt_201207241401_0013_m_06_0
2012-07-24 14:29:11,915 WARN org.apache.hadoop.mapred.Child: Error running
child
ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
at
org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:161)
at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:296)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
at org.apache.hadoop.mapred.Child.main(Child.java:229)
{quote}
The above error occurs in a JVM which runs tasks 6 and 27. Task6 goes
smoothly. Then Task27 starts. The directory
/var/log/hadoop/mapred/userlogs/job_201207241401_0013/attempt_201207241401_0013_m_027_0
is never created so when mapred.Child tries to write the log.index file for
Task27, it fails with ENOENT because the
attempt_201207241401_0013_m_027_0 directory does not exist. Therefore,
the second task in each JVM is guaranteed to fail (and then the JVM exits)
every time when using LinuxTaskController. Note that this problem does not
occur when using the DefaultTaskController because the userlogs directories
are created for each task (not just for each JVM as with LinuxTaskController).
For each task, the TaskRunner calls the TaskController's createLogDir method
before attempting to write out an index file.
* DefaultTaskController#createLogDir: creates log directory for each task
* LinuxTaskController#createLogDir: does nothing
** task-controller binary creates log directory [create_attempt_directories]
(but only for the first task)
Possible Solution: add a new command to task-controller *initialize task* to
create attempt directories. Call that command, with ShellCommandExecutor, in
the LinuxTaskController#createLogDir method
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5026) For shortening the time of TaskTracker heartbeat, decouple the statics collection operations
[ https://issues.apache.org/jira/browse/MAPREDUCE-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695213#comment-13695213 ] sam liu commented on MAPREDUCE-5026: In Hadoop 2.x, there is no TaskTracker and JobTracker any more, and we could not apply current fix on Hadoop 2.x code base directly. Further more, in Hadoop 2.x, the similar feature is NodeManager reports heartbeat to ResourceManager, and it uses NodeStatusUpdaterImpl.java for this feature. But, when get the slave resource info, it use NodeStatusUpdaterImpl.getNodeStatus() which collects containers info, not the machine resource info(cpu, memory, disk, ...) like Hadoop 1.x. So, I suggest only apply this fix on Hadoop 1.x, not Hadoop 2.x. Experts, any comments? For shortening the time of TaskTracker heartbeat, decouple the statics collection operations Key: MAPREDUCE-5026 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5026 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, tasktracker Affects Versions: 1.1.1 Reporter: sam liu Labels: patch Fix For: 1.1.1 Attachments: HDFS-4527.patch, HDFS-4527.patch Original Estimate: 24h Remaining Estimate: 24h In each heartbeat of TaskTracker, it will calculate some system statics, like the free disk space, available virtual/physical memory, cpu usage, etc. However, it's not necessary to calculate all the statics in every heartbeat, and this will consume many system resource and impace the performance of TaskTracker heartbeat. Furthermore, the characteristics of system properties(disk, memory, cpu) are different and it's better to collect their statics in different intervals. To reduce the latency of TaskTracker heartbeat, one solution is to decouple all the system statics collection operations from it, and issue separate threads to do the statics collection works when the TaskTracker starts. The threads could be three: the first one is to collect cpu related statics in a short interval; the second one is to collect memory related statics in a normal interval; the third one is to collect disk related statics in a long interval. And all the interval could be customized by the parameter mapred.stats.collection.interval in the mapred-site.xml. At last, the heartbeat could get values of system statics from the memory directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5026) For shortening the time of TaskTracker heartbeat, decouple the statics collection operations
[ https://issues.apache.org/jira/browse/MAPREDUCE-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13604555#comment-13604555 ] sam liu commented on MAPREDUCE-5026: Hi Tian Hong, As a general rule, their rates of change are different: cpu changes faster than memory, and memory changes faster than disk. So the frequency of fetching cpu should be more than memory, and memory should be more than disk. Finally, we could reasonably reduce resource consumption on the node running tasktracker. For shortening the time of TaskTracker heartbeat, decouple the statics collection operations Key: MAPREDUCE-5026 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5026 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, tasktracker Affects Versions: 1.1.1 Reporter: sam liu Labels: patch Fix For: 1.1.1 Attachments: HDFS-4527.patch Original Estimate: 24h Remaining Estimate: 24h In each heartbeat of TaskTracker, it will calculate some system statics, like the free disk space, available virtual/physical memory, cpu usage, etc. However, it's not necessary to calculate all the statics in every heartbeat, and this will consume many system resource and impace the performance of TaskTracker heartbeat. Furthermore, the characteristics of system properties(disk, memory, cpu) are different and it's better to collect their statics in different intervals. To reduce the latency of TaskTracker heartbeat, one solution is to decouple all the system statics collection operations from it, and issue separate threads to do the statics collection works when the TaskTracker starts. The threads could be three: the first one is to collect cpu related statics in a short interval; the second one is to collect memory related statics in a normal interval; the third one is to collect disk related statics in a long interval. And all the interval could be customized by the parameter mapred.stats.collection.interval in the mapred-site.xml. At last, the heartbeat could get values of system statics from the memory directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5026) For shortening the time of TaskTracker heartbeat, decouple the statics collection operations
[ https://issues.apache.org/jira/browse/MAPREDUCE-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-5026: --- Attachment: HDFS-4527.patch replace Statics with Statistics For shortening the time of TaskTracker heartbeat, decouple the statics collection operations Key: MAPREDUCE-5026 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5026 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, tasktracker Affects Versions: 1.1.1 Reporter: sam liu Labels: patch Fix For: 1.1.1 Attachments: HDFS-4527.patch, HDFS-4527.patch Original Estimate: 24h Remaining Estimate: 24h In each heartbeat of TaskTracker, it will calculate some system statics, like the free disk space, available virtual/physical memory, cpu usage, etc. However, it's not necessary to calculate all the statics in every heartbeat, and this will consume many system resource and impace the performance of TaskTracker heartbeat. Furthermore, the characteristics of system properties(disk, memory, cpu) are different and it's better to collect their statics in different intervals. To reduce the latency of TaskTracker heartbeat, one solution is to decouple all the system statics collection operations from it, and issue separate threads to do the statics collection works when the TaskTracker starts. The threads could be three: the first one is to collect cpu related statics in a short interval; the second one is to collect memory related statics in a normal interval; the third one is to collect disk related statics in a long interval. And all the interval could be customized by the parameter mapred.stats.collection.interval in the mapred-site.xml. At last, the heartbeat could get values of system statics from the memory directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5026) For shortening the time of TaskTracker heartbeat, decouple the statics collection operations
[ https://issues.apache.org/jira/browse/MAPREDUCE-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13604561#comment-13604561 ] sam liu commented on MAPREDUCE-5026: Hi Andrew, Sorry for replying late. In a simplest test: in one node cluster, the average time of executing 'HeartbeatResponse heartbeatResponse = transmitHeartBeat(now)' will spend 4.3 ms using the original TaskTracker.java, and the average time is from 1000 heartbeats. But, using the new TaskTracker.java, the time will be 2.1 ms in average. The efficiency improves a little more than 100%. For shortening the time of TaskTracker heartbeat, decouple the statics collection operations Key: MAPREDUCE-5026 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5026 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, tasktracker Affects Versions: 1.1.1 Reporter: sam liu Labels: patch Fix For: 1.1.1 Attachments: HDFS-4527.patch, HDFS-4527.patch Original Estimate: 24h Remaining Estimate: 24h In each heartbeat of TaskTracker, it will calculate some system statics, like the free disk space, available virtual/physical memory, cpu usage, etc. However, it's not necessary to calculate all the statics in every heartbeat, and this will consume many system resource and impace the performance of TaskTracker heartbeat. Furthermore, the characteristics of system properties(disk, memory, cpu) are different and it's better to collect their statics in different intervals. To reduce the latency of TaskTracker heartbeat, one solution is to decouple all the system statics collection operations from it, and issue separate threads to do the statics collection works when the TaskTracker starts. The threads could be three: the first one is to collect cpu related statics in a short interval; the second one is to collect memory related statics in a normal interval; the third one is to collect disk related statics in a long interval. And all the interval could be customized by the parameter mapred.stats.collection.interval in the mapred-site.xml. At last, the heartbeat could get values of system statics from the memory directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4798) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use'
[ https://issues.apache.org/jira/browse/MAPREDUCE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509435#comment-13509435 ] sam liu commented on MAPREDUCE-4798: Thanks Eric! Pls let me know if there is any further action TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use' - Key: MAPREDUCE-4798 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4798 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, test Affects Versions: 1.0.3 Environment: Red Hat Ent Server 6.2 Reporter: sam liu Priority: Minor Labels: patch Attachments: MAPREDUCE-4798_branch-1.patch, MAPREDUCE-4798.patch Original Estimate: 3h Remaining Estimate: 3h UT Failure in IHC 1.0.3: org.apache.hadoop.mapred.TestJobHistoryServer. This UT fails sometimes. The error message is: 'Testcase: testHistoryServerStandalone took 5.376 sec Caused an ERROR Address already in use java.lang.AssertionError: Address already in use at org.apache.hadoop.mapred.TestJobHistoryServer.testHistoryServerStandalone(TestJobHistoryServer.java:113)' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4798) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use'
[ https://issues.apache.org/jira/browse/MAPREDUCE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-4798: --- Attachment: (was: MAPREDUCE-4798.patch) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use' - Key: MAPREDUCE-4798 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4798 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, test Affects Versions: 1.0.3 Environment: Red Hat Ent Server 6.2 Reporter: sam liu Priority: Minor Labels: patch Fix For: 1.0.3 Original Estimate: 3h Remaining Estimate: 3h UT Failure in IHC 1.0.3: org.apache.hadoop.mapred.TestJobHistoryServer. This UT fails sometimes. The error message is: 'Testcase: testHistoryServerStandalone took 5.376 sec Caused an ERROR Address already in use java.lang.AssertionError: Address already in use at org.apache.hadoop.mapred.TestJobHistoryServer.testHistoryServerStandalone(TestJobHistoryServer.java:113)' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4798) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use'
[ https://issues.apache.org/jira/browse/MAPREDUCE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-4798: --- Attachment: MAPREDUCE-4798_branch-1.patch MAPREDUCE-4798.patch TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use' - Key: MAPREDUCE-4798 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4798 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, test Affects Versions: 1.0.3 Environment: Red Hat Ent Server 6.2 Reporter: sam liu Priority: Minor Labels: patch Fix For: 1.0.3 Attachments: MAPREDUCE-4798_branch-1.patch, MAPREDUCE-4798.patch Original Estimate: 3h Remaining Estimate: 3h UT Failure in IHC 1.0.3: org.apache.hadoop.mapred.TestJobHistoryServer. This UT fails sometimes. The error message is: 'Testcase: testHistoryServerStandalone took 5.376 sec Caused an ERROR Address already in use java.lang.AssertionError: Address already in use at org.apache.hadoop.mapred.TestJobHistoryServer.testHistoryServerStandalone(TestJobHistoryServer.java:113)' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4798) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use'
[ https://issues.apache.org/jira/browse/MAPREDUCE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505329#comment-13505329 ] sam liu commented on MAPREDUCE-4798: Eric, I updated the patch for 1.0.3 and uploaded the patch for branch-1. For trunk, there is no TestJobHistoryServer.java, so I did not generate another patch for trunk. Thanks! TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use' - Key: MAPREDUCE-4798 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4798 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, test Affects Versions: 1.0.3 Environment: Red Hat Ent Server 6.2 Reporter: sam liu Priority: Minor Labels: patch Fix For: 1.0.3 Attachments: MAPREDUCE-4798_branch-1.patch, MAPREDUCE-4798.patch Original Estimate: 3h Remaining Estimate: 3h UT Failure in IHC 1.0.3: org.apache.hadoop.mapred.TestJobHistoryServer. This UT fails sometimes. The error message is: 'Testcase: testHistoryServerStandalone took 5.376 sec Caused an ERROR Address already in use java.lang.AssertionError: Address already in use at org.apache.hadoop.mapred.TestJobHistoryServer.testHistoryServerStandalone(TestJobHistoryServer.java:113)' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4798) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use'
[ https://issues.apache.org/jira/browse/MAPREDUCE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-4798: --- Attachment: (was: MAPREDUCE-4798.patch) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use' - Key: MAPREDUCE-4798 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4798 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, test Affects Versions: 1.0.3 Environment: Red Hat Ent Server 6.2 Reporter: sam liu Priority: Minor Labels: patch Fix For: 1.0.3 Original Estimate: 3h Remaining Estimate: 3h UT Failure in IHC 1.0.3: org.apache.hadoop.mapred.TestJobHistoryServer. This UT fails sometimes. The error message is: 'Testcase: testHistoryServerStandalone took 5.376 sec Caused an ERROR Address already in use java.lang.AssertionError: Address already in use at org.apache.hadoop.mapred.TestJobHistoryServer.testHistoryServerStandalone(TestJobHistoryServer.java:113)' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4798) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use'
[ https://issues.apache.org/jira/browse/MAPREDUCE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-4798: --- Attachment: MAPREDUCE-4798.patch TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use' - Key: MAPREDUCE-4798 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4798 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, test Affects Versions: 1.0.3 Environment: Red Hat Ent Server 6.2 Reporter: sam liu Priority: Minor Labels: patch Fix For: 1.0.3 Attachments: MAPREDUCE-4798.patch Original Estimate: 3h Remaining Estimate: 3h UT Failure in IHC 1.0.3: org.apache.hadoop.mapred.TestJobHistoryServer. This UT fails sometimes. The error message is: 'Testcase: testHistoryServerStandalone took 5.376 sec Caused an ERROR Address already in use java.lang.AssertionError: Address already in use at org.apache.hadoop.mapred.TestJobHistoryServer.testHistoryServerStandalone(TestJobHistoryServer.java:113)' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4798) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use'
[ https://issues.apache.org/jira/browse/MAPREDUCE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-4798: --- Attachment: (was: MAPREDUCE-4798.patch) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use' - Key: MAPREDUCE-4798 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4798 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, test Affects Versions: 1.0.3 Environment: Red Hat Ent Server 6.2 Reporter: sam liu Priority: Minor Labels: patch Fix For: 1.0.3 Original Estimate: 3h Remaining Estimate: 3h UT Failure in IHC 1.0.3: org.apache.hadoop.mapred.TestJobHistoryServer. This UT fails sometimes. The error message is: 'Testcase: testHistoryServerStandalone took 5.376 sec Caused an ERROR Address already in use java.lang.AssertionError: Address already in use at org.apache.hadoop.mapred.TestJobHistoryServer.testHistoryServerStandalone(TestJobHistoryServer.java:113)' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4798) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use'
[ https://issues.apache.org/jira/browse/MAPREDUCE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-4798: --- Attachment: MAPREDUCE-4798.patch TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use' - Key: MAPREDUCE-4798 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4798 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, test Affects Versions: 1.0.3 Environment: Red Hat Ent Server 6.2 Reporter: sam liu Priority: Minor Labels: patch Fix For: 1.0.3 Attachments: MAPREDUCE-4798.patch Original Estimate: 3h Remaining Estimate: 3h UT Failure in IHC 1.0.3: org.apache.hadoop.mapred.TestJobHistoryServer. This UT fails sometimes. The error message is: 'Testcase: testHistoryServerStandalone took 5.376 sec Caused an ERROR Address already in use java.lang.AssertionError: Address already in use at org.apache.hadoop.mapred.TestJobHistoryServer.testHistoryServerStandalone(TestJobHistoryServer.java:113)' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4798) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use'
[ https://issues.apache.org/jira/browse/MAPREDUCE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-4798: --- Attachment: (was: MAPREDUCE-4798.patch) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use' - Key: MAPREDUCE-4798 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4798 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, test Affects Versions: 1.0.3 Environment: Red Hat Ent Server 6.2 Reporter: sam liu Priority: Minor Labels: patch Fix For: 1.0.3 Original Estimate: 3h Remaining Estimate: 3h UT Failure in IHC 1.0.3: org.apache.hadoop.mapred.TestJobHistoryServer. This UT fails sometimes. The error message is: 'Testcase: testHistoryServerStandalone took 5.376 sec Caused an ERROR Address already in use java.lang.AssertionError: Address already in use at org.apache.hadoop.mapred.TestJobHistoryServer.testHistoryServerStandalone(TestJobHistoryServer.java:113)' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4798) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use'
[ https://issues.apache.org/jira/browse/MAPREDUCE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-4798: --- Attachment: MAPREDUCE-4798.patch TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use' - Key: MAPREDUCE-4798 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4798 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, test Affects Versions: 1.0.3 Environment: Red Hat Ent Server 6.2 Reporter: sam liu Priority: Minor Labels: patch Fix For: 1.0.3 Attachments: MAPREDUCE-4798.patch Original Estimate: 3h Remaining Estimate: 3h UT Failure in IHC 1.0.3: org.apache.hadoop.mapred.TestJobHistoryServer. This UT fails sometimes. The error message is: 'Testcase: testHistoryServerStandalone took 5.376 sec Caused an ERROR Address already in use java.lang.AssertionError: Address already in use at org.apache.hadoop.mapred.TestJobHistoryServer.testHistoryServerStandalone(TestJobHistoryServer.java:113)' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4798) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use'
[ https://issues.apache.org/jira/browse/MAPREDUCE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13500060#comment-13500060 ] sam liu commented on MAPREDUCE-4798: Hi Eric, according to your suggestions, I updated the patch: - Indent the code properly - Set 4 retries of checking the URL redirect method, and add synthetic logic to pause 1, 2, 3, and 4 seconds upon each retry TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use' - Key: MAPREDUCE-4798 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4798 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, test Affects Versions: 1.0.3 Environment: Red Hat Ent Server 6.2 Reporter: sam liu Priority: Minor Labels: patch Fix For: 1.0.3 Attachments: MAPREDUCE-4798.patch Original Estimate: 3h Remaining Estimate: 3h UT Failure in IHC 1.0.3: org.apache.hadoop.mapred.TestJobHistoryServer. This UT fails sometimes. The error message is: 'Testcase: testHistoryServerStandalone took 5.376 sec Caused an ERROR Address already in use java.lang.AssertionError: Address already in use at org.apache.hadoop.mapred.TestJobHistoryServer.testHistoryServerStandalone(TestJobHistoryServer.java:113)' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4798) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use'
[ https://issues.apache.org/jira/browse/MAPREDUCE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-4798: --- Fix Version/s: 1.0.3 Labels: patch (was: ) Target Version/s: 1.0.3 Status: Patch Available (was: Open) The failures always happen in the step 'String redirectUrl = getRedirectUrl(job.getTrackingURL())' of testHistoryServerStandalone(). In the failure case, when executing into the method getRedirectUrl(job.getTrackingURL()), it always fails to execute 'Assert.assertEquals(status, HttpURLConnection.HTTP_MOVED_TEMP)' because the status value now is 200, but the HttpURLConnection.HTTP_MOVED_TEMP value is 302. In the successful cases., the status value will be 302. If add a break(sleep 10 sec) between 'int status = client.executeMethod(method)' and 'method.setFollowRedirects(false)', this case will be stable. I tried: 10 executions with 0 failure. TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use' - Key: MAPREDUCE-4798 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4798 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, test Affects Versions: 1.0.3 Environment: Red Hat Ent Server 6.2 Reporter: sam liu Priority: Minor Labels: patch Fix For: 1.0.3 Attachments: MAPREDUCE-4798.patch Original Estimate: 3h Remaining Estimate: 3h UT Failure in IHC 1.0.3: org.apache.hadoop.mapred.TestJobHistoryServer. This UT fails sometimes. The error message is: 'Testcase: testHistoryServerStandalone took 5.376 sec Caused an ERROR Address already in use java.lang.AssertionError: Address already in use at org.apache.hadoop.mapred.TestJobHistoryServer.testHistoryServerStandalone(TestJobHistoryServer.java:113)' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4798) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use'
[ https://issues.apache.org/jira/browse/MAPREDUCE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-4798: --- Attachment: MAPREDUCE-4798.patch TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use' - Key: MAPREDUCE-4798 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4798 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, test Affects Versions: 1.0.3 Environment: Red Hat Ent Server 6.2 Reporter: sam liu Priority: Minor Labels: patch Fix For: 1.0.3 Attachments: MAPREDUCE-4798.patch Original Estimate: 3h Remaining Estimate: 3h UT Failure in IHC 1.0.3: org.apache.hadoop.mapred.TestJobHistoryServer. This UT fails sometimes. The error message is: 'Testcase: testHistoryServerStandalone took 5.376 sec Caused an ERROR Address already in use java.lang.AssertionError: Address already in use at org.apache.hadoop.mapred.TestJobHistoryServer.testHistoryServerStandalone(TestJobHistoryServer.java:113)' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4798) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use'
[ https://issues.apache.org/jira/browse/MAPREDUCE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-4798: --- Status: Open (was: Patch Available) Just attached the patch now, but need the review of committer TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use' - Key: MAPREDUCE-4798 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4798 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, test Affects Versions: 1.0.3 Environment: Red Hat Ent Server 6.2 Reporter: sam liu Priority: Minor Labels: patch Fix For: 1.0.3 Attachments: MAPREDUCE-4798.patch Original Estimate: 3h Remaining Estimate: 3h UT Failure in IHC 1.0.3: org.apache.hadoop.mapred.TestJobHistoryServer. This UT fails sometimes. The error message is: 'Testcase: testHistoryServerStandalone took 5.376 sec Caused an ERROR Address already in use java.lang.AssertionError: Address already in use at org.apache.hadoop.mapred.TestJobHistoryServer.testHistoryServerStandalone(TestJobHistoryServer.java:113)' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4798) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use'
sam liu created MAPREDUCE-4798: -- Summary: TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use' Key: MAPREDUCE-4798 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4798 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, test Affects Versions: 1.0.3 Environment: Red Hat Ent Server 6.2 Reporter: sam liu Priority: Minor UT Failure in IHC 1.0.3: org.apache.hadoop.mapred.TestJobHistoryServer. This UT fails sometimes. The error message is: 'Testcase: testHistoryServerStandalone took 5.376 sec Caused an ERROR Address already in use java.lang.AssertionError: Address already in use at org.apache.hadoop.mapred.TestJobHistoryServer.testHistoryServerStandalone(TestJobHistoryServer.java:113)' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4798) TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use'
[ https://issues.apache.org/jira/browse/MAPREDUCE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13496816#comment-13496816 ] sam liu commented on MAPREDUCE-4798: Current issue is different from the existing TestJobHistoryServer related JIRAs, because it has a special failure reason: 'java.lang.AssertionError: Address already in use'. BTW, in my testing, this unit test has 1 failure in 6 executions. TestJobHistoryServer fails some times with 'java.lang.AssertionError: Address already in use' - Key: MAPREDUCE-4798 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4798 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, test Affects Versions: 1.0.3 Environment: Red Hat Ent Server 6.2 Reporter: sam liu Priority: Minor Original Estimate: 3h Remaining Estimate: 3h UT Failure in IHC 1.0.3: org.apache.hadoop.mapred.TestJobHistoryServer. This UT fails sometimes. The error message is: 'Testcase: testHistoryServerStandalone took 5.376 sec Caused an ERROR Address already in use java.lang.AssertionError: Address already in use at org.apache.hadoop.mapred.TestJobHistoryServer.testHistoryServerStandalone(TestJobHistoryServer.java:113)' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
