[jira] [Commented] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property "mapred.child.java.opts"
[
https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14268589#comment-14268589
]
Hadoop QA commented on MAPREDUCE-6205:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690653/MAPREDUCE-6205.003.patch
against trunk revision e13a484.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1219 javac
compiler warnings (more than the trunk's current 1218 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 13 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5099//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5099//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5099//artifact/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5099//console
This message is automatically generated.
> Update the value of the new version properties of the deprecated property
> "mapred.child.java.opts"
> --
>
> Key: MAPREDUCE-6205
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: sam liu
>Assignee: sam liu
>Priority: Minor
> Attachments: MAPREDUCE-6205.003.patch, MAPREDUCE-6205.patch,
> MAPREDUCE-6205.patch
>
>
> In current hadoop code, the old property "mapred.child.java.opts" is
> deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and
> MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the
> deprecated property "mapred.child.java.opts", hadoop won't automatically
> update its new versions properties
> MRJobConfig.MAP_JAVA_OPTS("mapreduce.map.java.opts") and
> MRJobConfig.REDUCE_JAVA_OPTS("mapreduce.reduce.java.opts"). As hadoop will
> update the new version properties for many other deprecated properties, we
> also should support such feature on the old property
> "mapred.child.java.opts", otherwise it might bring some imcompatible issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property "mapred.child.java.opts"
[
https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14268556#comment-14268556
]
Gera Shegalov commented on MAPREDUCE-6205:
--
I agree. I think the proposed deprecation chain handles this correctly but we
should definitely capture these requirements in unit tests. And I hope you can
review it [~jlowe].
> Update the value of the new version properties of the deprecated property
> "mapred.child.java.opts"
> --
>
> Key: MAPREDUCE-6205
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: sam liu
>Assignee: sam liu
>Priority: Minor
> Attachments: MAPREDUCE-6205.003.patch, MAPREDUCE-6205.patch,
> MAPREDUCE-6205.patch
>
>
> In current hadoop code, the old property "mapred.child.java.opts" is
> deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and
> MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the
> deprecated property "mapred.child.java.opts", hadoop won't automatically
> update its new versions properties
> MRJobConfig.MAP_JAVA_OPTS("mapreduce.map.java.opts") and
> MRJobConfig.REDUCE_JAVA_OPTS("mapreduce.reduce.java.opts"). As hadoop will
> update the new version properties for many other deprecated properties, we
> also should support such feature on the old property
> "mapred.child.java.opts", otherwise it might bring some imcompatible issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property "mapred.child.java.opts"
[
https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14268495#comment-14268495
]
Jason Lowe commented on MAPREDUCE-6205:
---
I think we need to be _very_ careful with a patch like this, or it could easily
create more problems than it solves. Something like this can readily break
backwards compatibility. For example, if a user does
"-Dmapreduce.map.java.opts=X -Dmapred.child.java.opts=Y", then maps should get
X and reduces should get Y. We need to make sure that this code doesn't end up
setting maps and reduces to Y because of the order we processed those settings
and how they're aliased. mapred.child.java.opts is indeed deprecated, but it
is not strictly an alias for the map/reduce-specific settings.
> Update the value of the new version properties of the deprecated property
> "mapred.child.java.opts"
> --
>
> Key: MAPREDUCE-6205
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: sam liu
>Assignee: sam liu
>Priority: Minor
> Attachments: MAPREDUCE-6205.003.patch, MAPREDUCE-6205.patch,
> MAPREDUCE-6205.patch
>
>
> In current hadoop code, the old property "mapred.child.java.opts" is
> deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and
> MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the
> deprecated property "mapred.child.java.opts", hadoop won't automatically
> update its new versions properties
> MRJobConfig.MAP_JAVA_OPTS("mapreduce.map.java.opts") and
> MRJobConfig.REDUCE_JAVA_OPTS("mapreduce.reduce.java.opts"). As hadoop will
> update the new version properties for many other deprecated properties, we
> also should support such feature on the old property
> "mapred.child.java.opts", otherwise it might bring some imcompatible issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property "mapred.child.java.opts"
[
https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267431#comment-14267431
]
Masatake Iwasaki commented on MAPREDUCE-6205:
-
bq. Uncomment both mapreduce.map|reduce.java.opts and set their value as
'-Xmx200m'.
[~sam liu], [the comment in
MAPREDUCE-5130|https://issues.apache.org/jira/browse/MAPREDUCE-5130?focusedCommentId=13623376&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13623376]
says it breaks compatibility.
> Update the value of the new version properties of the deprecated property
> "mapred.child.java.opts"
> --
>
> Key: MAPREDUCE-6205
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: sam liu
>Assignee: sam liu
>Priority: Minor
> Attachments: MAPREDUCE-6205.patch, MAPREDUCE-6205.patch
>
>
> In current hadoop code, the old property "mapred.child.java.opts" is
> deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and
> MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the
> deprecated property "mapred.child.java.opts", hadoop won't automatically
> update its new versions properties
> MRJobConfig.MAP_JAVA_OPTS("mapreduce.map.java.opts") and
> MRJobConfig.REDUCE_JAVA_OPTS("mapreduce.reduce.java.opts"). As hadoop will
> update the new version properties for many other deprecated properties, we
> also should support such feature on the old property
> "mapred.child.java.opts", otherwise it might bring some imcompatible issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property "mapred.child.java.opts"
[
https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267371#comment-14267371
]
sam liu commented on MAPREDUCE-6205:
Gera,
The root cause of above failure is that after deserializing both the deprecated
and the non-deprecated versions of a config are set. So, before deserializing,
mapreduce.reduce.java.opts is equals to $ {mapred.child.java.opts}; after
deserializing, mapreduce.reduce.java.opts is equals to '-Xmx200m'.
To resolve this issue, I have a proposal that in mapred-default.xml:
- Comment out the property 'mapred.child.java.opts'
- Uncomment both mapreduce.map|reduce.java.opts and set their value as
'-Xmx200m'.
I think this makes sense, because 'mapred.child.java.opts' is deprecated, but
'mapreduce.map|reduce.java.opts' are recommended by hadoop. With such
modifications, test TestWritableJobConf could pass. Also I ran some other
regarding tests, and I found only test TestJobConf should be updated till now.
What's your comments on the proposal?
Thanks!
> Update the value of the new version properties of the deprecated property
> "mapred.child.java.opts"
> --
>
> Key: MAPREDUCE-6205
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: sam liu
>Assignee: sam liu
>Priority: Minor
> Attachments: MAPREDUCE-6205.patch, MAPREDUCE-6205.patch
>
>
> In current hadoop code, the old property "mapred.child.java.opts" is
> deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and
> MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the
> deprecated property "mapred.child.java.opts", hadoop won't automatically
> update its new versions properties
> MRJobConfig.MAP_JAVA_OPTS("mapreduce.map.java.opts") and
> MRJobConfig.REDUCE_JAVA_OPTS("mapreduce.reduce.java.opts"). As hadoop will
> update the new version properties for many other deprecated properties, we
> also should support such feature on the old property
> "mapred.child.java.opts", otherwise it might bring some imcompatible issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property "mapred.child.java.opts"
[
https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265742#comment-14265742
]
sam liu commented on MAPREDUCE-6205:
Gera,
Thanks for your reminder!
After implementing suggestion 3(update mapred-default.xml), the unit test
failed on 'assertEquals(conf, deser)', but with different issue:
- the expected is: 'mapreduce.reduce.java.opts=${mapred.child.java.opts},
mapreduce.map.java.opts=${mapred.child.java.opts}'
- but is: 'mapreduce.reduce.java.opts=-Xmx200m,
mapreduce.map.java.opts=-Xmx200m'
However, I could get following output by 'conf.get(key)' or 'deser.get(key)'
during running this test:
1. The properties of 'conf':
mapred.child.java.opts=-Xmx200m
mapred.map.child.java.opts=-Xmx200m
mapred.reduce.child.java.opts=-Xmx200m
mapreduce.map.java.opts=-Xmx200m
mapreduce.reduce.java.opts=-Xmx200m
2. The properties of 'deser':
mapred.child.java.opts=-Xmx200m
mapred.map.child.java.opts=-Xmx200m
mapred.reduce.child.java.opts=-Xmx200m
mapreduce.map.java.opts=-Xmx200m
mapreduce.reduce.java.opts=-Xmx200m
Any comments?
> Update the value of the new version properties of the deprecated property
> "mapred.child.java.opts"
> --
>
> Key: MAPREDUCE-6205
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: sam liu
>Assignee: sam liu
>Priority: Minor
> Attachments: MAPREDUCE-6205.patch, MAPREDUCE-6205.patch
>
>
> In current hadoop code, the old property "mapred.child.java.opts" is
> deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and
> MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the
> deprecated property "mapred.child.java.opts", hadoop won't automatically
> update its new versions properties
> MRJobConfig.MAP_JAVA_OPTS("mapreduce.map.java.opts") and
> MRJobConfig.REDUCE_JAVA_OPTS("mapreduce.reduce.java.opts"). As hadoop will
> update the new version properties for many other deprecated properties, we
> also should support such feature on the old property
> "mapred.child.java.opts", otherwise it might bring some imcompatible issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property "mapred.child.java.opts"
[
https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265505#comment-14265505
]
Gera Shegalov commented on MAPREDUCE-6205:
--
[~sam liu], the test failure you see should go away if you implement suggestion
3 from my previous comment.
> Update the value of the new version properties of the deprecated property
> "mapred.child.java.opts"
> --
>
> Key: MAPREDUCE-6205
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: sam liu
>Assignee: sam liu
>Priority: Minor
> Attachments: MAPREDUCE-6205.patch, MAPREDUCE-6205.patch
>
>
> In current hadoop code, the old property "mapred.child.java.opts" is
> deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and
> MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the
> deprecated property "mapred.child.java.opts", hadoop won't automatically
> update its new versions properties
> MRJobConfig.MAP_JAVA_OPTS("mapreduce.map.java.opts") and
> MRJobConfig.REDUCE_JAVA_OPTS("mapreduce.reduce.java.opts"). As hadoop will
> update the new version properties for many other deprecated properties, we
> also should support such feature on the old property
> "mapred.child.java.opts", otherwise it might bring some imcompatible issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property "mapred.child.java.opts"
[
https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14262063#comment-14262063
]
sam liu commented on MAPREDUCE-6205:
Gera,
Yes, there is an existing deprecation deltas
"mapred.map|reduce.child.java.opts" -> "mapreduce.map|reduce.java.opts".
However, after I add the deprecation delta as " mapred.child.java.opts" ->
"mapred.map|reduce.child.java.opts", the test
TestWritableJobConf#testEmptyConfiguration() failed. In this test, it compares
two configurations(conf and deser):
-conf: JobConf conf = new JobConf();
-deser: Configuration deser = serDeser(conf);
[A] Before I added the new deprecation delta, I got following output during
running this test and the test passed.
[A.1] The properties of 'conf':
mapred.child.java.opts=-Xmx200m
2014-12-31 01:13:12,092 INFO [main] Configuration.deprecation
(Configuration.java:warnOnceIfDeprecated(1013)) - mapred.map.child.java.opts is
deprecated. Instead, use mapreduce.map.java.opts
mapred.map.child.java.opts=null
2014-12-31 01:13:12,107 INFO [main] Configuration.deprecation
(Configuration.java:warnOnceIfDeprecated(1013)) - mapred.reduce.child.java.opts
is deprecated. Instead, use mapreduce.reduce.java.opts
mapred.reduce.child.java.opts=null
mapreduce.map.java.opts=null
mapreduce.reduce.java.opts=null
[A.2] The properties of 'deser':
mapred.child.java.opts=-Xmx200m
mapred.map.child.java.opts=null
mapred.reduce.child.java.opts=null
mapreduce.map.java.opts=null
mapreduce.reduce.java.opts=null
[B] After I added the new deprecation delta, I got following output during
running this test and the test failed.
[B.1] The properties of 'conf':
2014-12-31 01:06:22,313 INFO [main] Configuration.deprecation
(Configuration.java:warnOnceIfDeprecated(1013)) - mapred.child.java.opts is
deprecated. Instead, use mapred.map.child.java.opts,
mapred.reduce.child.java.opts
mapred.child.java.opts=-Xmx200m
mapred.map.child.java.opts=null
mapred.reduce.child.java.opts=null
mapreduce.map.java.opts=null
mapreduce.reduce.java.opts=null
[B.2] The properties of 'deser':
mapred.child.java.opts=-Xmx200m
mapred.map.child.java.opts=-Xmx200m
mapred.reduce.child.java.opts=-Xmx200m
mapreduce.map.java.opts=-Xmx200m
mapreduce.reduce.java.opts=-Xmx200m
Furthermore, if I did not add the deprecation delta as "mapred.child.java.opts"
-> "mapred.map|reduce.child.java.opts", but add the deprecation delta as
"mapred.child.java.opts" -> "mapreduce.map|reduce.java.opts", the test
TestWritableJobConf#testEmptyConfiguration() could pass.
What's your opinion?
Thanks!
> Update the value of the new version properties of the deprecated property
> "mapred.child.java.opts"
> --
>
> Key: MAPREDUCE-6205
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: sam liu
>Assignee: sam liu
>Priority: Minor
> Attachments: MAPREDUCE-6205.patch, MAPREDUCE-6205.patch
>
>
> In current hadoop code, the old property "mapred.child.java.opts" is
> deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and
> MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the
> deprecated property "mapred.child.java.opts", hadoop won't automatically
> update its new versions properties
> MRJobConfig.MAP_JAVA_OPTS("mapreduce.map.java.opts") and
> MRJobConfig.REDUCE_JAVA_OPTS("mapreduce.reduce.java.opts"). As hadoop will
> update the new version properties for many other deprecated properties, we
> also should support such feature on the old property
> "mapred.child.java.opts", otherwise it might bring some imcompatible issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property "mapred.child.java.opts"
[
https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14260081#comment-14260081
]
Gera Shegalov commented on MAPREDUCE-6205:
--
[~sam liu], thanks for the patch.
# Please follow the guideline for including a version in patch names under
https://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch.
# The deprecation delta should be " mapred.child.java.opts" -> {
"mapred.map.child.java.opts", "mapred.reduce.child.java.opts" }. Otherwise you
introduce non-determinism due to existing deprecation deltas
"mapred.map|reduce.child.java.opts" -> "mapreduce.map|reduce.java.opts"
# mapred-default.xml should be updated. Currently
mapreduce.map|reduce.java.opts is commented out. Instead both should have thw
substitution value: {{$\{mapred.child.java.opts\}}}
> Update the value of the new version properties of the deprecated property
> "mapred.child.java.opts"
> --
>
> Key: MAPREDUCE-6205
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: sam liu
>Assignee: sam liu
>Priority: Minor
> Attachments: MAPREDUCE-6205.patch, MAPREDUCE-6205.patch
>
>
> In current hadoop code, the old property "mapred.child.java.opts" is
> deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and
> MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the
> deprecated property "mapred.child.java.opts", hadoop won't automatically
> update its new versions properties
> MRJobConfig.MAP_JAVA_OPTS("mapreduce.map.java.opts") and
> MRJobConfig.REDUCE_JAVA_OPTS("mapreduce.reduce.java.opts"). As hadoop will
> update the new version properties for many other deprecated properties, we
> also should support such feature on the old property
> "mapred.child.java.opts", otherwise it might bring some imcompatible issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property "mapred.child.java.opts"
[
https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14259141#comment-14259141
]
sam liu commented on MAPREDUCE-6205:
Yes, the MapReduceChildJVM#getChildJavaOpts makes the old property as fallback.
However, this solution has two disadvantages:
1. During map/reduce task execution, the program might get incorrect value of
new property MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS, if
only the value of "mapred.child.java.opts" is defined before.
2. Inconsistent handling style. For most of other deprecated properties, Hadoop
will automatically update the value of their new version with the value of old
version. However, hadoop does not use same style to handle the deprecation of
property "mapred.child.java.opts"
> Update the value of the new version properties of the deprecated property
> "mapred.child.java.opts"
> --
>
> Key: MAPREDUCE-6205
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: sam liu
>Assignee: sam liu
>Priority: Minor
> Attachments: MAPREDUCE-6205.patch, MAPREDUCE-6205.patch
>
>
> In current hadoop code, the old property "mapred.child.java.opts" is
> deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and
> MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the
> deprecated property "mapred.child.java.opts", hadoop won't automatically
> update its new versions properties
> MRJobConfig.MAP_JAVA_OPTS("mapreduce.map.java.opts") and
> MRJobConfig.REDUCE_JAVA_OPTS("mapreduce.reduce.java.opts"). As hadoop will
> update the new version properties for many other deprecated properties, we
> also should support such feature on the old property
> "mapred.child.java.opts", otherwise it might bring some imcompatible issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property "mapred.child.java.opts"
[
https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14259140#comment-14259140
]
sam liu commented on MAPREDUCE-6205:
Yes, the MapReduceChildJVM#getChildJavaOpts makes the old property as fallback.
However, this solution has two disadvantages:
1. During map/reduce task execution, the program might get incorrect value of
new property MRJobConfig.MAP_JAVA_OPTS and MRJobConfig.REDUCE_JAVA_OPTS, if
only the value of "mapred.child.java.opts" is defined before.
2. Inconsistent handling style. For most of other deprecated properties, Hadoop
will automatically update the value of their new version with the value of old
version. However, hadoop does not use same style to handle the deprecation of
property "mapred.child.java.opts"
> Update the value of the new version properties of the deprecated property
> "mapred.child.java.opts"
> --
>
> Key: MAPREDUCE-6205
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: sam liu
>Assignee: sam liu
>Priority: Minor
> Attachments: MAPREDUCE-6205.patch, MAPREDUCE-6205.patch
>
>
> In current hadoop code, the old property "mapred.child.java.opts" is
> deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and
> MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the
> deprecated property "mapred.child.java.opts", hadoop won't automatically
> update its new versions properties
> MRJobConfig.MAP_JAVA_OPTS("mapreduce.map.java.opts") and
> MRJobConfig.REDUCE_JAVA_OPTS("mapreduce.reduce.java.opts"). As hadoop will
> update the new version properties for many other deprecated properties, we
> also should support such feature on the old property
> "mapred.child.java.opts", otherwise it might bring some imcompatible issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property "mapred.child.java.opts"
[
https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258815#comment-14258815
]
Masatake Iwasaki commented on MAPREDUCE-6205:
-
The value of "mapred.child.java.opts" seems to be used as fallback in
MapReduceChildJVM#getChildJavaOpts. Is this not enough?
{code}
private static String getChildJavaOpts(JobConf jobConf, boolean isMapTask) {
String userClasspath = "";
String adminClasspath = "";
if (isMapTask) {
userClasspath =
jobConf.get(
JobConf.MAPRED_MAP_TASK_JAVA_OPTS,
jobConf.get(
JobConf.MAPRED_TASK_JAVA_OPTS,
JobConf.DEFAULT_MAPRED_TASK_JAVA_OPTS)
);
{code}
> Update the value of the new version properties of the deprecated property
> "mapred.child.java.opts"
> --
>
> Key: MAPREDUCE-6205
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: sam liu
>Assignee: sam liu
>Priority: Minor
> Attachments: MAPREDUCE-6205.patch, MAPREDUCE-6205.patch
>
>
> In current hadoop code, the old property "mapred.child.java.opts" is
> deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and
> MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the
> deprecated property "mapred.child.java.opts", hadoop won't automatically
> update its new versions properties
> MRJobConfig.MAP_JAVA_OPTS("mapreduce.map.java.opts") and
> MRJobConfig.REDUCE_JAVA_OPTS("mapreduce.reduce.java.opts"). As hadoop will
> update the new version properties for many other deprecated properties, we
> also should support such feature on the old property
> "mapred.child.java.opts", otherwise it might bring some imcompatible issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property "mapred.child.java.opts"
[
https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258158#comment-14258158
]
Hadoop QA commented on MAPREDUCE-6205:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689020/MAPREDUCE-6205.patch
against trunk revision 4f18018.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 16 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.ha.TestZKFailoverControllerStress
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5091//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5091//artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5091//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5091//console
This message is automatically generated.
> Update the value of the new version properties of the deprecated property
> "mapred.child.java.opts"
> --
>
> Key: MAPREDUCE-6205
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: sam liu
>Assignee: sam liu
>Priority: Minor
> Attachments: MAPREDUCE-6205.patch, MAPREDUCE-6205.patch
>
>
> In current hadoop code, the old property "mapred.child.java.opts" is
> deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and
> MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the
> deprecated property "mapred.child.java.opts", hadoop won't automatically
> update its new versions properties
> MRJobConfig.MAP_JAVA_OPTS("mapreduce.map.java.opts") and
> MRJobConfig.REDUCE_JAVA_OPTS("mapreduce.reduce.java.opts"). As hadoop will
> update the new version properties for many other deprecated properties, we
> also should support such feature on the old property
> "mapred.child.java.opts", otherwise it might bring some imcompatible issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
