[jira] [Commented] (MAPREDUCE-4168) Support multiple network interfaces
[ https://issues.apache.org/jira/browse/MAPREDUCE-4168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14262340#comment-14262340 ] Karthik Kambatla commented on MAPREDUCE-4168: - Too many discussion threads here, it is getting harder to follow. Just so we are all on the same page, MR/YARN already support multiple network interfaces. I left this JIRA open initially to address one inconvenience when submitting jobs using the RM conf. [~aw] - should we repurpose this JIRA to address the configuration issue? I think it might be better to open another JIRA for it. [~rajive] - are you proposing to not support multiple network interfaces? If so, could you please open another JIRA (or a discussion thread on yarn-dev) along with why the current approach is bad, and how the new proposed approach is better? > Support multiple network interfaces > --- > > Key: MAPREDUCE-4168 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4168 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Tom White > > Umbrella jira to track the MapReduce side of HADOOP-8198. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6149) Document override log4j.properties in MR job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14262254#comment-14262254 ] Hudson commented on MAPREDUCE-6149: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2009 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2009/]) MAPREDUCE-6149. Document override log4j.properties in MR job. Contributed by Junping Du. (harsh: rev 6621c3598e22279cde11eca73cfb5619a8bc8dee) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * hadoop-mapreduce-project/CHANGES.txt > Document override log4j.properties in MR job > > > Key: MAPREDUCE-6149 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6149 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Reporter: Junping Du >Assignee: Junping Du > Fix For: 2.7.0 > > Attachments: MAPREDUCE-6149-v2.patch, MAPREDUCE-6149-v3.patch, > MAPREDUCE-6149.patch > > > This new feature comes from MAPREDUCE-6052, some documentation requirements > from Vinod below: > Document the new config in mapred-default.xml > Mention in that documentation that if no-scheme is given in the path, it > defaults to a log4j file on the local FS. > Modify the documentation of log-level configs to say that if you override > to have your own log4j.properties file, the log-level configs may not work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6149) Document override log4j.properties in MR job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14262244#comment-14262244 ] Hudson commented on MAPREDUCE-6149: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #59 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/59/]) MAPREDUCE-6149. Document override log4j.properties in MR job. Contributed by Junping Du. (harsh: rev 6621c3598e22279cde11eca73cfb5619a8bc8dee) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml > Document override log4j.properties in MR job > > > Key: MAPREDUCE-6149 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6149 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Reporter: Junping Du >Assignee: Junping Du > Fix For: 2.7.0 > > Attachments: MAPREDUCE-6149-v2.patch, MAPREDUCE-6149-v3.patch, > MAPREDUCE-6149.patch > > > This new feature comes from MAPREDUCE-6052, some documentation requirements > from Vinod below: > Document the new config in mapred-default.xml > Mention in that documentation that if no-scheme is given in the path, it > defaults to a log4j file on the local FS. > Modify the documentation of log-level configs to say that if you override > to have your own log4j.properties file, the log-level configs may not work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6149) Document override log4j.properties in MR job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14262223#comment-14262223 ] Hudson commented on MAPREDUCE-6149: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #55 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/55/]) MAPREDUCE-6149. Document override log4j.properties in MR job. Contributed by Junping Du. (harsh: rev 6621c3598e22279cde11eca73cfb5619a8bc8dee) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml > Document override log4j.properties in MR job > > > Key: MAPREDUCE-6149 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6149 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Reporter: Junping Du >Assignee: Junping Du > Fix For: 2.7.0 > > Attachments: MAPREDUCE-6149-v2.patch, MAPREDUCE-6149-v3.patch, > MAPREDUCE-6149.patch > > > This new feature comes from MAPREDUCE-6052, some documentation requirements > from Vinod below: > Document the new config in mapred-default.xml > Mention in that documentation that if no-scheme is given in the path, it > defaults to a log4j file on the local FS. > Modify the documentation of log-level configs to say that if you override > to have your own log4j.properties file, the log-level configs may not work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6149) Document override log4j.properties in MR job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14262208#comment-14262208 ] Hudson commented on MAPREDUCE-6149: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #1990 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1990/]) MAPREDUCE-6149. Document override log4j.properties in MR job. Contributed by Junping Du. (harsh: rev 6621c3598e22279cde11eca73cfb5619a8bc8dee) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * hadoop-mapreduce-project/CHANGES.txt > Document override log4j.properties in MR job > > > Key: MAPREDUCE-6149 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6149 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Reporter: Junping Du >Assignee: Junping Du > Fix For: 2.7.0 > > Attachments: MAPREDUCE-6149-v2.patch, MAPREDUCE-6149-v3.patch, > MAPREDUCE-6149.patch > > > This new feature comes from MAPREDUCE-6052, some documentation requirements > from Vinod below: > Document the new config in mapred-default.xml > Mention in that documentation that if no-scheme is given in the path, it > defaults to a log4j file on the local FS. > Modify the documentation of log-level configs to say that if you override > to have your own log4j.properties file, the log-level configs may not work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6149) Document override log4j.properties in MR job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14262099#comment-14262099 ] Hudson commented on MAPREDUCE-6149: --- FAILURE: Integrated in Hadoop-Yarn-trunk #792 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/792/]) MAPREDUCE-6149. Document override log4j.properties in MR job. Contributed by Junping Du. (harsh: rev 6621c3598e22279cde11eca73cfb5619a8bc8dee) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * hadoop-mapreduce-project/CHANGES.txt > Document override log4j.properties in MR job > > > Key: MAPREDUCE-6149 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6149 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Reporter: Junping Du >Assignee: Junping Du > Fix For: 2.7.0 > > Attachments: MAPREDUCE-6149-v2.patch, MAPREDUCE-6149-v3.patch, > MAPREDUCE-6149.patch > > > This new feature comes from MAPREDUCE-6052, some documentation requirements > from Vinod below: > Document the new config in mapred-default.xml > Mention in that documentation that if no-scheme is given in the path, it > defaults to a log4j file on the local FS. > Modify the documentation of log-level configs to say that if you override > to have your own log4j.properties file, the log-level configs may not work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6149) Document override log4j.properties in MR job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14262094#comment-14262094 ] Hudson commented on MAPREDUCE-6149: --- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #58 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/58/]) MAPREDUCE-6149. Document override log4j.properties in MR job. Contributed by Junping Du. (harsh: rev 6621c3598e22279cde11eca73cfb5619a8bc8dee) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * hadoop-mapreduce-project/CHANGES.txt > Document override log4j.properties in MR job > > > Key: MAPREDUCE-6149 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6149 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Reporter: Junping Du >Assignee: Junping Du > Fix For: 2.7.0 > > Attachments: MAPREDUCE-6149-v2.patch, MAPREDUCE-6149-v3.patch, > MAPREDUCE-6149.patch > > > This new feature comes from MAPREDUCE-6052, some documentation requirements > from Vinod below: > Document the new config in mapred-default.xml > Mention in that documentation that if no-scheme is given in the path, it > defaults to a log4j file on the local FS. > Modify the documentation of log-level configs to say that if you override > to have your own log4j.properties file, the log-level configs may not work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4168) Support multiple network interfaces
[ https://issues.apache.org/jira/browse/MAPREDUCE-4168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14262085#comment-14262085 ] Rajiv Chittajallu commented on MAPREDUCE-4168: -- Restricting access to nodes managed by YARN by using proxies for outbound access: Cant this restriction be enforced via firewall, limiting external clients to only reach RM? Having separate physical network for RM to NM vs RM to clients assumes one can easily protect RM host instead of using a firewall to restrict access to nodes running nodemanagers. Reading through other comments and Hadoop-8198, requirement for multi homed network support is for redundancy and performance. This can't just be implemented by adding multiple IP's. Admin/oob vs cluster network: YARN and HDFS could function with out being aware of admin network. Non HDFS access: Changes proposed in this Jira are for core hadoop services. Network requirements for no hdfs access is up to respective client implementation. Compute node can have a Luster configured to use OFED infinband. This doesn't require changes to hdfs or yarn. Masters can be bound to one network. YARN could add support to manage interconnect as a resource. > Support multiple network interfaces > --- > > Key: MAPREDUCE-4168 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4168 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Tom White > > Umbrella jira to track the MapReduce side of HADOOP-8198. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6205) Update the value of the new version properties of the deprecated property "mapred.child.java.opts"
[
https://issues.apache.org/jira/browse/MAPREDUCE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14262063#comment-14262063
]
sam liu commented on MAPREDUCE-6205:
Gera,
Yes, there is an existing deprecation deltas
"mapred.map|reduce.child.java.opts" -> "mapreduce.map|reduce.java.opts".
However, after I add the deprecation delta as " mapred.child.java.opts" ->
"mapred.map|reduce.child.java.opts", the test
TestWritableJobConf#testEmptyConfiguration() failed. In this test, it compares
two configurations(conf and deser):
-conf: JobConf conf = new JobConf();
-deser: Configuration deser = serDeser(conf);
[A] Before I added the new deprecation delta, I got following output during
running this test and the test passed.
[A.1] The properties of 'conf':
mapred.child.java.opts=-Xmx200m
2014-12-31 01:13:12,092 INFO [main] Configuration.deprecation
(Configuration.java:warnOnceIfDeprecated(1013)) - mapred.map.child.java.opts is
deprecated. Instead, use mapreduce.map.java.opts
mapred.map.child.java.opts=null
2014-12-31 01:13:12,107 INFO [main] Configuration.deprecation
(Configuration.java:warnOnceIfDeprecated(1013)) - mapred.reduce.child.java.opts
is deprecated. Instead, use mapreduce.reduce.java.opts
mapred.reduce.child.java.opts=null
mapreduce.map.java.opts=null
mapreduce.reduce.java.opts=null
[A.2] The properties of 'deser':
mapred.child.java.opts=-Xmx200m
mapred.map.child.java.opts=null
mapred.reduce.child.java.opts=null
mapreduce.map.java.opts=null
mapreduce.reduce.java.opts=null
[B] After I added the new deprecation delta, I got following output during
running this test and the test failed.
[B.1] The properties of 'conf':
2014-12-31 01:06:22,313 INFO [main] Configuration.deprecation
(Configuration.java:warnOnceIfDeprecated(1013)) - mapred.child.java.opts is
deprecated. Instead, use mapred.map.child.java.opts,
mapred.reduce.child.java.opts
mapred.child.java.opts=-Xmx200m
mapred.map.child.java.opts=null
mapred.reduce.child.java.opts=null
mapreduce.map.java.opts=null
mapreduce.reduce.java.opts=null
[B.2] The properties of 'deser':
mapred.child.java.opts=-Xmx200m
mapred.map.child.java.opts=-Xmx200m
mapred.reduce.child.java.opts=-Xmx200m
mapreduce.map.java.opts=-Xmx200m
mapreduce.reduce.java.opts=-Xmx200m
Furthermore, if I did not add the deprecation delta as "mapred.child.java.opts"
-> "mapred.map|reduce.child.java.opts", but add the deprecation delta as
"mapred.child.java.opts" -> "mapreduce.map|reduce.java.opts", the test
TestWritableJobConf#testEmptyConfiguration() could pass.
What's your opinion?
Thanks!
> Update the value of the new version properties of the deprecated property
> "mapred.child.java.opts"
> --
>
> Key: MAPREDUCE-6205
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6205
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: sam liu
>Assignee: sam liu
>Priority: Minor
> Attachments: MAPREDUCE-6205.patch, MAPREDUCE-6205.patch
>
>
> In current hadoop code, the old property "mapred.child.java.opts" is
> deprecated and its new versions are MRJobConfig.MAP_JAVA_OPTS and
> MRJobConfig.REDUCE_JAVA_OPTS. However, when user set a value to the
> deprecated property "mapred.child.java.opts", hadoop won't automatically
> update its new versions properties
> MRJobConfig.MAP_JAVA_OPTS("mapreduce.map.java.opts") and
> MRJobConfig.REDUCE_JAVA_OPTS("mapreduce.reduce.java.opts"). As hadoop will
> update the new version properties for many other deprecated properties, we
> also should support such feature on the old property
> "mapred.child.java.opts", otherwise it might bring some imcompatible issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
