[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566326#comment-14566326
]
Chun Chen commented on YARN-3749:
-
Upload a patch to fix it.
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566388#comment-14566388
]
Hadoop QA commented on YARN-3749:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 18m 23s | Findbugs (version 3.0.0)
appears to be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 5 new or modified test files. |
| {color:green}+1{color} | javac | 7m 34s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 40s | The applied patch generated 2
new checkstyle issues (total was 13, now 15). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 36s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 25s | Tests passed in
hadoop-yarn-api. |
| {color:red}-1{color} | yarn tests | 48m 27s | Tests failed in
hadoop-yarn-client. |
| {color:green}+1{color} | yarn tests | 50m 12s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| {color:red}-1{color} | yarn tests | 1m 49s | Tests failed in
hadoop-yarn-server-tests. |
| | | 145m 50s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.yarn.client.TestApplicationClientProtocolOnHA |
| | hadoop.yarn.client.TestApplicationMasterServiceProtocolOnHA |
| | hadoop.yarn.client.TestRMFailover |
| | hadoop.yarn.server.TestMiniYARNClusterForHA |
| Timed out tests | org.apache.hadoop.yarn.client.api.impl.TestAMRMClient |
| | org.apache.hadoop.yarn.client.api.impl.TestYarnClient |
| | org.apache.hadoop.yarn.client.api.impl.TestNMClient |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12736394/YARN-3749.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a8acdd6 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8142/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8142/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8142/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8142/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-tests test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8142/artifact/patchprocess/testrun_hadoop-yarn-server-tests.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8142/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8142/console |
This message was automatically generated.
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567369#comment-14567369
]
Hadoop QA commented on YARN-3749:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 39s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 6 new or modified test files. |
| {color:green}+1{color} | javac | 7m 34s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 20s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 35s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 26s | Tests passed in
hadoop-yarn-api. |
| {color:red}-1{color} | yarn tests | 49m 27s | Tests failed in
hadoop-yarn-client. |
| {color:green}+1{color} | yarn tests | 50m 14s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| {color:green}+1{color} | yarn tests | 1m 52s | Tests passed in
hadoop-yarn-server-tests. |
| | | 147m 20s | |
\\
\\
|| Reason || Tests ||
| Timed out tests | org.apache.hadoop.yarn.client.api.impl.TestAMRMClient |
| | org.apache.hadoop.yarn.client.api.impl.TestYarnClient |
| | org.apache.hadoop.yarn.client.api.impl.TestNMClient |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12736544/YARN-3749.2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 5cc3fce |
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8148/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8148/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8148/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-tests test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8148/artifact/patchprocess/testrun_hadoop-yarn-server-tests.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8148/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8148/console |
This message was automatically generated.
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: YARN-3749.2.patch, YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568187#comment-14568187
]
zhihai xu commented on YARN-3749:
-
Hi [~chenchun], thanks for filing and working on this issue.
The patch seems reasonable to me.
Some nits:
1. It looks like setRpcAddressForRM and setConfForRM are only used by test
code. Should we create a new HA test utility file to include these functions?
2. Do we really need the following change at {{MiniYARNCluster#serviceInit}}
{{code}}
conf.set(YarnConfiguration.RM_HA_ID, "rm0");
{{code}}
Because I saw {{initResourceManager}} will also configure {{RM_HA_ID}}.
3. Is any particular reason to configure {{YarnConfiguration.RM_HA_ID}} as
{{RM2_NODE_ID}} instead of {{RM1_NODE_ID}} in ProtocolHATestBase?
{{code}}
conf.set(YarnConfiguration.RM_HA_ID, RM2_NODE_ID);
{{code}}
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: YARN-3749.2.patch, YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568429#comment-14568429
]
Masatake Iwasaki commented on YARN-3749:
Thanks for working on this, [~chenchun]. I would like this fix to comes in
because it seems to affect YARN-3752 I'm looking into.
{quote}
2. Do we really need the following change at MiniYARNCluster#serviceInit
conf.set(YarnConfiguration.RM_HA_ID, "rm0");
Because I saw initResourceManager will also configure RM_HA_ID.
{quote}
When I tried similar to the patch, I got error below because
{{HAUtil#getRMHAId}} called from {{YarnConfiguration#updateConnectAddr}}
expects that there is at most 1 RM id matching to the node.
{noformat}
2015-06-02 10:14:23,648 INFO [Thread-284] service.AbstractService
(AbstractService.java:noteFailure(272)) - Service
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
failed in state STARTED; cause:
org.apache.hadoop.HadoopIllegalArgumentException: The HA Configuration has
multiple addresses that match local node's address.
org.apache.hadoop.HadoopIllegalArgumentException: The HA Configuration has
multiple addresses that match local node's address.
at org.apache.hadoop.yarn.conf.HAUtil.getRMHAId(HAUtil.java:204)
at
org.apache.hadoop.yarn.conf.YarnConfiguration.updateConnectAddr(YarnConfiguration.java:1971)
at
org.apache.hadoop.conf.Configuration.updateConnectAddr(Configuration.java:2129)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.serviceStart(ResourceLocalizationService.java:357)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:467)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:321)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.MiniYARNCluster$NodeManagerWrapper$1.run(MiniYARNCluster.java:562)
{noformat}
The check can be bypassed by setting dummy value to
{{yarn.resourcemanager.ha.id}} in configuration *used by NodeManager instance*.
I think there should be a comment explain that it is a dummy for unit test at
least.
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: YARN-3749.2.patch, YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568448#comment-14568448
]
Chun Chen commented on YARN-3749:
-
Thanks for the review [~zxu] [~iwasakims].
Upload a new patch to address your comments.
bq. 1. It looks like setRpcAddressForRM and setConfForRM are only used by test
code. Should we create a new HA test utility file to include these functions?
Moved setRpcAddressForRM and setConfForRM to HATestUtil.java
bq. 2. Do we really need the following change at MiniYARNCluster#serviceInit
conf.set(YarnConfiguration.RM_HA_ID, "rm0");
This is indeed necessary, as [~iwasakims]'s comments, this is used to bypass
the check in `HAUtil#getRMHAId` used by NodeManager instance.
bq. 3. Is any particular reason to configure YarnConfiguration.RM_HA_ID as
RM2_NODE_ID instead of RM1_NODE_ID in ProtocolHATestBase?
Not really, changed it to RM1_NODE_ID.
bq. I think there should be a comment explain that it is a dummy for unit test
at least.
Added a comment in `MiniYARNCluster#serviceInit`
Also the newly uploaded patch YARN-3749.4.patch only make a copy of the
configuration in initResourceManager when there are multiple RMs. If there is
only one RM, many test case in yarn-client depends on the random ports after RM
starts.
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568458#comment-14568458
]
Chun Chen commented on YARN-3749:
-
Upload a YARN-3749.5.patch to set {{YarnConfiguration.RM_HA_ID}} only in
{{MiniYARNCluster#serviceInit}}, removed that in other tests.
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568492#comment-14568492
]
zhihai xu commented on YARN-3749:
-
[~iwasakims], thanks for the finding and explanations!
[~chenchun], thanks for updating the patch!
After looking deeper into the code, I am more convinced this is a bug in
{{YarnConfiguration#updateConnectAddr}}.
IMHO, we should change {{if (HAUtil.isHAEnabled(this))}} to {{if
(HAUtil.isHAEnabled(this) && getServiceAddressConfKeys(this).contains(name))}}
to match the code in {{YarnConfiguration#getSocketAddr}}. It doesn't sound
right to add RM_HA_ID suffix to NM service address
"yarn.nodemanager.localizer.address". Also there will be a problem if we call
{{getSocketAddr}} after {{updateConnectAddr}} for address Property
"yarn.nodemanager.localizer.address".
I think we can fix the HadoopIllegalArgumentException with this change.
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568509#comment-14568509
]
Hadoop QA commented on YARN-3749:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 27s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 7 new or modified test files. |
| {color:green}+1{color} | javac | 7m 32s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 34s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 29s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 4s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 7m 18s | Tests passed in
hadoop-yarn-client. |
| {color:red}-1{color} | yarn tests | 50m 8s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| {color:green}+1{color} | yarn tests | 1m 51s | Tests passed in
hadoop-yarn-server-tests. |
| | | 100m 56s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12736720/YARN-3749.3.patch |
| Optional Tests | javac unit findbugs checkstyle javadoc |
| git revision | trunk / 990078b |
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8155/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8155/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-tests test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8155/artifact/patchprocess/testrun_hadoop-yarn-server-tests.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8155/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8155/console |
This message was automatically generated.
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568518#comment-14568518
]
Hadoop QA commented on YARN-3749:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 42s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 7 new or modified test files. |
| {color:green}+1{color} | javac | 7m 37s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 43s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 27s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 1s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 7m 3s | Tests passed in
hadoop-yarn-client. |
| {color:green}+1{color} | yarn tests | 50m 17s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| {color:green}+1{color} | yarn tests | 1m 53s | Tests passed in
hadoop-yarn-server-tests. |
| | | 101m 23s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12736721/YARN-3749.4.patch |
| Optional Tests | javac unit findbugs checkstyle javadoc |
| git revision | trunk / 990078b |
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8156/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8156/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-tests test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8156/artifact/patchprocess/testrun_hadoop-yarn-server-tests.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8156/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8156/console |
This message was automatically generated.
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568521#comment-14568521
]
Chun Chen commented on YARN-3749:
-
[~zxu] ,thanks, I agree. Upload YARN-3749.6.patch to address your comments.
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.6.patch, YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568543#comment-14568543
]
Hadoop QA commented on YARN-3749:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 12s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 7 new or modified test files. |
| {color:green}+1{color} | javac | 7m 34s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 38s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 27s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 59s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 7m 3s | Tests passed in
hadoop-yarn-client. |
| {color:red}-1{color} | yarn tests | 50m 8s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| {color:green}+1{color} | yarn tests | 1m 52s | Tests passed in
hadoop-yarn-server-tests. |
| | | 100m 26s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12736724/YARN-3749.5.patch |
| Optional Tests | javac unit findbugs checkstyle javadoc |
| git revision | trunk / 990078b |
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8157/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8157/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-tests test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8157/artifact/patchprocess/testrun_hadoop-yarn-server-tests.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8157/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8157/console |
This message was automatically generated.
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.6.patch, YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568555#comment-14568555
]
zhihai xu commented on YARN-3749:
-
[~chenchun], thanks for the new patch, It looks like we need keep
{{conf.set(YarnConfiguration.RM_HA_ID, RM1_NODE_ID);}} in TestRMEmbeddedElector
to fix this test failure.
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.6.patch, YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568586#comment-14568586
]
Chun Chen commented on YARN-3749:
-
bq. It looks like we need keep conf.set(YarnConfiguration.RM_HA_ID,
RM1_NODE_ID); in TestRMEmbeddedElector to fix this test failure.
Sorry, my bad. Upload YARN-3749.7.patch to fix that and add a tests in
{{TestYarnConfiguration}} to make sure {{YarnConfiguration#updateConnectAddr}}
won't add suffix to NM service address configurations.
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.6.patch, YARN-3749.7.patch, YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568667#comment-14568667
]
Hadoop QA commented on YARN-3749:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 29s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 7 new or modified test files. |
| {color:green}+1{color} | javac | 7m 36s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 33s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 49s | The applied patch generated 1
new checkstyle issues (total was 212, now 213). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 34s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 25s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 7m 1s | Tests passed in
hadoop-yarn-client. |
| {color:red}-1{color} | yarn tests | 60m 25s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| {color:green}+1{color} | yarn tests | 1m 52s | Tests passed in
hadoop-yarn-server-tests. |
| | | 115m 5s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
| Timed out tests |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12736732/YARN-3749.6.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 990078b |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8158/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8158/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8158/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8158/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-tests test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8158/artifact/patchprocess/testrun_hadoop-yarn-server-tests.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8158/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8158/console |
This message was automatically generated.
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.6.patch, YARN-3749.7.patch, YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568735#comment-14568735
]
Hadoop QA commented on YARN-3749:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 20m 3s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 8 new or modified test files. |
| {color:green}+1{color} | javac | 7m 34s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 42s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 17s | The applied patch generated 1
new checkstyle issues (total was 212, now 213). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 6m 5s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 22s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 6m 58s | Tests passed in
hadoop-yarn-client. |
| {color:green}+1{color} | yarn tests | 1m 57s | Tests passed in
hadoop-yarn-common. |
| {color:red}-1{color} | yarn tests | 60m 34s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| {color:green}+1{color} | yarn tests | 1m 51s | Tests passed in
hadoop-yarn-server-tests. |
| | | 121m 2s | |
\\
\\
|| Reason || Tests ||
| Timed out tests |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12736753/YARN-3749.7.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 990078b |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8163/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8163/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8163/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8163/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8163/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-tests test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8163/artifact/patchprocess/testrun_hadoop-yarn-server-tests.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8163/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8163/console |
This message was automatically generated.
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.6.patch, YARN-3749.7.patch, YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddr
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14570131#comment-14570131
]
zhihai xu commented on YARN-3749:
-
Hi [~chenchun], thanks for updating the patch quickly
bq. only make a copy of the configuration in initResourceManager when there are
multiple RMs.
It is a nice optimization.
The lastest patch YARN-3749.7.patch LGTM
Also the test failure(TestNodeLabelContainerAllocation) is not related to the
patch.
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.6.patch, YARN-3749.7.patch, YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14570202#comment-14570202
]
Chun Chen commented on YARN-3749:
-
Thanks for reviewing the patch, [~zxu] !
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.6.patch, YARN-3749.7.patch, YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14571614#comment-14571614
]
Hadoop QA commented on YARN-3749:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 20m 2s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 8 new or modified test files. |
| {color:green}+1{color} | javac | 7m 35s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 14s | The applied patch generated 1
new checkstyle issues (total was 212, now 213). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 6m 11s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 25s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 7m 0s | Tests passed in
hadoop-yarn-client. |
| {color:green}+1{color} | yarn tests | 1m 57s | Tests passed in
hadoop-yarn-common. |
| {color:red}-1{color} | yarn tests | 60m 24s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| {color:green}+1{color} | yarn tests | 1m 52s | Tests passed in
hadoop-yarn-server-tests. |
| | | 120m 57s | |
\\
\\
|| Reason || Tests ||
| Timed out tests |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12737337/YARN-3749.7.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / c59e745 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8181/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8181/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8181/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8181/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8181/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-tests test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8181/artifact/patchprocess/testrun_hadoop-yarn-server-tests.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8181/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8181/console |
This message was automatically generated.
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.6.patch, YARN-3749.7.patch, YARN-3749.7.patch,
> YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14571879#comment-14571879
]
Xuan Gong commented on YARN-3749:
-
＋1. LGTM. Will commit
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.6.patch, YARN-3749.7.patch, YARN-3749.7.patch,
> YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14571886#comment-14571886
]
Xuan Gong commented on YARN-3749:
-
Committed into trunk/branch-2. Thanks, Chun Chen. And thanks for review, zhihai
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Fix For: 2.8.0
>
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.6.patch, YARN-3749.7.patch, YARN-3749.7.patch,
> YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14571896#comment-14571896
]
Hudson commented on YARN-3749:
--
FAILURE: Integrated in Hadoop-trunk-Commit #7958 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7958/])
YARN-3749. We should make a copy of configuration when init (xgong: rev
5766a04428f65bb008b5c451f6f09e61e1000300)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationMasterServiceProtocolOnHA.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/HATestUtil.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestMiniYarnCluster.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMEmbeddedElector.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestRMFailover.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Fix For: 2.8.0
>
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.6.patch, YARN-3749.7.patch, YARN-3749.7.patch,
> YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14571957#comment-14571957
]
Chun Chen commented on YARN-3749:
-
Thanks for reviewing and committing the patch, [~xgong].
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Fix For: 2.8.0
>
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.6.patch, YARN-3749.7.patch, YARN-3749.7.patch,
> YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572596#comment-14572596
]
Hudson commented on YARN-3749:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #218 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/218/])
YARN-3749. We should make a copy of configuration when init (xgong: rev
5766a04428f65bb008b5c451f6f09e61e1000300)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestMiniYarnCluster.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMEmbeddedElector.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/HATestUtil.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationMasterServiceProtocolOnHA.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestRMFailover.java
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Fix For: 2.8.0
>
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.6.patch, YARN-3749.7.patch, YARN-3749.7.patch,
> YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572617#comment-14572617
]
Hudson commented on YARN-3749:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #948 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/948/])
YARN-3749. We should make a copy of configuration when init (xgong: rev
5766a04428f65bb008b5c451f6f09e61e1000300)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestMiniYarnCluster.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestRMFailover.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationMasterServiceProtocolOnHA.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMEmbeddedElector.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/HATestUtil.java
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Fix For: 2.8.0
>
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.6.patch, YARN-3749.7.patch, YARN-3749.7.patch,
> YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572866#comment-14572866
]
Hudson commented on YARN-3749:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk #2146 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2146/])
YARN-3749. We should make a copy of configuration when init (xgong: rev
5766a04428f65bb008b5c451f6f09e61e1000300)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMEmbeddedElector.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationMasterServiceProtocolOnHA.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestMiniYarnCluster.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestRMFailover.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/HATestUtil.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Fix For: 2.8.0
>
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.6.patch, YARN-3749.7.patch, YARN-3749.7.patch,
> YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572894#comment-14572894
]
Hudson commented on YARN-3749:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #207 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/207/])
YARN-3749. We should make a copy of configuration when init (xgong: rev
5766a04428f65bb008b5c451f6f09e61e1000300)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestMiniYarnCluster.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationMasterServiceProtocolOnHA.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/HATestUtil.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestRMFailover.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMEmbeddedElector.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Fix For: 2.8.0
>
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.6.patch, YARN-3749.7.patch, YARN-3749.7.patch,
> YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573011#comment-14573011
]
Hudson commented on YARN-3749:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #216 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/216/])
YARN-3749. We should make a copy of configuration when init (xgong: rev
5766a04428f65bb008b5c451f6f09e61e1000300)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestMiniYarnCluster.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestRMFailover.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationMasterServiceProtocolOnHA.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/HATestUtil.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMEmbeddedElector.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Fix For: 2.8.0
>
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.6.patch, YARN-3749.7.patch, YARN-3749.7.patch,
> YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573039#comment-14573039
]
Hudson commented on YARN-3749:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2164 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2164/])
YARN-3749. We should make a copy of configuration when init (xgong: rev
5766a04428f65bb008b5c451f6f09e61e1000300)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/HATestUtil.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestMiniYarnCluster.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationMasterServiceProtocolOnHA.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestRMFailover.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMEmbeddedElector.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java
> We should make a copy of configuration when init MiniYARNCluster with
> multiple RMs
> --
>
> Key: YARN-3749
> URL: https://issues.apache.org/jira/browse/YARN-3749
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chun Chen
>Assignee: Chun Chen
> Fix For: 2.8.0
>
> Attachments: YARN-3749.2.patch, YARN-3749.3.patch, YARN-3749.4.patch,
> YARN-3749.5.patch, YARN-3749.6.patch, YARN-3749.7.patch, YARN-3749.7.patch,
> YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client
> trying to connect to both rm1 and rm2 with the same address 0.0.0.0:18032
> when RM failover. But I initially set
> yarn.resourcemanager.address.rm1=0.0.0.0:18032,
> yarn.resourcemanager.address.rm2=0.0.0.0:28032 After digging, I found it is
> in ClientRMService where the value of yarn.resourcemanager.address.rm2
> changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>YarnConfiguration.RM_ADDRESS,
>
> YarnConfiguration.DEFAULT_RM_ADDRESS,
>server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both
> RM before we start both RM, we will change yarn.resourcemanager.ha.id to rm2
> during init of rm2 and yarn.resourcemanager.ha.id will become rm2 during
> starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the
> rm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
