[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904477#comment-13904477 ] Xuan Gong commented on YARN-1734: - If FileSystemBasedConfigurationProvider is enabled, when RMs are initiated, they will read the latest Configuration files from remote file system. But It is very possible that the active RM refresh its configuration several times (update the configuration files several times), when the failover happens, the next active RM which will transit from standby to active can not get the latest configurations. My proposal: 1. When we call transitionToActive, we need to manually call all the refresh*s which can make sure that this active RM can update its configuration. 2. During this process, if there is any exceptions for these refresh*s, we will catch them and print out instead of bringing down the RM. 3. And we will return an inputStream from ConfigurationProvider. This may not be enough in future. I would like to create a new data structure named ConfigurationContext which can contain: * String configurationName * InputStream contentInputStream * long lastModifiedTime And we can add more information when we need, such as permission, user, groups, etc. And The configurationProvider can return ConfigurationContext instead of InputStream. > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Attachments: YARN-1734.1.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904483#comment-13904483 ] Xuan Gong commented on YARN-1734: - create a patch based on my proposal. I added empty yarn-site.xml and hadoop-policy.xml under hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/ and hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/. Because we are trying to read all the configuration files under the class path. (For using LocalConfiguraionProvider) > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Attachments: YARN-1734.1.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[
https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904557#comment-13904557
]
Hadoop QA commented on YARN-1734:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629606/YARN-1734.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests:
org.apache.hadoop.yarn.client.TestResourceManagerAdministrationProtocolPBClientImpl
org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
org.apache.hadoop.yarn.client.TestGetGroups
org.apache.hadoop.yarn.client.api.impl.TestNMClient
org.apache.hadoop.yarn.client.TestRMFailover
org.apache.hadoop.yarn.client.api.impl.TestYarnClient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3116//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3116//console
This message is automatically generated.
> RM should get the updated Configurations when it transits from Standby to
> Active
>
>
> Key: YARN-1734
> URL: https://issues.apache.org/jira/browse/YARN-1734
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Critical
> Attachments: YARN-1734.1.patch
>
>
> Currently, we have ConfigurationProvider which can support
> LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and
> FileSystemBasedConfiguration is enabled, RM can not get the updated
> Configurations when it transits from Standby to Active
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[
https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904599#comment-13904599
]
Steve Loughran commented on YARN-1734:
--
a long time ago I subclassed {{JobConfiguration}} to add one that picked up
its config from a directory service, not an XML object. The problem that
surfaced was that things copy the configs, and all it takes is one copy/clone
of a config to be cached somewhere and changes don't get picked up.
It may take some careful code review to make sure that every bit of the RM that
needs an updated config gets it, especially as some CompositeService
implementations override {{Service.init()}} to create a new
{{YarnConfiguration}} class
> RM should get the updated Configurations when it transits from Standby to
> Active
>
>
> Key: YARN-1734
> URL: https://issues.apache.org/jira/browse/YARN-1734
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Critical
> Attachments: YARN-1734.1.patch
>
>
> Currently, we have ConfigurationProvider which can support
> LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and
> FileSystemBasedConfiguration is enabled, RM can not get the updated
> Configurations when it transits from Standby to Active
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906224#comment-13906224 ] Xuan Gong commented on YARN-1734: - After YARN-1 is checked in, we do have InputStream object returned from ConfigurationProvider, so let us keep it. The new patch includes changes in AdminService. I create a set which includes function name, parameter type and parameter object for all refresh*s. And will manually call them after transitionToActive. In that case, the active RM can get the updated configuration. A test case is also included. > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Attachments: YARN-1734.1.patch, YARN-1734.2.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[
https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906352#comment-13906352
]
Hadoop QA commented on YARN-1734:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629914/YARN-1734.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3126//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3126//console
This message is automatically generated.
> RM should get the updated Configurations when it transits from Standby to
> Active
>
>
> Key: YARN-1734
> URL: https://issues.apache.org/jira/browse/YARN-1734
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Critical
> Attachments: YARN-1734.1.patch, YARN-1734.2.patch
>
>
> Currently, we have ConfigurationProvider which can support
> LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and
> FileSystemBasedConfiguration is enabled, RM can not get the updated
> Configurations when it transits from Standby to Active
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906445#comment-13906445 ] Xuan Gong commented on YARN-1734: - bq. Why is refreshAdminAcls() required to be done when transitioning state? It is possible that previous active rm has updated the AdminAcls. In that case, the current user may not have permission to do transitionToActive or transitionToStandby. That is why I want to do the checking before transitioning the state. > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Attachments: YARN-1734.1.patch, YARN-1734.2.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906462#comment-13906462 ] Xuan Gong commented on YARN-1734: - bq. Add code comments above the call if it is needed. Add comments bq. The user facing call refreshAdminAcls(RefreshAdminAclsRequest request) can invoke the same method to reuse code? Yes, we can do that. Modified bq. Validate that before the failover happens max-apps is still the default on the standby. Added bq. Fix formatting to not overflow 80 characters. fixed > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Attachments: YARN-1734.1.patch, YARN-1734.2.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[
https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906526#comment-13906526
]
Xuan Gong commented on YARN-1734:
-
throw out the IOException instead of just log exceptions.
{code}
try {
refreshAdminAcls(false);
} catch (YarnException ex) {
throw new IOException("Can not execute refreshAdminAcls", ex);
}
{code}
> RM should get the updated Configurations when it transits from Standby to
> Active
>
>
> Key: YARN-1734
> URL: https://issues.apache.org/jira/browse/YARN-1734
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Critical
> Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch,
> YARN-1734.4.patch
>
>
> Currently, we have ConfigurationProvider which can support
> LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and
> FileSystemBasedConfiguration is enabled, RM can not get the updated
> Configurations when it transits from Standby to Active
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[
https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906547#comment-13906547
]
Hadoop QA commented on YARN-1734:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629961/YARN-1734.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3129//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3129//console
This message is automatically generated.
> RM should get the updated Configurations when it transits from Standby to
> Active
>
>
> Key: YARN-1734
> URL: https://issues.apache.org/jira/browse/YARN-1734
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Critical
> Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch,
> YARN-1734.4.patch
>
>
> Currently, we have ConfigurationProvider which can support
> LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and
> FileSystemBasedConfiguration is enabled, RM can not get the updated
> Configurations when it transits from Standby to Active
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[
https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906588#comment-13906588
]
Hadoop QA commented on YARN-1734:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629982/YARN-1734.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3131//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3131//console
This message is automatically generated.
> RM should get the updated Configurations when it transits from Standby to
> Active
>
>
> Key: YARN-1734
> URL: https://issues.apache.org/jira/browse/YARN-1734
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Critical
> Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch,
> YARN-1734.4.patch
>
>
> Currently, we have ConfigurationProvider which can support
> LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and
> FileSystemBasedConfiguration is enabled, RM can not get the updated
> Configurations when it transits from Standby to Active
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906592#comment-13906592 ] Xuan Gong commented on YARN-1734: - should do the same (throw out the IOException instead of just log exceptions) for other refresh*s > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch, > YARN-1734.4.patch, YARN-1734.5.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[
https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906614#comment-13906614
]
Hadoop QA commented on YARN-1734:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629992/YARN-1734.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3133//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3133//console
This message is automatically generated.
> RM should get the updated Configurations when it transits from Standby to
> Active
>
>
> Key: YARN-1734
> URL: https://issues.apache.org/jira/browse/YARN-1734
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Critical
> Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch,
> YARN-1734.4.patch, YARN-1734.5.patch
>
>
> Currently, we have ConfigurationProvider which can support
> LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and
> FileSystemBasedConfiguration is enabled, RM can not get the updated
> Configurations when it transits from Standby to Active
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13907309#comment-13907309 ] Xuan Gong commented on YARN-1734: - bq. We reserve IOExceptions only for the RPC level errors. Please use YarnException for this. Can not throw YarnException in transitionToActive/transitionToStandby. They can only throw ServiceFailedException, AccessControlException,and IOException > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch, > YARN-1734.4.patch, YARN-1734.5.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[
https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13907432#comment-13907432
]
Hadoop QA commented on YARN-1734:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12630120/YARN-1734.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3137//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3137//console
This message is automatically generated.
> RM should get the updated Configurations when it transits from Standby to
> Active
>
>
> Key: YARN-1734
> URL: https://issues.apache.org/jira/browse/YARN-1734
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Critical
> Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch,
> YARN-1734.4.patch, YARN-1734.5.patch, YARN-1734.6.patch
>
>
> Currently, we have ConfigurationProvider which can support
> LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and
> FileSystemBasedConfiguration is enabled, RM can not get the updated
> Configurations when it transits from Standby to Active
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13910744#comment-13910744 ] Jian He commented on YARN-1734: --- ServiceFailedException is also one type of IOException that will be retried in RPC level by RMProxy > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch, > YARN-1734.4.patch, YARN-1734.5.patch, YARN-1734.6.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13910756#comment-13910756 ] Xuan Gong commented on YARN-1734: - bq. ServiceFailedException is also one type of IOException that will be retried in RPC level by RMProxy In HA, we provide different RetryPolicy which is failoverOnNetworkException > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch, > YARN-1734.4.patch, YARN-1734.5.patch, YARN-1734.6.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13911009#comment-13911009 ] Xuan Gong commented on YARN-1734: - bq. we will retry in the nonHA case? That also seems unwanted. AdminService#transitionToActive/transitionToStandby can only be called when HA is enabled. bq. One other comment related to the patch: The RefreshContext code is adding unnecessary complexity, let's just directly call each of the individual refresh methods? Sure. Removed. > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch, > YARN-1734.4.patch, YARN-1734.5.patch, YARN-1734.6.patch, YARN-1734.7.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[
https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13911052#comment-13911052
]
Hadoop QA commented on YARN-1734:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12630839/YARN-1734.7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3172//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3172//console
This message is automatically generated.
> RM should get the updated Configurations when it transits from Standby to
> Active
>
>
> Key: YARN-1734
> URL: https://issues.apache.org/jira/browse/YARN-1734
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Critical
> Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch,
> YARN-1734.4.patch, YARN-1734.5.patch, YARN-1734.6.patch, YARN-1734.7.patch
>
>
> Currently, we have ConfigurationProvider which can support
> LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and
> FileSystemBasedConfiguration is enabled, RM can not get the updated
> Configurations when it transits from Standby to Active
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1390#comment-1390 ] Vinod Kumar Vavilapalli commented on YARN-1734: --- bq. AdminService#transitionToActive/transitionToStandby can only be called when HA is enabled. Ah yes. That makes sense. The latest patch looks good. Checking this in. > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch, > YARN-1734.4.patch, YARN-1734.5.patch, YARN-1734.6.patch, YARN-1734.7.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13911135#comment-13911135 ] Hudson commented on YARN-1734: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5218 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5218/]) YARN-1734. Fixed ResourceManager to update the configurations when it transits from standby to active mode so as to assimilate any changes that happened while it was in standby mode. Contributed by Xuan Gong. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1571539) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Fix For: 2.4.0 > > Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch, > YARN-1734.4.patch, YARN-1734.5.patch, YARN-1734.6.patch, YARN-1734.7.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13911482#comment-13911482 ] Hudson commented on YARN-1734: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #492 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/492/]) YARN-1734. Fixed ResourceManager to update the configurations when it transits from standby to active mode so as to assimilate any changes that happened while it was in standby mode. Contributed by Xuan Gong. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1571539) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Fix For: 2.4.0 > > Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch, > YARN-1734.4.patch, YARN-1734.5.patch, YARN-1734.6.patch, YARN-1734.7.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13911582#comment-13911582 ] Hudson commented on YARN-1734: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1684 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1684/]) YARN-1734. Fixed ResourceManager to update the configurations when it transits from standby to active mode so as to assimilate any changes that happened while it was in standby mode. Contributed by Xuan Gong. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1571539) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Fix For: 2.4.0 > > Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch, > YARN-1734.4.patch, YARN-1734.5.patch, YARN-1734.6.patch, YARN-1734.7.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13911638#comment-13911638 ] Hudson commented on YARN-1734: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1709 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1709/]) YARN-1734. Fixed ResourceManager to update the configurations when it transits from standby to active mode so as to assimilate any changes that happened while it was in standby mode. Contributed by Xuan Gong. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1571539) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Fix For: 2.4.0 > > Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch, > YARN-1734.4.patch, YARN-1734.5.patch, YARN-1734.6.patch, YARN-1734.7.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918272#comment-13918272 ] Karthik Kambatla commented on YARN-1734: Sorry for jumping in late here. It appears the AdminService#refreshAll is called on transition to active. However, calling any of the refresh commands on the Standby throws StandbyException. This can lead to confusion - we throw an exception even though the refresh command takes affect when the RM transitions to Active. > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Fix For: 2.4.0 > > Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch, > YARN-1734.4.patch, YARN-1734.5.patch, YARN-1734.6.patch, YARN-1734.7.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918307#comment-13918307 ] Xuan Gong commented on YARN-1734: - bq. It appears the AdminService#refreshAll is called on transition to active. However, calling any of the refresh commands on the Standby throws StandbyException. This can lead to confusion - we throw an exception even though the refresh command takes affect when the RM transitions to Active. After rm.transitionToActive() is successfully executed, the rm is at Active state. So, it will not throw out StandbyException. > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Fix For: 2.4.0 > > Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch, > YARN-1734.4.patch, YARN-1734.5.patch, YARN-1734.6.patch, YARN-1734.7.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918365#comment-13918365 ] Karthik Kambatla commented on YARN-1734: In our case, we plan to use the LocalConfiguration and not the FileSystemBased one. So, in the HA case, we would update the local configs on both RMs and call the appropriate refresh command on both RMs - this is what we do for HDFS as well. The expectation is that the Active picks these up immediately, and the Standby picks them eventually when it becomes Active. In other words, the expectation is that these updates are not lost. With the current code, the Standby would throw a StandbyException, thereby telling the user that the config refresh has failed. This is not exactly true, because the Standby would actually pick the latest configs when transitioning to Active. No? Let me think more on this, but thought I should raise this concern. > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Fix For: 2.4.0 > > Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch, > YARN-1734.4.patch, YARN-1734.5.patch, YARN-1734.6.patch, YARN-1734.7.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918433#comment-13918433 ] Xuan Gong commented on YARN-1734: - So, if the Standby RM transits to Active, it will pick the latest configuration. For calling refresh* in standby RM, it will throw a standbyException and trigger the retry. In that case, even if we call refresh* in Standby RM, it actually do the refresh* in active RM. bq. With the current code, the Standby would throw a StandbyException, thereby telling the user that the config refresh has failed. This is not exactly true, because the Standby would actually pick the latest configs when transitioning to Active. No? When RM is at Standby state, all of the active services have already been stopped. I think this "pick the latest configs" should mean all the related services pick the latest configs, such as CapacityScheduler, NodesListManager, ClientRMService, ResourceTrackerService, etc. But since most of these services are stopped in standby mode, they can not get the latest configurations. > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Fix For: 2.4.0 > > Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch, > YARN-1734.4.patch, YARN-1734.5.patch, YARN-1734.6.patch, YARN-1734.7.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[
https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918437#comment-13918437
]
Karthik Kambatla commented on YARN-1734:
I guess the ambiguity stems from the definition of success for {{rmadmin
-refresh*}} commands.
I propose adding a config - yarn.resourcemanager.ha.refresh-all-rms. When set,
the refresh commands should attempt to refresh on all RMs and fail if it can't
- i.e., this should fail when called on the StandbyRM? When cleared, the
refresh command should attempt to refresh only on this RM and should succeed as
long as the configs are refreshed as early as they are required - i.e., it
should be okay to refresh on transition to active and the StandbyRM should also
succeed? [~xgong], [~vinodkv] - do you think this captures the behavior well
enough and is reasonable?
> RM should get the updated Configurations when it transits from Standby to
> Active
>
>
> Key: YARN-1734
> URL: https://issues.apache.org/jira/browse/YARN-1734
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Critical
> Fix For: 2.4.0
>
> Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch,
> YARN-1734.4.patch, YARN-1734.5.patch, YARN-1734.6.patch, YARN-1734.7.patch
>
>
> Currently, we have ConfigurationProvider which can support
> LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and
> FileSystemBasedConfiguration is enabled, RM can not get the updated
> Configurations when it transits from Standby to Active
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918441#comment-13918441 ] Karthik Kambatla commented on YARN-1734: bq. For calling refresh* in standby RM, it will throw a standbyException and trigger the retry. In that case, even if we call refresh* in Standby RM, it actually do the refresh* in active RM. Sorry, I missed this while browsing through the code. Let me try this on a cluster and report. > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Fix For: 2.4.0 > > Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch, > YARN-1734.4.patch, YARN-1734.5.patch, YARN-1734.6.patch, YARN-1734.7.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1734) RM should get the updated Configurations when it transits from Standby to Active
[ https://issues.apache.org/jira/browse/YARN-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13919031#comment-13919031 ] Karthik Kambatla commented on YARN-1734: Sorry for all the confusion caused here - forgot that the rmadmin command also uses ConfiguredRMFailoverProxyProvider. Played with a cluster with local configurations. It behaves as expected. refresh* refreshes the Active. The Standby refreshes everything on transition to active. Thanks [~xgong] for fixing the refresh commands, and for being patient with my questions/concerns. > RM should get the updated Configurations when it transits from Standby to > Active > > > Key: YARN-1734 > URL: https://issues.apache.org/jira/browse/YARN-1734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Fix For: 2.4.0 > > Attachments: YARN-1734.1.patch, YARN-1734.2.patch, YARN-1734.3.patch, > YARN-1734.4.patch, YARN-1734.5.patch, YARN-1734.6.patch, YARN-1734.7.patch > > > Currently, we have ConfigurationProvider which can support > LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and > FileSystemBasedConfiguration is enabled, RM can not get the updated > Configurations when it transits from Standby to Active -- This message was sent by Atlassian JIRA (v6.2#6252)
