[
https://issues.apache.org/jira/browse/SLIDER-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated SLIDER-1119:
--------------------------------
Description:
In some Hadoop cluster where Yarn registry is not enabled, we should not use it
in Slider. To check if Yarn registry is enabled or not, we can check an
existing hadoop configuration (hadoop.registry.rm.enabled).
Today the main use of Yarn registry is to securely publish Slider AM operation
URL, so that Slider agent can connect to Slider AM. Without Yarn registry, we
can directly pass in the AM operation URL as a command line parameter to
containers when launching the container.
Now admittedly without Yarn registry we lose Slider AM HA. This is because when
AM restarts, there is no way to tell containers where to find the new AM. In
this case AM would restart those containers. This may not be desirable for some
usage but is fine with some other usage scenarios.
Detailed changes needed:
1) In SliderAppMaster, disable service/component registration code if Yarn
registry is not enabled.
In SliderClient, disable zookeeper node creation/deletion if Yarn registry is
not enabled.
This is done in AbstractSliderLaunchedService.
2) In AbstractProviderService and AgentProviderService, do not use Yarn
registry if not enabled.
3) In AgentProviderService.buildContainerLaunchContext(), add "--am-ops-url"
parameter to the command line, pass in AM Ops URL from cluster info:
String amOpsUrl = getClusterInfoPropertyValue(StatusKeys.INFO_AM_AGENT_OPS_URL);
4) In agent python code, read "--am-ops-url" parameter and use it instead of
reading the value from Zookeeper
was:
In some Hadoop cluster where Yarn registry is not enabled, we should not use it
in Slider. To check if Yarn registry is enabled or not, we can check an
existing hadoop configuration (hadoop.registry.rm.enabled).
Today the main use of Yarn registry is to securely publish Slider AM operation
URL, so that Slider agent can connect to Slider AM. Without Yarn registry, we
can directly pass in the AM operation URL as a command line parameter to
containers when launching the container.
Now admittedly without Yarn registry we lose Slider AM HA. This is because when
AM restarts, there is no way to tell containers where to find the new AM. In
this case AM would restart those containers. This may not be desirable for some
usage but is fine with some other usage scenarios.
> Strip off Zookeeper dependency if Yarn Registry is not enabled
> --------------------------------------------------------------
>
> Key: SLIDER-1119
> URL: https://issues.apache.org/jira/browse/SLIDER-1119
> Project: Slider
> Issue Type: Bug
> Affects Versions: Slider 0.90.2
> Reporter: shanyu zhao
>
> In some Hadoop cluster where Yarn registry is not enabled, we should not use
> it in Slider. To check if Yarn registry is enabled or not, we can check an
> existing hadoop configuration (hadoop.registry.rm.enabled).
> Today the main use of Yarn registry is to securely publish Slider AM
> operation URL, so that Slider agent can connect to Slider AM. Without Yarn
> registry, we can directly pass in the AM operation URL as a command line
> parameter to containers when launching the container.
> Now admittedly without Yarn registry we lose Slider AM HA. This is because
> when AM restarts, there is no way to tell containers where to find the new
> AM. In this case AM would restart those containers. This may not be desirable
> for some usage but is fine with some other usage scenarios.
> Detailed changes needed:
> 1) In SliderAppMaster, disable service/component registration code if Yarn
> registry is not enabled.
> In SliderClient, disable zookeeper node creation/deletion if Yarn registry is
> not enabled.
> This is done in AbstractSliderLaunchedService.
> 2) In AbstractProviderService and AgentProviderService, do not use Yarn
> registry if not enabled.
> 3) In AgentProviderService.buildContainerLaunchContext(), add "--am-ops-url"
> parameter to the command line, pass in AM Ops URL from cluster info:
> String amOpsUrl =
> getClusterInfoPropertyValue(StatusKeys.INFO_AM_AGENT_OPS_URL);
> 4) In agent python code, read "--am-ops-url" parameter and use it instead of
> reading the value from Zookeeper
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
