YARN-7323. Data structure update in service REST API. Contributed by Jian He
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/68acd88d Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/68acd88d Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/68acd88d Branch: refs/heads/trunk Commit: 68acd88dcbfe03a0134c60b5398dfaa31ad2b786 Parents: 3c30b1a Author: Billie Rinaldi <bil...@apache.org> Authored: Tue Oct 17 11:50:16 2017 -0700 Committer: Jian He <jia...@apache.org> Committed: Mon Nov 6 13:30:18 2017 -0800 ---------------------------------------------------------------------- .../hadoop/yarn/service/webapp/ApiServer.java | 9 --- ...RN-Simplified-V1-API-Layer-For-Services.yaml | 67 ++++++++--------- .../hadoop/yarn/service/ServiceScheduler.java | 1 + .../yarn/service/api/records/Component.java | 22 +++++- .../service/api/records/ComponentState.java | 30 ++++++++ .../yarn/service/api/records/Container.java | 15 ++-- .../yarn/service/api/records/Service.java | 76 -------------------- .../yarn/service/api/records/ServiceState.java | 2 +- .../yarn/service/client/ServiceClient.java | 45 ++++++++---- .../yarn/service/component/Component.java | 12 ++++ .../component/instance/ComponentInstance.java | 48 +++++++++---- .../yarn/service/conf/RestApiConstants.java | 4 -- .../yarn/service/utils/ServiceApiUtil.java | 53 ++------------ .../hadoop/yarn/service/ServiceTestUtils.java | 4 +- .../hadoop/yarn/service/TestServiceApiUtil.java | 74 ++++++------------- .../yarn/service/TestYarnNativeServices.java | 4 +- .../service/conf/examples/app-override.json | 8 ++- .../hadoop/yarn/service/conf/examples/app.json | 8 ++- .../yarn/service/conf/examples/external0.json | 15 ++-- .../markdown/yarn-service/YarnServiceAPI.md | 22 +++--- 20 files changed, 232 insertions(+), 287 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/68acd88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java index 5773069..1bb6c93 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java @@ -242,15 +242,6 @@ public class ApiServer { return updateLifetime(appName, updateServiceData); } - // flex a single component app - if (updateServiceData.getNumberOfContainers() != null && !ServiceApiUtil - .hasComponent(updateServiceData)) { - Component defaultComp = ServiceApiUtil - .createDefaultComponent(updateServiceData); - return updateComponent(updateServiceData.getName(), defaultComp.getName(), - defaultComp); - } - // If nothing happens consider it a no-op return Response.status(Status.NO_CONTENT).build(); } http://git-wip-us.apache.org/repos/asf/hadoop/blob/68acd88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/resources/definition/YARN-Simplified-V1-API-Layer-For-Services.yaml ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/resources/definition/YARN-Simplified-V1-API-Layer-For-Services.yaml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/resources/definition/YARN-Simplified-V1-API-Layer-For-Services.yaml index b9b5b3a..cc76259 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/resources/definition/YARN-Simplified-V1-API-Layer-For-Services.yaml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/resources/definition/YARN-Simplified-V1-API-Layer-For-Services.yaml @@ -19,28 +19,17 @@ swagger: '2.0' info: title: "YARN Simplified API layer for services" description: | - Bringing a new service on YARN today is not a simple experience. The APIs of - existing frameworks are either too low level (native YARN), require writing - new code (for frameworks with programmatic APIs) or writing a complex spec - (for declarative frameworks). In addition to building critical building blocks - inside YARN (as part of other efforts at YARN-4692), there is a need for - simplifying the user facing story for building services. Experience of projects - like Apache Slider running real-life services like HBase, Storm, Accumulo, - Solr etc, gives us some very good insights on how simplified APIs for services - should look like. + Bringing a new service on YARN today is not a simple experience. The APIs of existing + frameworks are either too low level (native YARN), require writing new code (for frameworks with programmatic APIs) + or writing a complex spec (for declarative frameworks). - To this end, we should look at a new simple-services API layer backed by REST - interfaces. This API can be used to create and manage the lifecycle of YARN - services. Services here can range from simple single-component service to - complex multi-component assemblies needing orchestration. YARN-4793 tracks - this effort. + This simplified REST API can be used to create and manage the lifecycle of YARN services. + In most cases, the application owner will not be forced to make any changes to their applications. + This is primarily true if the application is packaged with containerization technologies like Docker. - This document spotlights on this specification. In most of the cases, the - application owner will not be forced to make any changes to their applications. - This is primarily true if the application is packaged with containerization - technologies like docker. Irrespective of how complex the application is, - there will be hooks provided at appropriate layers to allow pluggable and - customizable application behavior. + This document describes the API specifications (aka. YarnFile) for deploying/managing + containerized services on YARN. The same JSON spec can be used for both REST API + and CLI to manage the services. version: "1.0.0" license: @@ -216,22 +205,15 @@ definitions: type: string description: A unique service id. artifact: - description: Artifact of single-component service. + description: The default artifact for all components of the service except the components which has Artifact type set to SERVICE (optional). $ref: '#/definitions/Artifact' resource: - description: Resource of single-component service or the global default for multi-component services. Mandatory if it is a single-component service and if cpus and memory are not specified at the Service level. + description: The default resource for all components of the service (optional). $ref: '#/definitions/Resource' - launch_command: - type: string - description: The custom launch command of a service component (optional). If not specified for services with docker images say, it will default to the default start command of the image. If there is a single component in this service, you can specify this without the need to have a 'components' section. launch_time: type: string format: date description: The time when the service was created, e.g. 2016-03-16T01:01:49.000Z. - number_of_containers: - type: integer - format: int64 - description: Number of containers for each component in the service. Each component can further override this service-level global default. number_of_running_containers: type: integer format: int64 @@ -251,13 +233,8 @@ definitions: configuration: description: Config properties of a service. Configurations provided at the service/global level are available to all the components. Specific properties can be overridden at the component level. $ref: '#/definitions/Configuration' - containers: - description: Containers of a started service. Specifying a value for this attribute for the POST payload raises a validation error. This blob is available only in the GET response of a started service. - type: array - items: - $ref: '#/definitions/Container' state: - description: State of the service. Specifying a value for this attribute for the POST payload raises a validation error. This attribute is available only in the GET response of a started service. + description: State of the service. Specifying a value for this attribute for the PUT payload means update the service to this desired state. $ref: '#/definitions/ServiceState' quicklinks: type: object @@ -314,6 +291,9 @@ definitions: name: type: string description: Name of the service component (mandatory). If Registry DNS is enabled, the max length is 63 characters. If unique component support is enabled, the max length is lowered to 44 characters. + state: + description: The state of the component + $ref: "#/definitions/ComponentState" dependencies: type: array items: @@ -431,9 +411,11 @@ definitions: state: description: State of the container of a service. $ref: '#/definitions/ContainerState' - component_name: + component_instance_name: type: string - description: Name of the component that this container instance belongs to. + description: Name of the component instance that this container instance belongs to. Component instance name is named as $COMPONENT_NAME-i, where i is a + monotonically increasing integer. E.g. A componet called nginx can have multiple component instances named as nginx-0, nginx-1 etc. + Each component instance is backed by a container instance. resource: description: Resource used for this container. $ref: '#/definitions/Resource' @@ -452,7 +434,7 @@ definitions: enum: - ACCEPTED - STARTED - - READY + - STABLE - STOPPED - FAILED ContainerState: @@ -465,6 +447,15 @@ definitions: - INIT - STARTED - READY + ComponentState: + description: The state of the component + properties: + state: + type: string + description: enum of the state of the component + enum: + - FLEXING + - STABLE ServiceStatus: description: The current status of a submitted service, returned as a response to the GET API. properties: http://git-wip-us.apache.org/repos/asf/hadoop/blob/68acd88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/ServiceScheduler.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/ServiceScheduler.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/ServiceScheduler.java index f3824df..631f89e 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/ServiceScheduler.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/ServiceScheduler.java @@ -604,6 +604,7 @@ public class ServiceScheduler extends CompositeService { LOG.error("No component instance exists for " + containerId); return; } + LOG.error("Failed to start " + containerId, t); amRMClient.releaseAssignedContainer(containerId); // After container released, it'll get CONTAINER_COMPLETED event from RM // automatically which will trigger stopping COMPONENT INSTANCE http://git-wip-us.apache.org/repos/asf/hadoop/blob/68acd88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Component.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Component.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Component.java index b0a8e82..fe9c043 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Component.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Component.java @@ -59,6 +59,7 @@ public class Component implements Serializable { private Long numberOfContainers = null; private Boolean runPrivilegedContainer = false; private PlacementPolicy placementPolicy = null; + private ComponentState state = ComponentState.FLEXING; private Configuration configuration = new Configuration(); private List<String> quicklinks = new ArrayList<String>(); private List<Container> containers = @@ -305,6 +306,21 @@ public class Component implements Serializable { this.quicklinks = quicklinks; } + public Component state(ComponentState state) { + this.state = state; + return this; + } + + @ApiModelProperty(example = "null", value = "State of the component.") + @JsonProperty("state") + public ComponentState getState() { + return state; + } + + public void setState(ComponentState state) { + this.state = state; + } + @Override public boolean equals(java.lang.Object o) { if (this == o) { @@ -325,14 +341,15 @@ public class Component implements Serializable { component.runPrivilegedContainer) && Objects.equals(this.placementPolicy, component.placementPolicy) && Objects.equals(this.configuration, component.configuration) - && Objects.equals(this.quicklinks, component.quicklinks); + && Objects.equals(this.quicklinks, component.quicklinks) + && Objects.equals(this.state, component.state); } @Override public int hashCode() { return Objects.hash(name, dependencies, readinessCheck, artifact, launchCommand, resource, numberOfContainers, - runPrivilegedContainer, placementPolicy, configuration, quicklinks); + runPrivilegedContainer, placementPolicy, configuration, quicklinks, state); } @Override @@ -341,6 +358,7 @@ public class Component implements Serializable { sb.append("class Component {\n"); sb.append(" name: ").append(toIndentedString(name)).append("\n"); + sb.append(" state: ").append(toIndentedString(state)).append("\n"); sb.append(" dependencies: ").append(toIndentedString(dependencies)) .append("\n"); sb.append(" readinessCheck: ").append(toIndentedString(readinessCheck)) http://git-wip-us.apache.org/repos/asf/hadoop/blob/68acd88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ComponentState.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ComponentState.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ComponentState.java new file mode 100644 index 0000000..702a9ae --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ComponentState.java @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package org.apache.hadoop.yarn.service.api.records; + +import io.swagger.annotations.ApiModel; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; + +@InterfaceAudience.Public +@InterfaceStability.Unstable +@ApiModel(description = "The current state of a component.") +public enum ComponentState { + FLEXING, STABLE +} http://git-wip-us.apache.org/repos/asf/hadoop/blob/68acd88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Container.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Container.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Container.java index cf8cd79..af06542 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Container.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Container.java @@ -49,7 +49,7 @@ public class Container extends BaseResource { private String hostname = null; private String bareHost = null; private ContainerState state = null; - private String componentName = null; + private String componentInstanceName = null; private Resource resource = null; private Artifact artifact = null; private Boolean privilegedContainer = null; @@ -176,19 +176,19 @@ public class Container extends BaseResource { * Name of the component that this container instance belongs to. **/ public Container componentName(String componentName) { - this.componentName = componentName; + this.componentInstanceName = componentName; return this; } @ApiModelProperty(example = "null", value = "Name of the component that this container instance belongs to.") @JsonProperty("component_name") - public String getComponentName() { - return componentName; + public String getComponentInstanceName() { + return componentInstanceName; } @XmlElement(name = "component_name") - public void setComponentName(String componentName) { - this.componentName = componentName; + public void setComponentInstanceName(String componentInstanceName) { + this.componentInstanceName = componentInstanceName; } /** @@ -274,7 +274,8 @@ public class Container extends BaseResource { sb.append(" hostname: ").append(toIndentedString(hostname)).append("\n"); sb.append(" bareHost: ").append(toIndentedString(bareHost)).append("\n"); sb.append(" state: ").append(toIndentedString(state)).append("\n"); - sb.append(" componentName: ").append(toIndentedString(componentName)) + sb.append(" componentInstanceName: ").append(toIndentedString( + componentInstanceName)) .append("\n"); sb.append(" resource: ").append(toIndentedString(resource)).append("\n"); sb.append(" artifact: ").append(toIndentedString(artifact)).append("\n"); http://git-wip-us.apache.org/repos/asf/hadoop/blob/68acd88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Service.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Service.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Service.java index f3bbaa0..77a2610 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Service.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Service.java @@ -53,15 +53,12 @@ public class Service extends BaseResource { private String id = null; private Artifact artifact = null; private Resource resource = null; - private String launchCommand = null; private Date launchTime = null; - private Long numberOfContainers = null; private Long numberOfRunningContainers = null; private Long lifetime = null; private PlacementPolicy placementPolicy = null; private List<Component> components = new ArrayList<>(); private Configuration configuration = new Configuration(); - private List<Container> containers = new ArrayList<>(); private ServiceState state = null; private Map<String, String> quicklinks = new HashMap<>(); private String queue = null; @@ -143,29 +140,6 @@ public class Service extends BaseResource { } /** - * The custom launch command of an service component (optional). If not - * specified for services with docker images say, it will default to the - * default start command of the image. If there is a single component in this - * service, you can specify this without the need to have a 'components' - * section. - **/ - public Service launchCommand(String launchCommand) { - this.launchCommand = launchCommand; - return this; - } - - @ApiModelProperty(example = "null", value = "The custom launch command of an service component (optional). If not specified for services with docker images say, it will default to the default start command of the image. If there is a single component in this service, you can specify this without the need to have a 'components' section.") - @JsonProperty("launch_command") - public String getLaunchCommand() { - return launchCommand; - } - - @XmlElement(name = "launch_command") - public void setLaunchCommand(String launchCommand) { - this.launchCommand = launchCommand; - } - - /** * The time when the service was created, e.g. 2016-03-16T01:01:49.000Z. **/ public Service launchTime(Date launchTime) { @@ -185,26 +159,6 @@ public class Service extends BaseResource { } /** - * Number of containers for each component in the service. Each - * component can further override this service-level global default. - **/ - public Service numberOfContainers(Long numberOfContainers) { - this.numberOfContainers = numberOfContainers; - return this; - } - - @ApiModelProperty(example = "null", value = "Number of containers for each component in the service. Each component can further override this service-level global default.") - @JsonProperty("number_of_containers") - public Long getNumberOfContainers() { - return numberOfContainers; - } - - @XmlElement(name = "number_of_containers") - public void setNumberOfContainers(Long numberOfContainers) { - this.numberOfContainers = numberOfContainers; - } - - /** * In get response this provides the total number of running containers for * this service (across all components) at the time of request. Note, a * subsequent request can return a different number as and when more @@ -323,30 +277,6 @@ public class Service extends BaseResource { } /** - * Containers of a started service. Specifying a value for this attribute - * for the POST payload raises a validation error. This blob is available only - * in the GET response of a started service. - **/ - public Service containers(List<Container> containers) { - this.containers = containers; - return this; - } - - @ApiModelProperty(example = "null", value = "Containers of a started service. Specifying a value for this attribute for the POST payload raises a validation error. This blob is available only in the GET response of a started service.") - @JsonProperty("containers") - public List<Container> getContainers() { - return containers; - } - - public void setContainers(List<Container> containers) { - this.containers = containers; - } - - public void addContainer(Container container) { - this.containers.add(container); - } - - /** * State of the service. Specifying a value for this attribute for the * POST payload raises a validation error. This attribute is available only in * the GET response of a started service. @@ -428,12 +358,8 @@ public class Service extends BaseResource { sb.append(" id: ").append(toIndentedString(id)).append("\n"); sb.append(" artifact: ").append(toIndentedString(artifact)).append("\n"); sb.append(" resource: ").append(toIndentedString(resource)).append("\n"); - sb.append(" launchCommand: ").append(toIndentedString(launchCommand)) - .append("\n"); sb.append(" launchTime: ").append(toIndentedString(launchTime)) .append("\n"); - sb.append(" numberOfContainers: ") - .append(toIndentedString(numberOfContainers)).append("\n"); sb.append(" numberOfRunningContainers: ") .append(toIndentedString(numberOfRunningContainers)).append("\n"); sb.append(" lifetime: ").append(toIndentedString(lifetime)).append("\n"); @@ -443,8 +369,6 @@ public class Service extends BaseResource { .append("\n"); sb.append(" configuration: ").append(toIndentedString(configuration)) .append("\n"); - sb.append(" containers: ").append(toIndentedString(containers)) - .append("\n"); sb.append(" state: ").append(toIndentedString(state)).append("\n"); sb.append(" quicklinks: ").append(toIndentedString(quicklinks)) .append("\n"); http://git-wip-us.apache.org/repos/asf/hadoop/blob/68acd88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ServiceState.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ServiceState.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ServiceState.java index a4509bd..d2f5d06 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ServiceState.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ServiceState.java @@ -29,5 +29,5 @@ import org.apache.hadoop.classification.InterfaceStability; @ApiModel(description = "The current state of an service.") @javax.annotation.Generated(value = "class io.swagger.codegen.languages.JavaClientCodegen", date = "2016-06-02T08:15:05.615-07:00") public enum ServiceState { - ACCEPTED, STARTED, READY, STOPPED, FAILED; + ACCEPTED, STARTED, STABLE, STOPPED, FAILED; } http://git-wip-us.apache.org/repos/asf/hadoop/blob/68acd88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/client/ServiceClient.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/client/ServiceClient.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/client/ServiceClient.java index a3a9fd0..f6d35e4 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/client/ServiceClient.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/client/ServiceClient.java @@ -62,8 +62,8 @@ import org.apache.hadoop.yarn.proto.ClientAMProtocol.GetStatusResponseProto; import org.apache.hadoop.yarn.proto.ClientAMProtocol.StopRequestProto; import org.apache.hadoop.yarn.service.ClientAMProtocol; import org.apache.hadoop.yarn.service.ServiceMaster; -import org.apache.hadoop.yarn.service.api.records.Service; import org.apache.hadoop.yarn.service.api.records.Component; +import org.apache.hadoop.yarn.service.api.records.Service; import org.apache.hadoop.yarn.service.api.records.ServiceState; import org.apache.hadoop.yarn.service.client.params.AbstractClusterBuildingActionArgs; import org.apache.hadoop.yarn.service.client.params.ActionCreateArgs; @@ -73,23 +73,23 @@ import org.apache.hadoop.yarn.service.client.params.Arguments; import org.apache.hadoop.yarn.service.client.params.ClientArgs; import org.apache.hadoop.yarn.service.client.params.CommonArgs; import org.apache.hadoop.yarn.service.conf.SliderExitCodes; -import org.apache.hadoop.yarn.service.conf.YarnServiceConstants; import org.apache.hadoop.yarn.service.conf.YarnServiceConf; +import org.apache.hadoop.yarn.service.conf.YarnServiceConstants; +import org.apache.hadoop.yarn.service.containerlaunch.ClasspathConstructor; +import org.apache.hadoop.yarn.service.containerlaunch.JavaCommandLineBuilder; +import org.apache.hadoop.yarn.service.exceptions.BadClusterStateException; +import org.apache.hadoop.yarn.service.exceptions.BadConfigException; +import org.apache.hadoop.yarn.service.exceptions.SliderException; +import org.apache.hadoop.yarn.service.exceptions.UsageException; import org.apache.hadoop.yarn.service.provider.AbstractClientProvider; import org.apache.hadoop.yarn.service.provider.ProviderUtils; import org.apache.hadoop.yarn.service.utils.ServiceApiUtil; import org.apache.hadoop.yarn.service.utils.ServiceRegistryUtils; import org.apache.hadoop.yarn.service.utils.SliderFileSystem; import org.apache.hadoop.yarn.service.utils.SliderUtils; +import org.apache.hadoop.yarn.service.utils.ZookeeperUtils; import org.apache.hadoop.yarn.util.Records; import org.apache.hadoop.yarn.util.Times; -import org.apache.hadoop.yarn.service.exceptions.BadClusterStateException; -import org.apache.hadoop.yarn.service.exceptions.BadConfigException; -import org.apache.hadoop.yarn.service.exceptions.SliderException; -import org.apache.hadoop.yarn.service.exceptions.UsageException; -import org.apache.hadoop.yarn.service.containerlaunch.ClasspathConstructor; -import org.apache.hadoop.yarn.service.containerlaunch.JavaCommandLineBuilder; -import org.apache.hadoop.yarn.service.utils.ZookeeperUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -196,11 +196,7 @@ public class ServiceClient extends CompositeService serviceDef = loadAppJsonFromLocalFS(args); } else if (!StringUtils.isEmpty(args.example)) { // create an example service - String yarnHome = System - .getenv(ApplicationConstants.Environment.HADOOP_YARN_HOME.key()); - args.file = new File(MessageFormat - .format("{0}/share/hadoop/yarn/yarn-service-examples/{1}/{2}.json", - yarnHome, args.example, args.example)); + args.file = findExampleService(args); serviceDef = loadAppJsonFromLocalFS(args); } else { throw new YarnException("No service definition provided!"); @@ -209,6 +205,27 @@ public class ServiceClient extends CompositeService return EXIT_SUCCESS; } + private File findExampleService(ActionCreateArgs args) throws YarnException { + String yarnHome = System + .getenv(ApplicationConstants.Environment.HADOOP_YARN_HOME.key()); + // First look for the standard location. + File file = new File(MessageFormat + .format("{0}/share/hadoop/yarn/yarn-service-examples/{1}/{2}.json", + yarnHome, args.example, args.example)); + if (file.exists()) { + return file; + } + // Then look for secondary location. + file = new File(MessageFormat + .format("{0}/yarn-service-examples/{1}/{2}.json", yarnHome, + args.example, args.example)); + if (file.exists()) { + return file; + } + throw new YarnException( + "Example service " + args.example + " does not exist!"); + } + public ApplicationId actionCreate(Service service) throws IOException, YarnException { String serviceName = service.getName(); http://git-wip-us.apache.org/repos/asf/hadoop/blob/68acd88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/Component.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/Component.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/Component.java index 98bb238..a5dd39c 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/Component.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/Component.java @@ -202,6 +202,8 @@ public class Component implements EventHandler<ComponentEvent> { + before + " to " + event.getDesired()); component.requestContainers(delta); component.createNumCompInstances(delta); + component.componentSpec.setState( + org.apache.hadoop.yarn.service.api.records.ComponentState.FLEXING); return FLEXING; } else if (delta < 0){ delta = 0 - delta; @@ -224,10 +226,14 @@ public class Component implements EventHandler<ComponentEvent> { component.instanceIdCounter.decrementAndGet(); instance.destroy(); } + component.componentSpec.setState( + org.apache.hadoop.yarn.service.api.records.ComponentState.STABLE); return STABLE; } else { LOG.info("[FLEX COMPONENT " + component.getName() + "]: already has " + event.getDesired() + " instances, ignoring"); + component.componentSpec.setState( + org.apache.hadoop.yarn.service.api.records.ComponentState.STABLE); return STABLE; } } @@ -297,8 +303,12 @@ public class Component implements EventHandler<ComponentEvent> { // if desired == running if (component.componentMetrics.containersRunning.value() == component .getComponentSpec().getNumberOfContainers()) { + component.componentSpec.setState( + org.apache.hadoop.yarn.service.api.records.ComponentState.STABLE); return STABLE; } else { + component.componentSpec.setState( + org.apache.hadoop.yarn.service.api.records.ComponentState.FLEXING); return FLEXING; } } @@ -317,6 +327,8 @@ public class Component implements EventHandler<ComponentEvent> { component.compInstanceDispatcher.getEventHandler().handle( new ComponentInstanceEvent(event.getStatus().getContainerId(), STOP).setStatus(event.getStatus())); + component.componentSpec.setState( + org.apache.hadoop.yarn.service.api.records.ComponentState.FLEXING); } } http://git-wip-us.apache.org/repos/asf/hadoop/blob/68acd88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/instance/ComponentInstance.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/instance/ComponentInstance.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/instance/ComponentInstance.java index 68c0537..6e585aa 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/instance/ComponentInstance.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/instance/ComponentInstance.java @@ -99,6 +99,11 @@ public class ComponentInstance implements EventHandler<ComponentInstanceEvent>, ComponentInstanceEventType, ComponentInstanceEvent>(INIT) .addTransition(INIT, STARTED, START, new ContainerStartedTransition()) + .addTransition(INIT, INIT, STOP, + // container failed before launching, nothing to cleanup from registry + // This could happen if NMClient#startContainerAsync failed, container + // will be completed, but COMP_INSTANCE is still at INIT. + new ContainerStoppedTransition(true)) //From Running .addTransition(STARTED, INIT, STOP, @@ -159,7 +164,7 @@ public class ComponentInstance implements EventHandler<ComponentInstanceEvent>, container.setLaunchTime(new Date(containerStartTime)); container.setState(ContainerState.RUNNING_BUT_UNREADY); container.setBareHost(compInstance.container.getNodeId().getHost()); - container.setComponentName(compInstance.getCompInstanceName()); + container.setComponentInstanceName(compInstance.getCompInstanceName()); if (compInstance.containerSpec != null) { // remove the previous container. compInstance.getCompSpec().removeContainer(compInstance.containerSpec); @@ -194,6 +199,16 @@ public class ComponentInstance implements EventHandler<ComponentInstanceEvent>, } private static class ContainerStoppedTransition extends BaseTransition { + // whether the container failed before launched by AM or not. + boolean failedBeforeLaunching = false; + public ContainerStoppedTransition(boolean failedBeforeLaunching) { + this.failedBeforeLaunching = failedBeforeLaunching; + } + + public ContainerStoppedTransition() { + this(false); + } + @Override public void transition(ComponentInstance compInstance, ComponentInstanceEvent event) { @@ -225,24 +240,27 @@ public class ComponentInstance implements EventHandler<ComponentInstanceEvent>, shouldExit = true; } - // clean up registry - // hdfs dir content will be overwritten when a new container gets started, - // so no need remove. - compInstance.scheduler.executorService - .submit(compInstance::cleanupRegistry); + if (!failedBeforeLaunching) { + // clean up registry + // If the container failed before launching, no need to cleanup registry, + // because it was not registered before. + // hdfs dir content will be overwritten when a new container gets started, + // so no need remove. + compInstance.scheduler.executorService + .submit(compInstance::cleanupRegistry); + if (compInstance.timelineServiceEnabled) { + // record in ATS + compInstance.serviceTimelinePublisher + .componentInstanceFinished(compInstance, + event.getStatus().getExitStatus(), event.getStatus().getState(), + containerDiag); + } + compInstance.containerSpec.setState(ContainerState.STOPPED); + } // remove the failed ContainerId -> CompInstance mapping comp.getScheduler().removeLiveCompInstance(event.getContainerId()); - if (compInstance.timelineServiceEnabled) { - // record in ATS - compInstance.serviceTimelinePublisher - .componentInstanceFinished(compInstance, - event.getStatus().getExitStatus(), event.getStatus().getState(), - containerDiag); - } - - compInstance.containerSpec.setState(ContainerState.STOPPED); if (shouldExit) { // Sleep for 5 seconds in hope that the state can be recorded in ATS. // in case there's a client polling the comp state, it can be notified. http://git-wip-us.apache.org/repos/asf/hadoop/blob/68acd88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/conf/RestApiConstants.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/conf/RestApiConstants.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/conf/RestApiConstants.java index 6de2dc0..35e1980 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/conf/RestApiConstants.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/conf/RestApiConstants.java @@ -30,10 +30,6 @@ public interface RestApiConstants { String SERVICE_NAME = "service_name"; String COMPONENT_NAME = "component_name"; - String DEFAULT_COMPONENT_NAME = "default"; - - String PROPERTY_REST_SERVICE_HOST = "REST_SERVICE_HOST"; - String PROPERTY_REST_SERVICE_PORT = "REST_SERVICE_PORT"; Long DEFAULT_UNLIMITED_LIFETIME = -1l; Integer ERROR_CODE_APP_DOES_NOT_EXIST = 404001; http://git-wip-us.apache.org/repos/asf/hadoop/blob/68acd88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/ServiceApiUtil.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/ServiceApiUtil.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/ServiceApiUtil.java index de82580..68db0bb 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/ServiceApiUtil.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/ServiceApiUtil.java @@ -80,38 +80,14 @@ public class ServiceApiUtil { validateNameFormat(service.getName(), conf); - // If the service has no components do top-level checks + // If the service has no components, throw error if (!hasComponent(service)) { - // If artifact is of type SERVICE, read other service components - if (service.getArtifact() != null && service.getArtifact() - .getType() == Artifact.TypeEnum.SERVICE) { - if (StringUtils.isEmpty(service.getArtifact().getId())) { - throw new IllegalArgumentException( - RestApiErrorMessages.ERROR_ARTIFACT_ID_INVALID); - } - Service otherService = loadService(fs, - service.getArtifact().getId()); - service.setComponents(otherService.getComponents()); - service.setArtifact(null); - SliderUtils.mergeMapsIgnoreDuplicateKeys(service.getQuicklinks(), - otherService.getQuicklinks()); - } else { - // Since it is a simple service with no components, create a default - // component - Component comp = createDefaultComponent(service); - validateComponent(comp, fs.getFileSystem(), conf); - service.getComponents().add(comp); - if (service.getLifetime() == null) { - service.setLifetime(RestApiConstants.DEFAULT_UNLIMITED_LIFETIME); - } - return; - } + throw new IllegalArgumentException( + "No component specified for " + service.getName()); } // Validate there are no component name collisions (collisions are not // currently supported) and add any components from external services - // TODO allow name collisions? see AppState#roles - // TODO or add prefix to external component names? Configuration globalConf = service.getConfiguration(); Set<String> componentNames = new HashSet<>(); List<Component> componentsToRemove = new ArrayList<>(); @@ -174,8 +150,6 @@ public class ServiceApiUtil { // values are not provided Artifact globalArtifact = service.getArtifact(); Resource globalResource = service.getResource(); - Long globalNumberOfContainers = service.getNumberOfContainers(); - String globalLaunchCommand = service.getLaunchCommand(); for (Component comp : service.getComponents()) { // fill in global artifact unless it is type SERVICE if (comp.getArtifact() == null && service.getArtifact() != null @@ -187,14 +161,6 @@ public class ServiceApiUtil { if (comp.getResource() == null) { comp.setResource(globalResource); } - // fill in global container count - if (comp.getNumberOfContainers() == null) { - comp.setNumberOfContainers(globalNumberOfContainers); - } - // fill in global launch command - if (comp.getLaunchCommand() == null) { - comp.setLaunchCommand(globalLaunchCommand); - } // validate dependency existence if (comp.getDependencies() != null) { for (String dependency : comp.getDependencies()) { @@ -360,7 +326,7 @@ public class ServiceApiUtil { } } - public static boolean hasComponent(Service service) { + private static boolean hasComponent(Service service) { if (service.getComponents() == null || service.getComponents() .isEmpty()) { return false; @@ -368,17 +334,6 @@ public class ServiceApiUtil { return true; } - public static Component createDefaultComponent(Service service) { - Component comp = new Component(); - comp.setName(RestApiConstants.DEFAULT_COMPONENT_NAME); - comp.setArtifact(service.getArtifact()); - comp.setResource(service.getResource()); - comp.setNumberOfContainers(service.getNumberOfContainers()); - comp.setLaunchCommand(service.getLaunchCommand()); - comp.setConfiguration(service.getConfiguration()); - return comp; - } - public static Collection<Component> sortByDependencies(List<Component> components) { Map<String, Component> sortedComponents = http://git-wip-us.apache.org/repos/asf/hadoop/blob/68acd88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/ServiceTestUtils.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/ServiceTestUtils.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/ServiceTestUtils.java index aeb4341..19a5177 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/ServiceTestUtils.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/ServiceTestUtils.java @@ -84,11 +84,11 @@ public class ServiceTestUtils { return exampleApp; } - protected Component createComponent(String name) { + public static Component createComponent(String name) { return createComponent(name, 2L, "sleep 1000"); } - protected Component createComponent(String name, long numContainers, + protected static Component createComponent(String name, long numContainers, String command) { Component comp1 = new Component(); comp1.setNumberOfContainers(numContainers); http://git-wip-us.apache.org/repos/asf/hadoop/blob/68acd88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestServiceApiUtil.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestServiceApiUtil.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestServiceApiUtil.java index 959e4d6..55c096e 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestServiceApiUtil.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestServiceApiUtil.java @@ -20,11 +20,11 @@ package org.apache.hadoop.yarn.service; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.registry.client.api.RegistryConstants; import org.apache.hadoop.yarn.conf.YarnConfiguration; -import org.apache.hadoop.yarn.service.api.records.Service; -import org.apache.hadoop.yarn.service.exceptions.RestApiErrorMessages; import org.apache.hadoop.yarn.service.api.records.Artifact; import org.apache.hadoop.yarn.service.api.records.Component; import org.apache.hadoop.yarn.service.api.records.Resource; +import org.apache.hadoop.yarn.service.api.records.Service; +import org.apache.hadoop.yarn.service.exceptions.RestApiErrorMessages; import org.apache.hadoop.yarn.service.utils.ServiceApiUtil; import org.apache.hadoop.yarn.service.utils.SliderFileSystem; import org.junit.Assert; @@ -36,9 +36,9 @@ import org.slf4j.LoggerFactory; import java.io.IOException; import java.util.Arrays; import java.util.Collection; +import java.util.Collections; import java.util.List; -import static org.apache.hadoop.yarn.service.conf.RestApiConstants.DEFAULT_COMPONENT_NAME; import static org.apache.hadoop.yarn.service.conf.RestApiConstants.DEFAULT_UNLIMITED_LIFETIME; import static org.apache.hadoop.yarn.service.exceptions.RestApiErrorMessages.*; import static org.junit.Assert.assertEquals; @@ -99,6 +99,8 @@ public class TestServiceApiUtil { // launch command not specified app.setName(LEN_64_STR); + Component comp = new Component().name("comp1"); + app.addComponent(comp); try { ServiceApiUtil.validateAndResolveService(app, sfs, CONF_DEFAULT_DNS); Assert.fail(EXCEPTION_PREFIX + "service with no launch command"); @@ -118,18 +120,8 @@ public class TestServiceApiUtil { e.getMessage()); } - // resource not specified - app.setLaunchCommand("sleep 3600"); - try { - ServiceApiUtil.validateAndResolveService(app, sfs, CONF_DNS_ENABLED); - Assert.fail(EXCEPTION_PREFIX + "service with no resource"); - } catch (IllegalArgumentException e) { - assertEquals(String.format( - RestApiErrorMessages.ERROR_RESOURCE_FOR_COMP_INVALID, - DEFAULT_COMPONENT_NAME), e.getMessage()); - } - // memory not specified + comp.setLaunchCommand("sleep 1"); Resource res = new Resource(); app.setResource(res); try { @@ -138,7 +130,7 @@ public class TestServiceApiUtil { } catch (IllegalArgumentException e) { assertEquals(String.format( RestApiErrorMessages.ERROR_RESOURCE_MEMORY_FOR_COMP_INVALID, - DEFAULT_COMPONENT_NAME), e.getMessage()); + comp.getName()), e.getMessage()); } // invalid no of cpus @@ -151,7 +143,7 @@ public class TestServiceApiUtil { } catch (IllegalArgumentException e) { assertEquals(String.format( RestApiErrorMessages.ERROR_RESOURCE_CPUS_FOR_COMP_INVALID_RANGE, - DEFAULT_COMPONENT_NAME), e.getMessage()); + comp.getName()), e.getMessage()); } // number of containers not specified @@ -173,7 +165,7 @@ public class TestServiceApiUtil { } catch (IllegalArgumentException e) { assertEquals(String.format(RestApiErrorMessages .ERROR_RESOURCE_PROFILE_MULTIPLE_VALUES_FOR_COMP_NOT_SUPPORTED, - DEFAULT_COMPONENT_NAME), + comp.getName()), e.getMessage()); } @@ -202,25 +194,6 @@ public class TestServiceApiUtil { Assert.assertTrue(e.getMessage() .startsWith(ERROR_CONTAINERS_COUNT_INVALID)); } - - // negative number of containers - app.setNumberOfContainers(-1L); - try { - ServiceApiUtil.validateAndResolveService(app, sfs, CONF_DNS_ENABLED); - Assert.fail(EXCEPTION_PREFIX + "negative number of containers"); - } catch (IllegalArgumentException e) { - Assert.assertTrue(e.getMessage() - .startsWith(ERROR_CONTAINERS_COUNT_INVALID)); - } - - // everything valid here - app.setNumberOfContainers(5L); - try { - ServiceApiUtil.validateAndResolveService(app, sfs, CONF_DNS_ENABLED); - } catch (IllegalArgumentException e) { - LOG.error("service attributes specified should be valid here", e); - Assert.fail(NO_EXCEPTION_PREFIX + e.getMessage()); - } } @Test @@ -228,15 +201,17 @@ public class TestServiceApiUtil { SliderFileSystem sfs = ServiceTestUtils.initMockFs(); Service app = new Service(); - app.setName("name"); + app.setName("service1"); Resource res = new Resource(); app.setResource(res); res.setMemory("512M"); - app.setNumberOfContainers(3L); // no artifact id fails with default type Artifact artifact = new Artifact(); app.setArtifact(artifact); + Component comp = ServiceTestUtils.createComponent("comp1"); + + app.setComponents(Collections.singletonList(comp)); try { ServiceApiUtil.validateAndResolveService(app, sfs, CONF_DNS_ENABLED); Assert.fail(EXCEPTION_PREFIX + "service with no artifact id"); @@ -272,9 +247,6 @@ public class TestServiceApiUtil { Assert.fail(NO_EXCEPTION_PREFIX + e.getMessage()); } - // defaults assigned - assertEquals(app.getComponents().get(0).getName(), - DEFAULT_COMPONENT_NAME); assertEquals(app.getLifetime(), DEFAULT_UNLIMITED_LIFETIME); } @@ -289,15 +261,14 @@ public class TestServiceApiUtil { comp.setName(compName); comp.setResource(createValidResource()); comp.setNumberOfContainers(1L); + comp.setLaunchCommand("sleep 1"); return comp; } private static Service createValidApplication(String compName) { Service app = new Service(); - app.setLaunchCommand("sleep 3600"); app.setName("name"); app.setResource(createValidResource()); - app.setNumberOfContainers(1L); if (compName != null) { app.addComponent(createValidComponent(compName)); } @@ -315,7 +286,7 @@ public class TestServiceApiUtil { artifact.setType(Artifact.TypeEnum.SERVICE); artifact.setId("id"); app.setArtifact(artifact); - + app.addComponent(ServiceTestUtils.createComponent("comp2")); try { ServiceApiUtil.validateAndResolveService(app, sfs, CONF_DNS_ENABLED); } catch (IllegalArgumentException e) { @@ -323,7 +294,7 @@ public class TestServiceApiUtil { } assertEquals(1, app.getComponents().size()); - assertNotNull(app.getComponent("comp1")); + assertNotNull(app.getComponent("comp2")); } @Test @@ -410,12 +381,13 @@ public class TestServiceApiUtil { @Test public void testDependencySorting() throws IOException { - Component a = new Component().name("a"); - Component b = new Component().name("b"); - Component c = new Component().name("c"); - Component d = new Component().name("d").dependencies(Arrays.asList("c")); - Component e = new Component().name("e").dependencies(Arrays.asList("b", - "d")); + Component a = ServiceTestUtils.createComponent("a"); + Component b = ServiceTestUtils.createComponent("b"); + Component c = ServiceTestUtils.createComponent("c"); + Component d = + ServiceTestUtils.createComponent("d").dependencies(Arrays.asList("c")); + Component e = ServiceTestUtils.createComponent("e") + .dependencies(Arrays.asList("b", "d")); verifyDependencySorting(Arrays.asList(a, b, c), a, b, c); verifyDependencySorting(Arrays.asList(c, a, b), c, a, b); http://git-wip-us.apache.org/repos/asf/hadoop/blob/68acd88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestYarnNativeServices.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestYarnNativeServices.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestYarnNativeServices.java index d821b84..64fcf57 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestYarnNativeServices.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestYarnNativeServices.java @@ -176,7 +176,7 @@ public class TestYarnNativeServices extends ServiceTestUtils { for (String comp : compOrder) { long num = retrievedApp.getComponent(comp).getNumberOfContainers(); for (int i = 0; i < num; i++) { - String compInstanceName = containerList.get(index).getComponentName(); + String compInstanceName = containerList.get(index).getComponentInstanceName(); String compName = compInstanceName.substring(0, compInstanceName.lastIndexOf('-')); Assert.assertEquals(comp, compName); @@ -219,7 +219,7 @@ public class TestYarnNativeServices extends ServiceTestUtils { Assert.assertEquals(expectedNumInstances, component.getContainers().size()); TreeSet<String> instances = new TreeSet<>(); for (Container container : component.getContainers()) { - instances.add(container.getComponentName()); + instances.add(container.getComponentInstanceName()); } int i = 0; http://git-wip-us.apache.org/repos/asf/hadoop/blob/68acd88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/resources/org/apache/hadoop/yarn/service/conf/examples/app-override.json ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/resources/org/apache/hadoop/yarn/service/conf/examples/app-override.json b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/resources/org/apache/hadoop/yarn/service/conf/examples/app-override.json index d7e2fd0..753a9cd 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/resources/org/apache/hadoop/yarn/service/conf/examples/app-override.json +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/resources/org/apache/hadoop/yarn/service/conf/examples/app-override.json @@ -1,7 +1,6 @@ { "name": "app-1", "lifetime": "3600", - "launch_command": "sleep 3600", "configuration": { "properties": { "g1": "a", @@ -29,10 +28,11 @@ "cpus": 1, "memory": "512" }, - "number_of_containers": 2, "components": [ { "name": "simple", + "launch_command": "sleep 3600", + "number_of_containers": 2, "configuration": { "files": [ { @@ -47,6 +47,8 @@ }, { "name": "master", + "launch_command": "sleep 3600", + "number_of_containers": 2, "configuration": { "properties": { "name": "m", @@ -56,6 +58,8 @@ }, { "name": "worker", + "number_of_containers": 2, + "launch_command": "sleep 3600", "resource": { "cpus": 1, "memory": "1024" http://git-wip-us.apache.org/repos/asf/hadoop/blob/68acd88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/resources/org/apache/hadoop/yarn/service/conf/examples/app.json ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/resources/org/apache/hadoop/yarn/service/conf/examples/app.json b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/resources/org/apache/hadoop/yarn/service/conf/examples/app.json index 43d3c39..2eb477f 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/resources/org/apache/hadoop/yarn/service/conf/examples/app.json +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/resources/org/apache/hadoop/yarn/service/conf/examples/app.json @@ -2,7 +2,6 @@ "name": "app-1", "id" : "application_1503358878042_0011", "lifetime": "3600", - "launch_command": "sleep 3600", "configuration": { "properties": { "g1": "a", @@ -14,14 +13,16 @@ "cpus": 1, "memory": "512" }, - "number_of_containers": 2, "components": [ { - "name": "simple" + "name": "simple", + "number_of_containers": 2, + "launch_command": "sleep 3600" }, { "name": "master", "number_of_containers": 1, + "launch_command": "sleep 3600", "configuration": { "properties": { "g1": "overridden", @@ -33,6 +34,7 @@ { "name": "worker", "number_of_containers": 5, + "launch_command": "sleep 3600", "resource": { "cpus": 1, "memory": "1024" http://git-wip-us.apache.org/repos/asf/hadoop/blob/68acd88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/resources/org/apache/hadoop/yarn/service/conf/examples/external0.json ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/resources/org/apache/hadoop/yarn/service/conf/examples/external0.json b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/resources/org/apache/hadoop/yarn/service/conf/examples/external0.json index 0857f62..f0163bc 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/resources/org/apache/hadoop/yarn/service/conf/examples/external0.json +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/resources/org/apache/hadoop/yarn/service/conf/examples/external0.json @@ -1,8 +1,15 @@ { "name": "external-0", "lifetime": "3600", - "artifact": { - "type": "SERVICE", - "id": "app-1" - } + + "components" : [ + { + "name" : "comp1", + "artifact": { + "type": "SERVICE", + "id": "app-1" + } + } + ] + } http://git-wip-us.apache.org/repos/asf/hadoop/blob/68acd88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/YarnServiceAPI.md ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/YarnServiceAPI.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/YarnServiceAPI.md index 9022268..e9aaf7e 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/YarnServiceAPI.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/YarnServiceAPI.md @@ -201,7 +201,6 @@ Set a component's desired number of instanes |404|Service does not exist|No Content| |default|Unexpected error|ServiceStatus| - ## Definitions ### Artifact @@ -221,6 +220,7 @@ One or more components of the service. If the service is HBase say, then the com |Name|Description|Required|Schema|Default| |----|----|----|----|----| |name|Name of the service component (mandatory). If Registry DNS is enabled, the max length is 63 characters. If unique component support is enabled, the max length is lowered to 44 characters.|true|string|| +|state|The state of the component|false|ComponentState|| |dependencies|An array of service components which should be in READY state (as defined by readiness check), before this component can be started. The dependencies across all components of a service should be represented as a DAG.|false|string array|| |readiness_check|Readiness check for this component.|false|ReadinessCheck|| |artifact|Artifact of the component (optional). If not specified, the service level global artifact takes effect.|false|Artifact|| @@ -233,6 +233,15 @@ One or more components of the service. If the service is HBase say, then the com |quicklinks|A list of quicklink keys defined at the service level, and to be resolved by this component.|false|string array|| +### ComponentState + +The state of the component + +|Name|Description|Required|Schema|Default| +|----|----|----|----|----| +|state|enum of the state of the component|false|enum (FLEXING, STABLE)|| + + ### ConfigFile A config file that needs to be created and made available as a volume in a service component container. @@ -268,7 +277,7 @@ An instance of a running service container. |hostname|Fully qualified hostname of a running container, e.g. ctr-e3751-1458061340047-0008-01-000002.examplestg.site. The IP address and hostname attribute values are dependent on the cluster/docker network setup as per YARN-4007.|false|string|| |bare_host|The bare node or host in which the container is running, e.g. cn008.example.com.|false|string|| |state|State of the container of a service.|false|ContainerState|| -|component_name|Name of the component that this container instance belongs to.|false|string|| +|component_instance_name|Name of the component instance that this container instance belongs to. Component instance name is named as $COMPONENT_NAME-i, where i is a monotonically increasing integer. E.g. A componet called nginx can have multiple component instances named as nginx-0, nginx-1 etc. Each component instance is backed by a container instance.|false|string|| |resource|Resource used for this container.|false|Resource|| |artifact|Artifact used for this container.|false|Artifact|| |privileged_container|Container running in privileged mode or not.|false|boolean|| @@ -322,18 +331,15 @@ a service resource has the following attributes. |----|----|----|----|----| |name|A unique service name. If Registry DNS is enabled, the max length is 63 characters.|true|string|| |id|A unique service id.|false|string|| -|artifact|Artifact of single-component service.|false|Artifact|| -|resource|Resource of single-component service or the global default for multi-component services. Mandatory if it is a single-component service and if cpus and memory are not specified at the Service level.|false|Resource|| -|launch_command|The custom launch command of a service component (optional). If not specified for services with docker images say, it will default to the default start command of the image. If there is a single component in this service, you can specify this without the need to have a 'components' section.|false|string|| +|artifact|The default artifact for all components of the service except the components which has Artifact type set to SERVICE (optional).|false|Artifact|| +|resource|The default resource for all components of the service (optional).|false|Resource|| |launch_time|The time when the service was created, e.g. 2016-03-16T01:01:49.000Z.|false|string (date)|| -|number_of_containers|Number of containers for each component in the service. Each component can further override this service-level global default.|false|integer (int64)|| |number_of_running_containers|In get response this provides the total number of running containers for this service (across all components) at the time of request. Note, a subsequent request can return a different number as and when more containers get allocated until it reaches the total number of containers or if a flex request has been made between the two requests.|false|integer (int64)|| |lifetime|Life time (in seconds) of the service from the time it reaches the STARTED state (after which it is automatically destroyed by YARN). For unlimited lifetime do not set a lifetime value.|false|integer (int64)|| |placement_policy|(TBD) Advanced scheduling and placement policies. If not specified, it defaults to the default placement policy of the service owner. The design of placement policies are in the works. It is not very clear at this point, how policies in conjunction with labels be exposed to service owners. This is a placeholder for now. The advanced structure of this attribute will be determined by YARN-4902.|false|PlacementPolicy|| |components|Components of a service.|false|Component array|| |configuration|Config properties of a service. Configurations provided at the service/global level are available to all the components. Specific properties can be overridden at the component level.|false|Configuration|| -|containers|Containers of a started service. Specifying a value for this attribute for the POST payload raises a validation error. This blob is available only in the GET response of a started service.|false|Container array|| -|state|State of the service. Specifying a value for this attribute for the POST payload raises a validation error. This attribute is available only in the GET response of a started service.|false|ServiceState|| +|state|State of the service. Specifying a value for this attribute for the PUT payload means update the service to this desired state.|false|ServiceState|| |quicklinks|A blob of key-value pairs of quicklinks to be exported for a service.|false|object|| |queue|The YARN queue that this service should be submitted to.|false|string|| --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
