[jira] [Created] (YARN-9002) YARN Service keytab location is restricted to HDFS and local filesystem only
Gour Saha created YARN-9002: --- Summary: YARN Service keytab location is restricted to HDFS and local filesystem only Key: YARN-9002 URL: https://issues.apache.org/jira/browse/YARN-9002 Project: Hadoop YARN Issue Type: Bug Components: yarn-native-services Affects Versions: 3.1.1 Reporter: Gour Saha ServiceClient.java specifically checks if the keytab URI scheme is hdfs or file. This restricts it from supporting other FileSystem API conforming FSs like s3a, wasb, gs, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8779) Fix few discrepancies between YARN Service swagger spec and code
Gour Saha created YARN-8779:
---
Summary: Fix few discrepancies between YARN Service swagger spec
and code
Key: YARN-8779
URL: https://issues.apache.org/jira/browse/YARN-8779
Project: Hadoop YARN
Issue Type: Bug
Components: yarn-native-services
Affects Versions: 3.1.1, 3.1.0
Reporter: Gour Saha
Following issues were identified in YARN Service swagger definition during an
effort to integrate with a running service by generating Java and Go
client-side stubs from the spec -
1.
*restartPolicy* is wrong and should be *restart_policy*
2.
A DELETE request to a non-existing service (or a previously existing but
deleted service) throws an ApiException instead of something like
NotFoundException (the equivalent of 404). Note, DELETE of an existing service
behaves fine.
3.
The response code of DELETE request is 200. The spec says 204. Since the
response has a payload, the spec should be updated to 200 instead of 204.
4.
_DefaultApi.java_ client's _appV1ServicesServiceNameGetWithHttpInfo_ method
does not return a Service object. Swagger definition has the below bug in GET
response of */app/v1/services/\{service_name}* -
{code:java}
type: object
items:
$ref: '#/definitions/Service'
{code}
It should be -
{code:java}
$ref: '#/definitions/Service'
{code}
5.
Serialization issues were seen in all enum classes - ServiceState.java,
ContainerState.java, ComponentState.java, PlacementType.java and
PlacementScope.java.
Java client threw the below exception for ServiceState -
{code:java}
Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot
construct instance of `org.apache.cb.yarn.service.api.records.ServiceState`
(although at least one Creator exists): no String-argument constructor/factory
method to deserialize from String value ('ACCEPTED')
at [Source:
(org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream);
line: 1, column: 121] (through reference chain:
org.apache.cb.yarn.service.api.records.Service["state”])
{code}
For Golang we saw this for ContainerState -
{code:java}
ERRO[2018-08-12T23:32:31.851-07:00] During GET request: json: cannot unmarshal
string into Go struct field Container.state of type yarnmodel.ContainerState
{code}
6.
*launch_time* actually returns an integer but swagger definition says date.
Hence, the following exception is seen on the client side -
{code:java}
Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException:
Unexpected token (VALUE_NUMBER_INT), expected START_ARRAY: Expected array or
string.
at [Source:
(org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream);
line: 1, column: 477] (through reference chain:
org.apache.cb.yarn.service.api.records.Service["components"]->java.util.ArrayList[0]->org.apache.cb.yarn.service.api.records.Component["containers"]->java.util.ArrayList[0]->org.apache.cb.yarn.service.api.records.Container["launch_time”])
{code}
8.
*user.name* query param with a valid value is required for all API calls to an
unsecure cluster. This is not defined in the spec.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-8425) Yarn container getting killed due to running beyond physical memory limits
[ https://issues.apache.org/jira/browse/YARN-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gour Saha resolved YARN-8425. - Resolution: Not A Bug > Yarn container getting killed due to running beyond physical memory limits > -- > > Key: YARN-8425 > URL: https://issues.apache.org/jira/browse/YARN-8425 > Project: Hadoop YARN > Issue Type: Task > Components: applications, container-queuing, yarn >Affects Versions: 2.7.6 >Reporter: Tapas Sen >Priority: Major > Attachments: yarn_configuration_1.PNG, yarn_configuration_2.PNG, > yarn_configuration_3.PNG > > > Hi, > Getting these error. > > 2018-06-12 17:59:07,193 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics > report from attempt_1527758146858_45040_m_08_3: Container > [pid=15498,containerID=container_e60_1527758146858_45040_01_41] is > running beyond physical memory limits. Current usage: 8.1 GB of 8 GB physical > memory used; 12.2 GB of 16.8 GB virtual memory used. Killing container. > > Yarn resource configuration will in attachment. > > Any lead would be appreciated. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-8309) Diagnostic message for yarn service app failure due token renewal should be improved
[
https://issues.apache.org/jira/browse/YARN-8309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gour Saha resolved YARN-8309.
-
Resolution: Won't Do
> Diagnostic message for yarn service app failure due token renewal should be
> improved
>
>
> Key: YARN-8309
> URL: https://issues.apache.org/jira/browse/YARN-8309
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Priority: Major
>
> When Yarn service application failed due to token renewal issue , The
> diagonstic message was unclear .
> {code:java}
> Application application_1526413043392_0002 failed 20 times due to AM
> Container for appattempt_1526413043392_0002_20 exited with exitCode: 1
> Failing this attempt.Diagnostics: [2018-05-15 23:15:28.779]Exception from
> container-launch. Container id: container_e04_1526413043392_0002_20_01
> Exit code: 1 Exception message: Launch container failed Shell output: main :
> command provided 1 main : run as user is hbase main : requested yarn user is
> hbase Getting exit code file... Creating script paths... Writing pid file...
> Writing to tmp file
> /grid/0/hadoop/yarn/local/nmPrivate/application_1526413043392_0002/container_e04_1526413043392_0002_20_01/container_e04_1526413043392_0002_20_01.pid.tmp
> Writing to cgroup task files... Creating local dirs... Launching
> container... Getting exit code file... Creating script paths... [2018-05-15
> 23:15:28.806]Container exited with a non-zero exit code 1. Error file:
> prelaunch.err. Last 4096 bytes of prelaunch.err : [2018-05-15
> 23:15:28.807]Container exited with a non-zero exit code 1. Error file:
> prelaunch.err. Last 4096 bytes of prelaunch.err : For more detailed output,
> check the application tracking page:
> https://xxx:8090/cluster/app/application_1526413043392_0002 Then click on
> links to logs of each attempt. . Failing the application.{code}
> Here, diagnostic message should be improved to specify that AM is failing due
> to token renewal issues.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8367) 2 components, one with placement constraint and one without causes NPE in SingleConstraintAppPlacementAllocator
Gour Saha created YARN-8367:
---
Summary: 2 components, one with placement constraint and one
without causes NPE in SingleConstraintAppPlacementAllocator
Key: YARN-8367
URL: https://issues.apache.org/jira/browse/YARN-8367
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 3.1.0
Reporter: Gour Saha
While testing the fix for YARN-8350, [~billie.rinaldi] encountered this NPE in
AM log. Filling this on her behalf -
{noformat}
2018-05-25 21:11:54,006 [AMRM Heartbeater thread] ERROR
impl.AMRMClientAsyncImpl - Exception on heartbeat
java.lang.NullPointerException: java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.SingleConstraintAppPlacementAllocator.validateAndSetSchedulingRequest(SingleConstraintAppPlacementAllocator.java:245)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.SingleConstraintAppPlacementAllocator.internalUpdatePendingAsk(SingleConstraintAppPlacementAllocator.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.SingleConstraintAppPlacementAllocator.updatePendingAsk(SingleConstraintAppPlacementAllocator.java:207)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.addSchedulingRequests(AppSchedulingInfo.java:269)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.updateSchedulingRequests(AppSchedulingInfo.java:240)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.updateSchedulingRequests(SchedulerApplicationAttempt.java:469)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocate(CapacityScheduler.java:1154)
at
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:278)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.SchedulerPlacementProcessor.allocate(SchedulerPlacementProcessor.java:53)
at
org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
at
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:433)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
at
org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85)
at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at
[jira] [Created] (YARN-8243) Flex down should first remove pending container requests (if any) and then kill running containers
Gour Saha created YARN-8243:
---
Summary: Flex down should first remove pending container requests
(if any) and then kill running containers
Key: YARN-8243
URL: https://issues.apache.org/jira/browse/YARN-8243
Project: Hadoop YARN
Issue Type: Sub-task
Components: yarn-native-services
Affects Versions: 3.1.0
Reporter: Gour Saha
This is easy to test on a service with anti-affinity component, to simulate
pending container requests. It can be simulated by other means also (no
resource left in cluster, etc.).
Service yarnfile used to test this -
{code:java}
{
"name": "sleeper-service",
"version": "1",
"components" :
[
{
"name": "ping",
"number_of_containers": 2,
"resource": {
"cpus": 1,
"memory": "256"
},
"launch_command": "sleep 9000",
"placement_policy": {
"constraints": [
{
"type": "ANTI_AFFINITY",
"scope": "NODE",
"target_tags": [
"ping"
]
}
]
}
}
]
}
{code}
Launch a service with the above yarnfile as below -
{code:java}
yarn app -launch simple-aa-1 simple_AA.json
{code}
Let's assume there are only 5 nodes in this cluster. Now, flex the above
service to 1 extra container than the number of nodes (6 in my case).
{code:java}
yarn app -flex simple-aa-1 -component ping 6
{code}
Only 5 containers will be allocated and running for simple-aa-1. At this point,
flex it down to 5 containers -
{code:java}
yarn app -flex simple-aa-1 -component ping 5
{code}
This is what is seen in the serviceam log at this point -
{code:java}
2018-05-03 20:17:38,469 [IPC Server handler 0 on 38124] INFO
service.ClientAMService - Flexing component ping to 5
2018-05-03 20:17:38,469 [Component dispatcher] INFO component.Component -
[FLEX DOWN COMPONENT ping]: scaling down from 6 to 5
2018-05-03 20:17:38,470 [Component dispatcher] INFO
instance.ComponentInstance - [COMPINSTANCE ping-4 :
container_1525297086734_0013_01_06]: Flexed down by user, destroying.
2018-05-03 20:17:38,473 [Component dispatcher] INFO component.Component -
[COMPONENT ping] Transitioned from FLEXING to STABLE on FLEX event.
2018-05-03 20:17:38,474 [pool-5-thread-8] INFO
registry.YarnRegistryViewForProviders - [COMPINSTANCE ping-4 :
container_1525297086734_0013_01_06]: Deleting registry path
/users/root/services/yarn-service/simple-aa-1/components/ctr-1525297086734-0013-01-06
2018-05-03 20:17:38,476 [Component dispatcher] ERROR component.Component -
[COMPONENT ping]: Invalid event CHECK_STABLE at STABLE
org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
CHECK_STABLE at STABLE
at
org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:388)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
at
org.apache.hadoop.yarn.service.component.Component.handle(Component.java:913)
at
org.apache.hadoop.yarn.service.ServiceScheduler$ComponentEventHandler.handle(ServiceScheduler.java:574)
at
org.apache.hadoop.yarn.service.ServiceScheduler$ComponentEventHandler.handle(ServiceScheduler.java:563)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
at java.lang.Thread.run(Thread.java:745)
2018-05-03 20:17:38,480 [Component dispatcher] ERROR component.Component -
[COMPONENT ping]: Invalid event CHECK_STABLE at STABLE
org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
CHECK_STABLE at STABLE
at
org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:388)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
at
org.apache.hadoop.yarn.service.component.Component.handle(Component.java:913)
at
org.apache.hadoop.yarn.service.ServiceScheduler$ComponentEventHandler.handle(ServiceScheduler.java:574)
at
org.apache.hadoop.yarn.service.ServiceScheduler$ComponentEventHandler.handle(ServiceScheduler.java:563)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
at
[jira] [Created] (YARN-8203) Provide functional artifact & launch_command in examples of YARN-Services-Examples.md
Gour Saha created YARN-8203: --- Summary: Provide functional artifact & launch_command in examples of YARN-Services-Examples.md Key: YARN-8203 URL: https://issues.apache.org/jira/browse/YARN-8203 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha In YARN-7781, [~eyang] made the following suggestion. I think it is a good one and should be addressed. Copying Eric's comment verbatim - The examples are showing nginx, but nginx does not work until YARN-7654 is committed because nginx depends on ENTRY_POINT support and run privileged container. It would be good to change the example to use centos:httpd-24-centos7, and launch_command: /usr/bin/run-httpd for functional examples. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8161) ServiceState FLEX should be removed
Gour Saha created YARN-8161: --- Summary: ServiceState FLEX should be removed Key: YARN-8161 URL: https://issues.apache.org/jira/browse/YARN-8161 Project: Hadoop YARN Issue Type: Sub-task Components: yarn-native-services Affects Versions: 3.1.0 Reporter: Gour Saha ServiceState FLEX is not required to trigger flex up/down of containers and should be removed -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8136) Add version attribute to site doc examples and quickstart
Gour Saha created YARN-8136: --- Summary: Add version attribute to site doc examples and quickstart Key: YARN-8136 URL: https://issues.apache.org/jira/browse/YARN-8136 Project: Hadoop YARN Issue Type: Sub-task Components: site Reporter: Gour Saha version attribute is missing in the following 2 site doc files - src/site/markdown/yarn-service/Examples.md src/site/markdown/yarn-service/QuickStart.md -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8122) Component health threshold monitor
Gour Saha created YARN-8122: --- Summary: Component health threshold monitor Key: YARN-8122 URL: https://issues.apache.org/jira/browse/YARN-8122 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha Slider supported component health threshold monitoring with SLIDER-1246. It would be good to have this feature for YARN Service too. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8074) Support placement policy composite constraints in YARN Service
Gour Saha created YARN-8074: --- Summary: Support placement policy composite constraints in YARN Service Key: YARN-8074 URL: https://issues.apache.org/jira/browse/YARN-8074 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha This is a follow up of YARN-7142 where we support more advanced placement policy features like creating composite constraints by exposing expressions in YARN Service specification. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8070) Yarn Service API site doc broken due to unwanted character in YarnServiceAPI.md
Gour Saha created YARN-8070: --- Summary: Yarn Service API site doc broken due to unwanted character in YarnServiceAPI.md Key: YARN-8070 URL: https://issues.apache.org/jira/browse/YARN-8070 Project: Hadoop YARN Issue Type: Sub-task Components: site Affects Versions: 3.1.0 Reporter: Gour Saha The YARN Service API html page is not rendering properly in the yarn site documentation due to unnecessary # character in YarnServiceAPI.md. If possible, this should be fixed before we release 3.1.0 since it is the first release for YARN Service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7991) Use ServiceState values to publish to ATS
Gour Saha created YARN-7991: --- Summary: Use ServiceState values to publish to ATS Key: YARN-7991 URL: https://issues.apache.org/jira/browse/YARN-7991 Project: Hadoop YARN Issue Type: Sub-task Components: yarn-native-services Reporter: Gour Saha Add the state DELETED to ServiceState and then use ServiceState values to publish to ATS (instead of FinalApplicationStatus). Refer to parent issue for more details. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7955) Calling stop on an already stopped service says "Successfully stopped service"
Gour Saha created YARN-7955:
---
Summary: Calling stop on an already stopped service says
"Successfully stopped service"
Key: YARN-7955
URL: https://issues.apache.org/jira/browse/YARN-7955
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 3.1.0
Reporter: Gour Saha
If you invoke "yarn app -stop " on an already stopped service it
confusingly responds with message "Successfully stopped service
". It should say "Service is already stopped".
The same is seen with the REST API PUT request with data \{ "state":
"STOPPED"}, the response is 200 OK and diagnostics with same message
"Successfully stopped service ". It should return 400 Bad Request
with message "Service is already stopped".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7836) YARN Service component update PUT API should not use component name from JSON body
Gour Saha created YARN-7836:
---
Summary: YARN Service component update PUT API should not use
component name from JSON body
Key: YARN-7836
URL: https://issues.apache.org/jira/browse/YARN-7836
Project: Hadoop YARN
Issue Type: Sub-task
Components: api, yarn-native-services
Reporter: Gour Saha
The YARN Service PUT API for component update should not use component name
from the JSON body. The component update PUT URI is as follows -
[http://localhost:9191/app/v1/services/]/components/
e.g. [http://localhost:9191/app/v1/services/hello-world/components/hello]
The component name is already in the URI, hence the JSON body expected should
be only -
{noformat}
{
"number_of_containers": 3
}
{noformat}
It should not expect the name attribute in the JSON body. In fact, if the JSON
body contains a name attribute with value anything other than the
in the path param, we should send a 400 bad request saying they do not match.
If they are the same, it should be okay and we can process the request.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7816) YARN Service - Two different users are unable to launch a service of the same name
Gour Saha created YARN-7816: --- Summary: YARN Service - Two different users are unable to launch a service of the same name Key: YARN-7816 URL: https://issues.apache.org/jira/browse/YARN-7816 Project: Hadoop YARN Issue Type: Bug Components: applications Reporter: Gour Saha Now that YARN-7605 is committed, I am able to create a service in an unsecured cluster from cmd line as the logged in user. However when I login as a different user, I am unable to create a service of the exact same name. This feature should be supported in a multi-user setup. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7799) YARN Service dependency follow up work
Gour Saha created YARN-7799:
---
Summary: YARN Service dependency follow up work
Key: YARN-7799
URL: https://issues.apache.org/jira/browse/YARN-7799
Project: Hadoop YARN
Issue Type: Bug
Components: client, resourcemanager
Reporter: Gour Saha
As per [~jianhe] these are some followup items that make sense to do after
YARN-7766. Quoting Jian's comment below -
Currently, if user doesn't supply location when run yarn app -enableFastLaunch,
the jars will be put under this location
{code}
hdfs:///yarn-services//service-dep.tar.gz
{code}
Since API server is embedded in RM, should RM look for this location too if
"yarn.service.framework.path" is not specified ?
And if "yarn.service.framework.path" is not specified and still the file
doesn't exist at above default location, I think RM can try to upload the jars
to above default location instead, currently RM is uploading the jars to the
location defined by below code. This folder is per app and also inconsistent
with CLI location.
{code}
protected Path addJarResource(String serviceName,
Map localResources)
throws IOException, SliderException {
Path libPath = fs.buildClusterDirPath(serviceName);
{code}
By doing this, the next time a submission request comes, RM doesn't need to
upload the jars again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7781) Update YARN-Services-Examples.md to be in sync with the latest code
Gour Saha created YARN-7781:
---
Summary: Update YARN-Services-Examples.md to be in sync with the
latest code
Key: YARN-7781
URL: https://issues.apache.org/jira/browse/YARN-7781
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Gour Saha
Update YARN-Services-Examples.md to make the following additions/changes:
1. Add an additional URL and PUT Request JSON to support flex:
Update to flex up/down the no of containers (instances) of a component of a
service
PUT URL – http://localhost:9191/app/v1/services/hello-world
PUT Request JSON
{code}
{
"components" : [ {
"name" : "hello",
"number_of_containers" : 3
} ]
}
{code}
2. Modify all occurrences of /ws/ to /app/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7766) Introduce a new config property for YARN Service dependency tarball location
Gour Saha created YARN-7766:
---
Summary: Introduce a new config property for YARN Service
dependency tarball location
Key: YARN-7766
URL: https://issues.apache.org/jira/browse/YARN-7766
Project: Hadoop YARN
Issue Type: Bug
Components: applications, client, yarn-native-services
Reporter: Gour Saha
Introduce a new config property (something like _yarn.service.framework.path_
in-line with _mapreduce.application.framework.path_) for YARN Service
dependency tarball location. This will provide flexibility to the
user/cluster-admin to upload the dependency tarball to a location of their
choice. If this config property is not set, YARN Service client will default to
uploading all dependency jars from the client-host's classpath for every
service launch request (as it does today).
Also, accept an optional destination HDFS location for *-enableFastLaunch*
command, to specify the location where user/cluster-admin wants to upload the
tarball. If not specified, let's default it to the location we use today. The
cluster-admin still needs to set _yarn.service.framework.path_ to this default
location otherwise it will not be used. So the command-line will become
something like this -
{code:java}
yarn app -enableFastLaunch []{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7616) App status does not return state STABLE for a running and stable service
Gour Saha created YARN-7616: --- Summary: App status does not return state STABLE for a running and stable service Key: YARN-7616 URL: https://issues.apache.org/jira/browse/YARN-7616 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha state currently returns null for a running and stable service. Looks like the code does not return ServiceState.STABLE under any circumstance. Will need to wire this in. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7543) FileNotFoundException when creating a yarn service due to broken link under hadoop lib directory
Gour Saha created YARN-7543:
---
Summary: FileNotFoundException when creating a yarn service due to
broken link under hadoop lib directory
Key: YARN-7543
URL: https://issues.apache.org/jira/browse/YARN-7543
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Gour Saha
The hadoop lib dir had a broken link to a ojdb jar which was not really
required for a YARN service creation. The app submission failed with the below
FNFE. Ideally it should be handled and app should be successfully submitted and
let the app fail if it really needed the jar of the broken link -
{code}
[root@ctr-e134-1499953498516-324910-01-02 ~]# yarn app -launch gour-sleeper
sleeper
WARNING: YARN_LOG_DIR has been replaced by HADOOP_LOG_DIR. Using value of
YARN_LOG_DIR.
WARNING: YARN_LOGFILE has been replaced by HADOOP_LOGFILE. Using value of
YARN_LOGFILE.
WARNING: YARN_PID_DIR has been replaced by HADOOP_PID_DIR. Using value of
YARN_PID_DIR.
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
17/11/21 03:21:58 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
17/11/21 03:21:59 INFO client.RMProxy: Connecting to ResourceManager at
ctr-e134-1499953498516-324910-01-03.example.com/172.27.47.1:8050
17/11/21 03:22:00 WARN shortcircuit.DomainSocketFactory: The short-circuit
local reads feature cannot be used because libhadoop cannot be loaded.
17/11/21 03:22:00 INFO client.RMProxy: Connecting to ResourceManager at
ctr-e134-1499953498516-324910-01-03.example.com/172.27.47.1:8050
17/11/21 03:22:00 INFO client.ServiceClient: Loading service definition from
local FS:
/usr/hdp/3.0.0.0-493/hadoop-yarn/yarn-service-examples/sleeper/sleeper.json
17/11/21 03:22:01 INFO client.ServiceClient: Persisted service gour-sleeper at
hdfs://ctr-e134-1499953498516-324910-01-03.example.com:8020/user/hdfs/.yarn/services/gour-sleeper/gour-sleeper.json
17/11/21 03:22:01 INFO conf.Configuration: resource-types.xml not found
17/11/21 03:22:01 WARN client.ServiceClient: AM log4j property file doesn't
exist: /usr/hdp/3.0.0.0-493/hadoop/conf/yarnservice-log4j.properties
17/11/21 03:22:01 INFO client.ServiceClient: Uploading all dependency jars to
HDFS. For faster submission of apps, pre-upload dependency jars to HDFS using
command: yarn app -enableFastLaunch
Exception in thread "main" java.io.FileNotFoundException: File
/usr/hdp/3.0.0.0-493/hadoop/lib/ojdbc6.jar does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:867)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:365)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2399)
at
org.apache.hadoop.yarn.service.utils.CoreFileSystem.submitFile(CoreFileSystem.java:434)
at
org.apache.hadoop.yarn.service.utils.ServiceUtils.putAllJars(ServiceUtils.java:409)
at
org.apache.hadoop.yarn.service.provider.ProviderUtils.addAllDependencyJars(ProviderUtils.java:138)
at
org.apache.hadoop.yarn.service.client.ServiceClient.addJarResource(ServiceClient.java:695)
at
org.apache.hadoop.yarn.service.client.ServiceClient.submitApp(ServiceClient.java:553)
at
org.apache.hadoop.yarn.service.client.ServiceClient.actionCreate(ServiceClient.java:212)
at
org.apache.hadoop.yarn.service.client.ServiceClient.actionLaunch(ServiceClient.java:197)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:447)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:111)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7523) Introduce description and version field in Service record
Gour Saha created YARN-7523: --- Summary: Introduce description and version field in Service record Key: YARN-7523 URL: https://issues.apache.org/jira/browse/YARN-7523 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha YARN-7512 would need version field in Service record. It would be good to introduce a description field also to allow service owners to capture some details which can be used to display in Service catalog as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7519) Exception during RM shutdown - "No enum constant YarnApplicationAttemptState.FINAL_SAVING"
Gour Saha created YARN-7519:
---
Summary: Exception during RM shutdown - "No enum constant
YarnApplicationAttemptState.FINAL_SAVING"
Key: YARN-7519
URL: https://issues.apache.org/jira/browse/YARN-7519
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 3.1.0
Reporter: Gour Saha
Seeing the following exception while doing a shutdown of RM in a cluster
created off of the latest in trunk (3.1.0-SNAPSHOT) -
{code}
2017-11-16 22:17:28,956 WARN resourcemanager.DefaultAMSProcessor
(DefaultAMSProcessor.java:allocate(269)) - appattempt_1510699559234_0001_01
is in FINAL_SAVING state, ignore container allocate request.
2017-11-16 22:17:37,497 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(886)) -
application_1510699559234_0001 State change from RUNNING to KILLING on event =
KILL
2017-11-16 22:17:40,403 INFO rmcontainer.RMContainerImpl
(RMContainerImpl.java:handle(485)) - container_e01_1510699559234_0001_01_02
Container Transitioned from RUNNING to COMPLETED
2017-11-16 22:17:40,900 INFO rmcontainer.RMContainerImpl
(RMContainerImpl.java:handle(485)) - container_e01_1510699559234_0001_01_03
Container Transitioned from RUNNING to COMPLETED
2017-11-16 22:17:41,149 INFO rmcontainer.RMContainerImpl
(RMContainerImpl.java:handle(485)) - container_e01_1510699559234_0001_01_01
Container Transitioned from RUNNING to COMPLETED
2017-11-16 22:20:27,773 ERROR webapp.Dispatcher (Dispatcher.java:service(171))
- error handling URI: /cluster/appattempt/appattempt_1510699559234_0001_01
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:162)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
at
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:941)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875)
at
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:178)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
at
com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:110)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:304)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
at
org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.doFilter(RMAuthenticationFilter.java:82)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1578)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
[jira] [Created] (YARN-7512) Support service upgrade via YARN Service API and CLI
Gour Saha created YARN-7512: --- Summary: Support service upgrade via YARN Service API and CLI Key: YARN-7512 URL: https://issues.apache.org/jira/browse/YARN-7512 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha YARN Service API and CLI needs to support service (and containers) upgrade in line with what Slider supported in SLIDER-787 and http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7488) Make ServiceClient.getAppId method public to return ApplicationId for a service name
Gour Saha created YARN-7488:
---
Summary: Make ServiceClient.getAppId method public to return
ApplicationId for a service name
Key: YARN-7488
URL: https://issues.apache.org/jira/browse/YARN-7488
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Gour Saha
There are several APIs in YarnClient which can be used when the ApplicationId
is available. For a YARN service getting the ApplicationId corresponding to a
service name will open up all these APIs for the service. Hence making the
below API public will be very helpful.
{code}
private synchronized ApplicationId getAppId(String serviceName)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7161) Fix special chars in DNS documentation
Gour Saha created YARN-7161: --- Summary: Fix special chars in DNS documentation Key: YARN-7161 URL: https://issues.apache.org/jira/browse/YARN-7161 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha The yarn_dns_server.md file in YARN-5244 has special chars instead of hyphens and a few unwanted special chars. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7160) Create install doc for YARN native services
Gour Saha created YARN-7160: --- Summary: Create install doc for YARN native services Key: YARN-7160 URL: https://issues.apache.org/jira/browse/YARN-7160 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha Create a site installation guide for YARN native services -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7126) Create introductory site documentation for YARN native services
Gour Saha created YARN-7126: --- Summary: Create introductory site documentation for YARN native services Key: YARN-7126 URL: https://issues.apache.org/jira/browse/YARN-7126 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5244) Documentation required for DNS Server implementation
[ https://issues.apache.org/jira/browse/YARN-5244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gour Saha resolved YARN-5244. - Resolution: Fixed plugged into the site documentation and committed > Documentation required for DNS Server implementation > > > Key: YARN-5244 > URL: https://issues.apache.org/jira/browse/YARN-5244 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: dns overview.png, dns record creation.jpeg, dns record > removal.jpeg, yarn_dns_server.md > > > The DNS server requires documentation describing its functionality etc -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7073) Rest API site documentation
Gour Saha created YARN-7073: --- Summary: Rest API site documentation Key: YARN-7073 URL: https://issues.apache.org/jira/browse/YARN-7073 Project: Hadoop YARN Issue Type: Sub-task Components: documentation, site, yarn-native-services Reporter: Gour Saha Commit site documentation for REST API service, generated from the swagger definition as a MD file. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6626) Embed REST API service into RM
Gour Saha created YARN-6626: --- Summary: Embed REST API service into RM Key: YARN-6626 URL: https://issues.apache.org/jira/browse/YARN-6626 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha Fix For: yarn-native-services As of now the deployment model of the Native Services REST API service is standalone. There are several cross-cutting solutions that can be inherited for free (kerberos, HA, ACLs, trusted proxy support, etc.) by the REST API service if it is embedded into the RM process. In fact we can expose the REST API via the same port as RM UI (8088 default). The URI path /services/v1/applications will distinguish the REST API calls from other RM APIs. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6269) Pull into native services SLIDER-1185 - container/application diagnostics for enhanced debugging
Gour Saha created YARN-6269: --- Summary: Pull into native services SLIDER-1185 - container/application diagnostics for enhanced debugging Key: YARN-6269 URL: https://issues.apache.org/jira/browse/YARN-6269 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6187) Auto-generate REST API resources and server side stubs from swagger definition
Gour Saha created YARN-6187: --- Summary: Auto-generate REST API resources and server side stubs from swagger definition Key: YARN-6187 URL: https://issues.apache.org/jira/browse/YARN-6187 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha Fix For: yarn-native-services Currently the REST API resource package is generated offline using swagger codegen library and then formatted with basic eclipse formatter and then checked in. It is not entirely in line with YARN documentation and coding guidelines. We need to do these things to streamline this effort - # Auto-generate the resource package and the server side API interface/stubs using swagger codegen libraries # Use a template framework like jmustache or similar (or better) to align/add documentation and code-formatting in-line with Yarn project standards -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6067) API Service HA
Gour Saha created YARN-6067: --- Summary: API Service HA Key: YARN-6067 URL: https://issues.apache.org/jira/browse/YARN-6067 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha We need to start thinking about HA for API Service. How do we achieve it? Should API Service become part of the RM process to get a lot of things for free? Should there be some other strategy. We need to start the discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6009) RM fails to start during an upgrade - Failed to load/recover state (YarnException: Invalid application timeout, value=0 for type=LIFETIME)
Gour Saha created YARN-6009:
---
Summary: RM fails to start during an upgrade - Failed to
load/recover state (YarnException: Invalid application timeout, value=0 for
type=LIFETIME)
Key: YARN-6009
URL: https://issues.apache.org/jira/browse/YARN-6009
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Gour Saha
Priority: Critical
ResourceManager fails to start during an upgrade with the following exceptions
-
Exception 1:
{color:red}
{code}
2016-12-09 14:57:23,508 INFO capacity.CapacityScheduler
(CapacityScheduler.java:initScheduler(328)) - Initialized CapacityScheduler
with calculator=class
org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator,
minimumAllocation=<>, maximumAllocation=<>, asynchronousScheduling=false, asyncScheduleInterval=5ms
2016-12-09 14:57:23,509 WARN ha.ActiveStandbyElector
(ActiveStandbyElector.java:becomeActive(863)) - Exception handling the winning
of election
org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
at
org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:129)
at
org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:859)
at
org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:463)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:611)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
Caused by: org.apache.hadoop.ha.ServiceFailedException: Error when
transitioning to Active mode
at
org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:318)
at
org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:127)
... 4 more
Caused by: org.apache.hadoop.service.ServiceStateException:
org.apache.hadoop.yarn.exceptions.YarnException: Invalid application timeout,
value=0 for type=LIFETIME
at
org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:991)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1032)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1028)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1028)
at
org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:313)
... 5 more
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Invalid application
timeout, value=0 for type=LIFETIME
at
org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateApplicationTimeouts(RMServerUtils.java:305)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:365)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:330)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:463)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1184)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:594)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
... 13 more
{code}
{color}
Exception 2:
{color:red}
{code}
2016-12-09 14:57:26,162 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(790)) -
application_1477927786494_0008 State change from NEW to FINISHED
2016-12-09 14:57:26,162 ERROR resourcemanager.ResourceManager
(ResourceManager.java:serviceStart(599)) - Failed to load/recover state
org.apache.hadoop.yarn.exceptions.YarnException: Invalid application timeout,
value=0 for type=LIFETIME
at
org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateApplicationTimeouts(RMServerUtils.java:305)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:365)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:330)
at
[jira] [Created] (YARN-5945) Add Slider debug config in yarn log4j
Gour Saha created YARN-5945:
---
Summary: Add Slider debug config in yarn log4j
Key: YARN-5945
URL: https://issues.apache.org/jira/browse/YARN-5945
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Gour Saha
Fix For: yarn-native-services
We should add the Slider debug config property in the yarn log4j file
(commented by default). This will help us to point customers and end-users who
want to run "yarn slider ..." cli command lines in debug mode, to simply edit
the log4j and uncomment the line.
Here is the property that needs to be added (in commented form) -
{code}
#log4j.logger.org.apache.slider=DEBUG
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5883) Avoid or eliminate expensive YARN get all applications call
Gour Saha created YARN-5883: --- Summary: Avoid or eliminate expensive YARN get all applications call Key: YARN-5883 URL: https://issues.apache.org/jira/browse/YARN-5883 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha Fix For: yarn-native-services YARN _getAllApplications()_ call with no param or with application types _getApplications(Set applicationTypes)_ are very costly in a cluster with hundreds or thousands of Slider apps running. YARN now has the API to filter by application tags - YARN-4491. We should be able to use this to avoid these expensive calls. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5870) Expose getApplications API in YarnClient with GetApplicationsRequest parameter
Gour Saha created YARN-5870: --- Summary: Expose getApplications API in YarnClient with GetApplicationsRequest parameter Key: YARN-5870 URL: https://issues.apache.org/jira/browse/YARN-5870 Project: Hadoop YARN Issue Type: Bug Components: client Reporter: Gour Saha It would be best to expose getApplications API in YarnClient with GetApplicationsRequest parameter. That opens up all the filters and limits to the client. This will prevent us from the need to expose more getApplications API going forward, for every new parameter/filter like it was done in YARN-4491. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5817) Make yarn.cmd changes required for slider and servicesapi
Gour Saha created YARN-5817: --- Summary: Make yarn.cmd changes required for slider and servicesapi Key: YARN-5817 URL: https://issues.apache.org/jira/browse/YARN-5817 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha Fix For: yarn-native-services As per YARN-5808 and other changes made to yarn script, there are probably some corresponding changes required in _hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd_. We need to identify and make those changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5813) Slider should not try to set a negative lifetime timeout value
Gour Saha created YARN-5813:
---
Summary: Slider should not try to set a negative lifetime timeout
value
Key: YARN-5813
URL: https://issues.apache.org/jira/browse/YARN-5813
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Gour Saha
Fix For: yarn-native-services
YARN throws an exception if Slider sets the lifetime timeout to a negative
value. So it should be checked for a positive value before setting it. That
would mean that any negative value will be treated as unlimited lifetime which
should be ok. The REST API service internally relies on the value -1 to signify
unlimited lifetime anyway.
Exception thrown as below -
{noformat}
org.apache.hadoop.yarn.exceptions.YarnException: Invalid application timeout,
value=-3600 for type=LIFETIME
at
org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateApplicationTimeouts(RMServerUtils.java:299)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:362)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:287)
at
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:582)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:218)
at
org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:419)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:239)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy70.submitApplication(Unknown Source)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:254)
at
org.apache.slider.core.launch.AppMasterLauncher.submitApplication(AppMasterLauncher.java:212)
at
org.apache.slider.client.SliderClient.launchApplication(SliderClient.java:2406)
at
org.apache.slider.client.SliderClient.startCluster(SliderClient.java:1977)
at
org.apache.slider.client.SliderClient.actionCreate(SliderClient.java:739)
at
org.apache.hadoop.yarn.services.api.impl.ApplicationApiService$1.run(ApplicationApiService.java:364)
at
org.apache.hadoop.yarn.services.api.impl.ApplicationApiService$1.run(ApplicationApiService.java:360)
at
org.apache.hadoop.yarn.services.api.impl.ApplicationApiService$2.run(ApplicationApiService.java:693)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.services.api.impl.ApplicationApiService.invokeSliderClientRunnable(ApplicationApiService.java:690)
at
org.apache.hadoop.yarn.services.api.impl.ApplicationApiService.createSliderApp(ApplicationApiService.java:360)
at
[jira] [Created] (YARN-5812) Exception during GET call - "Failed to retrieve application: null"
Gour Saha created YARN-5812:
---
Summary: Exception during GET call - "Failed to retrieve
application: null"
Key: YARN-5812
URL: https://issues.apache.org/jira/browse/YARN-5812
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Gour Saha
Fix For: yarn-native-services
GET response spits out -
{code}
{"diagnostics":"Failed to retrieve application: null"}
{code}
This happens only while the app is in ACCEPTED state. After the app goes to the
running state the GET response works fine.
Exception in the server logs is -
{code}
2016-10-25 12:32:09,446 [236689627@qtp-848363848-185 -
/services/v1/applications/test-app-1] ERROR impl.ApplicationApiService -
Exception calling slider.actionRegistryGetConfig
org.apache.slider.core.exceptions.NotFoundException:
/registry/users/root/services/org-apache-slider/test-app-1
at org.apache.slider.client.ClientUtils.resolve(ClientUtils.java:70)
at
org.apache.slider.client.ClientUtils.lookupServiceRecord(ClientUtils.java:62)
at
org.apache.slider.client.ClientUtils.lookupServiceRecord(ClientUtils.java:46)
at
org.apache.slider.client.ClientUtils.getConfigFromRegistry(ClientUtils.java:78)
at
org.apache.slider.client.SliderClient.actionRegistryGetConfig(SliderClient.java:4216)
at
org.apache.hadoop.yarn.services.api.impl.ApplicationApiService$5.run(ApplicationApiService.java:1105)
at
org.apache.hadoop.yarn.services.api.impl.ApplicationApiService$5.run(ApplicationApiService.java:1099)
at
org.apache.hadoop.yarn.services.api.impl.ApplicationApiService$2.run(ApplicationApiService.java:700)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.services.api.impl.ApplicationApiService.invokeSliderClientRunnable(ApplicationApiService.java:697)
at
org.apache.hadoop.yarn.services.api.impl.ApplicationApiService.getSliderApplicationRegistry(ApplicationApiService.java:1098)
at
org.apache.hadoop.yarn.services.api.impl.ApplicationApiService.getApplication(ApplicationApiService.java:876)
at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1243)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1243)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
[jira] [Created] (YARN-5808) Add gc log options to the yarn daemon script when starting services-api
Gour Saha created YARN-5808:
---
Summary: Add gc log options to the yarn daemon script when
starting services-api
Key: YARN-5808
URL: https://issues.apache.org/jira/browse/YARN-5808
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Gour Saha
Fix For: yarn-native-services
We need to add the gc log options as below when starting services-api using the
yarn-daemon.sh script -
{code}
-XX:+PrintGC -Xloggc:$YARN_LOG_DIR/services-api-gc.log -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5796) Convert enums values in service code to upper case and special handling of an error
Gour Saha created YARN-5796: --- Summary: Convert enums values in service code to upper case and special handling of an error Key: YARN-5796 URL: https://issues.apache.org/jira/browse/YARN-5796 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha Fix For: yarn-native-services Bug fixes - - Convert enums values in service code to upper case in line with YARN-5775 - Elegantly handle the instance/directory exists error during create app (if the app was previously created but is in stopped/failed state) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5791) [YARN Native Service] Build application specific UI on top of data posted to timeline service V.2
Gour Saha created YARN-5791: --- Summary: [YARN Native Service] Build application specific UI on top of data posted to timeline service V.2 Key: YARN-5791 URL: https://issues.apache.org/jira/browse/YARN-5791 Project: Hadoop YARN Issue Type: New Feature Reporter: Gour Saha As per YARN-5780 we will start getting application-specific data in timeline service v2 for all YARN native services. Exposing a UI for these application-specific data would be very beneficial to application owners. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5775) Bug fixes in swagger definition
Gour Saha created YARN-5775:
---
Summary: Bug fixes in swagger definition
Key: YARN-5775
URL: https://issues.apache.org/jira/browse/YARN-5775
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Gour Saha
Fix For: yarn-native-services
All enums have been listed in lowercase. Need to convert all of them to
uppercase.
For e.g. ContainerState:
{noformat}
enum:
- init
- ready
{noformat}
needs to be changed to -
{noformat}
enum:
- INIT
- READY
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5770) Performance improvement of native-services REST API service
Gour Saha created YARN-5770: --- Summary: Performance improvement of native-services REST API service Key: YARN-5770 URL: https://issues.apache.org/jira/browse/YARN-5770 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha Fix For: yarn-native-services Make enhancements and bug-fixes to eliminate frequent full GC of the REST API Service. Dependent on few Slider fixes like SLIDER-1168 as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5769) Integrate update app lifetime using feature implemented in YARN-5611
Gour Saha created YARN-5769: --- Summary: Integrate update app lifetime using feature implemented in YARN-5611 Key: YARN-5769 URL: https://issues.apache.org/jira/browse/YARN-5769 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha Fix For: yarn-native-services The REST API PUT call provides capability to update the lifetime of a running application. Once YARN-5611 is available we need to integrate it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5768) Integrate remaining app lifetime using feature implemented in YARN-4206
Gour Saha created YARN-5768: --- Summary: Integrate remaining app lifetime using feature implemented in YARN-4206 Key: YARN-5768 URL: https://issues.apache.org/jira/browse/YARN-5768 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha Fix For: yarn-native-services -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5759) Capability to register for a notification/callback on the expiry of timeouts for an application
Gour Saha created YARN-5759: --- Summary: Capability to register for a notification/callback on the expiry of timeouts for an application Key: YARN-5759 URL: https://issues.apache.org/jira/browse/YARN-5759 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Gour Saha There is a need for the YARN native services REST-API service, to take certain actions once a timeout of an application expires. For example, an immediate requirement is to destroy a Slider application, once its lifetime timeout expires and YARN has stopped the application. Destroying a Slider application means cleanup of Slider HDFS state store and ZK paths for that application. Potentially, there will be advanced requirements from the REST-API service and other services in the future, which will make this feature very handy. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5740) Add a new field in Slider status output - lifetime (remaining)
Gour Saha created YARN-5740: --- Summary: Add a new field in Slider status output - lifetime (remaining) Key: YARN-5740 URL: https://issues.apache.org/jira/browse/YARN-5740 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha Fix For: yarn-native-services With YARN-5735, REST service is now setting lifetime to application during submission (YARN-4205 exposed application lifetime support). Now Slider status needs to expose this field so that the REST service can return it in its GET response. Note, the lifetime value that GET response intends to return is the remaining lifetime of the application. There is an ongoing discussion in YARN-4206, that the lifetime value returned in Application Report will be the remaining lifetime (at the time of request). So until it is finalized, the lifetime value might go through different connotations. But as long as we have the lifetime field in the status output, it will be a good start. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5729) Bug fixes identified during testing
Gour Saha created YARN-5729: --- Summary: Bug fixes identified during testing Key: YARN-5729 URL: https://issues.apache.org/jira/browse/YARN-5729 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha Fix For: yarn-native-services Use this to apply bug fixes identified during testing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5680) Add 2 new fields in Slider status output - image-name and is-privileged-container
Gour Saha created YARN-5680: --- Summary: Add 2 new fields in Slider status output - image-name and is-privileged-container Key: YARN-5680 URL: https://issues.apache.org/jira/browse/YARN-5680 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha We need to add 2 new fields in Slider status output for docker provider - image-name and is-privileged-container. The native services REST API needs to expose these 2 attribute values to the end-users. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5675) Checkin swagger definition in the repo
Gour Saha created YARN-5675: --- Summary: Checkin swagger definition in the repo Key: YARN-5675 URL: https://issues.apache.org/jira/browse/YARN-5675 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha This task will be used to submit the REST API swagger definition (yaml format) to be checked in to the repo -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5623) Apply SLIDER-1166 to yarn-native-services branch
Gour Saha created YARN-5623: --- Summary: Apply SLIDER-1166 to yarn-native-services branch Key: YARN-5623 URL: https://issues.apache.org/jira/browse/YARN-5623 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha Fix For: yarn-native-services SLIDER-1166 fixes a critical issue in SliderClient when used as a service. It needs to be merged into yarn-native-services branch as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5610) Initial code for native services REST API
Gour Saha created YARN-5610: --- Summary: Initial code for native services REST API Key: YARN-5610 URL: https://issues.apache.org/jira/browse/YARN-5610 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha This task will be used to submit and review patches for the initial code drop for the native services REST API -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5513) Move Java only tests from slider develop to yarn-native-services
Gour Saha created YARN-5513: --- Summary: Move Java only tests from slider develop to yarn-native-services Key: YARN-5513 URL: https://issues.apache.org/jira/browse/YARN-5513 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5498) [Usability] Make UI continue to work and render already loaded models even when there is no network connection
Gour Saha created YARN-5498:
---
Summary: [Usability] Make UI continue to work and render already
loaded models even when there is no network connection
Key: YARN-5498
URL: https://issues.apache.org/jira/browse/YARN-5498
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Gour Saha
I load the UI in my browser and traverse to all the tabs. Then I disconnect the
network. The tabs "Queues", "Applications" and "Nodes" continue to work even
when there is no network connection. However, the "Cluster Overview" tab does
not work. This tab should also continue to show the already loaded models, for
better usability.
We should also add a small message on the top of the UI when the network
connection is gone. It is very similar to what gmail or other modern
applications do today. An exception of type
{color:red}net::ERR_INTERNET_DISCONNECTED{color} is already thrown, which can
be caught and this small message can be marked visible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-3561) Non-AM Containers continue to run even after AM is stopped
Gour Saha created YARN-3561:
---
Summary: Non-AM Containers continue to run even after AM is stopped
Key: YARN-3561
URL: https://issues.apache.org/jira/browse/YARN-3561
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager, yarn
Affects Versions: 2.6.0
Reporter: Gour Saha
Priority: Critical
Non-AM containers continue to run even after application is stopped. This
occurred while deploying Storm 0.9.3 using Slider (0.60.0 and 0.70.1) in a
Hadoop 2.6 deployment.
Following are the NM logs from 2 different nodes:
*host-07* - where Slider AM was running
*host-03* - where Storm NIMBUS container was running.
*Note:* The logs are partial, starting with the time when the relevant Slider
AM and NIMBUS containers were allocated, till the time when the Slider AM was
stopped. Also, the large number of Memory usage log lines were removed
keeping only a few starts and ends of every segment.
*NM log from host-07 where Slider AM container was running:*
{noformat}
2015-04-29 00:39:24,614 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(356)) - Stopping resource-monitoring for
container_1428575950531_0020_02_01
2015-04-29 00:41:10,310 INFO ipc.Server (Server.java:saslProcess(1306)) - Auth
successful for appattempt_1428575950531_0021_01 (auth:SIMPLE)
2015-04-29 00:41:10,322 INFO containermanager.ContainerManagerImpl
(ContainerManagerImpl.java:startContainerInternal(803)) - Start request for
container_1428575950531_0021_01_01 by user yarn
2015-04-29 00:41:10,322 INFO containermanager.ContainerManagerImpl
(ContainerManagerImpl.java:startContainerInternal(843)) - Creating a new
application reference for app application_1428575950531_0021
2015-04-29 00:41:10,323 INFO application.Application
(ApplicationImpl.java:handle(464)) - Application application_1428575950531_0021
transitioned from NEW to INITING
2015-04-29 00:41:10,325 INFO nodemanager.NMAuditLogger
(NMAuditLogger.java:logSuccess(89)) - USER=yarn IP=10.84.105.162
OPERATION=Start Container Request TARGET=ContainerManageImpl
RESULT=SUCCESS APPID=application_1428575950531_0021
CONTAINERID=container_1428575950531_0021_01_01
2015-04-29 00:41:10,328 WARN logaggregation.LogAggregationService
(LogAggregationService.java:verifyAndCreateRemoteLogDir(195)) - Remote Root Log
Dir [/app-logs] already exist, but with incorrect permissions. Expected:
[rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple
users.
2015-04-29 00:41:10,328 WARN logaggregation.AppLogAggregatorImpl
(AppLogAggregatorImpl.java:init(182)) - rollingMonitorInterval is set as -1.
The log rolling mornitoring interval is disabled. The logs will be aggregated
after this application is finished.
2015-04-29 00:41:10,351 INFO application.Application
(ApplicationImpl.java:transition(304)) - Adding
container_1428575950531_0021_01_01 to application
application_1428575950531_0021
2015-04-29 00:41:10,352 INFO application.Application
(ApplicationImpl.java:handle(464)) - Application application_1428575950531_0021
transitioned from INITING to RUNNING
2015-04-29 00:41:10,356 INFO container.Container
(ContainerImpl.java:handle(999)) - Container
container_1428575950531_0021_01_01 transitioned from NEW to LOCALIZING
2015-04-29 00:41:10,357 INFO containermanager.AuxServices
(AuxServices.java:handle(196)) - Got event CONTAINER_INIT for appId
application_1428575950531_0021
2015-04-29 00:41:10,357 INFO localizer.LocalizedResource
(LocalizedResource.java:handle(203)) - Resource
hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/htrace-core-3.0.4.jar
transitioned from INIT to DOWNLOADING
2015-04-29 00:41:10,357 INFO localizer.LocalizedResource
(LocalizedResource.java:handle(203)) - Resource
hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jettison-1.1.jar
transitioned from INIT to DOWNLOADING
2015-04-29 00:41:10,358 INFO localizer.LocalizedResource
(LocalizedResource.java:handle(203)) - Resource
hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/api-util-1.0.0-M20.jar
transitioned from INIT to DOWNLOADING
2015-04-29 00:41:10,358 INFO localizer.LocalizedResource
(LocalizedResource.java:handle(203)) - Resource
hdfs://zsexp/user/yarn/.slider/cluster/storm1/confdir/log4j-server.properties
transitioned from INIT to DOWNLOADING
2015-04-29 00:41:10,358 INFO localizer.LocalizedResource
(LocalizedResource.java:handle(203)) - Resource
hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/httpcore-4.2.5.jar
transitioned from INIT to DOWNLOADING
2015-04-29 00:41:10,358 INFO localizer.LocalizedResource
(LocalizedResource.java:handle(203)) - Resource
[jira] [Created] (YARN-2823) NullPointerException in RM HA enabled 3-node cluster
Gour Saha created YARN-2823:
---
Summary: NullPointerException in RM HA enabled 3-node cluster
Key: YARN-2823
URL: https://issues.apache.org/jira/browse/YARN-2823
Project: Hadoop YARN
Issue Type: Bug
Reporter: Gour Saha
Branch:
2.6.0
Environment:
A 3-node cluster with RM HA enabled. The HA setup went pretty smooth (used
Ambari) and then installed HBase using Slider. After some time the RMs went
down and would not come back up anymore. Following is the NPE we see in both
the RM logs.
{noformat}
2014-09-16 01:36:28,037 FATAL resourcemanager.ResourceManager
(ResourceManager.java:run(612)) - Error in handling event type
APP_ATTEMPT_ADDED to the scheduler
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.transferStateFromPreviousAttempt(SchedulerApplicationAttempt.java:530)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplicationAttempt(CapacityScheduler.java:678)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1015)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:98)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:603)
at java.lang.Thread.run(Thread.java:744)
2014-09-16 01:36:28,042 INFO resourcemanager.ResourceManager
(ResourceManager.java:run(616)) - Exiting, bbye..
{noformat}
All the logs for this 3-node cluster has been uploaded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-2678) Recommended improvements to Yarn Registry
Gour Saha created YARN-2678:
---
Summary: Recommended improvements to Yarn Registry
Key: YARN-2678
URL: https://issues.apache.org/jira/browse/YARN-2678
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Gour Saha
In the process of binding to Slider AM from Slider agent python code here are
some of the items I stumbled upon and would recommend as improvements.
This is how the Slider's registry looks today -
{noformat}
jsonservicerec{
description : Slider Application Master,
external : [ {
api : org.apache.slider.appmaster,
addressType : host/port,
protocolType : hadoop/protobuf,
addresses : [ [ c6408.ambari.apache.org, 34837 ] ]
}, {
api : org.apache.http.UI,
addressType : uri,
protocolType : webui,
addresses : [ [ http://c6408.ambari.apache.org:43314; ] ]
}, {
api : org.apache.slider.management,
addressType : uri,
protocolType : REST,
addresses : [ [ http://c6408.ambari.apache.org:43314/ws/v1/slider/mgmt;
] ]
}, {
api : org.apache.slider.publisher,
addressType : uri,
protocolType : REST,
addresses : [ [
http://c6408.ambari.apache.org:43314/ws/v1/slider/publisher; ] ]
}, {
api : org.apache.slider.registry,
addressType : uri,
protocolType : REST,
addresses : [ [
http://c6408.ambari.apache.org:43314/ws/v1/slider/registry; ] ]
}, {
api : org.apache.slider.publisher.configurations,
addressType : uri,
protocolType : REST,
addresses : [ [
http://c6408.ambari.apache.org:43314/ws/v1/slider/publisher/slider; ] ]
} ],
internal : [ {
api : org.apache.slider.agents.secure,
addressType : uri,
protocolType : REST,
addresses : [ [
https://c6408.ambari.apache.org:46958/ws/v1/slider/agents; ] ]
}, {
api : org.apache.slider.agents.oneway,
addressType : uri,
protocolType : REST,
addresses : [ [
https://c6408.ambari.apache.org:57513/ws/v1/slider/agents; ] ]
} ],
yarn:persistence : application,
yarn:id : application_1412974695267_0015
}
{noformat}
Recommendations:
1. I would suggest to either remove the string {color:red}jsonservicerec{color}
or if it is desirable to have a non-null data at all times then loop the string
into the json structure as a top-level attribute to ensure that the registry
data is always a valid json document.
2. The {color:red}addresses{color} attribute is currently a list of list. I
would recommend to convert it to a list of dictionary objects. In the
dictionary object it would be nice to have the host and port portions of
objects of addressType uri as separate key-value pairs to avoid parsing on
the client side. The URI should also be retained as a key say uri to avoid
clients trying to generate it by concatenating host, port, resource-path, etc.
Here is a proposed structure -
{noformat}
{
...
internal : [ {
api : org.apache.slider.agents.secure,
addressType : uri,
protocolType : REST,
addresses : [
{ uri : https://c6408.ambari.apache.org:46958/ws/v1/slider/agents;,
host : c6408.ambari.apache.org,
port: 46958
}
]
}
],
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
