[
https://issues.apache.org/jira/browse/SLIDER-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209925#comment-14209925
]
ASF subversion and git services commented on SLIDER-646:
--------------------------------------------------------
Commit e3846d5c720172c9e63b377a70bff690f402b993 in incubator-slider's branch
refs/heads/develop from [~ste...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=e3846d5 ]
SLIDER-646 intermittent AgentLaunchFailureIT test failures
> AgentLaunchFailureIT test failing at times
> ------------------------------------------
>
> Key: SLIDER-646
> URL: https://issues.apache.org/jira/browse/SLIDER-646
> Project: Slider
> Issue Type: Bug
> Reporter: Gour Saha
> Assignee: Steve Loughran
>
> Chaos Monkey initial delay should be deterministic. It is currently set to 60
> seconds. Subsequent interval is also set to 60 secs. However
> AgentLaunchFailureIT fails at times because the AM does not get sufficient
> time to startup. In one failure scenario it has been seen to fail within 300
> ms of Chaos Monkey setup. This test fails about once in every 10 attempts.
> Here is the test output -
> {code}
> ------------------------------------------------------------------------------
> Test set: org.apache.slider.funtest.lifecycle.AgentLaunchFailureIT
> -------------------------------------------------------------------------------
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 43.503 sec
> <<< FAILURE! - in org.apache.slider.funtest.lifecycle.AgentLaunchFailureIT
> testAgentLaunchFailure(org.apache.slider.funtest.lifecycle.AgentLaunchFailureIT)
> Time elapsed: 40.903 sec <<< FAILURE!
> java.lang.AssertionError: Application Launch Failure, exit code 68
> Chaos monkey triggered launch failure
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.slider.funtest.framework.CommandTestBase.createTemplatedSliderApplication(CommandTestBase.groovy:676)
> at
> org.apache.slider.funtest.lifecycle.AgentLaunchFailureIT.testAgentLaunchFailure(AgentLaunchFailureIT.groovy:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
> Here is the AM log snippet -
> {code}
> 2014-11-12 09:29:34,989 [main] INFO appmaster.SliderAppMaster
> (SliderAppMaster.java:createAndRunCluster(764)) - Token YARN_AM_RM_TOKEN
> 2014-11-12 09:29:34,990 [main] INFO agent.AgentUtils
> (AgentUtils.java:getApplicationMetainfo(43)) - Reading metainfo at
> .slider/package/CMD_LOGGER/apache-slider-command-logger.zip
> 2014-11-12 09:29:35,014 [main] INFO tools.SliderUtils
> (SliderUtils.java:getApplicationResourceInputStream(1692)) - Reading
> metainfo.xml of size 1995
> 2014-11-12 09:29:35,096 [main] INFO agent.AgentUtils
> (AgentUtils.java:getDefaultConfig(64)) - Reading default config file
> configuration/cl-site.xml at
> .slider/package/CMD_LOGGER/apache-slider-command-logger.zip
> 2014-11-12 09:29:35,102 [main] INFO tools.SliderUtils
> (SliderUtils.java:getApplicationResourceInputStream(1692)) - Reading
> configuration/cl-site.xml of size 1270
> 2014-11-12 09:29:35,106 [main] INFO agent.HeartbeatMonitor
> (HeartbeatMonitor.java:start(46)) - Starting heartbeat monitor with interval
> 60000
> 2014-11-12 09:29:35,107 [Thread-36] DEBUG agent.HeartbeatMonitor
> (HeartbeatMonitor.java:run(65)) - Putting monitor to sleep for 60000
> milliseconds
> 2014-11-12 09:29:35,181 [main] INFO state.AppState
> (AppState.java:buildInstance(502)) - Adding role COMMAND_LOGGER
> 2014-11-12 09:29:35,181 [main] INFO state.AppState
> (AppState.java:createDynamicProviderRole(585)) - Role COMMAND_LOGGER assigned
> priority 1
> 2014-11-12 09:29:35,181 [main] INFO state.AppState
> (AppState.java:buildRoleRequirementsFromResources(687)) - Role COMMAND_LOGGER
> has 0 instances specified
> 2014-11-12 09:29:35,253 [main] DEBUG state.RoleHistory
> (RoleHistory.java:onBootstrap(370)) - Role history bootstrapped
> 2014-11-12 09:29:35,268 [main] INFO appmaster.SliderAppMaster
> (SliderAppMaster.java:maybeStartMonkey(2183)) - Adding Chaos Monkey scheduled
> every 60 seconds (0 hours -delay 60
> 2014-11-12 09:29:35,269 [main] INFO appmaster.SliderAppMaster
> (SliderAppMaster.java:maybeStartMonkey(2195)) - Chaos Monkey has triggered AM
> Launch failure
> 2014-11-12 09:29:35,269 [main] DEBUG actions.QueueService
> (QueueService.java:put(85)) - Queueing stop: exit code = -1, FAILED: Chaos
> monkey triggered launch failure;
> 2014-11-12 09:29:35,270 [main] DEBUG monkey.ChaosMonkeyService
> (ChaosMonkeyService.java:addTarget(66)) - Action AM killer not enabled
> 2014-11-12 09:29:35,270 [main] DEBUG monkey.ChaosMonkeyService
> (ChaosMonkeyService.java:addTarget(66)) - Action Container killer not enabled
> 2014-11-12 09:29:35,270 [main] INFO appmaster.SliderAppMaster
> (SliderAppMaster.java:maybeStartMonkey(2222)) - Chaos monkey not started
> 2014-11-12 09:29:35,271 [main] INFO appmaster.SliderAppMaster
> (SliderAppMaster.java:createAndRunCluster(879)) - HADOOP_USER_NAME='yarn'
> 2014-11-12 09:29:35,287 [main] INFO appmaster.SliderAppMaster
> (SliderAppMaster.java:createAndRunCluster(882)) - Registry service username
> =yarn
> 2014-11-12 09:29:35,308 [main] DEBUG tools.ConfigHelper
> (ConfigHelper.java:loadFromResource(511)) - loaded resources from
> file:/etc/hadoop/conf/yarn-site.xml
> 2014-11-12 09:29:35,342 [main] DEBUG tools.ConfigHelper
> (ConfigHelper.java:loadFromResource(511)) - loaded resources from
> file:/etc/hadoop/conf/core-site.xml
> 2014-11-12 09:29:35,375 [main] DEBUG tools.ConfigHelper
> (ConfigHelper.java:loadFromResource(511)) - loaded resources from
> file:/etc/hadoop/conf/hdfs-site.xml
> 2014-11-12 09:29:35,437 [main] INFO appmaster.SliderAppMaster
> (SliderAppMaster.java:registerServiceInstance(1083)) - Service Record
> ServiceRecord{description='Slider Application Master'; external endpoints: {{
> "api" : "http://";,
> "addressType" : "uri",
> "protocolType" : "webui",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091";
> } ]
> }; {
> "api" : "classpath:org.apache.slider.management",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091/ws/v1/slider/mgmt";
> } ]
> }; {
> "api" : "classpath:org.apache.slider.publisher",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher";
> } ]
> }; {
> "api" : "classpath:org.apache.slider.registry",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091/ws/v1/slider/registry";
> } ]
> }; {
> "api" : "classpath:org.apache.slider.publisher.configurations",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher/slider";
> } ]
> }; {
> "api" : "classpath:org.apache.slider.publisher.exports",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher/exports";
> } ]
> }; }; internal endpoints: {{
> "api" : "classpath:org.apache.slider.agents.secure",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "https://172.31.11.97:50007/ws/v1/slider/agents";
> } ]
> }; {
> "api" : "classpath:org.apache.slider.agents.oneway",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "https://172.31.11.97:52395/ws/v1/slider/agents";
> } ]
> }; }, attributes: {"yarn:persistence"="application"
> "yarn:id"="application_1415782602687_0008" }}
> 2014-11-12 09:29:35,473 [main] INFO zk.RegistryOperationsService
> (RegistryOperationsService.java:bind(110)) - Bound at
> /users/yarn/services/org-apache-slider/test-agent-launchfail :
> ServiceRecord{description='Slider Application Master'; external endpoints: {{
> "api" : "http://";,
> "addressType" : "uri",
> "protocolType" : "webui",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091";
> } ]
> }; {
> "api" : "classpath:org.apache.slider.management",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091/ws/v1/slider/mgmt";
> } ]
> }; {
> "api" : "classpath:org.apache.slider.publisher",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher";
> } ]
> }; {
> "api" : "classpath:org.apache.slider.registry",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091/ws/v1/slider/registry";
> } ]
> }; {
> "api" : "classpath:org.apache.slider.publisher.configurations",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher/slider";
> } ]
> }; {
> "api" : "classpath:org.apache.slider.publisher.exports",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher/exports";
> } ]
> }; }; internal endpoints: {{
> "api" : "classpath:org.apache.slider.agents.secure",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "https://172.31.11.97:50007/ws/v1/slider/agents";
> } ]
> }; {
> "api" : "classpath:org.apache.slider.agents.oneway",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "https://172.31.11.97:52395/ws/v1/slider/agents";
> } ]
> }; }, attributes: {"yarn:persistence"="application"
> "yarn:id"="application_1415782602687_0008" }}
> 2014-11-12 09:29:35,535 [main] INFO appmaster.SliderAppMaster
> (SliderAppMaster.java:registerServiceInstance(1085)) - Registered service
> under /users/yarn/services/org-apache-slider/test-agent-launchfail; absolute
> path /registry/users/yarn/services/org-apache-slider/test-agent-launchfail
> 2014-11-12 09:29:35,543 [main] INFO appmaster.SliderAppMaster
> (SliderAppMaster.java:createAndRunCluster(888)) - RM Webapp address
> 172.31.11.99:8088
> 2014-11-12 09:29:35,543 [main] INFO appmaster.SliderAppMaster
> (SliderAppMaster.java:createAndRunCluster(889)) - slider Webapp address
> http://172.31.11.97:37091
> 2014-11-12 09:29:35,543 [main] INFO appmaster.SliderAppMaster
> (SliderAppMaster.java:createAndRunCluster(892)) - Application Master
> Initialization Completed
> 2014-11-12 09:29:35,543 [main] INFO appmaster.SliderAppMaster
> (SliderAppMaster.java:startQueueProcessing(463)) - Queue Processing started
> 2014-11-12 09:29:35,544 [AmExecutor-005] INFO actions.QueueService
> (QueueService.java:run(171)) - QueueService processor started
> 2014-11-12 09:29:35,545 [AmExecutor-006] INFO actions.QueueExecutor
> (QueueExecutor.java:run(68)) - Queue Executor run() started
> 2014-11-12 09:29:35,545 [AmExecutor-006] DEBUG actions.QueueExecutor
> (QueueExecutor.java:run(71)) - Executing stop: exit code = -1, FAILED: Chaos
> monkey triggered launch failure;
> 2014-11-12 09:29:35,545 [AmExecutor-006] INFO appmaster.SliderAppMaster
> (ActionStopSlider.java:execute(118)) - SliderAppMasterApi.stopCluster: Chaos
> monkey triggered launch failure
> 2014-11-12 09:29:35,619 [main] DEBUG agent.AgentClientProvider
> (AgentClientProvider.java:validateInstanceDefinition(117)) - Validating conf
> {,
> "internal": {
> "schema" : "http://example.org/specification/v2.0.0";,
> "metadata" : {
> "create.hadoop.deployed.info" : "(no branch)
> @ae493e14a5f8a78bd6227e6d377bbef6",
> "create.application.build.info" : "Slider Core-0.60.0.2.2.0.0-1947 Built
> against commit# ${buildNumber} on Java 1.7.0_67 by yarn",
> "create.hadoop.build.info" : "2.6.0.2.2.0.0-1947",
> "create.time.millis" : "1415784562671",
> "create.time" : "12 Nov 2014 09:29:22 GMT"
> },
> "global" : {
> "internal.generated.conf.path" :
> "hdfs://172.31.11.99:8020/user/yarn/.slider/cluster/test-agent-launchfail/generated",
> "application.name" : "test-agent-launchfail",
> "slider.cluster.directory.permissions" : "0770",
> "internal.provider.name" : "agent",
> "internal.data.dir.path" :
> "hdfs://172.31.11.99:8020/user/yarn/.slider/cluster/test-agent-launchfail/database",
> "internal.tmp.dir" :
> "hdfs://172.31.11.99:8020/user/yarn/.slider/cluster/test-agent-launchfail/tmp",
> "internal.chaos.monkey.probability.amlaunchfailure" : "10000",
> "internal.snapshot.conf.path" :
> "hdfs://172.31.11.99:8020/user/yarn/.slider/cluster/test-agent-launchfail/snapshot",
> "internal.chaos.monkey.interval.seconds" : "60",
> "slider.data.directory.permissions" : "0770",
> "internal.container.failure.shortlife" : "60000",
> "internal.chaos.monkey.enabled" : "true",
> "internal.am.tmp.dir" :
> "hdfs://172.31.11.99:8020/user/yarn/.slider/cluster/test-agent-launchfail/tmp/appmaster",
> "internal.container.failure.threshold" : "5"
> },
> "credentials" : { },
> "components" : { }
> },
> "resources": {
> "schema" : "http://example.org/specification/v2.0.0";,
> "metadata" : { },
> "global" : { },
> "credentials" : { },
> "components" : {
> "slider-appmaster" : {
> "yarn.memory" : "256",
> "yarn.vcores" : "1",
> "yarn.component.instances" : "1"
> },
> "COMMAND_LOGGER" : {
> "yarn.memory" : "128",
> "yarn.role.priority" : "1",
> "yarn.component.instances" : "0"
> }
> }
> },
> "appConf" :{
> "schema" : "http://example.org/specification/v2.0.0";,
> "metadata" : { },
> "global" : {
> "site.dfs.namenode.kerberos.principal" : "nn/_h...@example.com",
> "site.fs.default.name" : "hdfs://172.31.11.99:8020",
> "site.cl-site.pattern.for.test.to.verify" : "verify this pattern",
> "site.cl-site.logfile.location" : "${AGENT_LOG_ROOT}/operations.log",
> "zookeeper.hosts" : "172.31.11.100,172.31.11.96,172.31.11.97",
> "java_home" : "/usr/jdk64/jdk1.7.0_67",
> "site.global.application_id" : "CommandLogger",
> "internal.chaos.monkey.probability.amlaunchfailure" : "10000",
> "site.fs.defaultFS" : "hdfs://172.31.11.99:8020",
> "env.MALLOC_ARENA_MAX" : "4",
> "zookeeper.path" : "/services/slider/users/yarn/test-agent-launchfail",
> "internal.chaos.monkey.interval.seconds" : "60",
> "internal.chaos.monkey.enabled" : "true",
> "zookeeper.quorum" : "172.31.11.100,172.31.11.96,172.31.11.97",
> "site.global.app_root" : "${AGENT_WORK_ROOT}/app/install/command-logger",
> "application.def" :
> ".slider/package/CMD_LOGGER/apache-slider-command-logger.zip",
> "site.cl-site.datetime.format" : "%A, %d. %B %Y %I:%M%p",
> "site.global.security_enabled" : "false"
> },
> "credentials" : { },
> "components" : {
> "slider-appmaster" : {
> "jvm.heapsize" : "256M",
> "site.dfs.namenode.kerberos.principal" : "nn/_h...@example.com",
> "site.fs.default.name" : "hdfs://172.31.11.99:8020",
> "site.cl-site.pattern.for.test.to.verify" : "verify this pattern",
> "site.cl-site.logfile.location" : "${AGENT_LOG_ROOT}/operations.log",
> "zookeeper.hosts" : "172.31.11.100,172.31.11.96,172.31.11.97",
> "java_home" : "/usr/jdk64/jdk1.7.0_67",
> "site.global.application_id" : "CommandLogger",
> "internal.chaos.monkey.probability.amlaunchfailure" : "10000",
> "site.fs.defaultFS" : "hdfs://172.31.11.99:8020",
> "env.MALLOC_ARENA_MAX" : "4",
> "zookeeper.path" : "/services/slider/users/yarn/test-agent-launchfail",
> "internal.chaos.monkey.interval.seconds" : "60",
> "internal.chaos.monkey.enabled" : "true",
> "zookeeper.quorum" : "172.31.11.100,172.31.11.96,172.31.11.97",
> "site.global.app_root" :
> "${AGENT_WORK_ROOT}/app/install/command-logger",
> "application.def" :
> ".slider/package/CMD_LOGGER/apache-slider-command-logger.zip",
> "site.cl-site.datetime.format" : "%A, %d. %B %Y %I:%M%p",
> "site.global.security_enabled" : "false"
> },
> "COMMAND_LOGGER" : {
> "site.dfs.namenode.kerberos.principal" : "nn/_h...@example.com",
> "site.fs.default.name" : "hdfs://172.31.11.99:8020",
> "site.cl-site.pattern.for.test.to.verify" : "verify this pattern",
> "site.cl-site.logfile.location" : "${AGENT_LOG_ROOT}/operations.log",
> "zookeeper.hosts" : "172.31.11.100,172.31.11.96,172.31.11.97",
> "java_home" : "/usr/jdk64/jdk1.7.0_67",
> "site.global.application_id" : "CommandLogger",
> "internal.chaos.monkey.probability.amlaunchfailure" : "10000",
> "site.fs.defaultFS" : "hdfs://172.31.11.99:8020",
> "env.MALLOC_ARENA_MAX" : "4",
> "zookeeper.path" : "/services/slider/users/yarn/test-agent-launchfail",
> "internal.chaos.monkey.interval.seconds" : "60",
> "internal.chaos.monkey.enabled" : "true",
> "zookeeper.quorum" : "172.31.11.100,172.31.11.96,172.31.11.97",
> "site.global.app_root" :
> "${AGENT_WORK_ROOT}/app/install/command-logger",
> "application.def" :
> ".slider/package/CMD_LOGGER/apache-slider-command-logger.zip",
> "site.cl-site.datetime.format" : "%A, %d. %B %Y %I:%M%p",
> "site.global.security_enabled" : "false"
> }
> }
> }}
> 2014-11-12 09:29:35,619 [main] INFO agent.AgentClientProvider
> (AgentClientProvider.java:validateInstanceDefinition(133)) - Validating app
> definition .slider/package/CMD_LOGGER/apache-slider-command-logger.zip
> 2014-11-12 09:29:35,620 [main] DEBUG state.AppState
> (AppState.java:updateResourceDefinitions(649)) - Updating resources to {
> "schema" : "http://example.org/specification/v2.0.0";,
> "metadata" : { },
> "global" : { },
> "credentials" : { },
> "components" : {
> "slider-appmaster" : {
> "yarn.memory" : "256",
> "yarn.vcores" : "1",
> "yarn.component.instances" : "1"
> },
> "COMMAND_LOGGER" : {
> "yarn.memory" : "128",
> "yarn.role.priority" : "1",
> "yarn.component.instances" : "0"
> }
> }
> }
> 2014-11-12 09:29:35,704 [main] INFO state.AppState
> (AppState.java:buildRoleRequirementsFromResources(687)) - Role COMMAND_LOGGER
> has 0 instances specified
> 2014-11-12 09:29:35,704 [main] DEBUG state.AppState
> (AppState.java:resetFailureCounts(1657)) - Resetting failure count of
> slider-appmaster; was 0
> 2014-11-12 09:29:35,704 [main] DEBUG state.AppState
> (AppState.java:resetFailureCounts(1657)) - Resetting failure count of
> COMMAND_LOGGER; was 0
> 2014-11-12 09:29:35,704 [main] DEBUG appmaster.SliderAppMaster
> (SliderAppMaster.java:reviewRequestAndReleaseNodes(1524)) -
> reviewRequestAndReleaseNodes(flexCluster)
> 2014-11-12 09:29:35,705 [main] DEBUG actions.QueueService
> (QueueService.java:put(85)) - Queueing
> org.apache.slider.server.appmaster.actions.ReviewAndFlexApplicationSize@27f143e9
> name='flexCluster', delay=0, attrs=4, sequenceNumber=2}
> 2014-11-12 09:29:35,705 [main] DEBUG appmaster.SliderAppMaster
> (SliderAppMaster.java:waitForAMCompletionSignal(1243)) - blocking until
> signalled to terminate
> 2014-11-12 09:29:35,705 [AmExecutor-006] DEBUG actions.QueueExecutor
> (QueueExecutor.java:run(74)) - Completed stop: exit code = -1, FAILED: Chaos
> monkey triggered launch failure;
> 2014-11-12 09:29:35,705 [main] INFO appmaster.SliderAppMaster
> (SliderAppMaster.java:finish(1288)) - Triggering shutdown of the AM: stop:
> exit code = -1, FAILED: Chaos monkey triggered launch failure;
> 2014-11-12 09:29:35,706 [AmExecutor-006] DEBUG actions.QueueExecutor
> (QueueExecutor.java:run(71)) - Executing
> org.apache.slider.server.appmaster.actions.ReviewAndFlexApplicationSize@27f143e9
> name='flexCluster', delay=0, attrs=4, sequenceNumber=2}
> 2014-11-12 09:29:35,706 [main] INFO appmaster.SliderAppMaster
> (SliderAppMaster.java:stateChanged(1963)) - Process has exited with exit code
> 0 mapped to 0 -ignoring
> 2014-11-12 09:29:35,706 [main] DEBUG appmaster.SliderAppMaster
> (SliderAppMaster.java:finish(1299)) - Stopped forked process: exit code=0
> 2014-11-12 09:29:35,707 [main] INFO workflow.WorkflowCompositeService
> (WorkflowCompositeService.java:stateChanged(123)) - Child service completed
> Service RoleLaunchService in state RoleLaunchService: STOPPED
> 2014-11-12 09:29:35,707 [main] INFO state.AppState
> (AppState.java:releaseAllContainers(1843)) - Releasing 0 containers
> 2014-11-12 09:29:35,707 [main] INFO appmaster.SliderAppMaster
> (SliderAppMaster.java:finish(1317)) - Application completed. Signalling
> finish to RM
> 2014-11-12 09:29:35,707 [main] INFO appmaster.SliderAppMaster
> (SliderAppMaster.java:finish(1320)) - Unregistering AM status=FAILED
> message=Chaos monkey triggered launch failure
> 2014-11-12 09:29:35,716 [main] INFO impl.AMRMClientImpl
> (AMRMClientImpl.java:unregisterApplicationMaster(383)) - Waiting for
> application to be successfully unregistered.
> 2014-11-12 09:29:35,818 [main] INFO appmaster.SliderAppMaster
> (SliderAppMaster.java:runService(529)) - Exiting AM; final exit code = 0
> 2014-11-12 09:29:35,818 [AmExecutor-006] DEBUG appmaster.SliderAppMaster
> (SliderAppMaster.java:executeNodeReview(1559)) - in
> executeNodeReview(flexCluster)
> 2014-11-12 09:29:35,818 [main] DEBUG main.ServiceLauncher
> (ServiceLauncher.java:launchService(189)) - Service exited with exit code 0
> 2014-11-12 09:29:35,818 [AmExecutor-006] INFO appmaster.SliderAppMaster
> (SliderAppMaster.java:executeNodeReview(1561)) - Ignoring node review
> operation: shutdown in progress
> 2014-11-12 09:29:35,818 [AmExecutor-006] DEBUG state.AppState
> (AppState.java:reviewRequestAndReleaseNodes(1599)) - in
> reviewRequestAndReleaseNodes()
> 2014-11-12 09:29:35,818 [AmExecutor-006] INFO state.AppState
> (AppState.java:reviewOneRole(1684)) - Reviewing
> RoleStatus{name='COMMAND_LOGGER', key=1, minimum=0, maximum=1, desired=0,
> actual=0, requested=0, releasing=0, failed=0, started=0, startFailed=0,
> completed=0, failureMessage=''} : expected 0
> 2014-11-12 09:29:35,819 [AmExecutor-006] DEBUG state.AppState
> (AppState.java:checkFailureThreshold(1620)) - Failure count of component:
> COMMAND_LOGGER: 0, threshold=5
> 2014-11-12 09:29:35,819 [AmExecutor-006] DEBUG actions.QueueExecutor
> (QueueExecutor.java:run(74)) - Completed
> org.apache.slider.server.appmaster.actions.ReviewAndFlexApplicationSize@27f143e9
> name='flexCluster', delay=0, attrs=4, sequenceNumber=2}
> 2014-11-12 09:29:35,821 [main] INFO util.ExitUtil
> (ExitUtil.java:terminate(124)) - Exiting with status 0
> 2014-11-12 09:29:35,821 [Shutdown] INFO mortbay.log (Slf4jLog.java:info(67))
> - Shutdown hook executing
> 2014-11-12 09:29:35,822 [Shutdown] INFO mortbay.log (Slf4jLog.java:info(67))
> - Stopped SslSelectChannelConnector@0.0.0.0:50007
> 2014-11-12 09:29:35,825 [Shutdown] INFO mortbay.log (Slf4jLog.java:info(67))
> - Stopped SslSelectChannelConnector@0.0.0.0:52395
> 2014-11-12 09:29:35,829 [Shutdown] INFO mortbay.log (Slf4jLog.java:info(67))
> - Shutdown hook complete
> 2014-11-12 09:29:35,834 [Thread-1] INFO mortbay.log (Slf4jLog.java:info(67))
> - Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:0
> 2014-11-12 09:29:36,037 [Thread-1] INFO ipc.Server (Server.java:stop(2437))
> - Stopping server on 40684
> 2014-11-12 09:29:36,039 [IPC Server listener on 40684] INFO ipc.Server
> (Server.java:run(706)) - Stopping IPC Server listener on 40684
> 2014-11-12 09:29:36,039 [IPC Server Responder] INFO ipc.Server
> (Server.java:run(832)) - Stopping IPC Server Responder
> 2014-11-12 09:29:36,040 [Thread-1] DEBUG actions.QueueService
> (QueueService.java:schedule(91)) - Scheduling
> org.apache.slider.server.appmaster.actions.ActionStopQueue@9731632
> name='serviceStop: Service Action Queue in state Action Queue: STOPPED',
> delay=0, attrs=0, sequenceNumber=3}
> 2014-11-12 09:29:36,040 [AMRM Callback Handler Thread] INFO
> impl.AMRMClientAsyncImpl (AMRMClientAsyncImpl.java:run(276)) - Interrupted
> while waiting for queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274)
> 2014-11-12 09:29:36,042 [AmExecutor-005] DEBUG actions.QueueService
> (QueueService.java:run(176)) - Propagating
> org.apache.slider.server.appmaster.actions.ActionStopQueue@9731632
> name='serviceStop: Service Action Queue in state Action Queue: STOPPED',
> delay=0, attrs=0, sequenceNumber=3}
> 2014-11-12 09:29:36,042 [AmExecutor-005] INFO actions.QueueService
> (QueueService.java:run(179)) - QueueService processor terminated
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
