[
https://issues.apache.org/jira/browse/AMBARI-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitry Lysnichenko updated AMBARI-17859:
----------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed
To https://git-wip-us.apache.org/repos/asf/ambari.git
de78fb4..8dcc37e branch-2.4 -> branch-2.4
0fb7885..33c10ef trunk -> trunk
> YARN service check failed during EU from HDP-2.4.0.0 to Erie
> ------------------------------------------------------------
>
> Key: AMBARI-17859
> URL: https://issues.apache.org/jira/browse/AMBARI-17859
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Affects Versions: 2.4.0
> Reporter: Dmitry Lysnichenko
> Assignee: Dmitry Lysnichenko
> Fix For: 2.4.0
>
> Attachments: AMBARI-17859.patch
>
>
> *Steps*
> # Deploy HDP-2.4.0.0 cluster with Ambari 2.2.1.1 (secure, non-HA cluster,
> customized service users)
> # Upgrade Ambari to 2.4.0.0
> # Perform EU to 2.5.0.0-934
> *Result*
> During EU, observed YARN service check reported below errors:
> {code}
> Traceback (most recent call last):\n File
> \"/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py\",
> line 159, in <module>\n ServiceCheck().execute()\n File
> \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\",
> line 280, in execute\n method(env)\n File
> \"/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py\",
> line 117, in service_check\n user=params.smokeuser,\n File
> \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line
> 71, in inner\n result = function(command, **kwargs)\n File
> \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line
> 93, in checked_call\n tries=tries, try_sleep=try_sleep)\n File
> \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line
> 141, in _call_wrapper\n result = _call(command, **kwargs_copy)\n File
> \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line
> 294, in _call\n raise
> Fail(err_msg)\nresource_management.core.exceptions.Fail: Execution of
> '/usr/bin/kinit -kt /etc/security/keytabs/smokeuser.headless.keytab
> smk_rndalxit7oqf2pq3erlqg6r...@hwqe.hortonworks.com; yarn
> org.apache.hadoop.yarn.applications.distributedshell.Client -shell_command ls
> -num_containers 1 -jar
> /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar
> -timeout 300000 --queue default' returned 2. ######## Hortonworks
> #############\nThis is MOTD message, added for testing in qe infra\n16/07/09
> 11:15:01 INFO impl.TimelineClientImpl: Timeline service address:
> http://host:8188/ws/v1/timeline/\n16/07/09 11:15:01 INFO
> distributedshell.Client: Initializing Client\n16/07/09 11:15:01 INFO
> distributedshell.Client: Running Client\n16/07/09 11:15:01 INFO
> client.RMProxy: Connecting to ResourceManager at
> host-5.domainlocal/10.0.113.157:8050\n16/07/09 11:15:03 INFO
> distributedshell.Client: Got Cluster metric info from ASM,
> numNodeManagers=3\n16/07/09 11:15:03 INFO distributedshell.Client: Got
> Cluster node info from ASM\n16/07/09 11:15:03 INFO distributedshell.Client:
> Got node report from ASM for, nodeId=host:25454, nodeAddresshost:8042,
> nodeRackName/default-rack, nodeNumContainers0\n16/07/09 11:15:03 INFO
> distributedshell.Client: Got node report from ASM for,
> nodeId=host-5.domainlocal:25454, nodeAddresshost-5.domainlocal:8042,
> nodeRackName/default-rack, nodeNumContainers0\n16/07/09 11:15:03 INFO
> distributedshell.Client: Got node report from ASM for,
> nodeId=host-1.domainlocal:25454, nodeAddresshost-1.domainlocal:8042,
> nodeRackName/default-rack, nodeNumContainers0\n16/07/09 11:15:03 INFO
> distributedshell.Client: Queue info, queueName=default,
> queueCurrentCapacity=0.0, queueMaxCapacity=1.0, queueApplicationCount=0,
> queueChildQueueCount=0\n16/07/09 11:15:04 INFO distributedshell.Client: User
> ACL Info for Queue, queueName=root, userAcl=SUBMIT_APPLICATIONS\n16/07/09
> 11:15:04 INFO distributedshell.Client: User ACL Info for Queue,
> queueName=default, userAcl=SUBMIT_APPLICATIONS\n16/07/09 11:15:04 INFO
> distributedshell.Client: Max mem capability of resources in this cluster
> 10240\n16/07/09 11:15:04 INFO distributedshell.Client: Max virtual cores
> capabililty of resources in this cluster 1\n16/07/09 11:15:04 INFO
> distributedshell.Client: Copy App Master jar from local filesystem and add to
> local environment\n16/07/09 11:15:04 INFO distributedshell.Client: Set the
> environment for the application master\n16/07/09 11:15:04 INFO
> distributedshell.Client: Setting up app master command\n16/07/09 11:15:04
> INFO distributedshell.Client: Completed setting up app master command
> {{JAVA_HOME}}/bin/java -Xmx10m
> org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster
> --container_memory 10 --container_vcores 1 --num_containers 1 --priority 0
> 1><LOG_DIR>/AppMaster.stdout 2><LOG_DIR>/AppMaster.stderr \n16/07/09 11:15:04
> INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 279 for ambari-qa on
> 10.0.113.145:8020\n16/07/09 11:15:04 INFO distributedshell.Client: Got dt for
> hdfs://host-1.domainlocal:8020; Kind: HDFS_DELEGATION_TOKEN, Service:
> 10.0.113.145:8020, Ident: (HDFS_DELEGATION_TOKEN token 279 for
> ambari-qa)\n16/07/09 11:15:04 INFO distributedshell.Client: Submitting
> application to ASM\n16/07/09 11:15:05 INFO impl.YarnClientImpl: Submitted
> application application_1468062580457_0001\n16/07/09 11:15:06 INFO
> distributedshell.Client: Got application report from ASM for, appId=1,
> clientToAMToken=Token { kind: YARN_CLIENT_TOKEN, service: },
> appDiagnostics=AM container is launched, waiting for AM container to Register
> with RM, appMasterHost=N/A, appQueue=default, appMasterRpcPort=-1,
> appStartTime=1468062904947, yarnAppState=ACCEPTED,
> distributedFinalState=UNDEFINED,
> appTrackingUrl=http://host-5.domainlocal:8088/proxy/application_1468062580457_0001/,
> appUser=ambari-qa\n16/07/09 11:15:07 INFO distributedshell.Client: Got
> application report from ASM for, appId=1, clientToAMToken=Token { kind:
> YARN_CLIENT_TOKEN, service: }, appDiagnostics=AM container is launched,
> waiting for AM container to Register with RM, appMasterHost=N/A,
> appQueue=default, appMasterRpcPort=-1, appStartTime=1468062904947,
> yarnAppState=ACCEPTED, distributedFinalState=UNDEFINED,
> appTrackingUrl=http://host-5.domainlocal:8088/proxy/application_1468062580457_0001/,
> appUser=ambari-qa\n16/07/09 11:15:08 INFO distributedshell.Client: Got
> application report from ASM for, appId=1, clientToAMToken=Token { kind:
> YARN_CLIENT_TOKEN, service: }, appDiagnostics=AM container is launched,
> waiting for AM container to Register with RM, appMasterHost=N/A,
> appQueue=default, appMasterRpcPort=-1, appStartTime=1468062904947,
> yarnAppState=ACCEPTED, distributedFinalState=UNDEFINED,
> appTrackingUrl=http://host-5.domainlocal:8088/proxy/application_1468062580457_0001/,
> appUser=ambari-qa\n16/07/09 11:15:09 INFO distributedshell.Client: Got
> application report from ASM for, appId=1, clientToAMToken=Token { kind:
> YARN_CLIENT_TOKEN, service: }, appDiagnostics=AM container is launched,
> waiting for AM container to Register with RM, appMasterHost=N/A,
> appQueue=default, appMasterRpcPort=-1, appStartTime=1468062904947,
> yarnAppState=ACCEPTED, distributedFinalState=UNDEFINED,
> appTrackingUrl=http://host-5.domainlocal:8088/proxy/application_1468062580457_0001/,
> appUser=ambari-qa\n16/07/09 11:15:10 INFO distributedshell.Client: Got
> application report from ASM for, appId=1, clientToAMToken=Token { kind:
> YARN_CLIENT_TOKEN, service: }, appDiagnostics=AM container is launched,
> waiting for AM container to Register with RM, appMasterHost=N/A,
> appQueue=default, appMasterRpcPort=-1, appStartTime=1468062904947,
> yarnAppState=ACCEPTED, distributedFinalState=UNDEFINED,
> appTrackingUrl=http://host-5.domainlocal:8088/proxy/application_1468062580457_0001/,
> appUser=ambari-qa\n16/07/09 11:15:11 INFO distributedshell.Client: Got
> application report from ASM for, appId=1, clientToAMToken=Token { kind:
> YARN_CLIENT_TOKEN, service: }, appDiagnostics=AM container is launched,
> waiting for AM container to Register with RM, appMasterHost=N/A,
> appQueue=default, appMasterRpcPort=-1, appStartTime=1468062904947,
> yarnAppState=ACCEPTED, distributedFinalState=UNDEFINED,
> appTrackingUrl=http://host-5.domainlocal:8088/proxy/application_1468062580457_0001/,
> appUser=ambari-qa\n16/07/09 11:15:12 INFO distributedshell.Client: Got
> application report from ASM for, appId=1, clientToAMToken=Token { kind:
> YARN_CLIENT_TOKEN, service: }, appDiagnostics=AM container is launched,
> waiting for AM container to Register with RM, appMasterHost=N/A,
> appQueue=default, appMasterRpcPort=-1, appStartTime=1468062904947,
> yarnAppState=ACCEPTED, distributedFinalState=UNDEFINED,
> appTrackingUrl=http://host-5.domainlocal:8088/proxy/application_1468062580457_0001/,
> appUser=ambari-qa\n16/07/09 11:15:13 INFO distributedshell.Client: Got
> application report from ASM for, appId=1, clientToAMToken=Token { kind:
> YARN_CLIENT_TOKEN, service: }, appDiagnostics=AM container is launched,
> waiting for AM container to Register with RM, appMasterHost=N/A,
> appQueue=default, appMasterRpcPort=-1, appStartTime=1468062904947,
> yarnAppState=ACCEPTED, distributedFinalState=UNDEFINED,
> appTrackingUrl=http://host-5.domainlocal:8088/proxy/application_1468062580457_0001/,
> appUser=ambari-qa\n16/07/09 11:15:14 INFO distributedshell.Client: Got
> application report from ASM for, appId=1, clientToAMToken=Token { kind:
> YARN_CLIENT_TOKEN, service: }, appDiagnostics=AM container is launched,
> waiting for AM container to Register with RM, appMasterHost=N/A,
> appQueue=default, appMasterRpcPort=-1, appStartTime=1468062904947,
> yarnAppState=ACCEPTED, distributedFinalState=UNDEFINED,
> appTrackingUrl=http://host-5.domainlocal:8088/proxy/application_1468062580457_0001/,
> appUser=ambari-qa\n16/07/09 11:15:15 INFO distributedshell.Client: Got
> application report from ASM for, appId=1, clientToAMToken=Token { kind:
> YARN_CLIENT_TOKEN, service: }, appDiagnostics=,
> appMasterHost=host-1/10.0.113.145, appQueue=default, appMasterRpcPort=-1,
> appStartTime=1468062904947, yarnAppState=RUNNING,
> distributedFinalState=UNDEFINED,
> appTrackingUrl=http://host-5.domainlocal:8088/proxy/application_1468062580457_0001/,
> appUser=ambari-qa\n16/07/09 11:15:16 INFO distributedshell.Client: Got
> application report from ASM for, appId=1, clientToAMToken=Token { kind:
> YARN_CLIENT_TOKEN, service: }, appDiagnostics=,
> appMasterHost=host-1/10.0.113.145, appQueue=default, appMasterRpcPort=-1,
> appStartTime=1468062904947, yarnAppState=RUNNING,
> distributedFinalState=UNDEFINED,
> appTrackingUrl=http://host-5.domainlocal:8088/proxy/application_1468062580457_0001/,
> appUser=ambari-qa\n16/07/09 11:15:17 INFO distributedshell.Client: Got
> application report from ASM for, appId=1, clientToAMToken=Token { kind:
> YARN_CLIENT_TOKEN, service: }, appDiagnostics=,
> appMasterHost=host-1/10.0.113.145, appQueue=default, appMasterRpcPort=-1,
> appStartTime=1468062904947, yarnAppState=RUNNING,
> distributedFinalState=UNDEFINED,
> appTrackingUrl=http://host-5.domainlocal:8088/proxy/application_1468062580457_0001/,
> appUser=ambari-qa\n16/07/09 11:15:18 INFO distributedshell.Client: Got
> application report from ASM for, appId=1, clientToAMToken=Token { kind:
> YARN_CLIENT_TOKEN, service: }, appDiagnostics=,
> appMasterHost=host-1/10.0.113.145, appQueue=default, appMasterRpcPort=-1,
> appStartTime=1468062904947, yarnAppState=RUNNING,
> distributedFinalState=UNDEFINED,
> appTrackingUrl=http://host-5.domainlocal:8088/proxy/application_1468062580457_0001/,
> appUser=ambari-qa\n16/07/09 11:15:19 INFO distributedshell.Client: Got
> application report from ASM for, appId=1, clientToAMToken=null,
> appDiagnostics=Diagnostics., total=1, completed=1, allocated=1, failed=1,
> appMasterHost=host-1/10.0.113.145, appQueue=default, appMasterRpcPort=-1,
> appStartTime=1468062904947, yarnAppState=FINISHED,
> distributedFinalState=FAILED,
> appTrackingUrl=http://host-5.domainlocal:8088/proxy/application_1468062580457_0001/,
> appUser=ambari-qa\n16/07/09 11:15:19 INFO distributedshell.Client:
> Application did finished unsuccessfully. YarnState=FINISHED,
> DSFinalStatus=FAILED. Breaking monitoring loop\n16/07/09 11:15:19 ERROR
> distributedshell.Client: Application failed to complete successfully"
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
