[ https://issues.apache.org/jira/browse/MAPREDUCE-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081754#comment-16081754 ]
翟玉勇 edited comment on MAPREDUCE-6863 at 7/11/17 6:38 AM: --------------------------------------------------------- {code} 2017-07-11 04:14:54,964 ERROR org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Can't handle event REGISTERED when appattempt_1499504190019_207162_000001 is ALLOCATED 2017-07-11 04:14:55,507 ERROR org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Can't handle event STATUS_UPDATE when appattempt_1499504190019_207162_000001 is ALLOCATED 2017-07-11 04:14:56,050 ERROR org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Can't handle event STATUS_UPDATE when appattempt_1499504190019_207162_000001 is ALLOCATED 2017-07-11 04:14:56,558 ERROR org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Can't handle event STATUS_UPDATE when appattempt_1499504190019_207162_000001 is ALLOCATED 2017-07-11 04:14:57,064 ERROR org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Can't handle event STATUS_UPDATE when appattempt_1499504190019_207162_000001 is ALLOCATED 2017-07-11 04:14:57,569 ERROR org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Can't handle event STATUS_UPDATE when appattempt_1499504190019_207162_000001 is ALLOCATED 2017-07-11 04:15:00,530 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Done launching container Container: [ContainerId: container_1499504190019_207162_01_000001, NodeId: sh-hadoop-datanode-139-21.elenet.me:21491, NodeHttpAddress: sh-hadoop-datanode-139-21.elenet.me:8042, Resource: <memory:1536, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.0.139.21:21491 }, ] for AM appattempt_1499504190019_207162_000001 2017-07-11 04:15:00,530 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1499504190019_207162_000001 State change from ALLOCATED to LAUNCHED 2017-07-11 04:15:00,575 ERROR org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Can't handle event STATUS_UPDATE when appattempt_1499504190019_207162_000001 is LAUNCHED 2017-07-11 04:15:00,755 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1499504190019_207162_01_000002 Container Transitioned from NEW to ALLOCATED {code} was (Author: zhaiyuyong): <code> 2017-07-11 04:14:54,964 ERROR org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Can't handle event REGISTERED when appattempt_1499504190019_207162_000001 is ALLOCATED 2017-07-11 04:14:55,507 ERROR org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Can't handle event STATUS_UPDATE when appattempt_1499504190019_207162_000001 is ALLOCATED 2017-07-11 04:14:56,050 ERROR org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Can't handle event STATUS_UPDATE when appattempt_1499504190019_207162_000001 is ALLOCATED 2017-07-11 04:14:56,558 ERROR org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Can't handle event STATUS_UPDATE when appattempt_1499504190019_207162_000001 is ALLOCATED 2017-07-11 04:14:57,064 ERROR org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Can't handle event STATUS_UPDATE when appattempt_1499504190019_207162_000001 is ALLOCATED 2017-07-11 04:14:57,569 ERROR org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Can't handle event STATUS_UPDATE when appattempt_1499504190019_207162_000001 is ALLOCATED 2017-07-11 04:15:00,530 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Done launching container Container: [ContainerId: container_1499504190019_207162_01_000001, NodeId: sh-hadoop-datanode-139-21.elenet.me:21491, NodeHttpAddress: sh-hadoop-datanode-139-21.elenet.me:8042, Resource: <memory:1536, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.0.139.21:21491 }, ] for AM appattempt_1499504190019_207162_000001 2017-07-11 04:15:00,530 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1499504190019_207162_000001 State change from ALLOCATED to LAUNCHED 2017-07-11 04:15:00,575 ERROR org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Can't handle event STATUS_UPDATE when appattempt_1499504190019_207162_000001 is LAUNCHED 2017-07-11 04:15:00,755 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1499504190019_207162_01_000002 Container Transitioned from NEW to ALLOCATED </code> > job finish but yarn list status is accepted and applicationmaster is hang > on Waiting for application to be successfully unregistered > -------------------------------------------------------------------------------------------------------------------------------------- > > Key: MAPREDUCE-6863 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6863 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, resourcemanager > Affects Versions: 2.6.0 > Reporter: 翟玉勇 > Priority: Minor > Attachments: jobhistory status.png, yarn resourcemanager job list > status.png > > > applicationmaster process log is loop on “Waiting for application to be > successfully unregistered.” > ApplicationMaster log > 2017-03-12 01:16:50,854 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: > job_1489067586592_112212Job Transitioned from RUNNING to COMMITTING > 2017-03-12 01:16:50,854 INFO [CommitterEvent Processor #2] > org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing > the event EventType: JOB_COMMIT > 2017-03-12 01:16:50,884 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for > JobFinishedEvent > 2017-03-12 01:16:50,884 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: > job_1489067586592_112212Job Transitioned from COMMITTING to SUCCEEDED > 2017-03-12 01:16:50,885 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so > this is the last retry > 2017-03-12 01:16:50,885 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator > isAMLastRetry: true > 2017-03-12 01:16:50,885 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator > notified that shouldUnregistered is: true > 2017-03-12 01:16:50,885 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: > true > 2017-03-12 01:16:50,885 INFO [Thread-402] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: > JobHistoryEventHandler notified that forceJobCompletion is true > 2017-03-12 01:16:50,885 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the > services > 2017-03-12 01:16:50,886 INFO [Thread-402] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping > JobHistoryEventHandler. Size of the outstanding queue size is 0 > 2017-03-12 01:16:50,959 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before > Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 > AssignedReds:0 CompletedMaps:260 CompletedReds:0 ContAlloc:261 ContRel:0 > HostLocal:115 RackLocal:146 > 2017-03-12 01:16:51,212 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying > hdfs://bipcluster:8020/data/yarn/stage/master/.staging/job_1489067586592_112212/job_1489067586592_112212_1.jhist > to > hdfs://bipcluster:8020/data/yarn/intermediate_done/master/job_1489067586592_112212-1489252563933-master-3934_7823976%3Adw_log_app_page_event_hour_inc.sql%3As4-1489252610863-260-0-SUCCEEDED-root.bigdata.etl.hourlyetl.veryhigh-1489252568534.jhist_tmp > 2017-03-12 01:16:51,255 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done > location: > hdfs://bipcluster:8020/data/yarn/intermediate_done/master/job_1489067586592_112212-1489252563933-master-3934_7823976%3Adw_log_app_page_event_hour_inc.sql%3As4-1489252610863-260-0-SUCCEEDED-root.bigdata.etl.hourlyetl.veryhigh-1489252568534.jhist_tmp > 2017-03-12 01:16:51,256 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying > hdfs://bipcluster:8020/data/yarn/stage/master/.staging/job_1489067586592_112212/job_1489067586592_112212_1_conf.xml > to > hdfs://bipcluster:8020/data/yarn/intermediate_done/master/job_1489067586592_112212_conf.xml_tmp > 2017-03-12 01:16:51,270 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done > location: > hdfs://bipcluster:8020/data/yarn/intermediate_done/master/job_1489067586592_112212_conf.xml_tmp > 2017-03-12 01:16:51,274 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to > done: > hdfs://bipcluster:8020/data/yarn/intermediate_done/master/job_1489067586592_112212.summary_tmp > to > hdfs://bipcluster:8020/data/yarn/intermediate_done/master/job_1489067586592_112212.summary > 2017-03-12 01:16:51,276 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to > done: > hdfs://bipcluster:8020/data/yarn/intermediate_done/master/job_1489067586592_112212_conf.xml_tmp > to > hdfs://bipcluster:8020/data/yarn/intermediate_done/master/job_1489067586592_112212_conf.xml > 2017-03-12 01:16:51,277 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to > done: > hdfs://bipcluster:8020/data/yarn/intermediate_done/master/job_1489067586592_112212-1489252563933-master-3934_7823976%3Adw_log_app_page_event_hour_inc.sql%3As4-1489252610863-260-0-SUCCEEDED-root.bigdata.etl.hourlyetl.veryhigh-1489252568534.jhist_tmp > to > hdfs://bipcluster:8020/data/yarn/intermediate_done/master/job_1489067586592_112212-1489252563933-master-3934_7823976%3Adw_log_app_page_event_hour_inc.sql%3As4-1489252610863-260-0-SUCCEEDED-root.bigdata.etl.hourlyetl.veryhigh-1489252568534.jhist > 2017-03-12 01:16:51,277 INFO [Thread-402] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped > JobHistoryEventHandler. super.stop() > 2017-03-12 01:16:51,278 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING > attempt_1489067586592_112212_m_000217_0 > 2017-03-12 01:16:51,279 INFO [Thread-402] > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: > Opening proxy : sh-hadoop-datanode-128-41.elenet.me:39175 > 2017-03-12 01:16:51,284 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: > attempt_1489067586592_112212_m_000217_0 TaskAttempt Transitioned from > SUCCESS_FINISHING_CONTAINER to SUCCEEDED > 2017-03-12 01:16:51,294 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job > diagnostics to > 2017-03-12 01:16:51,297 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is > http://bigdata-rsm.elenet.me:20020/jobhistory/job/job_1489067586592_112212 > 2017-03-12 01:16:51,302 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:16:51,803 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:16:52,305 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:16:52,806 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:16:53,306 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:16:53,808 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:16:54,309 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:16:54,810 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:16:55,311 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:16:55,821 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:16:56,322 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:16:56,823 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:16:57,324 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:16:57,825 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:16:58,326 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:16:58,828 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:16:59,329 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:16:59,830 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:17:00,331 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:17:00,832 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:17:01,333 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. > 2017-03-12 01:17:01,835 INFO [Thread-402] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for > application to be successfully unregistered. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org