[
https://issues.apache.org/jira/browse/SPARK-18994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15794090#comment-15794090
]
Apache Spark commented on SPARK-18994:
--------------------------------------
User 'liujianhuiouc' has created a pull request for this issue:
https://github.com/apache/spark/pull/16456
> worker clean up app directory block the heartbeat sending
> ----------------------------------------------------------
>
> Key: SPARK-18994
> URL: https://issues.apache.org/jira/browse/SPARK-18994
> Project: Spark
> Issue Type: Bug
> Components: Deploy
> Affects Versions: 1.6.1
> Reporter: liujianhui
>
> h2. scene
> # when an application stop, it will trigger master send ApplicationFinish
> event to worker
> # worker will clean up the directories for the application, but it takes a
> long time to delete them, as a result of the worker endpoint is Thread safe,
> and then it block the heartbeat sending to master
> # master find the heartbeat expired for that worker, and then remove the
> worker, it will re-launching the driver belong to that worker, actually that
> driver doesn't exit, finally this will exist two app with same driver-id
> the worker log
> {code}
> 2016-12-23 11:39:33,033 |INFO | dispatcher-event-loop-26 |
> org.apache.spark.deploy.worker.Worker | Executor app-20161208222600-0020/200
> finished with state KILLED exitStatus 137 |
> org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:39:33,033 |INFO | dispatcher-event-loop-26 |
> org.apache.spark.deploy.worker.Worker | Cleaning up local directories for
> application app-20161208222600-0020 |
> org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 |
> org.apache.spark.network.shuffle.ExternalShuffleBlockResolver | Application
> app-20161208222600-0020 removed, cleanupLocalDirs = true |
> org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.applicationRemoved(ExternalShuffleBlockResolver.java:206)
> 2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 |
> org.apache.spark.deploy.worker.Worker | Master with url
> spark://bigdata-sirius-ser02.gz01:8090 requested this worker to reconnect. |
> org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | worker-register-master-threadpool-2 |
> org.apache.spark.deploy.worker.Worker | Connecting to master
> bigdata-sirius-ser02.gz01:8090... |
> org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | worker-register-master-threadpool-3 |
> org.apache.spark.deploy.worker.Worker | Connecting to master
> bigdata-sirius-ser03.gz01:8090... |
> org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 |
> org.apache.spark.deploy.worker.Worker | Master with url
> spark://bigdata-sirius-ser02.gz01:8090 requested this worker to reconnect. |
> org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | shuffle-client-1 |
> org.apache.spark.deploy.worker.Worker | Successfully registered with master
> spark://bigdata-sirius-ser02.gz01:8090 |
> org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | shuffle-client-1 |
> org.apache.spark.deploy.worker.Worker | Worker cleanup enabled; old
> application directories will be deleted in: /data1/spark/work |
> org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 |
> org.apache.spark.deploy.worker.Worker | Not spawning another attempt to
> register with the master, since there is an attempt scheduled already. |
> org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 |
> org.apache.spark.deploy.worker.Worker | Master with url
> spark://bigdata-sirius-ser02.gz01:8090 requested this worker to reconnect. |
> org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 |
> org.apache.spark.deploy.worker.Worker | Master with url
> spark://bigdata-sirius-ser02.gz01:8090 requested this worker to reconnect. |
> org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | worker-register-master-threadpool-3 |
> org.apache.spark.deploy.worker.Worker | Connecting to master
> bigdata-sirius-ser03.gz01:8090... |
> org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 |
> org.apache.spark.deploy.worker.Worker | Not spawning another attempt to
> register with the master, since there is an attempt scheduled already. |
> org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | worker-register-master-threadpool-2 |
> org.apache.spark.deploy.worker.Worker | Connecting to master
> bigdata-sirius-ser02.gz01:8090... |
> org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> {code}
> *master* log
> {code}
> 2016-12-23 11:40:46,046 |WARN | dispatcher-event-loop-20 |
> org.apache.spark.deploy.master.Master | Removing
> worker-20161208233029-100.69.146.57-26874 because we got no heartbeat in 60
> seconds | org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 |
> org.apache.spark.deploy.master.Master | Got heartbeat from unregistered
> worker worker-20161208233029-100.69.146.57-26874. Asking it to re-register. |
> org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 |
> org.apache.spark.deploy.master.Master | Got heartbeat from unregistered
> worker worker-20161208233029-100.69.146.57-26874. Asking it to re-register. |
> org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 |
> org.apache.spark.deploy.master.Master | Got heartbeat from unregistered
> worker worker-20161208233029-100.69.146.57-26874. Asking it to re-register. |
> org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 |
> org.apache.spark.deploy.master.Master | Got heartbeat from unregistered
> worker worker-20161208233029-100.69.146.57-26874. Asking it to re-register. |
> org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 |
> org.apache.spark.deploy.master.Master | Got heartbeat from unregistered
> worker worker-20161208233029-100.69.146.57-26874. Asking it to re-register. |
> org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 |
> org.apache.spark.deploy.master.Master | Got heartbeat from unregistered
> worker worker-20161208233029-100.69.146.57-26874. Asking it to re-register. |
> org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 |
> org.apache.spark.deploy.master.Master | Got heartbeat from unregistered
> worker worker-20161208233029-100.69.146.57-26874. Asking it to re-register. |
> org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
