Repository: spark Updated Branches: refs/heads/master 1c9a386c6 -> 4ec5c360c
[SPARK-16868][WEB UI] Fix executor be both dead and alive on executor ui. ## What changes were proposed in this pull request? In a heavy pressure of the spark application, since the executor will register it to driver block manager twice(because of heart beats), the executor will show as picture show: ![image](https://cloud.githubusercontent.com/assets/7404824/17467245/c1359094-5d4e-11e6-843a-f6d6347e1bf6.png) ## How was this patch tested? NA Details in: [SPARK-16868](https://issues.apache.org/jira/browse/SPARK-16868) Author: huangzhaowei <carlmartin...@gmail.com> Closes #14530 from SaintBacchus/SPARK-16868. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4ec5c360 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/4ec5c360 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/4ec5c360 Branch: refs/heads/master Commit: 4ec5c360ce2045a9bdecb3c5277ba519bf0f44ae Parents: 1c9a386 Author: huangzhaowei <carlmartin...@gmail.com> Authored: Thu Aug 11 14:56:03 2016 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Thu Aug 11 14:56:03 2016 -0700 ---------------------------------------------------------------------- .../scala/org/apache/spark/storage/StorageStatusListener.scala | 4 ++++ 1 file changed, 4 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/4ec5c360/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala b/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala index 3008520..798658a 100644 --- a/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala +++ b/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala @@ -77,6 +77,10 @@ class StorageStatusListener(conf: SparkConf) extends SparkListener { val maxMem = blockManagerAdded.maxMem val storageStatus = new StorageStatus(blockManagerId, maxMem) executorIdToStorageStatus(executorId) = storageStatus + + // Try to remove the dead storage status if same executor register the block manager twice. + deadExecutorStorageStatus.zipWithIndex.find(_._1.blockManagerId.executorId == executorId) + .foreach(toRemoveExecutor => deadExecutorStorageStatus.remove(toRemoveExecutor._2)) } } --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org