Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/22209#discussion_r213779029 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala --- @@ -506,7 +516,16 @@ private[spark] class AppStatusListener( if (killedDelta > 0) { stage.killedSummary = killedTasksSummary(event.reason, stage.killedSummary) } - maybeUpdate(stage, now) + // Remove stage if there are no active tasks left and stage is already finished + val removeStage = + stage.activeTasks == 0 && + (v1.StageStatus.COMPLETE.equals(stage.status) || + v1.StageStatus.FAILED.equals(stage.status)) + if (removeStage) { + update(stage, now, last = true) --- End diff -- Shouldn't this be removing the stage from the live list too?
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org