Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/20781 @jerryshao Thanks for advice. I spent some time digging to find why multiple `kill` sent from Driver to AM, but didn't figure out a way to reproduce. I come to find that it's possible `YarnAllocator` process same completed container multiple times(https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala#L573 this log is printed multiple times for same container). And I made another change(see last commit in this pr) to propose my idea -- replace `numExecutorsRunning` with a set.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org