Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/23260 @srowen For now executor log url is **static** in Spark, which forces Node Manager to be alive even after application is finished, in order to provide executor log in SHS. This situation can be happen when decommission happens a bit frequently, or when end users want a kind of elasticity against YARN cluster (not only decommissioning nodes, but also elasticity on YARN cluster itself - YARN has cluster id for RM which classifies the cluster which can be leveraged when dealing with multiple YARN clusters.) There's also similar change applied on Hadoop side. https://github.com/apache/hadoop/commit/5fe1dbf1959976d0dc5a8e614dd74836cfbee04c We are experimenting central log service which resolves above situation. At least the log url for centralized log service can't be same URL as NM webapp, we have to get flexibility of executor log URL. Hope it explains the rationalization well.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org