[ https://issues.apache.org/jira/browse/SPARK-20624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603577#comment-17603577 ]
Juliusz Sompolski commented on SPARK-20624: ------------------------------------------- [~holden] Are these new APIs documented? I can't seem to find them in the official Spark documentation. Should they be mentioned e.g. in https://spark.apache.org/docs/latest/job-scheduling.html#graceful-decommission-of-executors ? > SPIP: Add better handling for node shutdown > ------------------------------------------- > > Key: SPARK-20624 > URL: https://issues.apache.org/jira/browse/SPARK-20624 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 3.0.0 > Reporter: Holden Karau > Priority: Major > > While we've done some good work with better handling when Spark is choosing > to decommission nodes (SPARK-7955), it might make sense in environments where > we get preempted without our own choice (e.g. YARN over-commit, EC2 spot > instances, GCE Preemptiable instances, etc.) to do something for the data on > the node (or at least not schedule any new tasks). -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org