At the recommendation of Hyukjin, I'm converting the graceful decommissioning work to an SPIP. The SPIP document is at https://docs.google.com/document/d/1EOei24ZpVvR7_w0BwBjOnrWRy4k-qTdIlx60FsHZSHA/edit?usp=sharing and the associated JIRA is at https://issues.apache.org/jira/browse/SPARK-20624. This work dates back to 2017 when an earlier design was brought up. Now in 2019 I've updated the design https://docs.google.com/document/d/1xVO1b6KAwdUhjEJBolVPl9C6sLj7oOveErwDSYdT-pE/edit?usp=sharing .
>From the SPIP requirements, I am willing to act as the shepherd & committer on this proposal, and have already been reviewing and creating PRs here. There are several folks who have contributed to design (you can see the discussion in the 2019 & 2017 documents) and I'm very thankful for those contributions, as well as the other code reviewers & PR authors who have been participating from a variety of vendors/users. Given the existence of multiple vendors' proprietary implementations (Databricks & AWS) of this feature, I think the user need is very clear. This begins the discussion phase of the SPIP process where the goal is to determine the need for the change and the general design outline. Once the discussion settles, we move on the voting stage. While there are WIP PRS attached to the design document to illustrate the implementation, approval of the SPIP does not necessarily mean those are the specific implementations that we will use. For more information about SPIPs in general see https://spark.apache.org/improvement-proposals.html -- Twitter: https://twitter.com/holdenkarau Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
