Hi Spark Development Community,
Since Spark 3 has shipped I've been going through some of the PRs and I've noticed some PRs have been merged with pending -1s with technical reasons, including those from committers. I'm bringing this up because I believe we, as PMC, committers and contributors, do not currently have a consistent understanding, and I believe we should develop one. The foundation level guidance is at https://www.apache.org/foundation/voting.html#votes-on-code-modification. It is my belief that we should not be merging code with -1s from active committers, -1 is a very strong signal and generally a sign that we should step back and try and build consensus around the change. Looking at how the httpd project (e.g. the original Apache project) handles it https://httpd.apache.org/dev/guidelines.html#voting it seems to be that once a -1 from a committer is received we can no longer use our regular lazy consensus mechanism for the change, and we then need to have a vote or get the -1 resolved with the person who placed it. Now of course, if the -1s aren't following the guidelines that's a different story (e.g. no technical reason provided), but regardless a -1 from a committer to me would require a public vote on dev@ following the foundation's voting guidelines. I believe, especially in the Spark project, committers have demonstrated sufficient merit and they are qualified to vote on code changes so a -1 should block merging, however in talking with other ASF projects there are a variety of ways of handling this. The unanimous opinion from the different folks I talked with is that any technical disagreement should be considered before moving on. Just waiting a few days and merging the code is not a valid solution. In the context of how much work and history we've chosen to require Spark committers to demonstrate, most folks seem to believe that committers in the Spark project would be qualified voters for these purposes. In general I expect -1s to continue to be relatively rare in the Spark community, and if this is no longer the case I believe we should take the areas where we are seeing more -1s and have a broader discussion to give us the opportunity to build consensus prior to making any further non-bugfix changes in those components/areas. Most folks from other projects seemed to share this concern as well. One of the things that was also brought up in this context is some projects require -1s to provide an alternative suggestion as well as the technical objection. That could be something we, as a project, could adopt if were concerned with over use of -1s. Sincerely, Holden P.S. I know this may be a sensitive topic, and we're all under more stress than usual right now. I appreciate everyone's patience while we discuss this. I know I find this a sensitive topic as well given the seemingly inconsistent standards. I'm tired of having people ask me to wait on PRs that have been open for more than a month and merging their own code with pending substantial issues raised by qualified members of the community. I'm not saying that the actions taken were necessarily wrong, just that I believe reaching a common understanding here is crucial to the healthy functioning of the project. I have not included any specific examples of this since this is the public list and I believe discussions involving individuals does not belong on dev@. If we want to discuss the specific situations that I noticed from the Spark 3 release, we can fork that conversation to private@. For any ASF members you can also find the discussion with folks from other projects and their views on that mailing list and get at the details. -- Twitter: https://twitter.com/holdenkarau Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
