Thank you for the discussion everyone! This vote passes. I'll work to get this posed on the website.
+1 Michael Armbrust Sean Owen Jules Damji 大啊 Ismaël Mejía Wenchen Fan Matei Zaharia Gengliang Wang Takeshi Yamamuro Denny Lee Xiao Li Xingbo Jiang Tkuya UESHIN Hichael Heuer John Zhuge Reynold Xin Burak Yavuz Holden Karau Dongjoon Hyun To respond to some of the questions on the interpretation of this policy: > Also, can we expand on 'when' an API change can occur ? Since we are > proposing to diverge from semver. > Patch release ? Minor release ? Only major release ? Based on 'impact' of > API ? Stability guarantees ? This is an addition to the existing semvar policy. We still do not break stable APIs at major versions. This new policy has a good indention, but can we narrow down on the > migration from Apache Spark 2.4.5 to Apache Spark 3.0+? I do not think that we should apply this policy to the 3.0 release any differently than we will for future releases. There is nothing special about 3.0 that means unnecessary breakages will not be costly to our users. If I had to summarize the policy in once sentence it would be "Think about users before you break APIs!". As I mentioned in my original email, I think in many cases this did not happen in the lead up to this release. Rather, the reasoning in some cases was that "This is a major release, so we can break things". Given that we all agree major versions are not sufficient justification to break APIs, I think its reasonable to revisit and discuss on a case-by-case basis, some of the more commonly used, broken APIs in the context of this rubric. We had better be more careful when we add a new policy and should aim not > to mislead the users and 3rd party library developers to say "older is > better". Nothing in the policy says "older is better". It only says that age is one factor to consider when trying to reason about usage. If an API has been around for a long time, its possible (but not always true) that it will have more usage than an old API. If usage is low, and the cost to keep it is high, get rid of it even if its very old. The policy also explicitly calls out updating docs to recommend the new/"correct" way of doing things. If you can convince all the users of Spark to switch, then you can remove any API you want in the future :) Is this only applying to stable apis? This is not explicitly called out, but I would argue you should still think about users, even when breaking experimental APIs. The bar is certainly lower here, we explicitly called out that these APIs might change. That said, I still would go though the exercise and decide if the benefits outweigh the costs before doing it. (Very similar to the discussion before our 1.0, before any promises of stability had been made). the way I read this proposal isn't really saying we can't break api's on > major releases, its just saying spend more time making sure its worth it. I agree with this interpretation! Michael On Tue, Mar 10, 2020 at 10:59 AM Tom Graves <tgraves...@yahoo.com> wrote: > Overall makes sense to me, but have same questions as others on the thread. > > Is this only applying to stable apis? > How are we going to apply to 3.0? > > the way I read this proposal isn't really saying we can't break api's on > major releases, its just saying spend more time making sure its worth it. > Tom > > On Friday, March 6, 2020, 08:59:03 PM CST, Michael Armbrust < > mich...@databricks.com> wrote: > > > I propose to add the following text to Spark's Semantic Versioning policy > <https://spark.apache.org/versioning-policy.html> and adopt it as the > rubric that should be used when deciding to break APIs (even at major > versions such as 3.0). > > > I'll leave the vote open until Tuesday, March 10th at 2pm. As this is a > procedural > vote <https://www.apache.org/foundation/voting.html>, the measure will > pass if there are more favourable votes than unfavourable ones. PMC votes > are binding, but the community is encouraged to add their voice to the > discussion. > > > [ ] +1 - Spark should adopt this policy. > > [ ] -1 - Spark should not adopt this policy. > > > <new policy> > > > Considerations When Breaking APIs > > The Spark project strives to avoid breaking APIs or silently changing > behavior, even at major versions. While this is not always possible, the > balance of the following factors should be considered before choosing to > break an API. > > Cost of Breaking an API > > Breaking an API almost always has a non-trivial cost to the users of > Spark. A broken API means that Spark programs need to be rewritten before > they can be upgraded. However, there are a few considerations when thinking > about what the cost will be: > > - > > Usage - an API that is actively used in many different places, is > always very costly to break. While it is hard to know usage for sure, there > are a bunch of ways that we can estimate: > - > > How long has the API been in Spark? > - > > Is the API common even for basic programs? > - > > How often do we see recent questions in JIRA or mailing lists? > - > > How often does it appear in StackOverflow or blogs? > - > > Behavior after the break - How will a program that works today, work > after the break? The following are listed roughly in order of increasing > severity: > - > > Will there be a compiler or linker error? > - > > Will there be a runtime exception? > - > > Will that exception happen after significant processing has been > done? > - > > Will we silently return different answers? (very hard to debug, > might not even notice!) > > > Cost of Maintaining an API > > Of course, the above does not mean that we will never break any APIs. We > must also consider the cost both to the project and to our users of keeping > the API in question. > > - > > Project Costs - Every API we have needs to be tested and needs to keep > working as other parts of the project changes. These costs are > significantly exacerbated when external dependencies change (the JVM, > Scala, etc). In some cases, while not completely technically infeasible, > the cost of maintaining a particular API can become too high. > - > > User Costs - APIs also have a cognitive cost to users learning Spark > or trying to understand Spark programs. This cost becomes even higher when > the API in question has confusing or undefined semantics. > > > Alternatives to Breaking an API > > In cases where there is a "Bad API", but where the cost of removal is also > high, there are alternatives that should be considered that do not hurt > existing users but do address some of the maintenance costs. > > > - > > Avoid Bad APIs - While this is a bit obvious, it is an important > point. Anytime we are adding a new interface to Spark we should consider > that we might be stuck with this API forever. Think deeply about how > new APIs relate to existing ones, as well as how you expect them to evolve > over time. > - > > Deprecation Warnings - All deprecation warnings should point to a > clear alternative and should never just say that an API is deprecated. > - > > Updated Docs - Documentation should point to the "best" recommended > way of performing a given task. In the cases where we maintain legacy > documentation, we should clearly point to newer APIs and suggest to users > the "right" way. > - > > Community Work - Many people learn Spark by reading blogs and other > sites such as StackOverflow. However, many of these resources are out of > date. Update them, to reduce the cost of eventually removing deprecated > APIs. > > > </new policy> >
