+1 (non-binding) Cheers,
Xingbo On Mon, Mar 9, 2020 at 9:35 AM Xiao Li <lix...@databricks.com> wrote: > +1 (binding) > > Xiao > > On Mon, Mar 9, 2020 at 8:33 AM Denny Lee <denny.g....@gmail.com> wrote: > >> +1 (non-binding) >> >> On Mon, Mar 9, 2020 at 1:59 AM Hyukjin Kwon <gurwls...@gmail.com> wrote: >> >>> The proposal itself seems good as the factors to consider, Thanks >>> Michael. >>> >>> Several concerns mentioned look good points, in particular: >>> >>> > ... assuming that this is for public stable APIs, not APIs that are >>> marked as unstable, evolving, etc. ... >>> I would like to confirm this. We already have API annotations such as >>> Experimental, Unstable, etc. and the implication of each is still >>> effective. If it's for stable APIs, it makes sense to me as well. >>> >>> > ... can we expand on 'when' an API change can occur ? Since we are >>> proposing to diverge from semver. ... >>> I think this is a good point. If we're proposing to divert from semver, >>> the delta compared to semver will have to be clarified to avoid different >>> personal interpretations of the somewhat general principles. >>> >>> > ... can we narrow down on the migration from Apache Spark 2.4.5 to >>> Apache Spark 3.0+? ... >>> >>> Assuming these concerns will be addressed, +1 (binding). >>> >>> >>> 2020년 3월 9일 (월) 오후 4:53, Takeshi Yamamuro <linguin....@gmail.com>님이 작성: >>> >>>> +1 (non-binding) >>>> >>>> Bests, >>>> Takeshi >>>> >>>> On Mon, Mar 9, 2020 at 4:52 PM Gengliang Wang < >>>> gengliang.w...@databricks.com> wrote: >>>> >>>>> +1 (non-binding) >>>>> >>>>> Gengliang >>>>> >>>>> On Mon, Mar 9, 2020 at 12:22 AM Matei Zaharia <matei.zaha...@gmail.com> >>>>> wrote: >>>>> >>>>>> +1 as well. >>>>>> >>>>>> Matei >>>>>> >>>>>> On Mar 9, 2020, at 12:05 AM, Wenchen Fan <cloud0...@gmail.com> wrote: >>>>>> >>>>>> +1 (binding), assuming that this is for public stable APIs, not APIs >>>>>> that are marked as unstable, evolving, etc. >>>>>> >>>>>> On Mon, Mar 9, 2020 at 1:10 AM Ismaël Mejía <ieme...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> +1 (non-binding) >>>>>>> >>>>>>> Michael's section on the trade-offs of maintaining / removing an API >>>>>>> are one of >>>>>>> the best reads I have seeing in this mailing list. Enthusiast +1 >>>>>>> >>>>>>> On Sat, Mar 7, 2020 at 8:28 PM Dongjoon Hyun < >>>>>>> dongjoon.h...@gmail.com> wrote: >>>>>>> > >>>>>>> > This new policy has a good indention, but can we narrow down on >>>>>>> the migration from Apache Spark 2.4.5 to Apache Spark 3.0+? >>>>>>> > >>>>>>> > I saw that there already exists a reverting PR to bring back Spark >>>>>>> 1.4 and 1.5 APIs based on this AS-IS suggestion. >>>>>>> > >>>>>>> > The AS-IS policy is clearly mentioning that JVM/Scala-level >>>>>>> difficulty, and it's nice. >>>>>>> > >>>>>>> > However, for the other cases, it sounds like `recommending older >>>>>>> APIs as much as possible` due to the following. >>>>>>> > >>>>>>> > > How long has the API been in Spark? >>>>>>> > >>>>>>> > We had better be more careful when we add a new policy and should >>>>>>> aim not to mislead the users and 3rd party library developers to say >>>>>>> "older >>>>>>> is better". >>>>>>> > >>>>>>> > Technically, I'm wondering who will use new APIs in their examples >>>>>>> (of books and StackOverflow) if they need to write an additional warning >>>>>>> like `this only works at 2.4.0+` always . >>>>>>> > >>>>>>> > Bests, >>>>>>> > Dongjoon. >>>>>>> > >>>>>>> > On Fri, Mar 6, 2020 at 7:10 PM Mridul Muralidharan < >>>>>>> mri...@gmail.com> wrote: >>>>>>> >> >>>>>>> >> I am in broad agreement with the prposal, as any developer, I >>>>>>> prefer >>>>>>> >> stable well designed API's :-) >>>>>>> >> >>>>>>> >> Can we tie the proposal to stability guarantees given by spark and >>>>>>> >> reasonable expectation from users ? >>>>>>> >> In my opinion, an unstable or evolving could change - while an >>>>>>> >> experimental api which has been around for ages should be more >>>>>>> >> conservatively handled. >>>>>>> >> Which brings in question what are the stability guarantees as >>>>>>> >> specified by annotations interacting with the proposal. >>>>>>> >> >>>>>>> >> Also, can we expand on 'when' an API change can occur ? Since we >>>>>>> are >>>>>>> >> proposing to diverge from semver. >>>>>>> >> Patch release ? Minor release ? Only major release ? Based on >>>>>>> 'impact' >>>>>>> >> of API ? Stability guarantees ? >>>>>>> >> >>>>>>> >> Regards, >>>>>>> >> Mridul >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> On Fri, Mar 6, 2020 at 7:01 PM Michael Armbrust < >>>>>>> mich...@databricks.com> wrote: >>>>>>> >> > >>>>>>> >> > I'll start off the vote with a strong +1 (binding). >>>>>>> >> > >>>>>>> >> > On Fri, Mar 6, 2020 at 1:01 PM Michael Armbrust < >>>>>>> mich...@databricks.com> wrote: >>>>>>> >> >> >>>>>>> >> >> I propose to add the following text to Spark's Semantic >>>>>>> Versioning policy and adopt it as the rubric that should be used when >>>>>>> deciding to break APIs (even at major versions such as 3.0). >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> I'll leave the vote open until Tuesday, March 10th at 2pm. As >>>>>>> this is a procedural vote, the measure will pass if there are more >>>>>>> favourable votes than unfavourable ones. PMC votes are binding, but the >>>>>>> community is encouraged to add their voice to the discussion. >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> [ ] +1 - Spark should adopt this policy. >>>>>>> >> >> >>>>>>> >> >> [ ] -1 - Spark should not adopt this policy. >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> <new policy> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> Considerations When Breaking APIs >>>>>>> >> >> >>>>>>> >> >> The Spark project strives to avoid breaking APIs or silently >>>>>>> changing behavior, even at major versions. While this is not always >>>>>>> possible, the balance of the following factors should be considered >>>>>>> before >>>>>>> choosing to break an API. >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> Cost of Breaking an API >>>>>>> >> >> >>>>>>> >> >> Breaking an API almost always has a non-trivial cost to the >>>>>>> users of Spark. A broken API means that Spark programs need to be >>>>>>> rewritten >>>>>>> before they can be upgraded. However, there are a few considerations >>>>>>> when >>>>>>> thinking about what the cost will be: >>>>>>> >> >> >>>>>>> >> >> Usage - an API that is actively used in many different places, >>>>>>> is always very costly to break. While it is hard to know usage for sure, >>>>>>> there are a bunch of ways that we can estimate: >>>>>>> >> >> >>>>>>> >> >> How long has the API been in Spark? >>>>>>> >> >> >>>>>>> >> >> Is the API common even for basic programs? >>>>>>> >> >> >>>>>>> >> >> How often do we see recent questions in JIRA or mailing lists? >>>>>>> >> >> >>>>>>> >> >> How often does it appear in StackOverflow or blogs? >>>>>>> >> >> >>>>>>> >> >> Behavior after the break - How will a program that works >>>>>>> today, work after the break? The following are listed roughly in order >>>>>>> of >>>>>>> increasing severity: >>>>>>> >> >> >>>>>>> >> >> Will there be a compiler or linker error? >>>>>>> >> >> >>>>>>> >> >> Will there be a runtime exception? >>>>>>> >> >> >>>>>>> >> >> Will that exception happen after significant processing has >>>>>>> been done? >>>>>>> >> >> >>>>>>> >> >> Will we silently return different answers? (very hard to >>>>>>> debug, might not even notice!) >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> Cost of Maintaining an API >>>>>>> >> >> >>>>>>> >> >> Of course, the above does not mean that we will never break >>>>>>> any APIs. We must also consider the cost both to the project and to our >>>>>>> users of keeping the API in question. >>>>>>> >> >> >>>>>>> >> >> Project Costs - Every API we have needs to be tested and needs >>>>>>> to keep working as other parts of the project changes. These costs are >>>>>>> significantly exacerbated when external dependencies change (the JVM, >>>>>>> Scala, etc). In some cases, while not completely technically infeasible, >>>>>>> the cost of maintaining a particular API can become too high. >>>>>>> >> >> >>>>>>> >> >> User Costs - APIs also have a cognitive cost to users learning >>>>>>> Spark or trying to understand Spark programs. This cost becomes even >>>>>>> higher >>>>>>> when the API in question has confusing or undefined semantics. >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> Alternatives to Breaking an API >>>>>>> >> >> >>>>>>> >> >> In cases where there is a "Bad API", but where the cost of >>>>>>> removal is also high, there are alternatives that should be considered >>>>>>> that >>>>>>> do not hurt existing users but do address some of the maintenance costs. >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> Avoid Bad APIs - While this is a bit obvious, it is an >>>>>>> important point. Anytime we are adding a new interface to Spark we >>>>>>> should >>>>>>> consider that we might be stuck with this API forever. Think deeply >>>>>>> about >>>>>>> how new APIs relate to existing ones, as well as how you expect them to >>>>>>> evolve over time. >>>>>>> >> >> >>>>>>> >> >> Deprecation Warnings - All deprecation warnings should point >>>>>>> to a clear alternative and should never just say that an API is >>>>>>> deprecated. >>>>>>> >> >> >>>>>>> >> >> Updated Docs - Documentation should point to the "best" >>>>>>> recommended way of performing a given task. In the cases where we >>>>>>> maintain >>>>>>> legacy documentation, we should clearly point to newer APIs and suggest >>>>>>> to >>>>>>> users the "right" way. >>>>>>> >> >> >>>>>>> >> >> Community Work - Many people learn Spark by reading blogs and >>>>>>> other sites such as StackOverflow. However, many of these resources are >>>>>>> out >>>>>>> of date. Update them, to reduce the cost of eventually removing >>>>>>> deprecated >>>>>>> APIs. >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> </new policy> >>>>>>> >> >>>>>>> >> >>>>>>> --------------------------------------------------------------------- >>>>>>> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>> >> >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>> >>>>>>> >>>>>> >>>> >>>> -- >>>> --- >>>> Takeshi Yamamuro >>>> >>> > > -- > <https://databricks.com/sparkaisummit/north-america> >
