*From: *Kenneth Knowles <[email protected]> *Date: *Wed, May 8, 2019 at 9:24 AM *To: *dev
> > On Fri, Apr 19, 2019 at 3:09 AM Ismaël Mejía <[email protected]> wrote: > >> It seems we mostly agree that @Experimental is important, and that API >> changes (removals) on experimental features should happen quickly but still >> give some time to users so the Experimental purpose is not lost. >> >> Ahmet proposal given our current release calendar is close to 2 releases. >> Can we settle this on 2 releases as a 'minimum time' before removal? (This >> will let maintainers the option to choose to support it more time if they >> want as discussed in the related KafkaIO thread but still be friendly with >> users). >> >> Do we agree? >> > > This sounds pretty good to me. > Sounds good to me too. > How can we manage this? Right now we tie most activities (like re-triaging > flakes) to the release process, since it is the only thing that happens > regularly for the community. If we don't have some forcing then I expect > the whole thing will just be forgotten. > Can we pre-create a list of future releases in JIRA, and for each experimental feature require that a JIRA issue is created for resolving the experimental status and tag it with the release that will happen after the minimum time period? > > Kenn > > >> >> Note: for the other subjects (e.g. when an Experimental feature should >> become not experimental) I think we will hardly find an agreement so I >> think this should be treated in a per case basis by the maintainers, but if >> you want to follow up on that discussion we can open another thread for >> this. >> >> >> >> On Sat, Apr 6, 2019 at 1:04 AM Ahmet Altay <[email protected]> wrote: >> >>> I agree that Experimental feature is still very useful. I was trying to >>> argue that we diluted its value so +1 to reclaim that. >>> >>> Back to the original question, in my opinion removing existing >>> "experimental and deprecated" features in n=1 release will confuse users. >>> This will likely be a surprise to them because we have been maintaining >>> this state release after release now. I would propose in the next release >>> warning users of such a change happening and give them at least 3 months to >>> upgrade to suggested newer paths. In the future we can have a shorter >>> timelines assuming that we will set the user expectations right. >>> >>> On Fri, Apr 5, 2019 at 3:01 PM Ismaël Mejía <[email protected]> wrote: >>> >>>> I agree 100% with Kenneth on the multiple advantages that the >>>> Experimental feature gave us. I also can count multiple places where this >>>> has been essential in other modules than core. I disagree on the fact that >>>> the @Experimental annotation has lost sense, it is simply ill defined, and >>>> probably it is by design because its advantages come from it. >>>> >>>> Most of the topics in this thread are a consequence of the this loose >>>> definition, e.g. (1) not defining how a feature becomes stable, and (2) >>>> what to do when we want to remove an experimental feature, are ideas that >>>> we need to decide if we define just continue to handle as we do today. >>>> >>>> Defining a target for graduating an Experimental feature is a bit too >>>> aggressive with not much benefit, in this case we could be losing the >>>> advantages of Experimental (save if we could change the proposed version in >>>> the future). This probably makes sense for the removal of features but >>>> makes less sense to decide when some feature becomes stable. Of course in >>>> the case of the core SDKs packages this is probably more critical but >>>> nothing guarantees that things will be ready when we expect too. When will >>>> we tag for stability things like SDF or portability APIs?. We cannot >>>> predict the future for completion of features. >>>> >>>> Nobody has mentioned the LTS releases couldn’t be these like the middle >>>> points for these decisions? That at least will give LTS some value because >>>> so far I still have issues to understand the value of this idea given that >>>> we can do a minor release of any pre-released version. >>>> >>>> This debate is super important and nice to have, but we lost focus on >>>> my initial question. I like the proposal to remove a deprecated >>>> experimental feature (or part of it) after one release, in particular if >>>> the feature has a clear replacement path, however for cases like the >>>> removal of previously supported versions of Kafka one release may be too >>>> short. Other opinions on this? (or the other topics). >>>> >>>> On Fri, Apr 5, 2019 at 10:52 AM Robert Bradshaw <[email protected]> >>>> wrote: >>>> >>>>> if it's technically feasible, I am also in favor of requiring >>>>> experimental features to be (per-tag, Python should be updated) opt-in >>>>> only. We should probably regularly audit the set of experimental features >>>>> we ship (I'd say as part of the release, but that process is laborious >>>>> enough, perhaps we should do it on a half-release cycle?) I think imposing >>>>> hard deadlines (chosen when a feature is introduced) is too extreme, but >>>>> might be valuable if opt-in plus regular audit is insufficient. >>>>> >>>>> On Thu, Apr 4, 2019 at 5:28 AM Kenneth Knowles <[email protected]> >>>>> wrote: >>>>> >>>>>> This all makes me think that we should rethink how we ship >>>>>> experimental features. My experience is also that (1) users don't know if >>>>>> something is experimental or don't think hard about it and (2) we don't >>>>>> use >>>>>> experimental time period to gather feedback and make changes. >>>>>> >>>>>> How can we change both of these? Perhaps we could require >>>>>> experimental features to be opt-in. Flags work and also clearly marked >>>>>> experimental dependencies that a user has to add. Changing the core is >>>>>> sometimes tricky to put behind a flag but rarely impossible. This way a >>>>>> contributor is also motivated to gather feedback to mature their feature >>>>>> to >>>>>> become default instead of opt-in. >>>>>> >>>>>> The need that @Experimental was trying to address is real. We *do* >>>>>> need a way to try things and get feedback prior to committing to forever >>>>>> support. We have discovered real problems far too late, or not had the >>>>>> will >>>>>> to fix the issue we did find: >>>>>> - many trigger combinators should probably be deleted >>>>>> - many triggers cannot meet a good spec with merging windows >>>>>> - the continuation trigger idea doesn't work well >>>>>> - CombineFn had to have its spec changed in order to be both correct >>>>>> and efficient >>>>>> - OutputTimeFn as a UDF is convenient for Java but it turns out an >>>>>> enum is better for portability >>>>>> - Coder contexts turned out to be a major usability problem >>>>>> - The built-in data types for schemas are evolving (luckily these >>>>>> are really being worked on!) >>>>>> >>>>>> That's just what I can think of off the top of my head. I expect the >>>>>> examples from IOs are more numerous; in that case it is pretty easy to >>>>>> fork >>>>>> and make a new and better IO. >>>>>> >>>>>> And as an extreme view, I would prefer if we add a deadline for >>>>>> experimental features, then our default action is to remove them, not >>>>>> declare them stable. If noone is trying to mature it and get it out of >>>>>> opt-in status, then it probably has not matured. And perhaps if noone >>>>>> care >>>>>> enough to do that work it also isn't that important. >>>>>> >>>>>> Kenn >>>>>> >>>>>> On Wed, Apr 3, 2019 at 5:57 PM Ahmet Altay <[email protected]> wrote: >>>>>> >>>>>>> I agree with Reuven that our experimental annotation is not useful >>>>>>> any more. For example Datastore IO in python sdk is experimental for 2 >>>>>>> years now. Even though it is marked as experimental an upgrade is >>>>>>> carefully >>>>>>> planned [1] as if it is not experimental. Given that I do not think we >>>>>>> can >>>>>>> remove features within a small number of minor releases. (Exception to >>>>>>> this >>>>>>> would be, if we have a clear knowledge of very low usage of a certain >>>>>>> IO.) >>>>>>> >>>>>>> I am worried that tagging experimental features with release >>>>>>> versions will add toil to the release process as mentioned and will also >>>>>>> add to the user confusion. What would be the signal to a user if they >>>>>>> see >>>>>>> an experimental feature target release bumped between releases? How >>>>>>> about >>>>>>> tagging experimental features with JIRAs (similar to TODOs) with an >>>>>>> action >>>>>>> to either promote them as supported features or remove them? These JIRAs >>>>>>> could have fix version targets as any other release blocking JIRAs. It >>>>>>> will >>>>>>> also clarify who is responsible for a given experimental feature. >>>>>>> >>>>>>> [1] >>>>>>> https://lists.apache.org/thread.html/5ec88967aa4a382db07a60e0101c4eb36165909076867155ab3546a6@%3Cdev.beam.apache.org%3E >>>>>>> >>>>>>> On Wed, Apr 3, 2019 at 5:24 PM Reuven Lax <[email protected]> wrote: >>>>>>> >>>>>>>> Experiments are already tagged with a Kind enum >>>>>>>> (e.g. @Experimental(Kind.Schemas)). >>>>>>>> >>>>>>> >>>>>>> This not the case for python's annotations. It will be a good idea >>>>>>> to add there as well. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> On Wed, Apr 3, 2019 at 4:56 PM Ankur Goenka <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I think a release version with Experimental flag makes sense. >>>>>>>>> In addition, I think many of our user start to rely on >>>>>>>>> experimental features because they are not even aware that these >>>>>>>>> features >>>>>>>>> are experimental and its really hard to find the experimental >>>>>>>>> features used >>>>>>>>> without giving a good look at the Beam code and having some knowledge >>>>>>>>> about >>>>>>>>> it. >>>>>>>>> >>>>>>>>> It will be good it we can have a step at the pipeline submission >>>>>>>>> time which can print all the experiments used in verbose mode. This >>>>>>>>> might >>>>>>>>> also require to add a meaningful group name for the experiment example >>>>>>>>> >>>>>>>>> @Experimental("SDF", 2.15.0) >>>>>>>>> >>>>>>>>> This will of-course add additional effort and require additional >>>>>>>>> context while tagging experiments. >>>>>>>>> >>>>>>>>> On Wed, Apr 3, 2019 at 4:43 PM Reuven Lax <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Our Experimental annotation has become almost useless. Many core, >>>>>>>>>> widely-used parts of the API (e.g. triggers) are still all marked as >>>>>>>>>> experimental. So many users use these features that we couldn't >>>>>>>>>> really >>>>>>>>>> change them (in a backwards-incompatible) without hurting many >>>>>>>>>> users, so >>>>>>>>>> the fact they are marked Experimental has become a fiction. >>>>>>>>>> >>>>>>>>>> Could we add a deadline to the Experimental tag - a release >>>>>>>>>> version when it will be removed? e.g. >>>>>>>>>> >>>>>>>>>> @Experimental(2.15.0) >>>>>>>>>> >>>>>>>>>> We can have a test that ensure that the tag is removed at this >>>>>>>>>> version. Of course if we're not ready to remove experimental by that >>>>>>>>>> version, it's fine - we can always bump the tagged version. However >>>>>>>>>> this >>>>>>>>>> forces us to think about each one. >>>>>>>>>> >>>>>>>>>> Downside - it might add more toil to the existing release process. >>>>>>>>>> >>>>>>>>>> Reuven >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Apr 3, 2019 at 4:00 PM Kyle Weaver <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> > We might also want to get in the habit of reviewing if >>>>>>>>>>> something should no longer be experimental. >>>>>>>>>>> >>>>>>>>>>> +1 >>>>>>>>>>> >>>>>>>>>>> Kyle Weaver | Software Engineer | [email protected] | >>>>>>>>>>> +16502035555 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Apr 3, 2019 at 3:53 PM Kenneth Knowles <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> I think option 2 with n=1 minor version seems OK. So users get >>>>>>>>>>>> the message for one release and it is gone the next. We should >>>>>>>>>>>> make sure >>>>>>>>>>>> the deprecation warning says "this is an experimental feature, so >>>>>>>>>>>> it will >>>>>>>>>>>> be removed after 1 minor version". And we need a process for doing >>>>>>>>>>>> it so it >>>>>>>>>>>> doesn't sit around. I think we should also leave room for using >>>>>>>>>>>> our own >>>>>>>>>>>> judgment about whether the user pain is very little and then it is >>>>>>>>>>>> not >>>>>>>>>>>> needed to have a deprecation cycle. >>>>>>>>>>>> >>>>>>>>>>>> We might also want to get in the habit of reviewing if >>>>>>>>>>>> something should no longer be experimental. >>>>>>>>>>>> >>>>>>>>>>>> Kenn >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Apr 3, 2019 at 2:33 PM Ismaël Mejía <[email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> When we did the first stable release of Beam (2.0.0) we >>>>>>>>>>>>> decided to >>>>>>>>>>>>> annotate most of the Beam IOs as @Experimental because we were >>>>>>>>>>>>> cautious about not getting the APIs right in the first try. >>>>>>>>>>>>> This was a >>>>>>>>>>>>> really good decision because we could do serious improvements >>>>>>>>>>>>> and >>>>>>>>>>>>> refactorings to them in the first releases without the hassle >>>>>>>>>>>>> of >>>>>>>>>>>>> keeping backwards compatibility. However after some more >>>>>>>>>>>>> releases >>>>>>>>>>>>> users started to rely on features and supported versions, so >>>>>>>>>>>>> we ended >>>>>>>>>>>>> up in a situation where we could not change them arbitrarily >>>>>>>>>>>>> without >>>>>>>>>>>>> consequences to the final users. >>>>>>>>>>>>> >>>>>>>>>>>>> So we started to deprecate some features and parts of the API >>>>>>>>>>>>> without >>>>>>>>>>>>> removing them, e.g. the introduction of HadoopFormatIO >>>>>>>>>>>>> deprecated >>>>>>>>>>>>> HadoopInputFormatIO, we deprecated methods of MongoDbIO and >>>>>>>>>>>>> MqttIO to >>>>>>>>>>>>> improve the APIs (in most cases with valid/improved >>>>>>>>>>>>> replacements), and >>>>>>>>>>>>> recently it was discussed to removal of support for older >>>>>>>>>>>>> versions in >>>>>>>>>>>>> KafkaIO. >>>>>>>>>>>>> >>>>>>>>>>>>> Keeping deprecated stuff in experimental APIs does not seem to >>>>>>>>>>>>> make >>>>>>>>>>>>> sense, but it is what he have started to do to be ‘user >>>>>>>>>>>>> friendly’, but >>>>>>>>>>>>> it is probably a good moment to define, what should be the >>>>>>>>>>>>> clear path >>>>>>>>>>>>> for removal and breaking changes of experimental features, some >>>>>>>>>>>>> options: >>>>>>>>>>>>> >>>>>>>>>>>>> 1. Stay as we were, do not mark things as deprecated and >>>>>>>>>>>>> remove them >>>>>>>>>>>>> at will because this is the contract of @Experimental. >>>>>>>>>>>>> 2. Deprecate stuff and remove it after n versions (where n >>>>>>>>>>>>> could be 3 releases). >>>>>>>>>>>>> 3. Deprecate stuff and remove it just after a new LTS is >>>>>>>>>>>>> decided to >>>>>>>>>>>>> ensure users who need these features may still have them for >>>>>>>>>>>>> some >>>>>>>>>>>>> time. >>>>>>>>>>>>> >>>>>>>>>>>>> I would like to know your opinions about this, or if you have >>>>>>>>>>>>> other >>>>>>>>>>>>> ideas. Notice that in discussion I refer only to @Experimental >>>>>>>>>>>>> features. >>>>>>>>>>>>> >>>>>>>>>>>>
