On Fri, Apr 14, 2023 at 1:15 PM Kenneth Knowles <k...@apache.org> wrote:
> > Thanks for the discussion. Many good points. Probably just removing all > the annotations is a noop to users, and will solve the "afraid to use > experimental features" problem. > > Regarding stability, the capabilities of Java (and Python is much much > worse) make it infeasible to produce quality software with the rule "once > it is public it is frozen forever". But on the other hand, there isn't much > of a practical alternative. Most projects just make breaking changes at > minor releases quite often, in my experience. I don't want to follow that > pattern, for sure. > > Regarding Danny's comment of not seeing this culture - check out any of > our more mature IOs, which all have very high cyclomatic complexity due to > never being significantly refactored. Adhering to in-place state > compatibility for update instead of focusing on blue/green deployment is > also a culprit here. I don't have examples to mind, but the point about the > culture of stagnation came from my recent experiences as code > reviewer where there was some idea that we couldn't change things even when > they were plainly wrong and the change was plainly a fix. > > Often, it comes from corners like triggered side inputs where we simply > never had a clear concept and so bringing things into alignment with a spec > will break someone, by necessity. To be clear: I have not received pushback > on that one (yet). Some other examples are > https://s.apache.org/finishing-triggers-drop-data (breaking change > necessary to eliminate data loss risk) > https://github.com/apache/beam/issues/20528 (fix was too slow because we > were hesitant to commit a breaking fix) > https://github.com/apache/beam/pull/8134#pullrequestreview-218592801 > (left unsafe API in place, applied doc-only fix). > > But indeed, of all the issues I raised, the customer concern with > `@Experimental` was the most important. We have had a few threads about it > in the past, too, and it hasn't gotten better. > > 1. It does not have the intended effect (making users OK with evolving > APIs and behavior to allow us to reach a high level of quality) > 2. It has an unintended effect (making users afraid to use things which > they should be happy to use) > 3. We don't use it consistently (many less-safe things are not > experimental, many totally stable things are experimental) > > Because of 3, if we don't have a feasible way to move to > "evolving/unstable by default" in a way that users know and are OK with, > then 1 is impossible. And so the only way to fix 2 is to just eliminate the > annotation approach entirely and go with language conventions. > +1 to eliminating @Experimental as a Beam level annotation. That is the simplest approach that will get us to a consistent state, and it will align the goals and intentions of us with users'. > > Kenn > > On Wed, Apr 12, 2023 at 5:10 PM Ahmet Altay via dev <dev@beam.apache.org> > wrote: > >> I agree with Alexey and Byron. >> 1. We do not have any concrete evidence of our users paying attention to >> any of those annotations. Experimental API that were in that state for a >> long while are good examples. A possible exception is a deprecated >> annotation. My preference would be to simplify annotations to nothing >> (stable enough for use and will evolve backward compatibility), and maybe >> deprecated annotations. >> 2. If you all think that Experimental annotation is needed, Byron's >> suggestion (more or less what we do today) but with some concrete life >> cycle definitions of those annotations would be useful to our users. (An >> example could be: experimental APIs either need to graduate or be removed >> in X releases.) >> >> >> >> On Tue, Apr 4, 2023 at 9:01 AM Alexey Romanenko <aromanenko....@gmail.com> >> wrote: >> >>> Great and long-to-wait topic to discuss. >>> >>> My personal opinion based on what I saw on different open-source >>> projects is that all such annotations, like @Experimental or @Stable, are >>> not usefull along the time and even rather useless and misleading. What >>> actually play roles is artifacts publishing and public API despite how it >>> was annotated. Once a class/method was published and available for users to >>> use, it should be considered as “stable" (even if it’s not yet stable from >>> its developers point of view) and can’t be easily removed/changed in the >>> next releases. >>> >>> At Beam, we have a “good" example with @Experimental that was used to >>> annotate many parts of code in the beginning of its creation but then >>> perhaps forgotten to be removed whenever this code is already used by many >>> users and API can’t be just changed despite of this annotation. >>> >>> So, I’m pro to dismiss such annotations and consider all public and >>> user-available API as “stable”. If it’s needed to change/remove a public >>> API then we should follow the procedure of API deprecation and final >>> removing, at least, after 3 major (x.y) Beam releases. It should help to >>> have the clear rules for API changes and avoiding breaking changes for >>> users. >>> >>> — >>> Alexey >>> >>> >>> On 3 Apr 2023, at 17:04, Byron Ellis via dev <dev@beam.apache.org> >>> wrote: >>> >>> Honestly, I think APIs could be pretty simply defined if you think of it >>> in terms of the user: >>> >>> @Deprecated = this was either stable or evolve but the >>> functionality/interface will go away at a future date >>> >>> @Stable = the user of this API opting out of changes to functionality >>> and interface. For example, default options don't change for a transform >>> annotated this way. >>> >>> Evolving (No Annotation) = the user is opting in to changes to >>> functionality but not to interface. We should generally try to write >>> backwards compatible code, but on the other hand the release model does not >>> force users into an upgrade >>> >>> @Experimental = this functionality / interface might be a bad idea and >>> could go away at any time >>> >>> >>> On Mon, Apr 3, 2023 at 7:22 AM Danny McCormick via dev < >>> dev@beam.apache.org> wrote: >>> >>>> *;tldr - I'd like "evolving" to be further defined, specifically around >>>> how we will make decisions about breaking behavior and API changes* >>>> >>>> I don't particularly care what tags we use as long as they're well >>>> documented. With that said, I think the following framing needs to be >>>> documented with more definition to flesh out the underlying philosophy: >>>> >>>> *> - new code is changeable/evolving by default (so we don't have to >>>> always remember to annotate it) but users have confidence they can use it >>>> in production (because we have good software engineering practices)* >>>> >>>> * > - Experimental would be reserved for more risky things* >>>> * > - after we are confident an API is stable, because it has been the >>>> same across a couple releases, we mark it* >>>> >>>> Here, we have 3 classes of APIs - "experimental", "stable", and >>>> "evolving" (or alternately "undefined"). >>>> >>>> "Experimental" seems clear - we can make any changes we want. "Stable" >>>> is reasonably straightforward as well - we will only make non-breaking >>>> changes except in exceptional cases (e.g. security hole, total failure of >>>> functionality, etc...) >>>> >>>> With "evolving" is the idea that we can still make any changes we want, >>>> but we think it's less likely we'll need to? Are silent behavior changes >>>> acceptable here (my vote would be no)? What about breaking API changes (my >>>> vote would be rarely)? >>>> >>>> I think being able to change our APIs is an ok goal, but outside of a >>>> true experimental context we should still be weighing the cost of API >>>> changes against the benefit; we have a problem of people not updating to >>>> newer SDKs, and introducing more breaking changes will just exacerbate that >>>> problem. Maybe my concerns are just a consequence of me not really seeing >>>> the same things that you're seeing, specifically: "*I'm seeing a >>>> culture of being afraid to change things, even when it would be good for >>>> users, because our API surface area is far too large and not explicitly >>>> chosen.*" Mostly what I've seen is a healthy concern about making it >>>> hard for users to upgrade versions, but my view is probably just limited >>>> here. >>>> >>>> My ideal framing for "evolving" is: an *evolving* API can make >>>> breaking API changes between versions, but this will be rare and weighed >>>> against the cost of slowing users' upgrade process. All breaking changes >>>> will be communicated in our change log. An *evolving* API will not >>>> make silent behavior changes except in exceptional cases (e.g. patching a >>>> security gap, fixing total failures of functionality). >>>> >>>> Thanks, >>>> Danny >>>> >>>> On Mon, Apr 3, 2023 at 9:02 AM Jan Lukavský <je...@seznam.cz> wrote: >>>> >>>>> Hi, >>>>> >>>>> removing @Experimental and adding explicit @Stable annotation makes >>>>> sense to me. FWIW, when we were designing Euphoria API, we adopted the >>>>> following convention: >>>>> >>>>> - the default stability of "evolving", @Experimental for really >>>>> experimental code [1] >>>>> >>>>> - target @Audience of API [2] (pipeline author, runner, internal, >>>>> test) >>>>> >>>>> - and @StateComplexity of operators (PTransforms) [3] >>>>> >>>>> The last part is something that was planned to be used by tools that >>>>> can >>>>> analyze the Pipeline for performance or visualize which transform(s) >>>>> are >>>>> most state-consuming. But this ended only as plans. :) >>>>> >>>>> Jan >>>>> >>>>> [1] >>>>> >>>>> https://github.com/apache/beam/blob/master/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/annotation/stability/Experimental.java >>>>> >>>>> [2] >>>>> >>>>> https://github.com/apache/beam/blob/master/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/annotation/audience/Audience.java >>>>> >>>>> [3] >>>>> >>>>> https://github.com/apache/beam/blob/master/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/annotation/operator/StateComplexity.java >>>>> >>>>> >>>>> On 3/31/23 23:05, Kenneth Knowles wrote: >>>>> > Hi all, >>>>> > >>>>> > Long ago, we adopted two annotations in Beam to communicate to users: >>>>> > >>>>> > - `@Experimental` indicates that an API might change >>>>> > - `@Internal` indicates that an API is not meant for users. >>>>> > >>>>> > I've seen some real problems with this approach: >>>>> > >>>>> > - Users are afraid to use `@Experimental` APIs, because they are >>>>> > worried they are not production-ready. But it really just means they >>>>> > might change, and has nothing to do with that. >>>>> > - People write new code and do not put `@Experimental` annotations >>>>> on >>>>> > it, even though it really should be able to change for a while, so >>>>> we >>>>> > can do a good job. >>>>> > - I'm seeing a culture of being afraid to change things, even when >>>>> it >>>>> > would be good for users, because our API surface area is far too >>>>> large >>>>> > and not explicitly chosen. >>>>> > - `@Internal` is not that well-known. And now we have many target >>>>> > audiences: Beam devs, PTransform devs, tool devs, pipeline authors. >>>>> > Some of them probably want to use `@Internal` stuff! >>>>> > >>>>> > I looked at a couple sibling projects and what they have >>>>> > - Flink: >>>>> > - Spark: >>>>> > >>>>> > They have many more tags, and some of them seem to have reverse >>>>> > defaults to Beam. >>>>> > >>>>> > Flink: >>>>> > >>>>> https://github.com/apache/flink/tree/master/flink-annotations/src/main/java/org/apache/flink/annotation >>>>> > >>>>> > - Experimental >>>>> > - Internal.java >>>>> > - Public >>>>> > - PublicEvolving >>>>> > - VisibleForTesting >>>>> > >>>>> > Spark: >>>>> > >>>>> https://github.com/apache/spark/tree/master/common/tags/src/main/java/org/apache/spark/annotation >>>>> and >>>>> >>>>> > >>>>> https://github.com/apache/spark/tree/master/common/tags/src/main/scala/org/apache/spark/annotation >>>>> > >>>>> > - AlphaComponent >>>>> > - DeveloperApi >>>>> > - Evolving >>>>> > - Experimental >>>>> > - Private >>>>> > - Stable >>>>> > - Unstable >>>>> > - Since >>>>> > >>>>> > I think it would help users to understand Beam with some simple, >>>>> > though possibly large-scale changes. My goal would be: >>>>> > >>>>> > - new code is changeable/evolving by default (so we don't have to >>>>> > always remember to annotate it) but users have confidence they can >>>>> use >>>>> > it in production (because we have good software engineering >>>>> practices) >>>>> > - Experimental would be reserved for more risky things >>>>> > - after we are confident an API is stable, because it has been the >>>>> > same across a couple releases, we mark it >>>>> > >>>>> > A concrete proposal to achieve this would be: >>>>> > >>>>> > - Add a @Stable annotation and use it as appropriate on our primary >>>>> APIs >>>>> > - [Possibly] add an @Evolving annotation that would also be the >>>>> default. >>>>> > - Remove most `@Experimental` annotations or change them to >>>>> `@Evolving` >>>>> > - Communicate about this (somehow). If possible, surface the >>>>> > `@Evolving` default in documentation. >>>>> > >>>>> > The last bit is the hardest. >>>>> > >>>>> > Kenn >>>>> >>>> >>>
