I've been thinking a similar thing for the Go SDK for a while, though Go doesn't have annotations in the way lJava does.
I agree that explicitly Stable/ default Evolving is probably the best way to go. Even for SDK devs, it's better the default isn't something that needs to be remembered to be added, and then removed at a later date, rather than promoting it to "this will continue to work the same way (modulo bug fixes)". The organic growth of the SDKs and their various packages don't necessarily make it clear what's intended for end users vs SDK internal use. It would be a fair amount of work to fix that in any v3, but this would be very clear work in that direction since a v3 could become a clean up and reorganization of anything that's deemed stable, while making that clear structurally for end users. On Fri, Mar 31, 2023, 2:05 PM Kenneth Knowles <k...@apache.org> wrote: > Hi all, > > Long ago, we adopted two annotations in Beam to communicate to users: > > - `@Experimental` indicates that an API might change > - `@Internal` indicates that an API is not meant for users. > > I've seen some real problems with this approach: > > - Users are afraid to use `@Experimental` APIs, because they are worried > they are not production-ready. But it really just means they might change, > and has nothing to do with that. > - People write new code and do not put `@Experimental` annotations on it, > even though it really should be able to change for a while, so we can do a > good job. > - I'm seeing a culture of being afraid to change things, even when it > would be good for users, because our API surface area is far too large and > not explicitly chosen. > - `@Internal` is not that well-known. And now we have many target > audiences: Beam devs, PTransform devs, tool devs, pipeline authors. Some of > them probably want to use `@Internal` stuff! > > I looked at a couple sibling projects and what they have > - Flink: > - Spark: > > They have many more tags, and some of them seem to have reverse defaults > to Beam. > > Flink: > https://github.com/apache/flink/tree/master/flink-annotations/src/main/java/org/apache/flink/annotation > > - Experimental > - Internal.java > - Public > - PublicEvolving > - VisibleForTesting > > Spark: > https://github.com/apache/spark/tree/master/common/tags/src/main/java/org/apache/spark/annotation > and > https://github.com/apache/spark/tree/master/common/tags/src/main/scala/org/apache/spark/annotation > > - AlphaComponent > - DeveloperApi > - Evolving > - Experimental > - Private > - Stable > - Unstable > - Since > > I think it would help users to understand Beam with some simple, though > possibly large-scale changes. My goal would be: > > - new code is changeable/evolving by default (so we don't have to always > remember to annotate it) but users have confidence they can use it in > production (because we have good software engineering practices) > - Experimental would be reserved for more risky things > - after we are confident an API is stable, because it has been the same > across a couple releases, we mark it > > A concrete proposal to achieve this would be: > > - Add a @Stable annotation and use it as appropriate on our primary APIs > - [Possibly] add an @Evolving annotation that would also be the default. > - Remove most `@Experimental` annotations or change them to `@Evolving` > - Communicate about this (somehow). If possible, surface the `@Evolving` > default in documentation. > > The last bit is the hardest. > > Kenn >
