Hi all, Long ago, we adopted two annotations in Beam to communicate to users:
- `@Experimental` indicates that an API might change - `@Internal` indicates that an API is not meant for users. I've seen some real problems with this approach: - Users are afraid to use `@Experimental` APIs, because they are worried they are not production-ready. But it really just means they might change, and has nothing to do with that. - People write new code and do not put `@Experimental` annotations on it, even though it really should be able to change for a while, so we can do a good job. - I'm seeing a culture of being afraid to change things, even when it would be good for users, because our API surface area is far too large and not explicitly chosen. - `@Internal` is not that well-known. And now we have many target audiences: Beam devs, PTransform devs, tool devs, pipeline authors. Some of them probably want to use `@Internal` stuff! I looked at a couple sibling projects and what they have - Flink: - Spark: They have many more tags, and some of them seem to have reverse defaults to Beam. Flink: https://github.com/apache/flink/tree/master/flink-annotations/src/main/java/org/apache/flink/annotation - Experimental - Internal.java - Public - PublicEvolving - VisibleForTesting Spark: https://github.com/apache/spark/tree/master/common/tags/src/main/java/org/apache/spark/annotation and https://github.com/apache/spark/tree/master/common/tags/src/main/scala/org/apache/spark/annotation - AlphaComponent - DeveloperApi - Evolving - Experimental - Private - Stable - Unstable - Since I think it would help users to understand Beam with some simple, though possibly large-scale changes. My goal would be: - new code is changeable/evolving by default (so we don't have to always remember to annotate it) but users have confidence they can use it in production (because we have good software engineering practices) - Experimental would be reserved for more risky things - after we are confident an API is stable, because it has been the same across a couple releases, we mark it A concrete proposal to achieve this would be: - Add a @Stable annotation and use it as appropriate on our primary APIs - [Possibly] add an @Evolving annotation that would also be the default. - Remove most `@Experimental` annotations or change them to `@Evolving` - Communicate about this (somehow). If possible, surface the `@Evolving` default in documentation. The last bit is the hardest. Kenn
