Hi,
removing @Experimental and adding explicit @Stable annotation makes
sense to me. FWIW, when we were designing Euphoria API, we adopted the
following convention:
- the default stability of "evolving", @Experimental for really
experimental code [1]
- target @Audience of API [2] (pipeline author, runner, internal, test)
- and @StateComplexity of operators (PTransforms) [3]
The last part is something that was planned to be used by tools that can
analyze the Pipeline for performance or visualize which transform(s) are
most state-consuming. But this ended only as plans. :)
Jan
[1]
https://github.com/apache/beam/blob/master/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/annotation/stability/Experimental.java
[2]
https://github.com/apache/beam/blob/master/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/annotation/audience/Audience.java
[3]
https://github.com/apache/beam/blob/master/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/annotation/operator/StateComplexity.java
On 3/31/23 23:05, Kenneth Knowles wrote:
Hi all,
Long ago, we adopted two annotations in Beam to communicate to users:
- `@Experimental` indicates that an API might change
- `@Internal` indicates that an API is not meant for users.
I've seen some real problems with this approach:
- Users are afraid to use `@Experimental` APIs, because they are
worried they are not production-ready. But it really just means they
might change, and has nothing to do with that.
- People write new code and do not put `@Experimental` annotations on
it, even though it really should be able to change for a while, so we
can do a good job.
- I'm seeing a culture of being afraid to change things, even when it
would be good for users, because our API surface area is far too large
and not explicitly chosen.
- `@Internal` is not that well-known. And now we have many target
audiences: Beam devs, PTransform devs, tool devs, pipeline authors.
Some of them probably want to use `@Internal` stuff!
I looked at a couple sibling projects and what they have
- Flink:
- Spark:
They have many more tags, and some of them seem to have reverse
defaults to Beam.
Flink:
https://github.com/apache/flink/tree/master/flink-annotations/src/main/java/org/apache/flink/annotation
- Experimental
- Internal.java
- Public
- PublicEvolving
- VisibleForTesting
Spark:
https://github.com/apache/spark/tree/master/common/tags/src/main/java/org/apache/spark/annotation and
https://github.com/apache/spark/tree/master/common/tags/src/main/scala/org/apache/spark/annotation
- AlphaComponent
- DeveloperApi
- Evolving
- Experimental
- Private
- Stable
- Unstable
- Since
I think it would help users to understand Beam with some simple,
though possibly large-scale changes. My goal would be:
- new code is changeable/evolving by default (so we don't have to
always remember to annotate it) but users have confidence they can use
it in production (because we have good software engineering practices)
- Experimental would be reserved for more risky things
- after we are confident an API is stable, because it has been the
same across a couple releases, we mark it
A concrete proposal to achieve this would be:
- Add a @Stable annotation and use it as appropriate on our primary APIs
- [Possibly] add an @Evolving annotation that would also be the default.
- Remove most `@Experimental` annotations or change them to `@Evolving`
- Communicate about this (somehow). If possible, surface the
`@Evolving` default in documentation.
The last bit is the hardest.
Kenn
