I still think we also need an annotation to mark public interfaces as
experimental. For example for the windowing/triggers I would like to use that.
> On 25 Nov 2015, at 01:23, Robert Metzger <rmetz...@apache.org> wrote:
>
> Thank you Nick. I'll look into the check_compatiblilty.sh script to see
> which tools are used.
> I think finding breaking API changes immediately is a better process then
> reworking the APIs before a release.
>
> As you can see from my email response times (2 days since your email), I'm
> probably too overloaded right now to participate in the Yetus project ...
> Sadly.
>
>
> @others: I know its not the most interesting thing to go through my list of
> stable interfaces, but keep in mind that we have to maintain the stuff for
> probably quite some time, so it would be good to have more than one pair of
> eyes looking at it :)
>
>
> On Mon, Nov 23, 2015 at 6:20 PM, Nick Dimiduk <ndimi...@gmail.com> wrote:
>
>>>
>>> Do you know if Hadoop/HBase is also using a maven plugin to fail a build
>> on
>>> breaking API changes? I would really like to have such a functionality in
>>> Flink, because we can spot breaking changes very early.
>>
>>
>> I don't think we have maven integration for this as of yet. We release
>> managers run a script $HBASE/dev-support/check_compatibility.sh that
>> generates a source and binary compatibility report. Issues are then
>> resolved during the period leading up to the release candidate.
>>
>> I think Hadoop is relying on a "QA bot" which reads patches from JIRA and
>>> then does these
>>> checks?
>>>
>>
>> The "QA bot" is just a collection of shell scripts used during "Patch
>> Available" status when a patch has been attached to JIRA or when a PR has
>> been submitted through github. The check_compatibility script could be
>> included in that automation, I don't see why not. Maybe you'd like to open
>> a YETUS ticket :)
>>
>> I've pushed a branch to my own GitHub account with all classes I would make
>>> public annotated:
>>>
>>>
>> https://github.com/apache/flink/compare/master...rmetzger:interface_stability_revapi?expand=1
>>> Since this is really hard to read, I (half-automated) generated the
>>> following list of annotated classes:
>>>
>>>
>> https://github.com/rmetzger/flink/blob/interface_stability_revapi/annotations.md
>>>
>>> Please let me know if you would like to include or exclude classes from
>>> that list.
>>> Also, let me know which methods (in stable classes) you would mark as
>>> experimental.
>>>
>>>
>>>
>>>
>>> On Wed, Nov 11, 2015 at 10:17 AM, Aljoscha Krettek <aljos...@apache.org>
>>> wrote:
>>>
>>>> +1 for some way of declaring public interfaces as experimental.
>>>>
>>>>> On 10 Nov 2015, at 22:24, Stephan Ewen <se...@apache.org> wrote:
>>>>>
>>>>> I think we need anyways an annotation "@PublicExperimental".
>>>>>
>>>>> We can make this annotation such that it can be added to methods and
>>> can
>>>>> use that to declare
>>>>> Methods in an otherwise public class (such as DataSet) as
>> experimental.
>>>>>
>>>>> On Tue, Nov 10, 2015 at 10:19 PM, Fabian Hueske <fhue...@gmail.com>
>>>> wrote:
>>>>>
>>>>>> I am not sure if we always should declare complete classes as
>>>>>> @PublicInterface.
>>>>>> This does definitely make sense for interfaces and abstract classes
>>>> such as
>>>>>> MapFunction or InputFormat but not necessarily for classes such as
>>>> DataSet
>>>>>> that we might want to extend by methods which should not immediately
>>> be
>>>>>> considered as stable.
>>>>>>
>>>>>>
>>>>>> 2015-11-10 21:36 GMT+01:00 Vasiliki Kalavri <
>>> vasilikikala...@gmail.com
>>>>> :
>>>>>>
>>>>>>> Yes, my opinion is that we shouldn't declare the Gelly API frozen
>>> yet.
>>>>>>> We can reconsider when we're closer to the 1.0 release, but if
>>>> possible,
>>>>>> I
>>>>>>> would give it some more time.
>>>>>>>
>>>>>>> -V.
>>>>>>>
>>>>>>> On 10 November 2015 at 21:06, Stephan Ewen <se...@apache.org>
>> wrote:
>>>>>>>
>>>>>>>> I think no component should be forced to be stable. It should be
>> an
>>>>>>>> individual decision for each component, and in some cases even for
>>>>>>>> individual classes.
>>>>>>>>
>>>>>>>> @Vasia If you think Gelly should not be declared interface-frozen,
>>>> then
>>>>>>>> this is a good point to raise and this should definitely be
>>> reflected.
>>>>>>>> There is no point in declaring certain APIs as frozen when we are
>>> not
>>>>>> yet
>>>>>>>> confident they have converged.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Nov 10, 2015 at 8:39 PM, Vasiliki Kalavri <
>>>>>>>> vasilikikala...@gmail.com
>>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Robert,
>>>>>>>>>
>>>>>>>>> thanks for bringing this up!
>>>>>>>>>
>>>>>>>>> I generally like the idea, but I wouldn't rush to annotate the
>>> Gelly
>>>>>>>>> classes yet. Gelly hasn't had that many users and I'm quite sure
>>>>>> we'll
>>>>>>>> find
>>>>>>>>> things to improve as it gets more exposure.
>>>>>>>>> TBH, I think it's quite unfair to force Gelly (also e.g. ML,
>> Table)
>>>>>> to
>>>>>>> a
>>>>>>>>> "1.0" status (from an API stability point of view) since they're
>>>>>> really
>>>>>>>>> young compared to the other Flink APIs.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Vasia.
>>>>>>>>> On Nov 10, 2015 8:04 PM, "Robert Metzger" <rmetz...@apache.org>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi everyone,
>>>>>>>>>>
>>>>>>>>>> I would like to bring this discussion back to your attention as
>> we
>>>>>>> seem
>>>>>>>>> to
>>>>>>>>>> approach the 1.0 release of Flink.
>>>>>>>>>>
>>>>>>>>>> My suggestion back in January was to annotate all classes, but I
>>>>>>> think
>>>>>>>>>> it'll be more feasible to just annotate public classes.
>>>>>>>>>> So how about adding an annotation @PublicInterface
>>>>>>>>>>
>>>>>>>>>> For @PublicInterface, I would annotate classes such as: DataSet,
>>>>>>>>>> DataStream, ExecutionEnvironment, InputFormat, MapFunction,
>>>>>>> FileSystems
>>>>>>>>> but
>>>>>>>>>> also Gelly for example.
>>>>>>>>>>
>>>>>>>>>> I would not annotate as public components such as ML, Storm
>>>>>>>>> compatibility,
>>>>>>>>>> internals from runtime, yarn, optimizer.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> From a tooling perspective, I've looked into different maven
>>>>>> plugins
>>>>>>>> and
>>>>>>>>>> java libraries and I found https://github.com/siom79/japicmp to
>>> be
>>>>>>> the
>>>>>>>>>> closest to our needs. I actually opened a pull request to the
>>>>>> project
>>>>>>>> to
>>>>>>>>>> allow inclusion/exclusion of classes based on annotations. Lets
>>>>>> hope
>>>>>>> it
>>>>>>>>>> gets merged.
>>>>>>>>>>
>>>>>>>>>> Does everybody agree with adding just the @PublicInterface
>>>>>>> annotation?
>>>>>>>>>>
>>>>>>>>>> Note that I'll add the annotation on a class-level, making the
>>>>>> entire
>>>>>>>>> class
>>>>>>>>>> either public or private (from a stability point of view). If we
>>>>>>> need a
>>>>>>>>>> more fine-grained annotation, we have to add a second
>>>>>>> @PrivateInterface
>>>>>>>>>> annotation which we'll only apply to certain methods.
>>>>>>>>>>
>>>>>>>>>> The next step is that I'm going to open a pull request with all
>>>>>>> classes
>>>>>>>>>> annotated that I consider public.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Jan 30, 2015 at 7:10 PM, Henry Saputra <
>>>>>>>> henry.sapu...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I like the idea. But would love to have different name for the
>>>>>>>>>>> "LimitedPrivate" to make it easier to distinguish.
>>>>>>>>>>> How about "Module" or "Package" ?
>>>>>>>>>>>
>>>>>>>>>>> - Henry
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jan 28, 2015 at 1:29 AM, Robert Metzger <
>>>>>>> rmetz...@apache.org
>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> I think in Hadoop they use LimitedPrivate for the different
>>>>>>>>> components
>>>>>>>>>> of
>>>>>>>>>>>> the project.
>>>>>>>>>>>> For example LimitedPrivate("yarn").
>>>>>>>>>>>> Here is a very good documentation on the topic:
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/InterfaceClassification.html
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Jan 27, 2015 at 3:58 PM, Alexander Alexandrov <
>>>>>>>>>>>> alexander.s.alexand...@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I don't get the difference between Private and
>> LimitedPrivate,
>>>>>>> but
>>>>>>>>>>>>> otherwise seems like quite a nice idea.
>>>>>>>>>>>>>
>>>>>>>>>>>>> It will be also good if we can agree upon what these tags
>>>>>>> actually
>>>>>>>>>> mean
>>>>>>>>>>> and
>>>>>>>>>>>>> add this meaning to the documentation.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2015-01-27 15:46 GMT+01:00 Robert Metzger <
>>>>>> rmetz...@apache.org
>>>>>>>> :
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hadoop has annotations for tagging the stability and
>>>>>> audience
>>>>>>> of
>>>>>>>>>>> classes
>>>>>>>>>>>>>> and methods.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Through that, you can have @InterfaceAudience.Public,
>>>>>> Private,
>>>>>>>>>>>>>> LimitedPrivate
>>>>>>>>>>>>>> and also @InterfaceStability.Evolving, Unstable, and Stable.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I guess there are tools which allow to automatically check
>>>>>> if
>>>>>>>>>>> interfaces,
>>>>>>>>>>>>>> which are marked as Stable have been changed between
>>>>>> releases
>>>>>>>> (or
>>>>>>>>> by
>>>>>>>>>>> pull
>>>>>>>>>>>>>> requests).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think such annotations are crucial if we want to guarantee
>>>>>>>>>> interface
>>>>>>>>>>>>>> stability between releases.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What do you think? Should we add those annotations? Which
>>>>>> one
>>>>>>>>> would
>>>>>>>>>>> you
>>>>>>>>>>>>>> like to add?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Robert
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>
