It would be good to engage with the Flink community and attempt to find stable API that the runner can depend on.
Thanks, Thomas On Mon, Sep 17, 2018 at 6:31 AM Maximilian Michels <m...@apache.org> wrote: > [Copying this also to the dev list] > > +1. A version compatibility table would be great! > > > I don't know if Flink could do something like this (become a provided > > dep) in particular for the current case where there seems not to be > > API breaking changes. > > That doesn't work. The Flink Runner is too tightly integrated with Flink > internals, and these internals are not always optimally decoupled. This > fails already at the client side, e.g. when submitting a Flink job via > Beam to a Flink cluster. Though it should be better now with the new > Rest-based clients. > > On 17.09.18 09:48, Ismaël Mejía wrote: > > In the Spark runner the user provides the core spark dependencies at > runtime and > > we assume that backwards compatibility is kept (in upstream Spark). We > support > > the whole 2.x line but we try to keep the version close to the latest > stable > > release. > > > > Notice however that we lack tests to validate that all versions do work, > I > > remember some issues with metrics during the migration to spark 2.x with > older > > versions of spark (<= 2.1). Those worked flawlessly with more recent > versions. > > > > I don't know if Flink could do something like this (become a provided > > dep) in particular for the current case where there seems not to be > > API breaking changes. > > > > In any case +1 to try to get a bit the act together on this. > > > > On Mon, Sep 17, 2018 at 9:31 AM Robert Bradshaw <rober...@google.com> > wrote: > >> > >> On Mon, Sep 17, 2018 at 2:02 AM Austin Bennett < > whatwouldausti...@gmail.com> wrote: > >>> > >>> Do we currently maintain a finer grained list of compatibility between > execution/runner versions and beam versions? Is this only really a concern > with recent Flink (sounded like at least Spark jump, too)? I see the > capability matrix: > https://beam.apache.org/documentation/runners/capability-matrix/, but > some sort of compatibility between runner versions with beam releases might > be useful. > >>> > >>> I see compatibility matrix as far as beam features, but not for > underlying runners. Ex: something like this would save a user trying to > get Beam working on recent Flink 1.6 and then subsequently hitting a > (potentially not well documented) wall given known issues. > >> > >> > >> +1. I was bitten by this as well. > >> > >> I don't know if it's worth having a compatibility matrix for each > version (as the overlap is likely to be all or nothing in most cases), but > it should be prominently displayed here and elsewhere. Want to send out a > PR? >
