[Copying this also to the dev list]
+1. A version compatibility table would be great!
> I don't know if Flink could do something like this (become a provided
> dep) in particular for the current case where there seems not to be
> API breaking changes.
That doesn't work. The Flink Runner is too tightly integrated with Flink
internals, and these internals are not always optimally decoupled. This
fails already at the client side, e.g. when submitting a Flink job via
Beam to a Flink cluster. Though it should be better now with the new
Rest-based clients.
On 17.09.18 09:48, Ismaël Mejía wrote:
In the Spark runner the user provides the core spark dependencies at runtime and
we assume that backwards compatibility is kept (in upstream Spark). We support
the whole 2.x line but we try to keep the version close to the latest stable
release.
Notice however that we lack tests to validate that all versions do work, I
remember some issues with metrics during the migration to spark 2.x with older
versions of spark (<= 2.1). Those worked flawlessly with more recent versions.
I don't know if Flink could do something like this (become a provided
dep) in particular for the current case where there seems not to be
API breaking changes.
In any case +1 to try to get a bit the act together on this.
On Mon, Sep 17, 2018 at 9:31 AM Robert Bradshaw <rober...@google.com> wrote:
On Mon, Sep 17, 2018 at 2:02 AM Austin Bennett <whatwouldausti...@gmail.com>
wrote:
Do we currently maintain a finer grained list of compatibility between
execution/runner versions and beam versions? Is this only really a concern
with recent Flink (sounded like at least Spark jump, too)? I see the
capability matrix:
https://beam.apache.org/documentation/runners/capability-matrix/, but some sort
of compatibility between runner versions with beam releases might be useful.
I see compatibility matrix as far as beam features, but not for underlying
runners. Ex: something like this would save a user trying to get Beam working
on recent Flink 1.6 and then subsequently hitting a (potentially not well
documented) wall given known issues.
+1. I was bitten by this as well.
I don't know if it's worth having a compatibility matrix for each version (as
the overlap is likely to be all or nothing in most cases), but it should be
prominently displayed here and elsewhere. Want to send out a PR?
