Do we currently maintain a finer grained list of compatibility between execution/runner versions and beam versions? Is this only really a concern with recent Flink (sounded like at least Spark jump, too)? I see the capability matrix: https://beam.apache.org/documentation/runners/capability-matrix/, but some sort of compatibility between runner versions with beam releases might be useful.
I see compatibility matrix as far as beam features, but not for underlying runners. Ex: something like this would save a user trying to get Beam working on recent Flink 1.6 and then subsequently hitting a (potentially not well documented) wall given known issues. On Sun, Sep 16, 2018 at 3:59 AM Maximilian Michels <m...@apache.org> wrote: > > If I understand the LTS proposal correctly, then it will be a release > line that continues to receive patches (as in semantic versioning), but no > new features as that would defeat the purpose (stability). > > It matters insofar, as execution engine upgrades could be performed in > the master but the LTS version won't receive them. So LTS is the go-to > if you want to ensure compatibility with your existing setup. > > > To limit the pain of dealing with incompatible runner changes and copies > within Beam, we should probably also work with the respective community to > improve the compatibility story. > > Absolutely. If we find that we can improve compatibility with upstream > changes, we should go that path. Even if we don't have a dedicated > compatibility layer upstream yet. > > On 13.09.18 19:34, Thomas Weise wrote: > > > > On Thu, Sep 13, 2018 at 9:49 AM Maximilian Michels <m...@apache.org > > <mailto:m...@apache.org>> wrote: > > > > Thank you for your comments. Let me try to summarize what has been > > discussed so far: > > > > 1. The Beam LTS version will ensure a stable execution engine for as > > long as the LTS life span. > > > > > > If I understand the LTS proposal correctly, then it will be a release > > line that continues to receive patches (as in semantic versioning), but > > no new features as that would defeat the purpose (stability). > > > > If so, then I don't think LTS matters for this discussion. > > > > 2. We agree that pushing updates to the execution engine for the > > Runners > > is only desirable if it results in a better integration with the Beam > > model or if it is necessary due security or performance reasons. > > > > 3. We might have to consider adding additional build targets for a > > Runner for whenever the execution engine gets upgraded. This might be > > really easy if the engine's API remains stable. It might also be > > desirable if the upgrade path is not easy and not completely > > foreseeable, e.g. Etienne mentioned Spark 1.x vs Spark 2.x Runner. > The > > Beam feature set could vary depending on the version. > > > > > > To limit the pain of dealing with incompatible runner changes and copies > > within Beam, we should probably also work with the respective community > > to improve the compatibility story. > > > > > > 4. In the long run, we want a stable abstraction layer for each > Runner > > that, ideally, is maintained by the upstream of the execution > > engine. In > > the short run, this is probably not realistic, as the shared > libraries > > of Beam are not stable enough. > > > > > > Yes, that will only become an option once we reach interface stability. > > Similar to how the runner projects maintain their IO connectors. > > > > On 13.09.18 14:39, Robert Bradshaw wrote: > > > The ideal long-term solution is, as Romain mentions, pushing the > > > runner-specific code up to be maintained by each runner with a > > stable > > > API to use to talk to Beam. Unfortunately, I think we're still a > > long > > > way from having this Stable API, or having the clout for > > > non-beam-developers to maintain these bindings externally (though > > > hopefully we'll get there). > > > > > > In the short term, we're stuck with either hurting users that > > want to > > > stick with Flink 1.5, hurting users that want to upgrade to Flink > > 1.6, > > > or supporting both. Is Beam's interaction with Flink such that we > > can't > > > simply have separate targets linking the same Beam code against > > one or > > > the other? (I.e. are code changes needed?) If so, we'll probably > > need a > > > flink-runner-1.5 module, a flink-runner-1.6, and a > > flink-runner-common > > > module. Or we hope that all users are happy with 1.5 until a > certain > > > point in time when they all want to simultaneously jump to 1.6 > > and Beam > > > at the same time. Maybe that's enough in the short term, but > > longer term > > > we need a more sustainable solution. > > > > > > > > > On Thu, Sep 13, 2018 at 7:13 AM Romain Manni-Bucau > > > <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>>> > wrote: > > > > > > Hi guys, > > > > > > Isnt the issue "only" that beam has this code instead of > engines? > > > > > > Assuming beam runner facing api is stable - which must be the > > case > > > anyway - and that each engine has its integration (flink-beam > > > instead of beam-runners-flink), then this issue disappears by > > > construction. > > > > > > It also has the advantage to have a better maintenance. > > > > > > Side note: this is what happent which arquillian, originally > the > > > community did all adapters impl then each vendor took it back > in > > > house to make it better. > > > > > > Any way to work in that direction maybe? > > > > > > Le jeu. 13 sept. 2018 00:49, Thomas Weise <t...@apache.org > > <mailto:t...@apache.org> > > > <mailto:t...@apache.org <mailto:t...@apache.org>>> a écrit : > > > > > > The main problem here is that users are forced to upgrade > > > infrastructure to obtain new features in Beam, even when > > those > > > features actually don't require such changes. As an > example, > > > another update to Flink 1.6.0 was proposed (without > > supporting > > > new functionality in Beam) and we already know that it > breaks > > > compatibility (again). > > > > > > I think that upgrading to a Flink X.Y.0 version isn't a > good > > > idea to start with. But besides that, if we want to grow > > > adoption, then we need to focus on stability and > delivering > > > improvements to Beam without disrupting users. > > > > > > In the specific case, ideally the surface of Flink would > be > > > backward compatible, allowing us to stick to a minimum > > version > > > and be able to submit pipelines to Flink endpoints of > higher > > > versions. Some work in that direction is underway (like > > > versioning the REST API). FYI, lowest common version is > what > > > most projects that depend on Hadoop 2.x follow. > > > > > > Since Beam with Flink 1.5.x client won't talk to Flink > > 1.6 and > > > there are code changes required to make it compile, we > would > > > need to come up with a more involved strategy to support > > > multiple Flink versions. Till then, I would prefer we > favor > > > existing users over short lived experiments, which would > mean > > > stick with 1.5.x and not support 1.6.0. > > > > > > Thanks, > > > Thomas > > > > > > > > > On Wed, Sep 12, 2018 at 1:15 PM Lukasz Cwik > > <lc...@google.com <mailto:lc...@google.com> > > > <mailto:lc...@google.com <mailto:lc...@google.com>>> > wrote: > > > > > > As others have already suggested, I also believe LTS > > > releases is the best we can do as a community right > now > > > until portability allows us to decouple what a user > > writes > > > with and how it runs (the SDK and the SDK > > environment) from > > > the runner (job service + shared common runner libs + > > > Flink/Spark/Dataflow/Apex/Samza/...). > > > > > > Dataflow would be highly invested in having the > > appropriate > > > tooling within Apache Beam to support multiple SDK > > versions > > > against a runner. This in turn would allow people to > > use any > > > SDK with any runner and as Robert had mentioned, > certain > > > optimizations and features would be disabled > depending on > > > the capabilities of the runner and the capabilities > > of the SDK. > > > > > > > > > > > > On Wed, Sep 12, 2018 at 6:38 AM Robert Bradshaw > > > <rober...@google.com <mailto:rober...@google.com> > > <mailto:rober...@google.com <mailto:rober...@google.com>>> wrote: > > > > > > The target audience is people who want to use the > > latest > > > Beam but do not want to use the latest version of > the > > > runner, right? > > > > > > I think this will be somewhat (though not > entirely) > > > addressed by Beam LTS releases, where those not > > wanting > > > to upgrade the runner at least have a > well-supported > > > version of Beam. In the long term, we have the > > division > > > > > > Runner <-> BeamRunnerSpecificCode <-> > > > CommonBeamRunnerLibs <-> SDK. > > > > > > (which applies to the job submission as well as > > execution). > > > > > > Insomuch as the BeamRunnerSpecificCode uses the > > public > > > APIs of the runner, hopefully upgrading the > > runner for > > > minor versions should be a no-op, and we can > > target the > > > lowest version of the runner that makes sense, > > allowing > > > the user to link against higher versions at his > > or her > > > discretion. We should provide built targets that > > allow > > > this. For major versions, it may make sense to > > have two > > > distinct BeamRunnerSpecificCode libraries (which > > may or > > > may not share some common code). I hope these > > wrappers > > > are not too thick. > > > > > > There is a tight coupling at > > the BeamRunnerSpecificCode > > > <-> CommonBeamRunnerLibs layer, but hopefully the > > bulk > > > of the code lives on the right hand side and can > be > > > updated as needed independent of the runner. > > There may > > > be code of the form "if the runner supports X, do > > this > > > fast path, otherwise, do this slow path (or > > reject the > > > pipeline). > > > > > > I hope the CommonBeamRunnerLibs <-> SDK coupling > is > > > fairly loose, to the point that one could use > > SDKs from > > > different versions of Beam (or even developed > > outside of > > > Beam) with an older/newer runner. We may need to > add > > > versioning to the Fn/Runner/Job API itself to > support > > > this. Right now of course we're still in a > pre-1.0, > > > rapid-development phase wrt this API. > > > > > > > > > > > > > > > On Wed, Sep 12, 2018 at 2:10 PM Etienne Chauchot > > > <echauc...@apache.org > > <mailto:echauc...@apache.org> <mailto:echauc...@apache.org > > <mailto:echauc...@apache.org>>> wrote: > > > > > > Hi Max, > > > > > > I totally agree with your points especially > the > > > users priorities (stick to the already working > > > version) , and the need to leverage important > new > > > features. It is indeed a difficult balance to > > find . > > > > > > I can talk for a part I know: for the Spark > > runner, > > > the aim was to support Dataset native spark > > API (in > > > place of RDD). For that we needed to upgrade > to > > > spark 2.x (and we will probably leverage Beam > > Row as > > > well). > > > But such an upgrade is a good amount of work > > which > > > makes it difficult to commit on a schedule > > such as > > > "if there is a major new feature on an > execution > > > engine that we want to leverage, then the > > upgrade in > > > Beam will be done within x months". > > > > > > Regarding your point on portability : > > decoupling SDK > > > from runner with runner harness and SDK > harness > > > might make pipeline authors work easy > regarding > > > pipeline maintenance. But, still, if we > upgrade > > > runner libs, then the users might have their > > runner > > > harness not work with their engine version. > > > If such SDK/runner decoupling is 100% > functional, > > > then we could imaging having multiple runner > > > harnesses shipping different versions of the > > runner > > > libs to solve this problem. > > > But we would need to support more than one > > version > > > of the runner libs. We chose not to do this > > on spark > > > runner. > > > > > > WDYT ? > > > > > > Best > > > Etienne > > > > > > > > > Le mardi 11 septembre 2018 à 15:42 +0200, > > Maximilian > > > Michels a écrit : > > >> Hi Beamers, > > >> > > >> In the light of the discussion about Beam > > LTS releases, I'd like to kick > > >> off a thread about how often we upgrade the > > execution engine of each > > >> Runner. By upgrade, I mean major/minor > > versions which typically break > > >> the binary compatibility of Beam pipelines. > > >> > > >> For the Flink Runner, we try to track the > > latest stable version. Some > > >> users reported that this can be problematic, > > as it requires them to > > >> potentially upgrade their Flink cluster with > > a new version of Beam. > > >> > > >> From a developer's perspective, it makes > > sense to migrate as early as > > >> possible to the newest version of the > > execution engine, e.g. to leverage > > >> the newest features. From a user's > > perspective, you don't care about the > > >> latest features if your use case still works > > with Beam. > > >> > > >> We have to please both parties. So I'd > > suggest to upgrade the execution > > >> engine whenever necessary (e.g. critical new > > features, end of life of > > >> current version). On the other hand, the > > upcoming Beam LTS releases will > > >> contain a longer-supported version. > > >> > > >> Maybe we don't need to discuss much about > > this but I wanted to hear what > > >> the community has to say about it. > > Particularly, I'd be interested in > > >> how the other Runner authors intend to do it. > > >> > > >> As far as I understand, with the portability > > being stable, we could > > >> theoretically upgrade the SDK without > > upgrading the runtime components. > > >> That would allow us to defer the upgrade for > > a longer time. > > >> > > >> Best, > > >> Max > > >> > > >
