On Thu, Sep 13, 2018 at 9:49 AM Maximilian Michels <m...@apache.org
<mailto:m...@apache.org>> wrote:
Thank you for your comments. Let me try to summarize what has been
discussed so far:
1. The Beam LTS version will ensure a stable execution engine for as
long as the LTS life span.
If I understand the LTS proposal correctly, then it will be a release
line that continues to receive patches (as in semantic versioning), but
no new features as that would defeat the purpose (stability).
If so, then I don't think LTS matters for this discussion.
2. We agree that pushing updates to the execution engine for the
Runners
is only desirable if it results in a better integration with the Beam
model or if it is necessary due security or performance reasons.
3. We might have to consider adding additional build targets for a
Runner for whenever the execution engine gets upgraded. This might be
really easy if the engine's API remains stable. It might also be
desirable if the upgrade path is not easy and not completely
foreseeable, e.g. Etienne mentioned Spark 1.x vs Spark 2.x Runner. The
Beam feature set could vary depending on the version.
To limit the pain of dealing with incompatible runner changes and copies
within Beam, we should probably also work with the respective community
to improve the compatibility story.
4. In the long run, we want a stable abstraction layer for each Runner
that, ideally, is maintained by the upstream of the execution
engine. In
the short run, this is probably not realistic, as the shared libraries
of Beam are not stable enough.
Yes, that will only become an option once we reach interface stability.
Similar to how the runner projects maintain their IO connectors.
On 13.09.18 14:39, Robert Bradshaw wrote:
> The ideal long-term solution is, as Romain mentions, pushing the
> runner-specific code up to be maintained by each runner with a
stable
> API to use to talk to Beam. Unfortunately, I think we're still a
long
> way from having this Stable API, or having the clout for
> non-beam-developers to maintain these bindings externally (though
> hopefully we'll get there).
>
> In the short term, we're stuck with either hurting users that
want to
> stick with Flink 1.5, hurting users that want to upgrade to Flink
1.6,
> or supporting both. Is Beam's interaction with Flink such that we
can't
> simply have separate targets linking the same Beam code against
one or
> the other? (I.e. are code changes needed?) If so, we'll probably
need a
> flink-runner-1.5 module, a flink-runner-1.6, and a
flink-runner-common
> module. Or we hope that all users are happy with 1.5 until a certain
> point in time when they all want to simultaneously jump to 1.6
and Beam
> at the same time. Maybe that's enough in the short term, but
longer term
> we need a more sustainable solution.
>
>
> On Thu, Sep 13, 2018 at 7:13 AM Romain Manni-Bucau
> <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>
<mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>>> wrote:
>
> Hi guys,
>
> Isnt the issue "only" that beam has this code instead of engines?
>
> Assuming beam runner facing api is stable - which must be the
case
> anyway - and that each engine has its integration (flink-beam
> instead of beam-runners-flink), then this issue disappears by
> construction.
>
> It also has the advantage to have a better maintenance.
>
> Side note: this is what happent which arquillian, originally the
> community did all adapters impl then each vendor took it back in
> house to make it better.
>
> Any way to work in that direction maybe?
>
> Le jeu. 13 sept. 2018 00:49, Thomas Weise <t...@apache.org
<mailto:t...@apache.org>
> <mailto:t...@apache.org <mailto:t...@apache.org>>> a écrit :
>
> The main problem here is that users are forced to upgrade
> infrastructure to obtain new features in Beam, even when
those
> features actually don't require such changes. As an example,
> another update to Flink 1.6.0 was proposed (without
supporting
> new functionality in Beam) and we already know that it breaks
> compatibility (again).
>
> I think that upgrading to a Flink X.Y.0 version isn't a good
> idea to start with. But besides that, if we want to grow
> adoption, then we need to focus on stability and delivering
> improvements to Beam without disrupting users.
>
> In the specific case, ideally the surface of Flink would be
> backward compatible, allowing us to stick to a minimum
version
> and be able to submit pipelines to Flink endpoints of higher
> versions. Some work in that direction is underway (like
> versioning the REST API). FYI, lowest common version is what
> most projects that depend on Hadoop 2.x follow.
>
> Since Beam with Flink 1.5.x client won't talk to Flink
1.6 and
> there are code changes required to make it compile, we would
> need to come up with a more involved strategy to support
> multiple Flink versions. Till then, I would prefer we favor
> existing users over short lived experiments, which would mean
> stick with 1.5.x and not support 1.6.0.
>
> Thanks,
> Thomas
>
>
> On Wed, Sep 12, 2018 at 1:15 PM Lukasz Cwik
<lc...@google.com <mailto:lc...@google.com>
> <mailto:lc...@google.com <mailto:lc...@google.com>>> wrote:
>
> As others have already suggested, I also believe LTS
> releases is the best we can do as a community right now
> until portability allows us to decouple what a user
writes
> with and how it runs (the SDK and the SDK
environment) from
> the runner (job service + shared common runner libs +
> Flink/Spark/Dataflow/Apex/Samza/...).
>
> Dataflow would be highly invested in having the
appropriate
> tooling within Apache Beam to support multiple SDK
versions
> against a runner. This in turn would allow people to
use any
> SDK with any runner and as Robert had mentioned, certain
> optimizations and features would be disabled depending on
> the capabilities of the runner and the capabilities
of the SDK.
>
>
>
> On Wed, Sep 12, 2018 at 6:38 AM Robert Bradshaw
> <rober...@google.com <mailto:rober...@google.com>
<mailto:rober...@google.com <mailto:rober...@google.com>>> wrote:
>
> The target audience is people who want to use the
latest
> Beam but do not want to use the latest version of the
> runner, right?
>
> I think this will be somewhat (though not entirely)
> addressed by Beam LTS releases, where those not
wanting
> to upgrade the runner at least have a well-supported
> version of Beam. In the long term, we have the
division
>
> Runner <-> BeamRunnerSpecificCode <->
> CommonBeamRunnerLibs <-> SDK.
>
> (which applies to the job submission as well as
execution).
>
> Insomuch as the BeamRunnerSpecificCode uses the
public
> APIs of the runner, hopefully upgrading the
runner for
> minor versions should be a no-op, and we can
target the
> lowest version of the runner that makes sense,
allowing
> the user to link against higher versions at his
or her
> discretion. We should provide built targets that
allow
> this. For major versions, it may make sense to
have two
> distinct BeamRunnerSpecificCode libraries (which
may or
> may not share some common code). I hope these
wrappers
> are not too thick.
>
> There is a tight coupling at
the BeamRunnerSpecificCode
> <-> CommonBeamRunnerLibs layer, but hopefully the
bulk
> of the code lives on the right hand side and can be
> updated as needed independent of the runner.
There may
> be code of the form "if the runner supports X, do
this
> fast path, otherwise, do this slow path (or
reject the
> pipeline).
>
> I hope the CommonBeamRunnerLibs <-> SDK coupling is
> fairly loose, to the point that one could use
SDKs from
> different versions of Beam (or even developed
outside of
> Beam) with an older/newer runner. We may need to add
> versioning to the Fn/Runner/Job API itself to support
> this. Right now of course we're still in a pre-1.0,
> rapid-development phase wrt this API.
>
>
>
>
> On Wed, Sep 12, 2018 at 2:10 PM Etienne Chauchot
> <echauc...@apache.org
<mailto:echauc...@apache.org> <mailto:echauc...@apache.org
<mailto:echauc...@apache.org>>> wrote:
>
> Hi Max,
>
> I totally agree with your points especially the
> users priorities (stick to the already working
> version) , and the need to leverage important new
> features. It is indeed a difficult balance to
find .
>
> I can talk for a part I know: for the Spark
runner,
> the aim was to support Dataset native spark
API (in
> place of RDD). For that we needed to upgrade to
> spark 2.x (and we will probably leverage Beam
Row as
> well).
> But such an upgrade is a good amount of work
which
> makes it difficult to commit on a schedule
such as
> "if there is a major new feature on an execution
> engine that we want to leverage, then the
upgrade in
> Beam will be done within x months".
>
> Regarding your point on portability :
decoupling SDK
> from runner with runner harness and SDK harness
> might make pipeline authors work easy regarding
> pipeline maintenance. But, still, if we upgrade
> runner libs, then the users might have their
runner
> harness not work with their engine version.
> If such SDK/runner decoupling is 100% functional,
> then we could imaging having multiple runner
> harnesses shipping different versions of the
runner
> libs to solve this problem.
> But we would need to support more than one
version
> of the runner libs. We chose not to do this
on spark
> runner.
>
> WDYT ?
>
> Best
> Etienne
>
>
> Le mardi 11 septembre 2018 à 15:42 +0200,
Maximilian
> Michels a écrit :
>> Hi Beamers,
>>
>> In the light of the discussion about Beam
LTS releases, I'd like to kick
>> off a thread about how often we upgrade the
execution engine of each
>> Runner. By upgrade, I mean major/minor
versions which typically break
>> the binary compatibility of Beam pipelines.
>>
>> For the Flink Runner, we try to track the
latest stable version. Some
>> users reported that this can be problematic,
as it requires them to
>> potentially upgrade their Flink cluster with
a new version of Beam.
>>
>> From a developer's perspective, it makes
sense to migrate as early as
>> possible to the newest version of the
execution engine, e.g. to leverage
>> the newest features. From a user's
perspective, you don't care about the
>> latest features if your use case still works
with Beam.
>>
>> We have to please both parties. So I'd
suggest to upgrade the execution
>> engine whenever necessary (e.g. critical new
features, end of life of
>> current version). On the other hand, the
upcoming Beam LTS releases will
>> contain a longer-supported version.
>>
>> Maybe we don't need to discuss much about
this but I wanted to hear what
>> the community has to say about it.
Particularly, I'd be interested in
>> how the other Runner authors intend to do it.
>>
>> As far as I understand, with the portability
being stable, we could
>> theoretically upgrade the SDK without
upgrading the runtime components.
>> That would allow us to defer the upgrade for
a longer time.
>>
>> Best,
>> Max
>>
