Cool, so we had the same ideas. I think this indicates that we're not completely on the wrong track with this! ;-)
Aljoscha > On 7. Mar 2018, at 21:14, Thomas Weise <t...@apache.org> wrote: > > Ben, > > Looks like we hit the send button at the same time. Is the plan the to derive > the Flink implementation of the various execution services from those under > org.apache.beam.runners.fnexecution ? > > Thanks > > On Wed, Mar 7, 2018 at 11:02 AM, Thomas Weise <t...@apache.org > <mailto:t...@apache.org>> wrote: > What's the plan for the endpoints that the Flink operator needs to provide > (control/data plane, state, logging)? Is the intention to provide base > implementations that can be shared across runners and then implement the > Flink specific parts on top of it? Has work started on those? > > If there are subtasks ready to be taken up I would be interested. > > Thanks, > Thomas > > > On Wed, Mar 7, 2018 at 9:35 AM, Ben Sidhom <sid...@google.com > <mailto:sid...@google.com>> wrote: > Yes, Axel has started work on such a shim. > > Our plan in the short term is to keep the old FlinkRunner around and to call > into it to process jobs from the job service itself. That way we can keep the > non-portable runner fully-functional while working on portability. > Eventually, I think it makes sense for this to go away, but we haven't given > much thought to that. The translator layer will likely stay the same, and the > FlinkRunner bits are a relatively simple wrapper around translation, so it > should be simple enough to factor this out. > > Much of the service code from the Universal Local Runner (ULR) should be > composed and reused with other runner implementations. Thomas and Axel have > more context around that. > > > On Wed, Mar 7, 2018 at 8:47 AM Aljoscha Krettek <aljos...@apache.org > <mailto:aljos...@apache.org>> wrote: > Hi, > > Has anyone started on https://issues.apache.org/jira/browse/BEAM-2588 > <https://issues.apache.org/jira/browse/BEAM-2588> (FlinkRunner shim for > serving Job API). If not I would start on that. > > My plan is to implement a FlinkJobService that implements JobServiceImplBase, > similar to ReferenceRunnerJobService. This would have a lot of the > functionality that FlinkRunner currently has. As a next step, I would add a > JobServiceRunner that can submit Pipelines to a JobService. > > For testing, I would probably add functionality that allows spinning up a > JobService in-process with the JobServiceRunner. I can imagine for testing we > could even eventually use something like: > "--runner=JobServiceRunner", "--streaming=true", > "--jobService=FlinkRunnerJobService". > > Once all of this is done, we only need the python component that talks to the > JobService to submit a pipeline. > > What do you think about the plan? > > Btw, I feel that the thing currently called Runner, i.e. FlinkRunner will go > way in the long run and we will have FlinkJobService, SparkJobService and > whatnot, what do you think? > > Aljoscha > > >> On 9. Feb 2018, at 01:31, Ben Sidhom <sid...@google.com >> <mailto:sid...@google.com>> wrote: >> >> Hey all, >> >> We're working on getting the portability framework plumbed through the Flink >> runner. The first iteration will likely only support batch and will be >> limited in its deployment flexibility, but hopefully it shouldn't be too >> painful to expand this. >> >> We have the start of a tracking doc here: >> https://s.apache.org/portable-beam-on-flink >> <https://s.apache.org/portable-beam-on-flink>. >> >> We've documented the general deployment strategy here: >> https://s.apache.org/portable-flink-runner-overview >> <https://s.apache.org/portable-flink-runner-overview>. >> >> Feel free to provide comments on the docs or jump in on any of the >> referenced bugs. >> >> -- >> -Ben > > > > -- > -Ben > >
