Ben, Looks like we hit the send button at the same time. Is the plan the to derive the Flink implementation of the various execution services from those under org.apache.beam.runners.fnexecution ?
Thanks On Wed, Mar 7, 2018 at 11:02 AM, Thomas Weise <t...@apache.org> wrote: > What's the plan for the endpoints that the Flink operator needs to provide > (control/data plane, state, logging)? Is the intention to provide base > implementations that can be shared across runners and then implement the > Flink specific parts on top of it? Has work started on those? > > If there are subtasks ready to be taken up I would be interested. > > Thanks, > Thomas > > > On Wed, Mar 7, 2018 at 9:35 AM, Ben Sidhom <sid...@google.com> wrote: > >> Yes, Axel has started work on such a shim. >> >> Our plan in the short term is to keep the old FlinkRunner around and to >> call into it to process jobs from the job service itself. That way we can >> keep the non-portable runner fully-functional while working on portability. >> Eventually, I think it makes sense for this to go away, but we haven't >> given much thought to that. The translator layer will likely stay the same, >> and the FlinkRunner bits are a relatively simple wrapper around >> translation, so it should be simple enough to factor this out. >> >> Much of the service code from the Universal Local Runner (ULR) should be >> composed and reused with other runner implementations. Thomas and Axel have >> more context around that. >> >> >> On Wed, Mar 7, 2018 at 8:47 AM Aljoscha Krettek <aljos...@apache.org> >> wrote: >> >>> Hi, >>> >>> Has anyone started on https://issues.apache.org/jira/browse/BEAM-2588 >>> (FlinkRunner shim for serving Job API). If not I would start on that. >>> >>> My plan is to implement a FlinkJobService that implements >>> JobServiceImplBase, >>> similar to ReferenceRunnerJobService. This would have a lot of the >>> functionality that FlinkRunner currently has. As a next step, I would add a >>> JobServiceRunner that can submit Pipelines to a JobService. >>> >>> For testing, I would probably add functionality that allows spinning up >>> a JobService in-process with the JobServiceRunner. I can imagine for >>> testing we could even eventually use something like: >>> "--runner=JobServiceRunner", "--streaming=true", >>> "--jobService=FlinkRunnerJobService". >>> >>> Once all of this is done, we only need the python component that talks >>> to the JobService to submit a pipeline. >>> >>> What do you think about the plan? >>> >>> Btw, I feel that the thing currently called Runner, i.e. FlinkRunner >>> will go way in the long run and we will have FlinkJobService, >>> SparkJobService and whatnot, what do you think? >>> >>> Aljoscha >>> >>> >>> On 9. Feb 2018, at 01:31, Ben Sidhom <sid...@google.com> wrote: >>> >>> Hey all, >>> >>> We're working on getting the portability framework plumbed through the >>> Flink runner. The first iteration will likely only support batch and will >>> be limited in its deployment flexibility, but hopefully it shouldn't be too >>> painful to expand this. >>> >>> We have the start of a tracking doc here: https://s.apache.org/por >>> table-beam-on-flink. >>> >>> We've documented the general deployment strategy here: >>> https://s.apache.org/portable-flink-runner-overview. >>> >>> Feel free to provide comments on the docs or jump in on any of the >>> referenced bugs. >>> >>> -- >>> -Ben >>> >>> >>> >> >> -- >> -Ben >> > >
