Open question: did you think to a way to run the portable api on top of any runner to implement it once? Since runners have primitive it should be doable and avoid a per runner codebase, no? Other benefit: no direct portable api code in runners, yeah :). (Im thinking to a runner decorator or orchestrator or previsitor which rewrites the pipeline before the actual translations)
Le 7 mars 2018 18:35, "Ben Sidhom" <[email protected]> a écrit : > Yes, Axel has started work on such a shim. > > Our plan in the short term is to keep the old FlinkRunner around and to > call into it to process jobs from the job service itself. That way we can > keep the non-portable runner fully-functional while working on portability. > Eventually, I think it makes sense for this to go away, but we haven't > given much thought to that. The translator layer will likely stay the same, > and the FlinkRunner bits are a relatively simple wrapper around > translation, so it should be simple enough to factor this out. > > Much of the service code from the Universal Local Runner (ULR) should be > composed and reused with other runner implementations. Thomas and Axel have > more context around that. > > > On Wed, Mar 7, 2018 at 8:47 AM Aljoscha Krettek <[email protected]> > wrote: > >> Hi, >> >> Has anyone started on https://issues.apache.org/jira/browse/BEAM-2588 >> (FlinkRunner >> shim for serving Job API). If not I would start on that. >> >> My plan is to implement a FlinkJobService that implements JobServiceImplBase, >> similar to ReferenceRunnerJobService. This would have a lot of the >> functionality that FlinkRunner currently has. As a next step, I would add a >> JobServiceRunner that can submit Pipelines to a JobService. >> >> For testing, I would probably add functionality that allows spinning up a >> JobService in-process with the JobServiceRunner. I can imagine for testing >> we could even eventually use something like: >> "--runner=JobServiceRunner", "--streaming=true", "--jobService= >> FlinkRunnerJobService". >> >> Once all of this is done, we only need the python component that talks to >> the JobService to submit a pipeline. >> >> What do you think about the plan? >> >> Btw, I feel that the thing currently called Runner, i.e. FlinkRunner will >> go way in the long run and we will have FlinkJobService, SparkJobService >> and whatnot, what do you think? >> >> Aljoscha >> >> >> On 9. Feb 2018, at 01:31, Ben Sidhom <[email protected]> wrote: >> >> Hey all, >> >> We're working on getting the portability framework plumbed through the >> Flink runner. The first iteration will likely only support batch and will >> be limited in its deployment flexibility, but hopefully it shouldn't be too >> painful to expand this. >> >> We have the start of a tracking doc here: https://s.apache.org/ >> portable-beam-on-flink. >> >> We've documented the general deployment strategy here: >> https://s.apache.org/portable-flink-runner-overview. >> >> Feel free to provide comments on the docs or jump in on any of the >> referenced bugs. >> >> -- >> -Ben >> >> >> > > -- > -Ben >
