On Tue, Feb 12, 2019 at 8:59 AM Thomas Weise <t...@apache.org> wrote:
> The Java ULR initially provided some value for the portability effort as > Max mentions. It helped to develop the shared library for all Java runners > and the job server functionality. > > However, I think the same could have been accomplished by developing the > Flink runner instead of the Java ULR from the get go. This is also what > happened later last year when support for state, timers and metrics was > added to the portable Flink runner first and the ULR still does not support > those features [1]. > > Since all (or most) Java based runners that are based on another ASF > project support embedded execution, I think it might make sense to > discontinue separate direct runners for Java and instead focus efforts on > making the runners that folks would also use in production better? > Caveat: if people only test using embedded execution of a production runner, they are quite likely to depend on quirks of that runner, such as bundle size, fusion, whether shuffle is also checkpoint, etc. I think there's a lot of value in an antagonistic testing runner, which is something the Java DirectRunner tried to do with GBK random ordering, checking illegal mutations, checking encodability. These were all driven by real user needs and each caught a lot of user bugs. That said, I wouldn't want to maintain an extra runner, but would like to put these into a portable runner, whichever it is. Kenn > > As for Python (and hopefully soon Go), it makes a lot of sense to have a > simple to use and stable runner that can be used for local development. At > the moment, the Py FnApiRunner seems the best candidate to serve as > reference for portability. > > On a related note, we should probably also consider making pure Java > pipeline execution via portability framework on a Java runner simpler and > more efficient. We already use embedded environment for testing. If we also > inline/embed the job server and this becomes readily available and easy to > use, it might improve chances of other runners migrating to portability > sooner. > > Thomas > > [1] https://s.apache.org/apache-beam-portability-support-table > > > > On Tue, Feb 12, 2019 at 3:34 AM Maximilian Michels <m...@apache.org> wrote: > >> Do you consider job submission and artifact staging part of the >> ReferenceRunner? If so, these parts have been reused or served as a >> model for the portable FlinkRunner. So they had some value. >> >> A reference implementation helps Runner authors to understand and reuse >> the code. However, I agree that the Flink implementation is more helpful >> to Runners authors than a ReferenceRunner which was designed for single >> node testing. >> >> I think there are three parts which help to push forward portability: >> >> 1) Good library support for new portable Runners (Java) >> 2) A reference implementation of a distributed Runner (Flink) >> 3) An easy way for users to run/test portable Pipelines (Python via >> FnApiRunner) >> >> The main motivation for the portability layer is supporting additional >> language to Java. Most users will be using Python, so focusing on a good >> reference Runner in Python is key. >> >> -Max >> >> On 12.02.19 10:11, Robert Bradshaw wrote: >> > This is certainly an interesting question, and I definitely have my >> > opinions, but am curious as to what others think as well. >> > >> > One thing that I think wasn't as clear from the outset is >> distinguishing >> > between the development of runners/core-java and development of a Java >> > reference runner itself. With the work on work on moving Flink to >> > portability, it turned out that work on the latter was not a >> > prerequisite for work on the former, and runners/core-java is the >> > artifact that other runners want to build on. I think that it is also >> > the case, as suggested, that a distributed runner's use of this shared >> > library is a better reference point (for other distributed runners) >> than >> > one using the direct runner (e.g. there is a much more obvious >> > delineation between the runner's responsibility and Beam code than in >> > the direct runner where the boundaries between orchestration, >> execution, >> > and other concerns are not as clear). >> > >> > As well as serving as a reference to runner implementers, the reference >> > runner can also be useful for prototyping (here I think Python holds an >> > advantage, but we're getting into subjective areas now), documenting >> (or >> > ideally augmenting the documentation of) the spec (here I'd say a >> > smaller advantage to Python, but neither runner clean, straightforward, >> > and documented enough to serve this purpose well yet), and serving as a >> > lightweight universal local runner against which to develop (and, >> > possibly use long term in place of a direct runner) new SDKs (here >> > you'll get a wide variety of answers whether Python or Java is easier >> to >> > take on as a dependency for a third language, or we could just package >> > it up in a docker image and take docker as a dependency). >> > >> > Another more pragmatic note is that one thing that helped both the >> Flink >> > and FnApiRunner forwards is that they were driven forward by actual >> > usecases--Lyft has actual Python (necessitating portable) pipelines >> they >> > want to run on Flink, and the FnApiRunner is the direct runner for >> > Python. The Java ULR (at least where it is now) sits in an awkward >> place >> > where its only role is to be a reference rather than be used, which (in >> > a world of limited resources) makes it harder to justify investment. >> > >> > - Robert >> > >> > >> > >> > On Tue, Feb 12, 2019 at 3:53 AM Kenneth Knowles <k...@apache.org >> > <mailto:k...@apache.org>> wrote: >> > >> > Interesting silence here. You've got it right that the reason we >> > initially chose Java was because of the cross-runner sharing. The >> > reference runner could be the first target runner for any new >> > feature and then its work could be directly (or indirectly via >> > copy/paste/modify if it works better) be used in other runners. >> > Examples: >> > >> > - The implementations of (pre-portability) state & timers in >> > runners/core-java and prototyped in the Java DirectRunner made it a >> > matter of a couple of days to implement on other runners, and they >> > saw pretty quick adoption. >> > - Probably the same could be said for the first drafts of the >> > runners, which re-used a bunch of runners/core-java and had each >> > others' translation code as a reference. >> > >> > I'm interested if anyone would be willing to confirm if it is >> > because the FlinkRunner has forged ahead and the Dataflow worker is >> > open source. It makes sense that the code from a distributed runner >> > is an even better reference point if you are building another >> > distributed runner. From the look of it, the SamzaRunner had no >> > trouble getting started on portability. >> > >> > Kenn >> > >> > On Mon, Feb 11, 2019 at 6:04 PM Daniel Oliveira >> > <danolive...@google.com <mailto:danolive...@google.com>> wrote: >> > >> > Yeah, the FnApiRunner is what I'm leaning towards too. I wasn't >> > sure how much demand there was for an actual reference >> > implementation in Java though, so I was hoping there were runner >> > authors that would want to chime in. >> > >> > On the other hand, the Flink runner could serve as a reference >> > implementation for portable features since it's further along, >> > so maybe it's not an issue regardless. >> > >> > On Mon, Feb 11, 2019 at 1:09 PM Sam Rohde <sro...@google.com >> > <mailto:sro...@google.com>> wrote: >> > >> > Thanks for starting this thread. If I had to guess, I would >> > say there is more of a demand for Python as it's more widely >> > used for data scientists/ analytics. Being pragmatic, the >> > FnApiRunner already has more feature work than the Java so >> > we should go with that. >> > >> > -Sam >> > >> > On Fri, Feb 8, 2019 at 10:07 AM Daniel Oliveira >> > <danolive...@google.com <mailto:danolive...@google.com>> >> wrote: >> > >> > Hello Beam dev community, >> > >> > For those who don't know me, I work for Google and I've >> > been working on the Java reference runner, which is a >> > portable, local Java runner (it's basically the direct >> > runner with the portability APIs implemented). Our goal >> > in working on this was to have a portable runner which >> > ran locally so it could be used by users for testing >> > portable pipelines, devs for testing new features with >> > portability, and for runner authors to provide a simple >> > reference implementation of a portable runner. >> > >> > Due to various circumstances though, progress on the >> > Java reference runner has been pretty slow, and a Python >> > runner which does pretty much the same things was made >> > to aid portability development in Python (called the >> > FnApiRunner). This runner is currently further along in >> > feature work than the Java reference runner, so we've >> > been reevaluating if we should switch to investing in it >> > instead. >> > >> > My question to the community is: Which runner do you >> > think would be more valuable to the dev community and >> > Beam users? For those of you who are runner authors, do >> > you have a preference for what language you'd like to >> > see a reference implementation in? >> > >> > Thanks, >> > Daniel Oliveira >> > >> >
