Thanks all for the great discussion. My take on the current status of Beam support in Nemo from the perspective of the Beam Capability Matrix[1]: - What: Full support except for SDF / Metrics / Stateful Processing (Side Inputs for streaming pipelines are not supported yet though) - Where: Probably full support - When: I think we support event-time/processing-time/count triggers, but I am not sure we have tested the other triggers. - How: I don't have a good idea on this one, as we haven't had tests for this.
I would say that we have good fundamentals to be on par with many of the existing official Beam runners. Taegeon and I are experimenting with the NEXMark benchmark, which I believe cover almost all of the Beam features, on Nemo to understand in more detail and fix the remaining issues. Regarding the governance I would also like to hear from other members. Taegeon and I are also happy to prioritize things differently depending on the plan. Thanks, John [1] Beam Capability Matrix: https://beam.apache.org/documentation/runners/capability-matrix/#cap-summary-what On Sun, Nov 11, 2018 at 1:44 PM Byung-Gon Chun <bgc...@gmail.com> wrote: > Thanks for the valuable input, Davor! > > Nemo PPMC members, which direction would you like to take? > > -Gon > > On Thu, Nov 8, 2018 at 3:06 AM Davor Bonaci <da...@apache.org> wrote: > > > I think the first decision you have to make is around the governance of > the > > runner. It can live in the Beam project (and be governed by the Beam > PMC), > > or in the Nemo project (and be governed by the Nemo PPMC). Both are > viable > > paths, and different folks have chosen different paths. > > > > On the technical side, the discussion revolves around API stability and > > versioning: (1) whether the API surface between the runner and Nemo > > internals is more stable than the API surface between the runner and > Beam's > > runner-facing APIs; and (2) how to version Beam and Nemo, which pairs can > > work together, etc. > > > > On the organizational side, delegating governance to Beam PMC would mean > > that they control release cadence, contribution process, and committer > > access. None of it is a particular issue, I think. These are reasonable > > people, but certainly requires more consensus building and process. > > > > * * * > > > > I don't want to steer you in any direction, and happy to support > whichever > > decision you make. Also, happy to help on the Beam side and make things > > happen quickly. (But, I'd love to see a meaningful discussion and > consensus > > decision before proceeding.) > > > > On Wed, Nov 7, 2018 at 1:19 AM Jean-Baptiste Onofré <j...@nanthrax.net> > > wrote: > > > > > Hi guys, > > > > > > definitely happy to help on that front. > > > > > > Regards > > > JB > > > > > > On 07/11/2018 09:54, Byung-Gon Chun wrote: > > > > Thanks for initiating this discussion, Wonwook! > > > > > > > > Davor and JB, it’d be great to get your guide. > > > > > > > > Thanks. > > > > - Gon > > > > > > > > Sent from my iPhone > > > > > > > > 2018. 11. 7. 오후 3:57, 송원욱 <won...@apache.org> 작성: > > > > > > > >> Hi all, > > > >> > > > >> It's nice to hear that the first release is coming up pretty soon > with > > > the > > > >> progress that we are making! > > > >> With the first release and the current development for supporting > > stream > > > >> processing, I think it's time for us to consider sending a request > to > > > >> the *Apache > > > >> Beam* community to include the *support for the Nemo Runner* for > Beam > > > >> applications, as our frontend provides the support for running Beam > > > >> applications. > > > >> > > > >> Any opinions regarding the issue are welcome! > > > >> I think a word from Davor would greatly help this issue, as he is a > > PMC > > > >> member of the Apache Beam community and our mentor. Would there some > > > >> information that you could provide us with? > > > >> > > > >> Thanks a lot! > > > >> Wonook > > > > > > -- > > > Jean-Baptiste Onofré > > > jbono...@apache.org > > > http://blog.nanthrax.net > > > Talend - http://www.talend.com > > > > > > > > -- > Byung-Gon Chun >
