That also sounds good to me. +1 Best, Jangho
On 11/15/18 7:29 PM, Byung-Gon Chun wrote: > Sounds good! > > On Thu, Nov 15, 2018 at 6:09 PM Joo Yeon Kim <[email protected]> wrote: > >> This sounds great to me :) +1 >> >> On Tue, Nov 13, 2018 at 1:13 PM 송원욱 <[email protected]> wrote: >> >>> Regarding the discussion, I agree with JB to keep the runner in the Nemo >>> codebase for now, as there are a number of ongoing developments related >> to >>> supporting the streaming functionalities. Also, as Nemo relies on >>> programming layers like Apache Beam and Apache Spark, it feels better to >>> let the runner live in the Nemo codebase, if both ways work to list the >>> Nemo runner as an official runner in Beam. >>> >>> With the directions set, I'll send an email to the Beam-dev mailing list >>> inquiring the process required to list the Nemo runner as an official >>> runner in Beam, and let you know how it goes on the thread! 😀 >>> Wonook >>> >>> >>> 2018년 11월 12일 (월) 오후 3:32, John Yang <[email protected]>님이 작성: >>> >>>> Thanks JB for your mail. >>>> >>>> I think the Beam portability framework is an exciting feature that Nemo >>>> wants to support in the future like Flink, although my personal focus >> at >>>> the moment is on streaming and overall performance/reliability. >>>> >>>> A quick update on the "When"/"How" parts of the capability matrix: >>> Taegeon >>>> confirmed (on his private development branch) that NEXMark Q0-Q7 >>> streaming >>>> queries run on Nemo except for Q3(timer, stateful) and Q7(sideinput), >> so >>> I >>>> think we have a good coverage, although we haven't yet compared the >>>> outputs/performance with Flink. >>>> >>>> Best, >>>> John >>>> >>>> >>>> On Mon, Nov 12, 2018 at 2:06 PM Jean-Baptiste Onofré <[email protected]> >>>> wrote: >>>> >>>>> Thanks for the update. >>>>> >>>>> Regarding the governance, it's also related to visibility. >>>>> >>>>> My preference would be to have the runner in nemo codebase. It gives >> us >>>>> a complete control and managed our own release cycle. >>>>> >>>>> However, it reduces the visibility for the beam community (or we have >>> to >>>>> send update on the beam mailing list). The nemo runner can be listed >> as >>>>> official runner in Beam wherever the code is located. >>>>> >>>>> I have a question about the runner: do we plan to support the Beam >>>>> portability layer (Job API, ...) in the nemo runner ? >>>>> >>>>> Regards >>>>> JB >>>>> >>>>> On 12/11/2018 01:23, John Yang wrote: >>>>>> Thanks all for the great discussion. >>>>>> >>>>>> My take on the current status of Beam support in Nemo from the >>>>> perspective >>>>>> of the Beam Capability Matrix[1]: >>>>>> - What: Full support except for SDF / Metrics / Stateful Processing >>>> (Side >>>>>> Inputs for streaming pipelines are not supported yet though) >>>>>> - Where: Probably full support >>>>>> - When: I think we support event-time/processing-time/count >> triggers, >>>>> but I >>>>>> am not sure we have tested the other triggers. >>>>>> - How: I don't have a good idea on this one, as we haven't had >> tests >>>> for >>>>>> this. >>>>>> >>>>>> I would say that we have good fundamentals to be on par with many >> of >>>> the >>>>>> existing official Beam runners. Taegeon and I are experimenting >> with >>>> the >>>>>> NEXMark benchmark, which I believe cover almost all of the Beam >>>> features, >>>>>> on Nemo to understand in more detail and fix the remaining issues. >>>>>> >>>>>> Regarding the governance I would also like to hear from other >>> members. >>>>>> Taegeon and I are also happy to prioritize things differently >>> depending >>>>> on >>>>>> the plan. >>>>>> >>>>>> Thanks, >>>>>> John >>>>>> >>>>>> [1] Beam Capability Matrix: >>>>>> >> https://beam.apache.org/documentation/runners/capability-matrix/#cap-summary-what >>>>>> >>>>>> On Sun, Nov 11, 2018 at 1:44 PM Byung-Gon Chun <[email protected]> >>>> wrote: >>>>>>> Thanks for the valuable input, Davor! >>>>>>> >>>>>>> Nemo PPMC members, which direction would you like to take? >>>>>>> >>>>>>> -Gon >>>>>>> >>>>>>> On Thu, Nov 8, 2018 at 3:06 AM Davor Bonaci <[email protected]> >>> wrote: >>>>>>>> I think the first decision you have to make is around the >>> governance >>>> of >>>>>>> the >>>>>>>> runner. It can live in the Beam project (and be governed by the >>> Beam >>>>>>> PMC), >>>>>>>> or in the Nemo project (and be governed by the Nemo PPMC). Both >> are >>>>>>> viable >>>>>>>> paths, and different folks have chosen different paths. >>>>>>>> >>>>>>>> On the technical side, the discussion revolves around API >> stability >>>> and >>>>>>>> versioning: (1) whether the API surface between the runner and >> Nemo >>>>>>>> internals is more stable than the API surface between the runner >>> and >>>>>>> Beam's >>>>>>>> runner-facing APIs; and (2) how to version Beam and Nemo, which >>> pairs >>>>> can >>>>>>>> work together, etc. >>>>>>>> >>>>>>>> On the organizational side, delegating governance to Beam PMC >> would >>>>> mean >>>>>>>> that they control release cadence, contribution process, and >>>> committer >>>>>>>> access. None of it is a particular issue, I think. These are >>>> reasonable >>>>>>>> people, but certainly requires more consensus building and >> process. >>>>>>>> * * * >>>>>>>> >>>>>>>> I don't want to steer you in any direction, and happy to support >>>>>>> whichever >>>>>>>> decision you make. Also, happy to help on the Beam side and make >>>> things >>>>>>>> happen quickly. (But, I'd love to see a meaningful discussion and >>>>>>> consensus >>>>>>>> decision before proceeding.) >>>>>>>> >>>>>>>> On Wed, Nov 7, 2018 at 1:19 AM Jean-Baptiste Onofré < >>> [email protected] >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi guys, >>>>>>>>> >>>>>>>>> definitely happy to help on that front. >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> JB >>>>>>>>> >>>>>>>>> On 07/11/2018 09:54, Byung-Gon Chun wrote: >>>>>>>>>> Thanks for initiating this discussion, Wonwook! >>>>>>>>>> >>>>>>>>>> Davor and JB, it’d be great to get your guide. >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> - Gon >>>>>>>>>> >>>>>>>>>> Sent from my iPhone >>>>>>>>>> >>>>>>>>>> 2018. 11. 7. 오후 3:57, 송원욱 <[email protected]> 작성: >>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> It's nice to hear that the first release is coming up pretty >>> soon >>>>>>> with >>>>>>>>> the >>>>>>>>>>> progress that we are making! >>>>>>>>>>> With the first release and the current development for >>> supporting >>>>>>>> stream >>>>>>>>>>> processing, I think it's time for us to consider sending a >>> request >>>>>>> to >>>>>>>>>>> the *Apache >>>>>>>>>>> Beam* community to include the *support for the Nemo Runner* >> for >>>>>>> Beam >>>>>>>>>>> applications, as our frontend provides the support for running >>>> Beam >>>>>>>>>>> applications. >>>>>>>>>>> >>>>>>>>>>> Any opinions regarding the issue are welcome! >>>>>>>>>>> I think a word from Davor would greatly help this issue, as he >>> is >>>> a >>>>>>>> PMC >>>>>>>>>>> member of the Apache Beam community and our mentor. Would >> there >>>> some >>>>>>>>>>> information that you could provide us with? >>>>>>>>>>> >>>>>>>>>>> Thanks a lot! >>>>>>>>>>> Wonook >>>>>>>>> -- >>>>>>>>> Jean-Baptiste Onofré >>>>>>>>> [email protected] >>>>>>>>> http://blog.nanthrax.net >>>>>>>>> Talend - http://www.talend.com >>>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Byung-Gon Chun >>>>>>> >>>>> -- >>>>> Jean-Baptiste Onofré >>>>> [email protected] >>>>> http://blog.nanthrax.net >>>>> Talend - http://www.talend.com >>>>> >
