I think it's great to go ahead and merge it, so it can continue evolving. As with all things, it'll adopt new stuff as it becomes ready (in fact, it may even prove to be a great example of how to port an existing "legacy" runner to the portability stuff when ready).
It seems the immediate blocker (gradle) was addressed, and there is great future work planned. Exciting! On Thu, Jun 21, 2018 at 8:00 PM Kenneth Knowles <k...@google.com> wrote: > *Contributors* > Agree with Robert's concern. But this is a nice opportunity for Beam to > connect. It is a different sort of backend and a different sort of > community that we are linking in. > > Consider the Gearpump and Apex runners: both had resumes that met the > requirements, but might not today. But they haven't been a burden. I have > some hope the Samza runner might have a better chance recruiting users and > contributors, since the value add for Samza users is unique among Beam > runners, and likewise the Samza community is unique among backend > communities. > > *Portability* > My take is that we shouldn't _start_ any runner down the legacy path. But > this is runner predates portability. I don't think the Java SDK is ready to > provide feature parity, much less adequate performance, so it doesn't seem > reasonable to require using it. Community > code as well. > > Kenn > > On Thu, Jun 21, 2018 at 3:34 PM Robert Bradshaw <rober...@google.com> > wrote: > >> Neat to see a new runner on board! >> >> I would like to make it a requirement for all new runners to support >> the portability API, but given that it's still somewhat of a moving >> target, and you have ongoing work in this direction, that may not be a >> hard requirement. >> >> I'm a bit concerned that there is are only two contributors (but the >> git logs): you and Kenn. But you do indicate there are others >> interested in working on this. >> >> Other than that, this looks great. >> >> - Robert >> >> >> On Thu, Jun 21, 2018 at 3:14 PM Xinyu Liu <xinyuliu...@gmail.com> wrote: >> > >> > I updated the merge PR with the gradle integration (there was some >> Jenkins Java tests failure with google cloud quota issues. It seems not >> related to this patch). Please feel free to ping me if anything else is >> needed. >> > >> > Thanks, >> > Xinyu >> > >> > On Mon, Jun 18, 2018 at 5:44 PM, Xinyu Liu <xinyuliu...@gmail.com> >> wrote: >> >> >> >> @Kenn: I am going to add the build.gradle. Is there anything else? >> >> >> >> @Ahmet, @Robert: here are more details about the samza runner right >> now: >> >> >> >> - Missing pieces: timer support in ParDo is not there yet and I plan >> to add it soon. SplittableParDo is missing but we don't have a use case so >> far. We are on par with the other runners for the rest of the Java features. >> >> - Work in Progress: implement the portable pipeline runner logic. >> >> - Future plans: support Python is our next goal. Hopefully we will get >> a prototype working sometime next quarter :). >> >> >> >> Btw, thanks everyone for the comments! >> >> >> >> Thanks, >> >> Xinyu >> >> >> >> On Mon, Jun 18, 2018 at 4:59 PM, Robert Burke <rob...@frantil.com> >> wrote: >> >>> >> >>> This is exciting! Is it implemented as a portability framework runner >> too? >> >>> >> >>> >> >>> On Mon, Jun 18, 2018, 4:36 PM Pablo Estrada <pabl...@google.com> >> wrote: >> >>>> >> >>>> It's very exciting to see a new runner making it into master. : ) >> >>>> >> >>>> Best >> >>>> -P. >> >>>> >> >>>> On Mon, Jun 18, 2018 at 3:38 PM Rafael Fernandez < >> rfern...@google.com> wrote: >> >>>>> >> >>>>> I've just read this and wanted to share my excitement :D >> >>>>> >> >>>>> >> >>>>> >> >>>>> On Mon, Jun 18, 2018 at 3:10 PM Kenneth Knowles <k...@google.com> >> wrote: >> >>>>>> >> >>>>>> One thing that will be necessary is porting the build to Gradle. >> >>>>>> >> >>>>>> Kenn >> >>>>>> >> >>>>>> On Mon, Jun 18, 2018 at 11:57 AM Xinyu Liu <xinyuliu...@gmail.com> >> wrote: >> >>>>>>> >> >>>>>>> Hi, Folks, >> >>>>>>> >> >>>>>>> On behalf of the Samza team, I would like to propose to merge the >> samza-runner branch into master. The branch was created on Jan when we >> first introduced the Samza Runner [1], and we've been adding features and >> refining it afterwards. Now the runner satisfies the criteria outlined in >> [2], and merging it to master will give more visibility to other >> contributors and users. >> >>>>>>> >> >>>>>>> 1. Have at least 2 contributors interested in maintaining it, and >> 1 committer interested in supporting it: *Both Chris and me have been >> making contributions and I am going to sign up for the support. There are >> more folks in the Samza team interested in contributing to it. Thanks Kenn >> for all the help and reviews for the runner!* >> >>>>>>> 2. Provide both end-user and developer-facing documentation: *The >> PR for the samza-runner doc has runner user guide, capability matrix, and >> tutorial using WordCount examples.* >> >>>>>>> 3. Have at least a basic level of unit test coverage: *Unit tests >> are here [3].* >> >>>>>>> 4. Run all existing applicable integration tests with other Beam >> components and create additional tests as appropriate: Enabled >> ValidatesRunner tests.* >> >>>>>>> 5. Be able to handle a subset of the model that addresses a >> significant set of use cases, such as ‘traditional batch’ or ‘processing >> time streaming’: *We have test Beam jobs running in Yarn using event-time >> processing of Kafka streams.* >> >>>>>>> 6. Update the capability matrix with the current status. *Same as >> #2.* >> >>>>>>> 7. Add a webpage under documentation/runners. *Same as #2.* >> >>>>>>> >> >>>>>>> The PR for the samza-runner merge: >> https://github.com/apache/beam/pull/5668 >> >>>>>>> The PR for the samza-runner doc: >> https://github.com/apache/beam-site/pull/471 >> >>>>>>> >> >>>>>>> Thanks, >> >>>>>>> Xinyu >> >>>>>>> >> >>>>>>> [1] https://issues.apache.org/jira/browse/BEAM-3079 >> >>>>>>> [2] https://beam.apache.org/contribute/ >> >>>>>>> [3] >> https://github.com/apache/beam/tree/samza-runner/runners/samza/src/test >> >>>> >> >>>> -- >> >>>> Got feedback? go/pabloem-feedback >> <https://goto.google.com/pabloem-feedback> >> >> >> >> >> > >> >
smime.p7s
Description: S/MIME Cryptographic Signature
