*Contributors* Agree with Robert's concern. But this is a nice opportunity for Beam to connect. It is a different sort of backend and a different sort of community that we are linking in.
Consider the Gearpump and Apex runners: both had resumes that met the requirements, but might not today. But they haven't been a burden. I have some hope the Samza runner might have a better chance recruiting users and contributors, since the value add for Samza users is unique among Beam runners, and likewise the Samza community is unique among backend communities. *Portability* My take is that we shouldn't _start_ any runner down the legacy path. But this is runner predates portability. I don't think the Java SDK is ready to provide feature parity, much less adequate performance, so it doesn't seem reasonable to require using it. Community > code as well. Kenn On Thu, Jun 21, 2018 at 3:34 PM Robert Bradshaw <rober...@google.com> wrote: > Neat to see a new runner on board! > > I would like to make it a requirement for all new runners to support > the portability API, but given that it's still somewhat of a moving > target, and you have ongoing work in this direction, that may not be a > hard requirement. > > I'm a bit concerned that there is are only two contributors (but the > git logs): you and Kenn. But you do indicate there are others > interested in working on this. > > Other than that, this looks great. > > - Robert > > > On Thu, Jun 21, 2018 at 3:14 PM Xinyu Liu <xinyuliu...@gmail.com> wrote: > > > > I updated the merge PR with the gradle integration (there was some > Jenkins Java tests failure with google cloud quota issues. It seems not > related to this patch). Please feel free to ping me if anything else is > needed. > > > > Thanks, > > Xinyu > > > > On Mon, Jun 18, 2018 at 5:44 PM, Xinyu Liu <xinyuliu...@gmail.com> > wrote: > >> > >> @Kenn: I am going to add the build.gradle. Is there anything else? > >> > >> @Ahmet, @Robert: here are more details about the samza runner right now: > >> > >> - Missing pieces: timer support in ParDo is not there yet and I plan to > add it soon. SplittableParDo is missing but we don't have a use case so > far. We are on par with the other runners for the rest of the Java features. > >> - Work in Progress: implement the portable pipeline runner logic. > >> - Future plans: support Python is our next goal. Hopefully we will get > a prototype working sometime next quarter :). > >> > >> Btw, thanks everyone for the comments! > >> > >> Thanks, > >> Xinyu > >> > >> On Mon, Jun 18, 2018 at 4:59 PM, Robert Burke <rob...@frantil.com> > wrote: > >>> > >>> This is exciting! Is it implemented as a portability framework runner > too? > >>> > >>> > >>> On Mon, Jun 18, 2018, 4:36 PM Pablo Estrada <pabl...@google.com> > wrote: > >>>> > >>>> It's very exciting to see a new runner making it into master. : ) > >>>> > >>>> Best > >>>> -P. > >>>> > >>>> On Mon, Jun 18, 2018 at 3:38 PM Rafael Fernandez <rfern...@google.com> > wrote: > >>>>> > >>>>> I've just read this and wanted to share my excitement :D > >>>>> > >>>>> > >>>>> > >>>>> On Mon, Jun 18, 2018 at 3:10 PM Kenneth Knowles <k...@google.com> > wrote: > >>>>>> > >>>>>> One thing that will be necessary is porting the build to Gradle. > >>>>>> > >>>>>> Kenn > >>>>>> > >>>>>> On Mon, Jun 18, 2018 at 11:57 AM Xinyu Liu <xinyuliu...@gmail.com> > wrote: > >>>>>>> > >>>>>>> Hi, Folks, > >>>>>>> > >>>>>>> On behalf of the Samza team, I would like to propose to merge the > samza-runner branch into master. The branch was created on Jan when we > first introduced the Samza Runner [1], and we've been adding features and > refining it afterwards. Now the runner satisfies the criteria outlined in > [2], and merging it to master will give more visibility to other > contributors and users. > >>>>>>> > >>>>>>> 1. Have at least 2 contributors interested in maintaining it, and > 1 committer interested in supporting it: *Both Chris and me have been > making contributions and I am going to sign up for the support. There are > more folks in the Samza team interested in contributing to it. Thanks Kenn > for all the help and reviews for the runner!* > >>>>>>> 2. Provide both end-user and developer-facing documentation: *The > PR for the samza-runner doc has runner user guide, capability matrix, and > tutorial using WordCount examples.* > >>>>>>> 3. Have at least a basic level of unit test coverage: *Unit tests > are here [3].* > >>>>>>> 4. Run all existing applicable integration tests with other Beam > components and create additional tests as appropriate: Enabled > ValidatesRunner tests.* > >>>>>>> 5. Be able to handle a subset of the model that addresses a > significant set of use cases, such as ‘traditional batch’ or ‘processing > time streaming’: *We have test Beam jobs running in Yarn using event-time > processing of Kafka streams.* > >>>>>>> 6. Update the capability matrix with the current status. *Same as > #2.* > >>>>>>> 7. Add a webpage under documentation/runners. *Same as #2.* > >>>>>>> > >>>>>>> The PR for the samza-runner merge: > https://github.com/apache/beam/pull/5668 > >>>>>>> The PR for the samza-runner doc: > https://github.com/apache/beam-site/pull/471 > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Xinyu > >>>>>>> > >>>>>>> [1] https://issues.apache.org/jira/browse/BEAM-3079 > >>>>>>> [2] https://beam.apache.org/contribute/ > >>>>>>> [3] > https://github.com/apache/beam/tree/samza-runner/runners/samza/src/test > >>>> > >>>> -- > >>>> Got feedback? go/pabloem-feedback > <https://goto.google.com/pabloem-feedback> > >> > >> > > >
