Neat to see a new runner on board! I would like to make it a requirement for all new runners to support the portability API, but given that it's still somewhat of a moving target, and you have ongoing work in this direction, that may not be a hard requirement.
I'm a bit concerned that there is are only two contributors (but the git logs): you and Kenn. But you do indicate there are others interested in working on this. Other than that, this looks great. - Robert On Thu, Jun 21, 2018 at 3:14 PM Xinyu Liu <xinyuliu...@gmail.com> wrote: > > I updated the merge PR with the gradle integration (there was some Jenkins > Java tests failure with google cloud quota issues. It seems not related to > this patch). Please feel free to ping me if anything else is needed. > > Thanks, > Xinyu > > On Mon, Jun 18, 2018 at 5:44 PM, Xinyu Liu <xinyuliu...@gmail.com> wrote: >> >> @Kenn: I am going to add the build.gradle. Is there anything else? >> >> @Ahmet, @Robert: here are more details about the samza runner right now: >> >> - Missing pieces: timer support in ParDo is not there yet and I plan to add >> it soon. SplittableParDo is missing but we don't have a use case so far. We >> are on par with the other runners for the rest of the Java features. >> - Work in Progress: implement the portable pipeline runner logic. >> - Future plans: support Python is our next goal. Hopefully we will get a >> prototype working sometime next quarter :). >> >> Btw, thanks everyone for the comments! >> >> Thanks, >> Xinyu >> >> On Mon, Jun 18, 2018 at 4:59 PM, Robert Burke <rob...@frantil.com> wrote: >>> >>> This is exciting! Is it implemented as a portability framework runner too? >>> >>> >>> On Mon, Jun 18, 2018, 4:36 PM Pablo Estrada <pabl...@google.com> wrote: >>>> >>>> It's very exciting to see a new runner making it into master. : ) >>>> >>>> Best >>>> -P. >>>> >>>> On Mon, Jun 18, 2018 at 3:38 PM Rafael Fernandez <rfern...@google.com> >>>> wrote: >>>>> >>>>> I've just read this and wanted to share my excitement :D >>>>> >>>>> >>>>> >>>>> On Mon, Jun 18, 2018 at 3:10 PM Kenneth Knowles <k...@google.com> wrote: >>>>>> >>>>>> One thing that will be necessary is porting the build to Gradle. >>>>>> >>>>>> Kenn >>>>>> >>>>>> On Mon, Jun 18, 2018 at 11:57 AM Xinyu Liu <xinyuliu...@gmail.com> wrote: >>>>>>> >>>>>>> Hi, Folks, >>>>>>> >>>>>>> On behalf of the Samza team, I would like to propose to merge the >>>>>>> samza-runner branch into master. The branch was created on Jan when we >>>>>>> first introduced the Samza Runner [1], and we've been adding features >>>>>>> and refining it afterwards. Now the runner satisfies the criteria >>>>>>> outlined in [2], and merging it to master will give more visibility to >>>>>>> other contributors and users. >>>>>>> >>>>>>> 1. Have at least 2 contributors interested in maintaining it, and 1 >>>>>>> committer interested in supporting it: *Both Chris and me have been >>>>>>> making contributions and I am going to sign up for the support. There >>>>>>> are more folks in the Samza team interested in contributing to it. >>>>>>> Thanks Kenn for all the help and reviews for the runner!* >>>>>>> 2. Provide both end-user and developer-facing documentation: *The PR >>>>>>> for the samza-runner doc has runner user guide, capability matrix, and >>>>>>> tutorial using WordCount examples.* >>>>>>> 3. Have at least a basic level of unit test coverage: *Unit tests are >>>>>>> here [3].* >>>>>>> 4. Run all existing applicable integration tests with other Beam >>>>>>> components and create additional tests as appropriate: Enabled >>>>>>> ValidatesRunner tests.* >>>>>>> 5. Be able to handle a subset of the model that addresses a significant >>>>>>> set of use cases, such as ‘traditional batch’ or ‘processing time >>>>>>> streaming’: *We have test Beam jobs running in Yarn using event-time >>>>>>> processing of Kafka streams.* >>>>>>> 6. Update the capability matrix with the current status. *Same as #2.* >>>>>>> 7. Add a webpage under documentation/runners. *Same as #2.* >>>>>>> >>>>>>> The PR for the samza-runner merge: >>>>>>> https://github.com/apache/beam/pull/5668 >>>>>>> The PR for the samza-runner doc: >>>>>>> https://github.com/apache/beam-site/pull/471 >>>>>>> >>>>>>> Thanks, >>>>>>> Xinyu >>>>>>> >>>>>>> [1] https://issues.apache.org/jira/browse/BEAM-3079 >>>>>>> [2] https://beam.apache.org/contribute/ >>>>>>> [3] >>>>>>> https://github.com/apache/beam/tree/samza-runner/runners/samza/src/test >>>> >>>> -- >>>> Got feedback? go/pabloem-feedback >> >> >
