Hi Kenn Thanks for the information, Will add information accordingly and update the community.
Best Regards, Pulasthi On Wed, Jan 29, 2020 at 8:28 AM Kenneth Knowles <k...@apache.org> wrote: > In my opinion it is fine to add the documentation after the runner is > added. I do think we should have input from more members of the community > about accepting the donation. Since there is time here are places where you > should add information about the runner: > > https://beam.apache.org/documentation/runners/capability-matrix/ source > data at > https://github.com/apache/beam/blob/master/website/src/_data/capability-matrix.yml > > https://beam.apache.org/documentation/runners/twister2 (new page - see > the pages for other runners) > > https://beam.apache.org/get-started/quickstart-java/ has some snippets > with toggles per runner > > https://beam.apache.org/roadmap/ has the roadmaps for different runners. > For a new runner especially this could be helpful for users. > > Kenn > > On Sun, Jan 12, 2020 at 9:36 AM Pulasthi Supun Wickramasinghe < > pulasthi...@gmail.com> wrote: > >> Hi Kenn, >> >> Is there any documentation that needs to accompany the new runner in the >> pull request or is the documentation added after the pull request is >> approved?. I would be great if you can point me in the right direction >> regarding this. >> >> Best Regards, >> Pulasthi >> >> On Mon, Jan 6, 2020 at 9:56 PM Pulasthi Supun Wickramasinghe < >> pulasthi...@gmail.com> wrote: >> >>> Hi Kenn, >>> >>> >>> >>> On Mon, Jan 6, 2020 at 9:09 PM Kenneth Knowles <k...@apache.org> wrote: >>> >>>> >>>> >>>> On Mon, Jan 6, 2020 at 8:30 AM Pulasthi Supun Wickramasinghe < >>>> pulasthi...@gmail.com> wrote: >>>> >>>>> Hi Kenn, >>>>> >>>>> I was able to solve the problem mentioned above, I am currently >>>>> running the "ValidatesRunner" tests, I have around 4-5 tests that are >>>>> failing that I should be able to fix in a couple of days. I wanted to >>>>> check >>>>> the next steps I would need to take after all the "ValidatesRunner" tests >>>>> are passing. I assume that the runner does not need to pass all the >>>>> "NeedsRunner" tests. >>>>> >>>> >>>> That's correct. You don't need to run the NeedsRunner tests. Those are >>>> tests of the core SDK's functionality, not the runner. The annotation is a >>>> workaround for the false cycle in deps "SDK tests" -> "direct runner" -> >>>> "SDK", which to maven looks like a cyclic dependency. You should run on the >>>> ValidatesRunner tests. It is fine to also disable some of them, either by >>>> excluding categories or test classes, or adding new categories to exclude. >>>> >>>> >>>>> The runner is only implemented for the batch mode at the moment >>>>> because the higher-level API's for streaming on Twister2 are still being >>>>> finalized. Once that work is done we will add streaming support for the >>>>> runner as well. >>>>> >>>> >>>> Nice! Batch-only is perfectly fine for a runner. You should be able to >>>> detect and reject pipelines that the runner cannot execute. >>>> >>> >>> I will make sure that the capability is there. >>> >>> >>>> I re-read the old thread, but I may have missed the answer to a >>>> fundamental question. Just to get it clear on the mailing list: are you >>>> intending to submit the runner's code to Apache Beam and ask the community >>>> to maintain it? >>>> >>>> To answer the original question the issue was with forwarding >>> exceptions that happen during execution since Twister2 has a >>> distributed model for execution, I added the ability in the Twister2 side >>> so that the Twister2 job submitting client will receive a job state object >>> that contains any exceptions thrown during runtime once the job is >>> completed. >>> >>> And about maintaining the twister2 runner, we would like to submit the >>> runner to the beam codebase but the Twister2 team will maintain and update >>> it continuously, in that case, we would become part of the Beam community I >>> suppose. And any contributions from other members of the community are more >>> than welcome. I hope that answers your question. >>> >>> Best Regards, >>> Pulasthi >>> >>> >>>> Kenn >>>> >>>> >>>> Best Regards, >>>>> Pulasthi >>>>> >>>>> On Thu, Dec 12, 2019 at 11:27 AM Pulasthi Supun Wickramasinghe < >>>>> pulasthi...@gmail.com> wrote: >>>>> >>>>>> Hi Kenn >>>>>> >>>>>> We are still working on aspects like automated job monitoring so >>>>>> currently do not have those capabilities built-in. I discussed with the >>>>>> Twister2 team on a way we can forward failure information from the >>>>>> workers >>>>>> to the Jobmaster which would be a solution to this problem. It might >>>>>> take a >>>>>> little time to develop and test. I will update you after looking into >>>>>> that >>>>>> solution in a little more detail. >>>>>> >>>>>> Best Regards, >>>>>> Pulasthi >>>>>> >>>>>> On Wed, Dec 11, 2019 at 10:51 PM Kenneth Knowles <k...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> I dug in to Twister2 a little bit to understand the question better, >>>>>>> checking how the various resource managers / launchers are plumbed. >>>>>>> >>>>>>> How would a user set up automated monitoring for a job? If that is >>>>>>> scraping the logs, then it seems unfortunate for users, but I think the >>>>>>> Beam runner would naturally use whatever a user might use. >>>>>>> >>>>>>> Kenn >>>>>>> >>>>>>> On Wed, Dec 11, 2019 at 10:45 AM Pulasthi Supun Wickramasinghe < >>>>>>> pulasthi...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Dev's >>>>>>>> >>>>>>>> I have been making some progress on the Twister2 runner for the >>>>>>>> beam that I mentioned before on the mailing list. The runner is able >>>>>>>> to run >>>>>>>> the wordcount example and produce correct results. So I am currently >>>>>>>> trying >>>>>>>> to run the runner validation tests. >>>>>>>> >>>>>>>> From what I understood looking at a couple examples is that tests >>>>>>>> are validated based on the exceptions that are thrown (or not) during >>>>>>>> test >>>>>>>> runtime. However in Twister2 currently the job submission client does >>>>>>>> not >>>>>>>> get failure information such as exceptions back once the job is >>>>>>>> submitted. >>>>>>>> These are however recorded in the worker log files. >>>>>>>> >>>>>>>> So in order to validate the tests for Twister2 I would have to >>>>>>>> parse the worker logfile and check what exceptions are in the logs. >>>>>>>> Would >>>>>>>> that be an acceptable solution for the validation tests? >>>>>>>> >>>>>>>> Best Regards, >>>>>>>> Pulasthi >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Pulasthi S. Wickramasinghe >>>>>>>> PhD Candidate | Research Assistant >>>>>>>> School of Informatics and Computing | Digital Science Center >>>>>>>> Indiana University, Bloomington >>>>>>>> cell: 224-386-9035 <(224)%20386-9035> >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Pulasthi S. Wickramasinghe >>>>>> PhD Candidate | Research Assistant >>>>>> School of Informatics and Computing | Digital Science Center >>>>>> Indiana University, Bloomington >>>>>> cell: 224-386-9035 <(224)%20386-9035> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Pulasthi S. Wickramasinghe >>>>> PhD Candidate | Research Assistant >>>>> School of Informatics and Computing | Digital Science Center >>>>> Indiana University, Bloomington >>>>> cell: 224-386-9035 <(224)%20386-9035> >>>>> >>>> >>> >>> -- >>> Pulasthi S. Wickramasinghe >>> PhD Candidate | Research Assistant >>> School of Informatics and Computing | Digital Science Center >>> Indiana University, Bloomington >>> cell: 224-386-9035 <(224)%20386-9035> >>> >> >> >> -- >> Pulasthi S. Wickramasinghe >> PhD Candidate | Research Assistant >> School of Informatics and Computing | Digital Science Center >> Indiana University, Bloomington >> cell: 224-386-9035 <(224)%20386-9035> >> > -- Pulasthi S. Wickramasinghe PhD Candidate | Research Assistant School of Informatics and Computing | Digital Science Center Indiana University, Bloomington cell: 224-386-9035
