Hi Kenn, I was able to solve the problem mentioned above, I am currently running the "ValidatesRunner" tests, I have around 4-5 tests that are failing that I should be able to fix in a couple of days. I wanted to check the next steps I would need to take after all the "ValidatesRunner" tests are passing. I assume that the runner does not need to pass all the "NeedsRunner" tests.
The runner is only implemented for the batch mode at the moment because the higher-level API's for streaming on Twister2 are still being finalized. Once that work is done we will add streaming support for the runner as well. Best Regards, Pulasthi On Thu, Dec 12, 2019 at 11:27 AM Pulasthi Supun Wickramasinghe < pulasthi...@gmail.com> wrote: > Hi Kenn > > We are still working on aspects like automated job monitoring so currently > do not have those capabilities built-in. I discussed with the Twister2 team > on a way we can forward failure information from the workers to the > Jobmaster which would be a solution to this problem. It might take a little > time to develop and test. I will update you after looking into that > solution in a little more detail. > > Best Regards, > Pulasthi > > On Wed, Dec 11, 2019 at 10:51 PM Kenneth Knowles <k...@apache.org> wrote: > >> I dug in to Twister2 a little bit to understand the question better, >> checking how the various resource managers / launchers are plumbed. >> >> How would a user set up automated monitoring for a job? If that is >> scraping the logs, then it seems unfortunate for users, but I think the >> Beam runner would naturally use whatever a user might use. >> >> Kenn >> >> On Wed, Dec 11, 2019 at 10:45 AM Pulasthi Supun Wickramasinghe < >> pulasthi...@gmail.com> wrote: >> >>> Hi Dev's >>> >>> I have been making some progress on the Twister2 runner for the beam >>> that I mentioned before on the mailing list. The runner is able to run the >>> wordcount example and produce correct results. So I am currently trying to >>> run the runner validation tests. >>> >>> From what I understood looking at a couple examples is that tests are >>> validated based on the exceptions that are thrown (or not) during test >>> runtime. However in Twister2 currently the job submission client does not >>> get failure information such as exceptions back once the job is submitted. >>> These are however recorded in the worker log files. >>> >>> So in order to validate the tests for Twister2 I would have to parse the >>> worker logfile and check what exceptions are in the logs. Would that be an >>> acceptable solution for the validation tests? >>> >>> Best Regards, >>> Pulasthi >>> >>> >>> >>> >>> -- >>> Pulasthi S. Wickramasinghe >>> PhD Candidate | Research Assistant >>> School of Informatics and Computing | Digital Science Center >>> Indiana University, Bloomington >>> cell: 224-386-9035 <(224)%20386-9035> >>> >> > > -- > Pulasthi S. Wickramasinghe > PhD Candidate | Research Assistant > School of Informatics and Computing | Digital Science Center > Indiana University, Bloomington > cell: 224-386-9035 > -- Pulasthi S. Wickramasinghe PhD Candidate | Research Assistant School of Informatics and Computing | Digital Science Center Indiana University, Bloomington cell: 224-386-9035
