Hi Kenn We are still working on aspects like automated job monitoring so currently do not have those capabilities built-in. I discussed with the Twister2 team on a way we can forward failure information from the workers to the Jobmaster which would be a solution to this problem. It might take a little time to develop and test. I will update you after looking into that solution in a little more detail.
Best Regards, Pulasthi On Wed, Dec 11, 2019 at 10:51 PM Kenneth Knowles <k...@apache.org> wrote: > I dug in to Twister2 a little bit to understand the question better, > checking how the various resource managers / launchers are plumbed. > > How would a user set up automated monitoring for a job? If that is > scraping the logs, then it seems unfortunate for users, but I think the > Beam runner would naturally use whatever a user might use. > > Kenn > > On Wed, Dec 11, 2019 at 10:45 AM Pulasthi Supun Wickramasinghe < > pulasthi...@gmail.com> wrote: > >> Hi Dev's >> >> I have been making some progress on the Twister2 runner for the beam that >> I mentioned before on the mailing list. The runner is able to run the >> wordcount example and produce correct results. So I am currently trying to >> run the runner validation tests. >> >> From what I understood looking at a couple examples is that tests are >> validated based on the exceptions that are thrown (or not) during test >> runtime. However in Twister2 currently the job submission client does not >> get failure information such as exceptions back once the job is submitted. >> These are however recorded in the worker log files. >> >> So in order to validate the tests for Twister2 I would have to parse the >> worker logfile and check what exceptions are in the logs. Would that be an >> acceptable solution for the validation tests? >> >> Best Regards, >> Pulasthi >> >> >> >> >> -- >> Pulasthi S. Wickramasinghe >> PhD Candidate | Research Assistant >> School of Informatics and Computing | Digital Science Center >> Indiana University, Bloomington >> cell: 224-386-9035 <(224)%20386-9035> >> > -- Pulasthi S. Wickramasinghe PhD Candidate | Research Assistant School of Informatics and Computing | Digital Science Center Indiana University, Bloomington cell: 224-386-9035
