On Thu, Sep 3, 2020 at 11:05 AM Brian Hulette <[email protected]> wrote:
> The new GitHub Actions workflows that run Java and Python tests against > different targets (macos, ubuntu, windows) are great! But just like our > Jenkins infra they flake occasionally. Should we be re-running all of these > jobs until we get green runs? > > Unfortunately it's not possible to re-run an individual job in a workflow > [1], the only option is to re-run all jobs, so flaky tests become even more > problematic. > > I see two options: > 1) Consider it "good enough" if just Jenkins CI passes and any GitHub > actions failures appear to be flakes. > 2) Require that all Jenkins and GitHub checks pass. > > My vote is for (2). (1) risks merging legitimate breakages, and one could > argue that making flaky tests extra painful is a good thing. Also we can > always make an exception if an obvious flake is blocking a critical PR. > +1 for (2) given that it might be not so easy to figure out whether the failure is flaky (or how critical it is). BTW, I see it's impossible to re-run a specific test but how do we re-run all tests then? Is there a menu item for it or needs to force update the commits? > > > Also FYI - at first I thought these workflows only had the stdout > available, but the test report directory is also zipped and uploaded as an > artifact. When a failure occurs you can download it to get the full output: > [image: image.png] > > > Brian > > [1] > https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234 >
