Hey everyone,
I've been taking a look at some flakey tests as a way to get more familiar
with the codebase. The first thing I want to do when investigating a
particular test is find all the builds where it failed - but unfortunately
this has proven rather challenging.
For example, when I was looking at BEAM-6512 [1], I wanted to find all
builds where `testMessageReceivedBySingleClientWhenThereAreMultipleClients`
failed. Fortunately this test was (is) flaking relatively often, so it
didn't take too long to just step through the
Java_PreCommit_{Commit,Phrase,Cron} builds in Jenkins and find a couple of
examples, like Cron #900 [2]. But what should I do for a flake that doesn't
occur so often, or a flake that has actually already been resolved? Ideally
I could confirm it hasn't happened in any recent builds and close the
corresponding Jira in the latter case. Or find a couple of occurrences in
the former case.
klk@ suggested searching builds@ archives which was a great idea, but sadly
I don't think it will work. When I searched for the BEAM-6512 failure [3],
it did turn up a couple of builds, but note that Cron #900 is not present
in the results. It looks like the relevant line was truncated in that
build's email, so the results are not complete. Also Commit and Phrase
build failures don't seem to go to builds@, which makes a lot of sense
since I'm sure there's a lot of churn there, but failures in those builds
could certainly be relevant for this search.
I would think there would be some way to search Jenkins by test name, but I
can't find it if it's there. Is anyone aware of a way to do this? I'm sure
I could hack something together with curl and the Jenkins API but I'd
rather not go that route if there's something easier I'm missing.
[1] https://issues.apache.org/jira/browse/BEAM-6512
[2] https://builds.apache.org/job/beam_PreCommit_Java_Cron/900/
[3]
https://lists.apache.org/[email protected]:lte=1M:testMessageReceivedBySingleClientWhenThereAreMultipleClients
