Hi Bruno, Yes, you're right. Sorry for the typo.
Hi Ismael, You're right. Jenkins does not support the flakyFailure element and hence the information is not at all in the Jenkins report. I am still experimenting with printing the flaky tests somewhere. I will update this thread if I get something working. In the meantime, I wanted to gauge whether there is support for it. Cheers, David On Mon, Feb 12, 2024 at 3:59 PM Ismael Juma <[email protected]> wrote: > Hi David, > > Your message didn't make this clear, but you are saying that Jenkins does > _not_ support the flakyFailure element and hence this information will be > completely missing from the Jenkins report. Have we considered including > the flakyFailure information ourselves? I have seen that being done and it > seems strictly better than totally ignoring it. > > Ismael > > On Mon, Feb 12, 2024 at 12:11 AM David Jacot <[email protected]> > wrote: > > > Hi folks, > > > > I have been playing with `reports.junitXml.mergeReruns` setting in gradle > > [1]. From the gradle doc: > > > > > When mergeReruns is enabled, if a test fails but is then retried and > > succeeds, its failures will be recorded as <flakyFailure> instead of > > <failure>, within one <testcase>. This is effectively the reporting > > produced by the surefire plugin of Apache Maven™ when enabling reruns. If > > your CI server understands this format, it will indicate that the test > was > > flaky. If it does not, it will indicate that the test succeeded as it > will > > ignore the <flakyFailure> information. If the test does not succeed (i.e. > > it fails for every retry), it will be indicated as having failed whether > > your tool understands this format or not. > > > > With this, we get really close to having green builds [2] all the time. > > There are only a few tests which are too flaky. We should address or > > disable those. > > > > I think that this would help us a lot because it would reduce the noise > > that we get in pull requests. At the moment, there are just too many > failed > > tests reported so it is really hard to know whether a pull request is > > actually fine or not. > > > > [1] applies it to both unit and integration tests. Following the > discussion > > in the `github build queue` thread, it may be better to only apply it to > > the integration tests. Being stricter with unit tests would make sense. > > > > This does not mean that we should continue our effort to reduce the > number > > of flaky tests. For this, I propose to keep using Gradle Entreprise. It > > provides a nice report for them that we can leverage. > > > > Thoughts? > > > > Best, > > David > > > > [1] https://github.com/apache/kafka/pull/14862 > > [2] > > > > > https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14862/19/tests > > >
