Blocking commits to master on test flaps seems critical here. The test flaps won't get the attention they deserve as long as people are just spamming their PRs with 'Run Java Precommit' until they turn green. I'm guilty of this behavior and I know it masks new flaky tests.
I added a comment to your doc about detecting flaky tests. This can easily be done by rerunning the postcommits during times when Jenkins would otherwise be idle. You'll easily get a few dozen runs every weekend, you just need a process to triage all the flakes and ensure there are bugs. I worked on a project that did this along with blocking master on any post commit failure. It was painful for the first few weeks, but things got significantly better once most of the bugs were fixed. Andrew On Fri, May 18, 2018 at 10:39 AM Kenneth Knowles <k...@google.com> wrote: > Love it. I would pull out from the doc also the key point: make the > postcommit status constantly visible to everyone. > > Kenn > > On Fri, May 18, 2018 at 10:17 AM Mikhail Gryzykhin <mig...@google.com> > wrote: > >> Hi everyone, >> >> I'm Mikhail and started working on Google Dataflow several months ago. >> I'm really excited to work with Beam opensource community. >> >> I have a proposal to improve contributor experience by keeping >> post-commit tests green. >> >> I'm looking to get community consensus and approval about the process for >> keeping post-commit tests green and addressing post-commit test failures. >> >> Find full list of ideas brought in for discussion in this document: >> >> https://docs.google.com/document/d/1sczGwnCvdHiboVajGVdnZL0rfnr7ViXXAebBAf_uQME >> >> Key points are: >> 1. Add explicit tracking of failures via JIRA >> 2. No-Commit policy when post-commit tests are red >> >> --Mikhail >> >>
