Thanks for the thoughtful proposal Mikhail. I've left some comments in the doc.
I encourage others to take a look: the proposal adds some strong policies about dealing with post-commit failures (rollback policy, locking master). Currently our post-commits are frequently red, and we're missing out on a valuable quality signal. I'm in favor of such policies to help get the test signals back to a healthy state. On Mon, May 21, 2018 at 2:48 PM Mikhail Gryzykhin <[email protected]> wrote: > Hi Everyone, > > I've updated design doc according to comments. > > https://docs.google.com/document/d/1sczGwnCvdHiboVajGVdnZL0rfnr7ViXXAebBAf_uQME > > In general, ideas proposed seem to be appreciated. Still, some of sections > require more discussion. > > Changes highlight: > * Added roll-back first policy to best practices. This includes process on > how to handle roll-back. > * Marked topics that I'd like to have more input on. [cyan color] > > --Mikhail > > Have feedback <http://go/migryz-feedback>? > > > On Fri, May 18, 2018 at 10:56 AM Andrew Pilloud <[email protected]> > wrote: > >> Blocking commits to master on test flaps seems critical here. The test >> flaps won't get the attention they deserve as long as people are just >> spamming their PRs with 'Run Java Precommit' until they turn green. I'm >> guilty of this behavior and I know it masks new flaky tests. >> >> I added a comment to your doc about detecting flaky tests. This can >> easily be done by rerunning the postcommits during times when Jenkins would >> otherwise be idle. You'll easily get a few dozen runs every weekend, you >> just need a process to triage all the flakes and ensure there are bugs. I >> worked on a project that did this along with blocking master on any post >> commit failure. It was painful for the first few weeks, but things got >> significantly better once most of the bugs were fixed. >> >> Andrew >> >> On Fri, May 18, 2018 at 10:39 AM Kenneth Knowles <[email protected]> wrote: >> >>> Love it. I would pull out from the doc also the key point: make the >>> postcommit status constantly visible to everyone. >>> >>> Kenn >>> >>> On Fri, May 18, 2018 at 10:17 AM Mikhail Gryzykhin <[email protected]> >>> wrote: >>> >>>> Hi everyone, >>>> >>>> I'm Mikhail and started working on Google Dataflow several months ago. >>>> I'm really excited to work with Beam opensource community. >>>> >>>> I have a proposal to improve contributor experience by keeping >>>> post-commit tests green. >>>> >>>> I'm looking to get community consensus and approval about the process >>>> for keeping post-commit tests green and addressing post-commit test >>>> failures. >>>> >>>> Find full list of ideas brought in for discussion in this document: >>>> >>>> https://docs.google.com/document/d/1sczGwnCvdHiboVajGVdnZL0rfnr7ViXXAebBAf_uQME >>>> >>>> Key points are: >>>> 1. Add explicit tracking of failures via JIRA >>>> 2. No-Commit policy when post-commit tests are red >>>> >>>> --Mikhail >>>> >>>>
