Hello everyone, I have addressed comments on the proposal doc and updated it accordingly. I have also added section on metrics that we want to track for pre-commit tests and contents for dashboard.
Please, take a second look at the document. Highlights: * Sections that I feel require more discussion are marked with *[More opinions wanted]* ** I've kept original comments open for this iteration. Please, close them if you feel those resolved, or elaborate more on the topic.* * Added information on metrics to track * Moved “Split test jobs into automatically and manually triggered” to “Other ideas to consider” * Prioritized automated JIRA ticket creation over manual * Prioritized roll-back first policy * Added process for enforcing proposed policies. --Mikhail Have feedback <http://go/migryz-feedback>? On Tue, May 22, 2018 at 10:11 AM Scott Wegner <sweg...@google.com> wrote: > Thanks for the thoughtful proposal Mikhail. I've left some comments in the > doc. > > I encourage others to take a look: the proposal adds some strong policies > about dealing with post-commit failures (rollback policy, locking master). > Currently our post-commits are frequently red, and we're missing out on a > valuable quality signal. I'm in favor of such policies to help get the test > signals back to a healthy state. > > On Mon, May 21, 2018 at 2:48 PM Mikhail Gryzykhin <mig...@google.com> > wrote: > >> Hi Everyone, >> >> I've updated design doc according to comments. >> >> https://docs.google.com/document/d/1sczGwnCvdHiboVajGVdnZL0rfnr7ViXXAebBAf_uQME >> >> In general, ideas proposed seem to be appreciated. Still, some of >> sections require more discussion. >> >> Changes highlight: >> * Added roll-back first policy to best practices. This includes process >> on how to handle roll-back. >> * Marked topics that I'd like to have more input on. [cyan color] >> >> --Mikhail >> >> Have feedback <http://go/migryz-feedback>? >> >> >> On Fri, May 18, 2018 at 10:56 AM Andrew Pilloud <apill...@google.com> >> wrote: >> >>> Blocking commits to master on test flaps seems critical here. The test >>> flaps won't get the attention they deserve as long as people are just >>> spamming their PRs with 'Run Java Precommit' until they turn green. I'm >>> guilty of this behavior and I know it masks new flaky tests. >>> >>> I added a comment to your doc about detecting flaky tests. This can >>> easily be done by rerunning the postcommits during times when Jenkins would >>> otherwise be idle. You'll easily get a few dozen runs every weekend, you >>> just need a process to triage all the flakes and ensure there are bugs. I >>> worked on a project that did this along with blocking master on any post >>> commit failure. It was painful for the first few weeks, but things got >>> significantly better once most of the bugs were fixed. >>> >>> Andrew >>> >>> On Fri, May 18, 2018 at 10:39 AM Kenneth Knowles <k...@google.com> wrote: >>> >>>> Love it. I would pull out from the doc also the key point: make the >>>> postcommit status constantly visible to everyone. >>>> >>>> Kenn >>>> >>>> On Fri, May 18, 2018 at 10:17 AM Mikhail Gryzykhin <mig...@google.com> >>>> wrote: >>>> >>>>> Hi everyone, >>>>> >>>>> I'm Mikhail and started working on Google Dataflow several months ago. >>>>> I'm really excited to work with Beam opensource community. >>>>> >>>>> I have a proposal to improve contributor experience by keeping >>>>> post-commit tests green. >>>>> >>>>> I'm looking to get community consensus and approval about the process >>>>> for keeping post-commit tests green and addressing post-commit test >>>>> failures. >>>>> >>>>> Find full list of ideas brought in for discussion in this document: >>>>> >>>>> https://docs.google.com/document/d/1sczGwnCvdHiboVajGVdnZL0rfnr7ViXXAebBAf_uQME >>>>> >>>>> Key points are: >>>>> 1. Add explicit tracking of failures via JIRA >>>>> 2. No-Commit policy when post-commit tests are red >>>>> >>>>> --Mikhail >>>>> >>>>>
