I think this looks good, we should enable the plugin and try it out. Concrete details of the follow-up tasks (auto-assignment, triage, and dashboarding) will probably depend on how functional the plugin is and what the test failures data looks like.
Regards, Anton On Wed, Jul 11, 2018 at 5:00 PM Mikhail Gryzykhin <mig...@google.com> wrote: > @Yifan Zou <yifan...@google.com> > > I believe that we should test-drive the system with tickets + PR first and > decide on email notification later. We already have tests failure emails > sent to commits@, I believe most people filter out or not signed up for > that list though. > > It creates only one ticket, and keeps it for recurring test failures. > > @Andrew Pilloud <apill...@google.com> > Thank you for the suggestion. I'll add it to design doc. > > --Mikhail > > > > On Wed, Jul 11, 2018 at 4:52 PM Yifan Zou <yifan...@google.com> wrote: > >> +1 to Andrew's concerns. Leaving the tickets unassigned will cause the >> ticket being ignored and no actions being taken. >> >> I can see the challenges on ticket assignment. Like Mikhail mentioned, >> the plugin does not support dynamic assignments. We have to implement >> custom script to determine the assignees and do some tricks to the jenkins >> job. Also, the post-commits tests usually cover tons of stuffs that it is >> difficult to find which part was broken and ask the right person to look >> into within the Auto JIRA process. Some naive thoughts: Are we able to send >> emails to the dev@ to ask people to take care of the JIRA issues? Are we >> able to find component leads and ask them triage the test failure tickets? >> >> Another nitpick comment. Does the jenkins job file the JIRA issue in >> every test failure? Sometimes the test continuously fails in a time period >> due to the same reason. In this case, we will get some duplicate issues >> filed by Jenkins. I think it could be better if we can avoid filing issues >> if the previous one has not been resolved. >> >> Thanks. >> Yifan >> >> >> On Wed, Jul 11, 2018 at 4:37 PM Andrew Pilloud <apill...@google.com> >> wrote: >> >>> That sounds great. You should add this detail to the doc. >>> >>> On Wed, Jul 11, 2018 at 4:29 PM Mikhail Gryzykhin <mig...@google.com> >>> wrote: >>> >>>> We already have component for this purpose: "test-failures". All >>>> tickets created will go to that component. As an option, we can add link to >>>> view list of open JIRA tickets to PR template. >>>> >>>> We also would want to create graph on dashboard with amount of >>>> unassigned and assigned bugs. >>>> >>>> I believe that we can also add counter of unassigned bugs to PR >>>> template. This way it will be easier for everyone to know when there's some >>>> tests issue not attended. >>>> >>>> --Mikhail >>>> >>>> >>>> On Wed, Jul 11, 2018 at 4:24 PM Andrew Pilloud <apill...@google.com> >>>> wrote: >>>> >>>>> So it sounds like you will want to create a component for untriaged >>>>> issues so they are easy to find. I like the idea of distributing the work >>>>> of triaging post commit failures to new PR authors as a condition of >>>>> merging. I feel like we will just be filling JIRA with spam if the issues >>>>> are automatically created without a plan for triage. >>>>> >>>>> Andrew >>>>> >>>>> On Wed, Jul 11, 2018 at 4:12 PM Rui Wang <ruw...@google.com> wrote: >>>>> >>>>>> Maybe this is also a good thread to start the discussion that if we >>>>>> want to enforce postcommit test for every PR. >>>>>> >>>>>> Can we afford the cost of longer waiting time to catch potential >>>>>> bugs? >>>>>> >>>>>> -Rui >>>>>> >>>>>> On Wed, Jul 11, 2018 at 4:04 PM Mikhail Gryzykhin <mig...@google.com> >>>>>> wrote: >>>>>> >>>>>>> That's a valid point. >>>>>>> >>>>>>> Unfortunately, the JiraTestResultReporter plugin does not have >>>>>>> features to dynamically assign owners. Additionally, I don't think it is >>>>>>> always easy to find proper owner for post-commit tests at first glance, >>>>>>> since they usually cover broad specter of issues. >>>>>>> >>>>>>> My assumption is that we need someone to triage new issues. >>>>>>> >>>>>>> Ideally, any contributor, who sees failing test, should check >>>>>>> unassigned tickets and either do triage, or assign them to someone who >>>>>>> can. >>>>>>> I strongly encourage this approach. >>>>>>> >>>>>>> We have couple other ready-made options to consider: >>>>>>> 1. We can configure JIRA component owner who would be assigned to >>>>>>> created tickets. >>>>>>> 2. JiraTestReporterPlugin can assign tickets to specific user. This >>>>>>> is configured per Jenkins job. We can utilize this if someone >>>>>>> volunteers. >>>>>>> 3. Dynamic assignment will most likely require custom solution. >>>>>>> >>>>>>> --Mikhail >>>>>>> >>>>>>> >>>>>>> On Wed, Jul 11, 2018 at 3:34 PM Andrew Pilloud <apill...@google.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Mikhail, >>>>>>>> >>>>>>>> I like the proposal! Hopefully this can replace the constant stream >>>>>>>> of build failure emails. I noticed one detail seems to be missing: How >>>>>>>> will new issues be assigned to the proper owner? Will the tool do this >>>>>>>> automatically or will we need someone to triage new issues? >>>>>>>> >>>>>>>> Andrew >>>>>>>> >>>>>>>> On Wed, Jul 11, 2018 at 3:07 PM Mikhail Gryzykhin < >>>>>>>> mig...@google.com> wrote: >>>>>>>> >>>>>>>>> Hi everyone, >>>>>>>>> >>>>>>>>> I want to add an automatic JIRA tickets creation for failing >>>>>>>>> post-commit tests. >>>>>>>>> >>>>>>>>> I wrote up design proposal doc with more details on this: >>>>>>>>> >>>>>>>>> https://docs.google.com/document/d/1kpsTy0sYJkLqlZvkPalDkqzBbpu-Wug0z-oWIVPo6UI >>>>>>>>> >>>>>>>>> Quick summary: >>>>>>>>> I suggest to utilize JiraTestResultReporter plugin. >>>>>>>>> Since this plugin is not installed on our Jenkins yet, we have to >>>>>>>>> request to Infra team to add it. >>>>>>>>> >>>>>>>>> Please, comment if this approach sounds good to you. >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> --Mikhail >>>>>>>>> >>>>>>>>>
