I like the idea of auto-creating bugs. I think that if we do this we should make sure to set a time period after which we will evaluate whether it has succeeded, i.e. whether bugs filed as a result of this have been triaged, owned, fixed, and closed.
On Fri, Jul 13, 2018 at 10:20 AM Mikhail Gryzykhin <mig...@google.com> wrote: > Can we get some PMC opinions on this? > > According to https://www.apache.org/dev/infra-contact#request-checklist, > it is highly recommended to have your opinions when filing a request. > > --Mikhail > > Have feedback <http://go/migryz-feedback>? > > > On Wed, Jul 11, 2018 at 6:01 PM Anton Kedin <ke...@google.com> wrote: > >> I think this looks good, we should enable the plugin and try it out. >> Concrete details of the follow-up tasks (auto-assignment, triage, and >> dashboarding) will probably depend on how functional the plugin is and what >> the test failures data looks like. >> >> Regards, >> Anton >> >> On Wed, Jul 11, 2018 at 5:00 PM Mikhail Gryzykhin <mig...@google.com> >> wrote: >> >>> @Yifan Zou <yifan...@google.com> >>> >>> I believe that we should test-drive the system with tickets + PR first >>> and decide on email notification later. We already have tests failure >>> emails sent to commits@, I believe most people filter out or not signed >>> up for that list though. >>> >>> It creates only one ticket, and keeps it for recurring test failures. >>> >>> @Andrew Pilloud <apill...@google.com> >>> Thank you for the suggestion. I'll add it to design doc. >>> >>> --Mikhail >>> >>> >>> >>> On Wed, Jul 11, 2018 at 4:52 PM Yifan Zou <yifan...@google.com> wrote: >>> >>>> +1 to Andrew's concerns. Leaving the tickets unassigned will cause the >>>> ticket being ignored and no actions being taken. >>>> >>>> I can see the challenges on ticket assignment. Like Mikhail mentioned, >>>> the plugin does not support dynamic assignments. We have to implement >>>> custom script to determine the assignees and do some tricks to the jenkins >>>> job. Also, the post-commits tests usually cover tons of stuffs that it is >>>> difficult to find which part was broken and ask the right person to look >>>> into within the Auto JIRA process. Some naive thoughts: Are we able to send >>>> emails to the dev@ to ask people to take care of the JIRA issues? Are >>>> we able to find component leads and ask them triage the test failure >>>> tickets? >>>> >>>> Another nitpick comment. Does the jenkins job file the JIRA issue in >>>> every test failure? Sometimes the test continuously fails in a time period >>>> due to the same reason. In this case, we will get some duplicate issues >>>> filed by Jenkins. I think it could be better if we can avoid filing issues >>>> if the previous one has not been resolved. >>>> >>>> Thanks. >>>> Yifan >>>> >>>> >>>> On Wed, Jul 11, 2018 at 4:37 PM Andrew Pilloud <apill...@google.com> >>>> wrote: >>>> >>>>> That sounds great. You should add this detail to the doc. >>>>> >>>>> On Wed, Jul 11, 2018 at 4:29 PM Mikhail Gryzykhin <mig...@google.com> >>>>> wrote: >>>>> >>>>>> We already have component for this purpose: "test-failures". All >>>>>> tickets created will go to that component. As an option, we can add link >>>>>> to >>>>>> view list of open JIRA tickets to PR template. >>>>>> >>>>>> We also would want to create graph on dashboard with amount of >>>>>> unassigned and assigned bugs. >>>>>> >>>>>> I believe that we can also add counter of unassigned bugs to PR >>>>>> template. This way it will be easier for everyone to know when there's >>>>>> some >>>>>> tests issue not attended. >>>>>> >>>>>> --Mikhail >>>>>> >>>>>> >>>>>> On Wed, Jul 11, 2018 at 4:24 PM Andrew Pilloud <apill...@google.com> >>>>>> wrote: >>>>>> >>>>>>> So it sounds like you will want to create a component for untriaged >>>>>>> issues so they are easy to find. I like the idea of distributing the >>>>>>> work >>>>>>> of triaging post commit failures to new PR authors as a condition of >>>>>>> merging. I feel like we will just be filling JIRA with spam if the >>>>>>> issues >>>>>>> are automatically created without a plan for triage. >>>>>>> >>>>>>> Andrew >>>>>>> >>>>>>> On Wed, Jul 11, 2018 at 4:12 PM Rui Wang <ruw...@google.com> wrote: >>>>>>> >>>>>>>> Maybe this is also a good thread to start the discussion that if we >>>>>>>> want to enforce postcommit test for every PR. >>>>>>>> >>>>>>>> Can we afford the cost of longer waiting time to catch potential >>>>>>>> bugs? >>>>>>>> >>>>>>>> -Rui >>>>>>>> >>>>>>>> On Wed, Jul 11, 2018 at 4:04 PM Mikhail Gryzykhin < >>>>>>>> mig...@google.com> wrote: >>>>>>>> >>>>>>>>> That's a valid point. >>>>>>>>> >>>>>>>>> Unfortunately, the JiraTestResultReporter plugin does not have >>>>>>>>> features to dynamically assign owners. Additionally, I don't think it >>>>>>>>> is >>>>>>>>> always easy to find proper owner for post-commit tests at first >>>>>>>>> glance, >>>>>>>>> since they usually cover broad specter of issues. >>>>>>>>> >>>>>>>>> My assumption is that we need someone to triage new issues. >>>>>>>>> >>>>>>>>> Ideally, any contributor, who sees failing test, should check >>>>>>>>> unassigned tickets and either do triage, or assign them to someone >>>>>>>>> who can. >>>>>>>>> I strongly encourage this approach. >>>>>>>>> >>>>>>>>> We have couple other ready-made options to consider: >>>>>>>>> 1. We can configure JIRA component owner who would be assigned to >>>>>>>>> created tickets. >>>>>>>>> 2. JiraTestReporterPlugin can assign tickets to specific user. >>>>>>>>> This is configured per Jenkins job. We can utilize this if someone >>>>>>>>> volunteers. >>>>>>>>> 3. Dynamic assignment will most likely require custom solution. >>>>>>>>> >>>>>>>>> --Mikhail >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Jul 11, 2018 at 3:34 PM Andrew Pilloud < >>>>>>>>> apill...@google.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Mikhail, >>>>>>>>>> >>>>>>>>>> I like the proposal! Hopefully this can replace the constant >>>>>>>>>> stream of build failure emails. I noticed one detail seems to be >>>>>>>>>> missing: >>>>>>>>>> How will new issues be assigned to the proper owner? Will the tool >>>>>>>>>> do this >>>>>>>>>> automatically or will we need someone to triage new issues? >>>>>>>>>> >>>>>>>>>> Andrew >>>>>>>>>> >>>>>>>>>> On Wed, Jul 11, 2018 at 3:07 PM Mikhail Gryzykhin < >>>>>>>>>> mig...@google.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi everyone, >>>>>>>>>>> >>>>>>>>>>> I want to add an automatic JIRA tickets creation for failing >>>>>>>>>>> post-commit tests. >>>>>>>>>>> >>>>>>>>>>> I wrote up design proposal doc with more details on this: >>>>>>>>>>> >>>>>>>>>>> https://docs.google.com/document/d/1kpsTy0sYJkLqlZvkPalDkqzBbpu-Wug0z-oWIVPo6UI >>>>>>>>>>> >>>>>>>>>>> Quick summary: >>>>>>>>>>> I suggest to utilize JiraTestResultReporter plugin. >>>>>>>>>>> Since this plugin is not installed on our Jenkins yet, we have >>>>>>>>>>> to request to Infra team to add it. >>>>>>>>>>> >>>>>>>>>>> Please, comment if this approach sounds good to you. >>>>>>>>>>> >>>>>>>>>>> Best regards, >>>>>>>>>>> --Mikhail >>>>>>>>>>> >>>>>>>>>>> -- ------- Jason Kuster Apache Beam / Google Cloud Dataflow See something? Say something. go/jasonkuster-feedback
