Re: Automatically create JIRA tickets for failing post-commit tests

[Ahmet Altay]

This proposal looks good to me.

On Fri, Jul 13, 2018 at 10:49 AM, Jason Kuster 
wrote:

> I like the idea of auto-creating bugs. I think that if we do this we
> should make sure to set a time period after which we will evaluate whether
> it has succeeded, i.e. whether bugs filed as a result of this have been
> triaged, owned, fixed, and closed.
>
> On Fri, Jul 13, 2018 at 10:20 AM Mikhail Gryzykhin 
> wrote:
>
>> Can we get some PMC opinions on this?
>>
>> According to https://www.apache.org/dev/infra-contact#request-checklist,
>> it is highly recommended to have your opinions when filing a request.
>>
>
Do you need a specific comment or a LGTM is good enough?


>
>> --Mikhail
>>
>> Have feedback ?
>>
>>
>> On Wed, Jul 11, 2018 at 6:01 PM Anton Kedin  wrote:
>>
>>> I think this looks good, we should enable the plugin and try it out.
>>> Concrete details of the follow-up tasks (auto-assignment, triage, and
>>> dashboarding) will probably depend on how functional the plugin is and what
>>> the test failures data looks like.
>>>
>>> Regards,
>>> Anton
>>>
>>> On Wed, Jul 11, 2018 at 5:00 PM Mikhail Gryzykhin 
>>> wrote:
>>>
 @Yifan Zou 

 I believe that we should test-drive the system with tickets + PR first
 and decide on email notification later. We already have tests failure
 emails sent to commits@, I believe most people filter out or not
 signed up for that list though.

 It creates only one ticket, and keeps it for recurring test failures.

 @Andrew Pilloud 
 Thank you for the suggestion. I'll add it to design doc.

 --Mikhail



 On Wed, Jul 11, 2018 at 4:52 PM Yifan Zou  wrote:

> +1 to Andrew's concerns. Leaving the tickets unassigned will cause the
> ticket being ignored and no actions being taken.
>
> I can see the challenges on ticket assignment. Like Mikhail mentioned,
> the plugin does not support dynamic assignments. We have to implement
> custom script to determine the assignees and do some tricks to the jenkins
> job. Also, the post-commits tests usually cover tons of stuffs that it is
> difficult to find which part was broken and ask the right person to look
> into within the Auto JIRA process. Some naive thoughts: Are we able to 
> send
> emails to the dev@ to ask people to take care of the JIRA issues? Are
> we able to find component leads and ask them triage the test failure
> tickets?
>
> Another nitpick comment. Does the jenkins job file the JIRA issue in
> every test failure? Sometimes the test continuously fails in a time period
> due to the same reason. In this case, we will get some duplicate issues
> filed by Jenkins. I think it could be better if we can avoid filing issues
> if the previous one has not been resolved.
>
> Thanks.
> Yifan
>
>
> On Wed, Jul 11, 2018 at 4:37 PM Andrew Pilloud 
> wrote:
>
>> That sounds great. You should add this detail to the doc.
>>
>> On Wed, Jul 11, 2018 at 4:29 PM Mikhail Gryzykhin 
>> wrote:
>>
>>> We already have component for this purpose: "test-failures". All
>>> tickets created will go to that component. As an option, we can add 
>>> link to
>>> view list of open JIRA tickets to PR template.
>>>
>>> We also would want to create graph on dashboard with amount of
>>> unassigned and assigned bugs.
>>>
>>> I believe that we can also add counter of unassigned bugs to PR
>>> template. This way it will be easier for everyone to know when there's 
>>> some
>>> tests issue not attended.
>>>
>>> --Mikhail
>>>
>>>
>>> On Wed, Jul 11, 2018 at 4:24 PM Andrew Pilloud 
>>> wrote:
>>>
 So it sounds like you will want to create a component for untriaged
 issues so they are easy to find. I like the idea of distributing the 
 work
 of triaging post commit failures to new PR authors as a condition of
 merging. I feel like we will just be filling JIRA with spam if the 
 issues
 are automatically created without a plan for triage.

 Andrew

 On Wed, Jul 11, 2018 at 4:12 PM Rui Wang  wrote:

> Maybe this is also a good thread to start the discussion that if
> we want to enforce postcommit test for every PR.
>
> Can we afford the cost of longer waiting time to catch potential
> bugs?
>
> -Rui
>
> On Wed, Jul 11, 2018 at 4:04 PM Mikhail Gryzykhin <
> mig...@google.com> wrote:
>
>> That's a valid point.
>>
>> Unfortunately, the JiraTestResultReporter plugin does not have
>> features to dynamically assign owners. Additionally, I don't think 
>> it is
>> always easy to find proper owner for post-commit tests at first 
>> glance,
>>>

Re: Automatically create JIRA tickets for failing post-commit tests

[Jason Kuster]

I like the idea of auto-creating bugs. I think that if we do this we should
make sure to set a time period after which we will evaluate whether it has
succeeded, i.e. whether bugs filed as a result of this have been triaged,
owned, fixed, and closed.

On Fri, Jul 13, 2018 at 10:20 AM Mikhail Gryzykhin 
wrote:

> Can we get some PMC opinions on this?
>
> According to https://www.apache.org/dev/infra-contact#request-checklist,
> it is highly recommended to have your opinions when filing a request.
>
> --Mikhail
>
> Have feedback ?
>
>
> On Wed, Jul 11, 2018 at 6:01 PM Anton Kedin  wrote:
>
>> I think this looks good, we should enable the plugin and try it out.
>> Concrete details of the follow-up tasks (auto-assignment, triage, and
>> dashboarding) will probably depend on how functional the plugin is and what
>> the test failures data looks like.
>>
>> Regards,
>> Anton
>>
>> On Wed, Jul 11, 2018 at 5:00 PM Mikhail Gryzykhin 
>> wrote:
>>
>>> @Yifan Zou 
>>>
>>> I believe that we should test-drive the system with tickets + PR first
>>> and decide on email notification later. We already have tests failure
>>> emails sent to commits@, I believe most people filter out or not signed
>>> up for that list though.
>>>
>>> It creates only one ticket, and keeps it for recurring test failures.
>>>
>>> @Andrew Pilloud 
>>> Thank you for the suggestion. I'll add it to design doc.
>>>
>>> --Mikhail
>>>
>>>
>>>
>>> On Wed, Jul 11, 2018 at 4:52 PM Yifan Zou  wrote:
>>>
 +1 to Andrew's concerns. Leaving the tickets unassigned will cause the
 ticket being ignored and no actions being taken.

 I can see the challenges on ticket assignment. Like Mikhail mentioned,
 the plugin does not support dynamic assignments. We have to implement
 custom script to determine the assignees and do some tricks to the jenkins
 job. Also, the post-commits tests usually cover tons of stuffs that it is
 difficult to find which part was broken and ask the right person to look
 into within the Auto JIRA process. Some naive thoughts: Are we able to send
 emails to the dev@ to ask people to take care of the JIRA issues? Are
 we able to find component leads and ask them triage the test failure
 tickets?

 Another nitpick comment. Does the jenkins job file the JIRA issue in
 every test failure? Sometimes the test continuously fails in a time period
 due to the same reason. In this case, we will get some duplicate issues
 filed by Jenkins. I think it could be better if we can avoid filing issues
 if the previous one has not been resolved.

 Thanks.
 Yifan


 On Wed, Jul 11, 2018 at 4:37 PM Andrew Pilloud 
 wrote:

> That sounds great. You should add this detail to the doc.
>
> On Wed, Jul 11, 2018 at 4:29 PM Mikhail Gryzykhin 
> wrote:
>
>> We already have component for this purpose: "test-failures". All
>> tickets created will go to that component. As an option, we can add link 
>> to
>> view list of open JIRA tickets to PR template.
>>
>> We also would want to create graph on dashboard with amount of
>> unassigned and assigned bugs.
>>
>> I believe that we can also add counter of unassigned bugs to PR
>> template. This way it will be easier for everyone to know when there's 
>> some
>> tests issue not attended.
>>
>> --Mikhail
>>
>>
>> On Wed, Jul 11, 2018 at 4:24 PM Andrew Pilloud 
>> wrote:
>>
>>> So it sounds like you will want to create a component for untriaged
>>> issues so they are easy to find. I like the idea of distributing the 
>>> work
>>> of triaging post commit failures to new PR authors as a condition of
>>> merging. I feel like we will just be filling JIRA with spam if the 
>>> issues
>>> are automatically created without a plan for triage.
>>>
>>> Andrew
>>>
>>> On Wed, Jul 11, 2018 at 4:12 PM Rui Wang  wrote:
>>>
 Maybe this is also a good thread to start the discussion that if we
 want to enforce postcommit test for every PR.

 Can we afford the cost of longer waiting time to catch potential
 bugs?

 -Rui

 On Wed, Jul 11, 2018 at 4:04 PM Mikhail Gryzykhin <
 mig...@google.com> wrote:

> That's a valid point.
>
> Unfortunately, the JiraTestResultReporter plugin does not have
> features to dynamically assign owners. Additionally, I don't think it 
> is
> always easy to find proper owner for post-commit tests at first 
> glance,
> since they usually cover broad specter of issues.
>
> My assumption is that we need someone to triage new issues.
>
> Ideally, any contributor, who sees failing test, should check
> unassigned tickets and either do triage, or assign them to someone 
>

Re: Automatically create JIRA tickets for failing post-commit tests

[Mikhail Gryzykhin]

Can we get some PMC opinions on this?

According to https://www.apache.org/dev/infra-contact#request-checklist, it
is highly recommended to have your opinions when filing a request.

--Mikhail

Have feedback ?


On Wed, Jul 11, 2018 at 6:01 PM Anton Kedin  wrote:

> I think this looks good, we should enable the plugin and try it out.
> Concrete details of the follow-up tasks (auto-assignment, triage, and
> dashboarding) will probably depend on how functional the plugin is and what
> the test failures data looks like.
>
> Regards,
> Anton
>
> On Wed, Jul 11, 2018 at 5:00 PM Mikhail Gryzykhin 
> wrote:
>
>> @Yifan Zou 
>>
>> I believe that we should test-drive the system with tickets + PR first
>> and decide on email notification later. We already have tests failure
>> emails sent to commits@, I believe most people filter out or not signed
>> up for that list though.
>>
>> It creates only one ticket, and keeps it for recurring test failures.
>>
>> @Andrew Pilloud 
>> Thank you for the suggestion. I'll add it to design doc.
>>
>> --Mikhail
>>
>>
>>
>> On Wed, Jul 11, 2018 at 4:52 PM Yifan Zou  wrote:
>>
>>> +1 to Andrew's concerns. Leaving the tickets unassigned will cause the
>>> ticket being ignored and no actions being taken.
>>>
>>> I can see the challenges on ticket assignment. Like Mikhail mentioned,
>>> the plugin does not support dynamic assignments. We have to implement
>>> custom script to determine the assignees and do some tricks to the jenkins
>>> job. Also, the post-commits tests usually cover tons of stuffs that it is
>>> difficult to find which part was broken and ask the right person to look
>>> into within the Auto JIRA process. Some naive thoughts: Are we able to send
>>> emails to the dev@ to ask people to take care of the JIRA issues? Are
>>> we able to find component leads and ask them triage the test failure
>>> tickets?
>>>
>>> Another nitpick comment. Does the jenkins job file the JIRA issue in
>>> every test failure? Sometimes the test continuously fails in a time period
>>> due to the same reason. In this case, we will get some duplicate issues
>>> filed by Jenkins. I think it could be better if we can avoid filing issues
>>> if the previous one has not been resolved.
>>>
>>> Thanks.
>>> Yifan
>>>
>>>
>>> On Wed, Jul 11, 2018 at 4:37 PM Andrew Pilloud 
>>> wrote:
>>>
 That sounds great. You should add this detail to the doc.

 On Wed, Jul 11, 2018 at 4:29 PM Mikhail Gryzykhin 
 wrote:

> We already have component for this purpose: "test-failures". All
> tickets created will go to that component. As an option, we can add link 
> to
> view list of open JIRA tickets to PR template.
>
> We also would want to create graph on dashboard with amount of
> unassigned and assigned bugs.
>
> I believe that we can also add counter of unassigned bugs to PR
> template. This way it will be easier for everyone to know when there's 
> some
> tests issue not attended.
>
> --Mikhail
>
>
> On Wed, Jul 11, 2018 at 4:24 PM Andrew Pilloud 
> wrote:
>
>> So it sounds like you will want to create a component for untriaged
>> issues so they are easy to find. I like the idea of distributing the work
>> of triaging post commit failures to new PR authors as a condition of
>> merging. I feel like we will just be filling JIRA with spam if the issues
>> are automatically created without a plan for triage.
>>
>> Andrew
>>
>> On Wed, Jul 11, 2018 at 4:12 PM Rui Wang  wrote:
>>
>>> Maybe this is also a good thread to start the discussion that if we
>>> want to enforce postcommit test for every PR.
>>>
>>> Can we afford the cost of longer waiting time to catch potential
>>> bugs?
>>>
>>> -Rui
>>>
>>> On Wed, Jul 11, 2018 at 4:04 PM Mikhail Gryzykhin 
>>> wrote:
>>>
 That's a valid point.

 Unfortunately, the JiraTestResultReporter plugin does not have
 features to dynamically assign owners. Additionally, I don't think it 
 is
 always easy to find proper owner for post-commit tests at first glance,
 since they usually cover broad specter of issues.

 My assumption is that we need someone to triage new issues.

 Ideally, any contributor, who sees failing test, should check
 unassigned tickets and either do triage, or assign them to someone who 
 can.
 I strongly encourage this approach.

 We have couple other ready-made options to consider:
 1. We can configure JIRA component owner who would be assigned to
 created tickets.
 2. JiraTestReporterPlugin can assign tickets to specific user. This
 is configured per Jenkins job. We can utilize this if someone 
 volunteers.
 3. Dynamic assignment will most likely require custom solution.

>

Re: Automatically create JIRA tickets for failing post-commit tests

[Anton Kedin]

I think this looks good, we should enable the plugin and try it out.
Concrete details of the follow-up tasks (auto-assignment, triage, and
dashboarding) will probably depend on how functional the plugin is and what
the test failures data looks like.

Regards,
Anton

On Wed, Jul 11, 2018 at 5:00 PM Mikhail Gryzykhin  wrote:

> @Yifan Zou 
>
> I believe that we should test-drive the system with tickets + PR first and
> decide on email notification later. We already have tests failure emails
> sent to commits@, I believe most people filter out or not signed up for
> that list though.
>
> It creates only one ticket, and keeps it for recurring test failures.
>
> @Andrew Pilloud 
> Thank you for the suggestion. I'll add it to design doc.
>
> --Mikhail
>
>
>
> On Wed, Jul 11, 2018 at 4:52 PM Yifan Zou  wrote:
>
>> +1 to Andrew's concerns. Leaving the tickets unassigned will cause the
>> ticket being ignored and no actions being taken.
>>
>> I can see the challenges on ticket assignment. Like Mikhail mentioned,
>> the plugin does not support dynamic assignments. We have to implement
>> custom script to determine the assignees and do some tricks to the jenkins
>> job. Also, the post-commits tests usually cover tons of stuffs that it is
>> difficult to find which part was broken and ask the right person to look
>> into within the Auto JIRA process. Some naive thoughts: Are we able to send
>> emails to the dev@ to ask people to take care of the JIRA issues? Are we
>> able to find component leads and ask them triage the test failure tickets?
>>
>> Another nitpick comment. Does the jenkins job file the JIRA issue in
>> every test failure? Sometimes the test continuously fails in a time period
>> due to the same reason. In this case, we will get some duplicate issues
>> filed by Jenkins. I think it could be better if we can avoid filing issues
>> if the previous one has not been resolved.
>>
>> Thanks.
>> Yifan
>>
>>
>> On Wed, Jul 11, 2018 at 4:37 PM Andrew Pilloud 
>> wrote:
>>
>>> That sounds great. You should add this detail to the doc.
>>>
>>> On Wed, Jul 11, 2018 at 4:29 PM Mikhail Gryzykhin 
>>> wrote:
>>>
 We already have component for this purpose: "test-failures". All
 tickets created will go to that component. As an option, we can add link to
 view list of open JIRA tickets to PR template.

 We also would want to create graph on dashboard with amount of
 unassigned and assigned bugs.

 I believe that we can also add counter of unassigned bugs to PR
 template. This way it will be easier for everyone to know when there's some
 tests issue not attended.

 --Mikhail


 On Wed, Jul 11, 2018 at 4:24 PM Andrew Pilloud 
 wrote:

> So it sounds like you will want to create a component for untriaged
> issues so they are easy to find. I like the idea of distributing the work
> of triaging post commit failures to new PR authors as a condition of
> merging. I feel like we will just be filling JIRA with spam if the issues
> are automatically created without a plan for triage.
>
> Andrew
>
> On Wed, Jul 11, 2018 at 4:12 PM Rui Wang  wrote:
>
>> Maybe this is also a good thread to start the discussion that if we
>> want to enforce postcommit test for every PR.
>>
>> Can we afford the cost of longer waiting time to catch potential
>> bugs?
>>
>> -Rui
>>
>> On Wed, Jul 11, 2018 at 4:04 PM Mikhail Gryzykhin 
>> wrote:
>>
>>> That's a valid point.
>>>
>>> Unfortunately, the JiraTestResultReporter plugin does not have
>>> features to dynamically assign owners. Additionally, I don't think it is
>>> always easy to find proper owner for post-commit tests at first glance,
>>> since they usually cover broad specter of issues.
>>>
>>> My assumption is that we need someone to triage new issues.
>>>
>>> Ideally, any contributor, who sees failing test, should check
>>> unassigned tickets and either do triage, or assign them to someone who 
>>> can.
>>> I strongly encourage this approach.
>>>
>>> We have couple other ready-made options to consider:
>>> 1. We can configure JIRA component owner who would be assigned to
>>> created tickets.
>>> 2. JiraTestReporterPlugin can assign tickets to specific user. This
>>> is configured per Jenkins job. We can utilize this if someone 
>>> volunteers.
>>> 3. Dynamic assignment will most likely require custom solution.
>>>
>>> --Mikhail
>>>
>>>
>>> On Wed, Jul 11, 2018 at 3:34 PM Andrew Pilloud 
>>> wrote:
>>>
 Hi Mikhail,

 I like the proposal! Hopefully this can replace the constant stream
 of build failure emails. I noticed one detail seems to be missing:  How
 will new issues be assigned to the proper owner? Will the tool do this
 automatically or will we need someone to triage new issues?
>>

Re: Automatically create JIRA tickets for failing post-commit tests

[Mikhail Gryzykhin]

@Yifan Zou 

I believe that we should test-drive the system with tickets + PR first and
decide on email notification later. We already have tests failure emails
sent to commits@, I believe most people filter out or not signed up for
that list though.

It creates only one ticket, and keeps it for recurring test failures.

@Andrew Pilloud 
Thank you for the suggestion. I'll add it to design doc.

--Mikhail



On Wed, Jul 11, 2018 at 4:52 PM Yifan Zou  wrote:

> +1 to Andrew's concerns. Leaving the tickets unassigned will cause the
> ticket being ignored and no actions being taken.
>
> I can see the challenges on ticket assignment. Like Mikhail mentioned, the
> plugin does not support dynamic assignments. We have to implement custom
> script to determine the assignees and do some tricks to the jenkins job.
> Also, the post-commits tests usually cover tons of stuffs that it is
> difficult to find which part was broken and ask the right person to look
> into within the Auto JIRA process. Some naive thoughts: Are we able to send
> emails to the dev@ to ask people to take care of the JIRA issues? Are we
> able to find component leads and ask them triage the test failure tickets?
>
> Another nitpick comment. Does the jenkins job file the JIRA issue in every
> test failure? Sometimes the test continuously fails in a time period due to
> the same reason. In this case, we will get some duplicate issues filed by
> Jenkins. I think it could be better if we can avoid filing issues if the
> previous one has not been resolved.
>
> Thanks.
> Yifan
>
>
> On Wed, Jul 11, 2018 at 4:37 PM Andrew Pilloud 
> wrote:
>
>> That sounds great. You should add this detail to the doc.
>>
>> On Wed, Jul 11, 2018 at 4:29 PM Mikhail Gryzykhin 
>> wrote:
>>
>>> We already have component for this purpose: "test-failures". All tickets
>>> created will go to that component. As an option, we can add link to view
>>> list of open JIRA tickets to PR template.
>>>
>>> We also would want to create graph on dashboard with amount of
>>> unassigned and assigned bugs.
>>>
>>> I believe that we can also add counter of unassigned bugs to PR
>>> template. This way it will be easier for everyone to know when there's some
>>> tests issue not attended.
>>>
>>> --Mikhail
>>>
>>>
>>> On Wed, Jul 11, 2018 at 4:24 PM Andrew Pilloud 
>>> wrote:
>>>
 So it sounds like you will want to create a component for untriaged
 issues so they are easy to find. I like the idea of distributing the work
 of triaging post commit failures to new PR authors as a condition of
 merging. I feel like we will just be filling JIRA with spam if the issues
 are automatically created without a plan for triage.

 Andrew

 On Wed, Jul 11, 2018 at 4:12 PM Rui Wang  wrote:

> Maybe this is also a good thread to start the discussion that if we
> want to enforce postcommit test for every PR.
>
> Can we afford the cost of longer waiting time to catch potential bugs?
>
> -Rui
>
> On Wed, Jul 11, 2018 at 4:04 PM Mikhail Gryzykhin 
> wrote:
>
>> That's a valid point.
>>
>> Unfortunately, the JiraTestResultReporter plugin does not have
>> features to dynamically assign owners. Additionally, I don't think it is
>> always easy to find proper owner for post-commit tests at first glance,
>> since they usually cover broad specter of issues.
>>
>> My assumption is that we need someone to triage new issues.
>>
>> Ideally, any contributor, who sees failing test, should check
>> unassigned tickets and either do triage, or assign them to someone who 
>> can.
>> I strongly encourage this approach.
>>
>> We have couple other ready-made options to consider:
>> 1. We can configure JIRA component owner who would be assigned to
>> created tickets.
>> 2. JiraTestReporterPlugin can assign tickets to specific user. This
>> is configured per Jenkins job. We can utilize this if someone volunteers.
>> 3. Dynamic assignment will most likely require custom solution.
>>
>> --Mikhail
>>
>>
>> On Wed, Jul 11, 2018 at 3:34 PM Andrew Pilloud 
>> wrote:
>>
>>> Hi Mikhail,
>>>
>>> I like the proposal! Hopefully this can replace the constant stream
>>> of build failure emails. I noticed one detail seems to be missing:  How
>>> will new issues be assigned to the proper owner? Will the tool do this
>>> automatically or will we need someone to triage new issues?
>>>
>>> Andrew
>>>
>>> On Wed, Jul 11, 2018 at 3:07 PM Mikhail Gryzykhin 
>>> wrote:
>>>
 Hi everyone,

 I want to add an automatic JIRA tickets creation for failing
 post-commit tests.

 I wrote up design proposal doc with more details on this:

 https://docs.google.com/document/d/1kpsTy0sYJkLqlZvkPalDkqzBbpu-Wug0z-oWIVPo6UI

 Quick summary:
 I suggest to u

Re: Automatically create JIRA tickets for failing post-commit tests

[Yifan Zou]

+1 to Andrew's concerns. Leaving the tickets unassigned will cause the
ticket being ignored and no actions being taken.

I can see the challenges on ticket assignment. Like Mikhail mentioned, the
plugin does not support dynamic assignments. We have to implement custom
script to determine the assignees and do some tricks to the jenkins job.
Also, the post-commits tests usually cover tons of stuffs that it is
difficult to find which part was broken and ask the right person to look
into within the Auto JIRA process. Some naive thoughts: Are we able to send
emails to the dev@ to ask people to take care of the JIRA issues? Are we
able to find component leads and ask them triage the test failure tickets?

Another nitpick comment. Does the jenkins job file the JIRA issue in every
test failure? Sometimes the test continuously fails in a time period due to
the same reason. In this case, we will get some duplicate issues filed by
Jenkins. I think it could be better if we can avoid filing issues if the
previous one has not been resolved.

Thanks.
Yifan


On Wed, Jul 11, 2018 at 4:37 PM Andrew Pilloud  wrote:

> That sounds great. You should add this detail to the doc.
>
> On Wed, Jul 11, 2018 at 4:29 PM Mikhail Gryzykhin 
> wrote:
>
>> We already have component for this purpose: "test-failures". All tickets
>> created will go to that component. As an option, we can add link to view
>> list of open JIRA tickets to PR template.
>>
>> We also would want to create graph on dashboard with amount of unassigned
>> and assigned bugs.
>>
>> I believe that we can also add counter of unassigned bugs to PR template.
>> This way it will be easier for everyone to know when there's some tests
>> issue not attended.
>>
>> --Mikhail
>>
>>
>> On Wed, Jul 11, 2018 at 4:24 PM Andrew Pilloud 
>> wrote:
>>
>>> So it sounds like you will want to create a component for untriaged
>>> issues so they are easy to find. I like the idea of distributing the work
>>> of triaging post commit failures to new PR authors as a condition of
>>> merging. I feel like we will just be filling JIRA with spam if the issues
>>> are automatically created without a plan for triage.
>>>
>>> Andrew
>>>
>>> On Wed, Jul 11, 2018 at 4:12 PM Rui Wang  wrote:
>>>
 Maybe this is also a good thread to start the discussion that if we
 want to enforce postcommit test for every PR.

 Can we afford the cost of longer waiting time to catch potential bugs?

 -Rui

 On Wed, Jul 11, 2018 at 4:04 PM Mikhail Gryzykhin 
 wrote:

> That's a valid point.
>
> Unfortunately, the JiraTestResultReporter plugin does not have
> features to dynamically assign owners. Additionally, I don't think it is
> always easy to find proper owner for post-commit tests at first glance,
> since they usually cover broad specter of issues.
>
> My assumption is that we need someone to triage new issues.
>
> Ideally, any contributor, who sees failing test, should check
> unassigned tickets and either do triage, or assign them to someone who 
> can.
> I strongly encourage this approach.
>
> We have couple other ready-made options to consider:
> 1. We can configure JIRA component owner who would be assigned to
> created tickets.
> 2. JiraTestReporterPlugin can assign tickets to specific user. This is
> configured per Jenkins job. We can utilize this if someone volunteers.
> 3. Dynamic assignment will most likely require custom solution.
>
> --Mikhail
>
>
> On Wed, Jul 11, 2018 at 3:34 PM Andrew Pilloud 
> wrote:
>
>> Hi Mikhail,
>>
>> I like the proposal! Hopefully this can replace the constant stream
>> of build failure emails. I noticed one detail seems to be missing:  How
>> will new issues be assigned to the proper owner? Will the tool do this
>> automatically or will we need someone to triage new issues?
>>
>> Andrew
>>
>> On Wed, Jul 11, 2018 at 3:07 PM Mikhail Gryzykhin 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> I want to add an automatic JIRA tickets creation for failing
>>> post-commit tests.
>>>
>>> I wrote up design proposal doc with more details on this:
>>>
>>> https://docs.google.com/document/d/1kpsTy0sYJkLqlZvkPalDkqzBbpu-Wug0z-oWIVPo6UI
>>>
>>> Quick summary:
>>> I suggest to utilize JiraTestResultReporter plugin.
>>> Since this plugin is not installed on our Jenkins yet, we have to
>>> request to Infra team to add it.
>>>
>>> Please, comment if this approach sounds good to you.
>>>
>>> Best regards,
>>> --Mikhail
>>>
>>>

Re: Automatically create JIRA tickets for failing post-commit tests

[Andrew Pilloud]

That sounds great. You should add this detail to the doc.

On Wed, Jul 11, 2018 at 4:29 PM Mikhail Gryzykhin  wrote:

> We already have component for this purpose: "test-failures". All tickets
> created will go to that component. As an option, we can add link to view
> list of open JIRA tickets to PR template.
>
> We also would want to create graph on dashboard with amount of unassigned
> and assigned bugs.
>
> I believe that we can also add counter of unassigned bugs to PR template.
> This way it will be easier for everyone to know when there's some tests
> issue not attended.
>
> --Mikhail
>
>
> On Wed, Jul 11, 2018 at 4:24 PM Andrew Pilloud 
> wrote:
>
>> So it sounds like you will want to create a component for untriaged
>> issues so they are easy to find. I like the idea of distributing the work
>> of triaging post commit failures to new PR authors as a condition of
>> merging. I feel like we will just be filling JIRA with spam if the issues
>> are automatically created without a plan for triage.
>>
>> Andrew
>>
>> On Wed, Jul 11, 2018 at 4:12 PM Rui Wang  wrote:
>>
>>> Maybe this is also a good thread to start the discussion that if we want
>>> to enforce postcommit test for every PR.
>>>
>>> Can we afford the cost of longer waiting time to catch potential bugs?
>>>
>>> -Rui
>>>
>>> On Wed, Jul 11, 2018 at 4:04 PM Mikhail Gryzykhin 
>>> wrote:
>>>
 That's a valid point.

 Unfortunately, the JiraTestResultReporter plugin does not have features
 to dynamically assign owners. Additionally, I don't think it is always easy
 to find proper owner for post-commit tests at first glance, since they
 usually cover broad specter of issues.

 My assumption is that we need someone to triage new issues.

 Ideally, any contributor, who sees failing test, should check
 unassigned tickets and either do triage, or assign them to someone who can.
 I strongly encourage this approach.

 We have couple other ready-made options to consider:
 1. We can configure JIRA component owner who would be assigned to
 created tickets.
 2. JiraTestReporterPlugin can assign tickets to specific user. This is
 configured per Jenkins job. We can utilize this if someone volunteers.
 3. Dynamic assignment will most likely require custom solution.

 --Mikhail


 On Wed, Jul 11, 2018 at 3:34 PM Andrew Pilloud 
 wrote:

> Hi Mikhail,
>
> I like the proposal! Hopefully this can replace the constant stream of
> build failure emails. I noticed one detail seems to be missing:  How will
> new issues be assigned to the proper owner? Will the tool do this
> automatically or will we need someone to triage new issues?
>
> Andrew
>
> On Wed, Jul 11, 2018 at 3:07 PM Mikhail Gryzykhin 
> wrote:
>
>> Hi everyone,
>>
>> I want to add an automatic JIRA tickets creation for failing
>> post-commit tests.
>>
>> I wrote up design proposal doc with more details on this:
>>
>> https://docs.google.com/document/d/1kpsTy0sYJkLqlZvkPalDkqzBbpu-Wug0z-oWIVPo6UI
>>
>> Quick summary:
>> I suggest to utilize JiraTestResultReporter plugin.
>> Since this plugin is not installed on our Jenkins yet, we have to
>> request to Infra team to add it.
>>
>> Please, comment if this approach sounds good to you.
>>
>> Best regards,
>> --Mikhail
>>
>>

Re: Automatically create JIRA tickets for failing post-commit tests

[Mikhail Gryzykhin]

We already have component for this purpose: "test-failures". All tickets
created will go to that component. As an option, we can add link to view
list of open JIRA tickets to PR template.

We also would want to create graph on dashboard with amount of unassigned
and assigned bugs.

I believe that we can also add counter of unassigned bugs to PR template.
This way it will be easier for everyone to know when there's some tests
issue not attended.

--Mikhail


On Wed, Jul 11, 2018 at 4:24 PM Andrew Pilloud  wrote:

> So it sounds like you will want to create a component for untriaged issues
> so they are easy to find. I like the idea of distributing the work of
> triaging post commit failures to new PR authors as a condition of merging.
> I feel like we will just be filling JIRA with spam if the issues are
> automatically created without a plan for triage.
>
> Andrew
>
> On Wed, Jul 11, 2018 at 4:12 PM Rui Wang  wrote:
>
>> Maybe this is also a good thread to start the discussion that if we want
>> to enforce postcommit test for every PR.
>>
>> Can we afford the cost of longer waiting time to catch potential bugs?
>>
>> -Rui
>>
>> On Wed, Jul 11, 2018 at 4:04 PM Mikhail Gryzykhin 
>> wrote:
>>
>>> That's a valid point.
>>>
>>> Unfortunately, the JiraTestResultReporter plugin does not have features
>>> to dynamically assign owners. Additionally, I don't think it is always easy
>>> to find proper owner for post-commit tests at first glance, since they
>>> usually cover broad specter of issues.
>>>
>>> My assumption is that we need someone to triage new issues.
>>>
>>> Ideally, any contributor, who sees failing test, should check unassigned
>>> tickets and either do triage, or assign them to someone who can. I strongly
>>> encourage this approach.
>>>
>>> We have couple other ready-made options to consider:
>>> 1. We can configure JIRA component owner who would be assigned to
>>> created tickets.
>>> 2. JiraTestReporterPlugin can assign tickets to specific user. This is
>>> configured per Jenkins job. We can utilize this if someone volunteers.
>>> 3. Dynamic assignment will most likely require custom solution.
>>>
>>> --Mikhail
>>>
>>>
>>> On Wed, Jul 11, 2018 at 3:34 PM Andrew Pilloud 
>>> wrote:
>>>
 Hi Mikhail,

 I like the proposal! Hopefully this can replace the constant stream of
 build failure emails. I noticed one detail seems to be missing:  How will
 new issues be assigned to the proper owner? Will the tool do this
 automatically or will we need someone to triage new issues?

 Andrew

 On Wed, Jul 11, 2018 at 3:07 PM Mikhail Gryzykhin 
 wrote:

> Hi everyone,
>
> I want to add an automatic JIRA tickets creation for failing
> post-commit tests.
>
> I wrote up design proposal doc with more details on this:
>
> https://docs.google.com/document/d/1kpsTy0sYJkLqlZvkPalDkqzBbpu-Wug0z-oWIVPo6UI
>
> Quick summary:
> I suggest to utilize JiraTestResultReporter plugin.
> Since this plugin is not installed on our Jenkins yet, we have to
> request to Infra team to add it.
>
> Please, comment if this approach sounds good to you.
>
> Best regards,
> --Mikhail
>
>

Re: Automatically create JIRA tickets for failing post-commit tests

[Mikhail Gryzykhin]

Hi Rui,

I would suggest to start another thread for that discussion.

Lets keep this discussion focused on automated JIRA tickets creation.

--Mikhail


On Wed, Jul 11, 2018 at 4:12 PM Rui Wang  wrote:

> Maybe this is also a good thread to start the discussion that if we want
> to enforce postcommit test for every PR.
>
> Can we afford the cost of longer waiting time to catch potential bugs?
>
> -Rui
>
> On Wed, Jul 11, 2018 at 4:04 PM Mikhail Gryzykhin 
> wrote:
>
>> That's a valid point.
>>
>> Unfortunately, the JiraTestResultReporter plugin does not have features
>> to dynamically assign owners. Additionally, I don't think it is always easy
>> to find proper owner for post-commit tests at first glance, since they
>> usually cover broad specter of issues.
>>
>> My assumption is that we need someone to triage new issues.
>>
>> Ideally, any contributor, who sees failing test, should check unassigned
>> tickets and either do triage, or assign them to someone who can. I strongly
>> encourage this approach.
>>
>> We have couple other ready-made options to consider:
>> 1. We can configure JIRA component owner who would be assigned to created
>> tickets.
>> 2. JiraTestReporterPlugin can assign tickets to specific user. This is
>> configured per Jenkins job. We can utilize this if someone volunteers.
>> 3. Dynamic assignment will most likely require custom solution.
>>
>> --Mikhail
>>
>>
>> On Wed, Jul 11, 2018 at 3:34 PM Andrew Pilloud 
>> wrote:
>>
>>> Hi Mikhail,
>>>
>>> I like the proposal! Hopefully this can replace the constant stream of
>>> build failure emails. I noticed one detail seems to be missing:  How will
>>> new issues be assigned to the proper owner? Will the tool do this
>>> automatically or will we need someone to triage new issues?
>>>
>>> Andrew
>>>
>>> On Wed, Jul 11, 2018 at 3:07 PM Mikhail Gryzykhin 
>>> wrote:
>>>
 Hi everyone,

 I want to add an automatic JIRA tickets creation for failing
 post-commit tests.

 I wrote up design proposal doc with more details on this:

 https://docs.google.com/document/d/1kpsTy0sYJkLqlZvkPalDkqzBbpu-Wug0z-oWIVPo6UI

 Quick summary:
 I suggest to utilize JiraTestResultReporter plugin.
 Since this plugin is not installed on our Jenkins yet, we have to
 request to Infra team to add it.

 Please, comment if this approach sounds good to you.

 Best regards,
 --Mikhail

Re: Automatically create JIRA tickets for failing post-commit tests

[Andrew Pilloud]

So it sounds like you will want to create a component for untriaged issues
so they are easy to find. I like the idea of distributing the work of
triaging post commit failures to new PR authors as a condition of merging.
I feel like we will just be filling JIRA with spam if the issues are
automatically created without a plan for triage.

Andrew

On Wed, Jul 11, 2018 at 4:12 PM Rui Wang  wrote:

> Maybe this is also a good thread to start the discussion that if we want
> to enforce postcommit test for every PR.
>
> Can we afford the cost of longer waiting time to catch potential bugs?
>
> -Rui
>
> On Wed, Jul 11, 2018 at 4:04 PM Mikhail Gryzykhin 
> wrote:
>
>> That's a valid point.
>>
>> Unfortunately, the JiraTestResultReporter plugin does not have features
>> to dynamically assign owners. Additionally, I don't think it is always easy
>> to find proper owner for post-commit tests at first glance, since they
>> usually cover broad specter of issues.
>>
>> My assumption is that we need someone to triage new issues.
>>
>> Ideally, any contributor, who sees failing test, should check unassigned
>> tickets and either do triage, or assign them to someone who can. I strongly
>> encourage this approach.
>>
>> We have couple other ready-made options to consider:
>> 1. We can configure JIRA component owner who would be assigned to created
>> tickets.
>> 2. JiraTestReporterPlugin can assign tickets to specific user. This is
>> configured per Jenkins job. We can utilize this if someone volunteers.
>> 3. Dynamic assignment will most likely require custom solution.
>>
>> --Mikhail
>>
>>
>> On Wed, Jul 11, 2018 at 3:34 PM Andrew Pilloud 
>> wrote:
>>
>>> Hi Mikhail,
>>>
>>> I like the proposal! Hopefully this can replace the constant stream of
>>> build failure emails. I noticed one detail seems to be missing:  How will
>>> new issues be assigned to the proper owner? Will the tool do this
>>> automatically or will we need someone to triage new issues?
>>>
>>> Andrew
>>>
>>> On Wed, Jul 11, 2018 at 3:07 PM Mikhail Gryzykhin 
>>> wrote:
>>>
 Hi everyone,

 I want to add an automatic JIRA tickets creation for failing
 post-commit tests.

 I wrote up design proposal doc with more details on this:

 https://docs.google.com/document/d/1kpsTy0sYJkLqlZvkPalDkqzBbpu-Wug0z-oWIVPo6UI

 Quick summary:
 I suggest to utilize JiraTestResultReporter plugin.
 Since this plugin is not installed on our Jenkins yet, we have to
 request to Infra team to add it.

 Please, comment if this approach sounds good to you.

 Best regards,
 --Mikhail

Re: Automatically create JIRA tickets for failing post-commit tests

[Rui Wang]

Maybe this is also a good thread to start the discussion that if we want to
enforce postcommit test for every PR.

Can we afford the cost of longer waiting time to catch potential bugs?

-Rui

On Wed, Jul 11, 2018 at 4:04 PM Mikhail Gryzykhin  wrote:

> That's a valid point.
>
> Unfortunately, the JiraTestResultReporter plugin does not have features to
> dynamically assign owners. Additionally, I don't think it is always easy to
> find proper owner for post-commit tests at first glance, since they usually
> cover broad specter of issues.
>
> My assumption is that we need someone to triage new issues.
>
> Ideally, any contributor, who sees failing test, should check unassigned
> tickets and either do triage, or assign them to someone who can. I strongly
> encourage this approach.
>
> We have couple other ready-made options to consider:
> 1. We can configure JIRA component owner who would be assigned to created
> tickets.
> 2. JiraTestReporterPlugin can assign tickets to specific user. This is
> configured per Jenkins job. We can utilize this if someone volunteers.
> 3. Dynamic assignment will most likely require custom solution.
>
> --Mikhail
>
>
> On Wed, Jul 11, 2018 at 3:34 PM Andrew Pilloud 
> wrote:
>
>> Hi Mikhail,
>>
>> I like the proposal! Hopefully this can replace the constant stream of
>> build failure emails. I noticed one detail seems to be missing:  How will
>> new issues be assigned to the proper owner? Will the tool do this
>> automatically or will we need someone to triage new issues?
>>
>> Andrew
>>
>> On Wed, Jul 11, 2018 at 3:07 PM Mikhail Gryzykhin 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> I want to add an automatic JIRA tickets creation for failing post-commit
>>> tests.
>>>
>>> I wrote up design proposal doc with more details on this:
>>>
>>> https://docs.google.com/document/d/1kpsTy0sYJkLqlZvkPalDkqzBbpu-Wug0z-oWIVPo6UI
>>>
>>> Quick summary:
>>> I suggest to utilize JiraTestResultReporter plugin.
>>> Since this plugin is not installed on our Jenkins yet, we have to
>>> request to Infra team to add it.
>>>
>>> Please, comment if this approach sounds good to you.
>>>
>>> Best regards,
>>> --Mikhail
>>>
>>>

Re: Automatically create JIRA tickets for failing post-commit tests

[Mikhail Gryzykhin]

That's a valid point.

Unfortunately, the JiraTestResultReporter plugin does not have features to
dynamically assign owners. Additionally, I don't think it is always easy to
find proper owner for post-commit tests at first glance, since they usually
cover broad specter of issues.

My assumption is that we need someone to triage new issues.

Ideally, any contributor, who sees failing test, should check unassigned
tickets and either do triage, or assign them to someone who can. I strongly
encourage this approach.

We have couple other ready-made options to consider:
1. We can configure JIRA component owner who would be assigned to created
tickets.
2. JiraTestReporterPlugin can assign tickets to specific user. This is
configured per Jenkins job. We can utilize this if someone volunteers.
3. Dynamic assignment will most likely require custom solution.

--Mikhail


On Wed, Jul 11, 2018 at 3:34 PM Andrew Pilloud  wrote:

> Hi Mikhail,
>
> I like the proposal! Hopefully this can replace the constant stream of
> build failure emails. I noticed one detail seems to be missing:  How will
> new issues be assigned to the proper owner? Will the tool do this
> automatically or will we need someone to triage new issues?
>
> Andrew
>
> On Wed, Jul 11, 2018 at 3:07 PM Mikhail Gryzykhin 
> wrote:
>
>> Hi everyone,
>>
>> I want to add an automatic JIRA tickets creation for failing post-commit
>> tests.
>>
>> I wrote up design proposal doc with more details on this:
>>
>> https://docs.google.com/document/d/1kpsTy0sYJkLqlZvkPalDkqzBbpu-Wug0z-oWIVPo6UI
>>
>> Quick summary:
>> I suggest to utilize JiraTestResultReporter plugin.
>> Since this plugin is not installed on our Jenkins yet, we have to request
>> to Infra team to add it.
>>
>> Please, comment if this approach sounds good to you.
>>
>> Best regards,
>> --Mikhail
>>
>>

Re: Automatically create JIRA tickets for failing post-commit tests

[Andrew Pilloud]

Hi Mikhail,

I like the proposal! Hopefully this can replace the constant stream of
build failure emails. I noticed one detail seems to be missing:  How will
new issues be assigned to the proper owner? Will the tool do this
automatically or will we need someone to triage new issues?

Andrew

On Wed, Jul 11, 2018 at 3:07 PM Mikhail Gryzykhin  wrote:

> Hi everyone,
>
> I want to add an automatic JIRA tickets creation for failing post-commit
> tests.
>
> I wrote up design proposal doc with more details on this:
>
> https://docs.google.com/document/d/1kpsTy0sYJkLqlZvkPalDkqzBbpu-Wug0z-oWIVPo6UI
>
> Quick summary:
> I suggest to utilize JiraTestResultReporter plugin.
> Since this plugin is not installed on our Jenkins yet, we have to request
> to Infra team to add it.
>
> Please, comment if this approach sounds good to you.
>
> Best regards,
> --Mikhail
>
>

Automatically create JIRA tickets for failing post-commit tests

[Mikhail Gryzykhin]

Hi everyone,

I want to add an automatic JIRA tickets creation for failing post-commit
tests.

I wrote up design proposal doc with more details on this:
https://docs.google.com/document/d/1kpsTy0sYJkLqlZvkPalDkqzBbpu-Wug0z-oWIVPo6UI

Quick summary:
I suggest to utilize JiraTestResultReporter plugin.
Since this plugin is not installed on our Jenkins yet, we have to request
to Infra team to add it.

Please, comment if this approach sounds good to you.

Best regards,
--Mikhail