Re: [openstack-dev] [infra] redo gate jobs only

[Eric K]

Thanks a lot Jeremy and Andreas for the reference and the added context!

On 8/3/17, 10:49 AM, "Jeremy Stanley"  wrote:

>On 2017-08-03 08:15:36 +0200 (+0200), Andreas Jaeger wrote:
>[...]
>> "A patchset has to be approved to run tests in the gate pipeline. If the
>> patchset has failed in the gate pipeline (it will have been approved to
>> get into the gate pipeline) a recheck will first run the check jobs and
>> if those pass, it will again run the gate jobs. There is no way to only
>> run the gate jobs, the check jobs will first be run again."
>
>The reasons being:
>
>1. There's no good way to decide how long is too long to wait
>between passing jobs in check and running jobs in the gate. We used
>to not enforce this "clean check" policy and developers would
>repeatedly reverify broken changes back into the gate pipeline over
>and over creating a significant amount of additional disruption
>because their change had passed check jobs once (perhaps many months
>earlier). Now they at least only get to disrupt the gate once when
>that change gets approved, but after that it won't be able to make
>it back into the gate until a fixed revision is uploaded and so
>doesn't further slow down the merging of unrelated changes.
>
>2. If a change passes jobs once (in check) and then fails later (in
>the gate) then there's a fair chance it's introducing a
>nondeterministic bug (one which only manifests sometimes but not on
>every run). Back when we used to allow reverification directly in
>the gate pipeline for changes which passed check, we had people
>rechecking flaky changes until they passed and then reverifying them
>over and over after approval until they made them through the gate.
>Under these conditions a recheck followed by a reverify could merge
>changes which failed jobs 50% of the time; 9 rechecks and 9
>reverifies could merge a change which failed jobs 90% of the time on
>average. With the current requirements to pass both check and gate
>in series, it takes on average 3 rechecks to merge a 50% failing
>change and 99 rechecks to merge a 90% failing change.
>
>So basically if a change fails in the gate pipeline, there's good
>reason for it to get increased scrutiny at least in the form of
>trying the jobs again in the check pipeline before going back to the
>gate once more.
>-- 
>Jeremy Stanley
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra] redo gate jobs only

[Jeremy Stanley]

On 2017-08-03 08:15:36 +0200 (+0200), Andreas Jaeger wrote:
[...]
> "A patchset has to be approved to run tests in the gate pipeline. If the
> patchset has failed in the gate pipeline (it will have been approved to
> get into the gate pipeline) a recheck will first run the check jobs and
> if those pass, it will again run the gate jobs. There is no way to only
> run the gate jobs, the check jobs will first be run again."

The reasons being:

1. There's no good way to decide how long is too long to wait
between passing jobs in check and running jobs in the gate. We used
to not enforce this "clean check" policy and developers would
repeatedly reverify broken changes back into the gate pipeline over
and over creating a significant amount of additional disruption
because their change had passed check jobs once (perhaps many months
earlier). Now they at least only get to disrupt the gate once when
that change gets approved, but after that it won't be able to make
it back into the gate until a fixed revision is uploaded and so
doesn't further slow down the merging of unrelated changes.

2. If a change passes jobs once (in check) and then fails later (in
the gate) then there's a fair chance it's introducing a
nondeterministic bug (one which only manifests sometimes but not on
every run). Back when we used to allow reverification directly in
the gate pipeline for changes which passed check, we had people
rechecking flaky changes until they passed and then reverifying them
over and over after approval until they made them through the gate.
Under these conditions a recheck followed by a reverify could merge
changes which failed jobs 50% of the time; 9 rechecks and 9
reverifies could merge a change which failed jobs 90% of the time on
average. With the current requirements to pass both check and gate
in series, it takes on average 3 rechecks to merge a 50% failing
change and 99 rechecks to merge a 90% failing change.

So basically if a change fails in the gate pipeline, there's good
reason for it to get increased scrutiny at least in the form of
trying the jobs again in the check pipeline before going back to the
gate once more.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra] redo gate jobs only

[Andreas Jaeger]

On 2017-08-03 07:37, Eric K wrote:
> Hi all is there a command on review.openstack.org that triggers a redo of
> gate jobs but not check jobs (already passed)?
> `remerge` doesn¹t seem to do anything. Thanks!



This is explained in
https://docs.openstack.org/infra/manual/developers.html#automated-testing
"A patchset has to be approved to run tests in the gate pipeline. If the
patchset has failed in the gate pipeline (it will have been approved to
get into the gate pipeline) a recheck will first run the check jobs and
if those pass, it will again run the gate jobs. There is no way to only
run the gate jobs, the check jobs will first be run again."

Andreas
-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi
  SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
   GF: Felix Imendörffer, Jane Smithard, Graham Norton,
   HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [infra] redo gate jobs only

[Eric K]

Hi all is there a command on review.openstack.org that triggers a redo of
gate jobs but not check jobs (already passed)?
`remerge` doesn¹t seem to do anything. Thanks!



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev