[OMPI devel] Jenkins testing - what purpose are we striving to achieve?

[Ralph Castain]

Hi folks

I’m trying to get a handle on our use of Jenkins testing for PRs prior to 
committing them. When we first discussed this, it was my impression that our 
objective was to screen PRs to catch any errors caused by differences in 
environment and to avoid regressions. However, it appears that the tests keep 
changing without warning, leading to the impression that we are using Jenkins 
as a “mini-MTT” testing device.

So I think we need to come to consensus on the purpose of the Jenkins testing. 
If it is to screen for regressions, then the tests need to remain stable. A PR 
that does not introduce any new problems might not address old ones, but that 
is no reason to flag it as an “error”.

On the other hand, if the objective is to use Jenkins as a “mini-MTT”, then we 
need to agree on how/when a PR is ready to be merged. Insisting that nothing be 
merged until even a mini-MTT is perfectly clean is probably excessively 
prohibitive - it would require that the entire community (and not just the one 
proposing the PR) take responsibility for cleaning up the code base against any 
and all imposed tests.

So I would welcome opinions on this: are we using Jenkins as a screening tool 
on changes, or as a test for overall correctness of the code base?

Ralph

Re: [OMPI devel] Jenkins testing - what purpose are we striving to achieve?

[Howard Pritchard]

HI Ralph,

We briefly discussed this some today.  I would like to avoid the mini-MTT
approach for PR checking.
At the same time, one can also see why it might be useful from time to time
to make changes to
the script a given jenkins project runs on PRs.

An idea we discussed was to have jenkins folks support a "stable" version
of their jenkins script.  If they would
like to make changes,  they would create an experimental, temporary jenkins
project to run the new script.
If the new project's script runs clean against open PRs, the new script can
be swapped in to the
original jenkins project.  The experimental project could then be
deactivated.  If the new script showed failures in the
open PRs, or against master or other branch, issues can be opened to track
the problem(s) found by the
script.  The experimental, temporary jenkins project can continue to run,
but its  "failure" status can be ignored
until the underlying bug(s) is fixed.

I don't think it makes much sense to run a jenkins script against PRs if it
fails when run against master.
The purpose of jenkins PR testing is to trap new problems, not to keep
reminding us there are problems
with the underlying branch which the PR targets.

Howard


2016-06-07 13:33 GMT-06:00 Ralph Castain :

> Hi folks
>
> I’m trying to get a handle on our use of Jenkins testing for PRs prior to
> committing them. When we first discussed this, it was my impression that
> our objective was to screen PRs to catch any errors caused by differences
> in environment and to avoid regressions. However, it appears that the tests
> keep changing without warning, leading to the impression that we are using
> Jenkins as a “mini-MTT” testing device.
>
> So I think we need to come to consensus on the purpose of the Jenkins
> testing. If it is to screen for regressions, then the tests need to remain
> stable. A PR that does not introduce any new problems might not address old
> ones, but that is no reason to flag it as an “error”.
>
> On the other hand, if the objective is to use Jenkins as a “mini-MTT”,
> then we need to agree on how/when a PR is ready to be merged. Insisting
> that nothing be merged until even a mini-MTT is perfectly clean is probably
> excessively prohibitive - it would require that the entire community (and
> not just the one proposing the PR) take responsibility for cleaning up the
> code base against any and all imposed tests.
>
> So I would welcome opinions on this: are we using Jenkins as a screening
> tool on changes, or as a test for overall correctness of the code base?
>
> Ralph
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/06/19087.php

Re: [OMPI devel] Jenkins testing - what purpose are we striving to achieve?

[Ralph Castain]

I would agree with all those points 

> On Jun 7, 2016, at 2:12 PM, Howard Pritchard  wrote:
> 
> HI Ralph,
> 
> We briefly discussed this some today.  I would like to avoid the mini-MTT 
> approach for PR checking.
> At the same time, one can also see why it might be useful from time to time 
> to make changes to
> the script a given jenkins project runs on PRs.  
> 
> An idea we discussed was to have jenkins folks support a "stable" version of 
> their jenkins script.  If they would
> like to make changes,  they would create an experimental, temporary jenkins 
> project to run the new script.
> If the new project's script runs clean against open PRs, the new script can 
> be swapped in to the
> original jenkins project.  The experimental project could then be 
> deactivated.  If the new script showed failures in the
> open PRs, or against master or other branch, issues can be opened to track 
> the problem(s) found by the
> script.  The experimental, temporary jenkins project can continue to run, but 
> its  "failure" status can be ignored
> until the underlying bug(s) is fixed.
> 
> I don't think it makes much sense to run a jenkins script against PRs if it 
> fails when run against master.
> The purpose of jenkins PR testing is to trap new problems, not to keep 
> reminding us there are problems
> with the underlying branch which the PR targets.
> 
> Howard
> 
> 
> 2016-06-07 13:33 GMT-06:00 Ralph Castain  >:
> Hi folks
> 
> I’m trying to get a handle on our use of Jenkins testing for PRs prior to 
> committing them. When we first discussed this, it was my impression that our 
> objective was to screen PRs to catch any errors caused by differences in 
> environment and to avoid regressions. However, it appears that the tests keep 
> changing without warning, leading to the impression that we are using Jenkins 
> as a “mini-MTT” testing device.
> 
> So I think we need to come to consensus on the purpose of the Jenkins 
> testing. If it is to screen for regressions, then the tests need to remain 
> stable. A PR that does not introduce any new problems might not address old 
> ones, but that is no reason to flag it as an “error”.
> 
> On the other hand, if the objective is to use Jenkins as a “mini-MTT”, then 
> we need to agree on how/when a PR is ready to be merged. Insisting that 
> nothing be merged until even a mini-MTT is perfectly clean is probably 
> excessively prohibitive - it would require that the entire community (and not 
> just the one proposing the PR) take responsibility for cleaning up the code 
> base against any and all imposed tests.
> 
> So I would welcome opinions on this: are we using Jenkins as a screening tool 
> on changes, or as a test for overall correctness of the code base?
> 
> Ralph
> 
> ___
> devel mailing list
> de...@open-mpi.org 
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel 
> 
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2016/06/19087.php 
> 
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2016/06/19088.php

Re: [OMPI devel] Jenkins testing - what purpose are we striving to achieve?

[Gilles Gouaillardet]

my 0.02 US$


from an implementation point of view, the canonical way of using Jenkins 
with Github is


1) receive/poll a new PR

2) create a "check" and mark it pending

3) run a script

4) update the "check" status (OK/Failed) based on the exit status of the 
script.



that being said, it is possible to be "creative" in the script and use 
the github api.


For example, one script could create several checks, or post comments 
and set tags in the PR.



as far as i am concerned, i think (all) the check(s) should be expected 
to success.


and i would also consider as a plus if some comments are posted or tags 
are set when some non standard tests fail, and/or to remind some issues 
still exists.


for example, the check status could be OK if OpenMPI builds 
successfully, and a comment could be posted to indicate valgrind test fails.



that would save some resources (only one Jenkins server, OpenMPI is 
built once per PR) and make PR statuses easier to interpret (e.g. all 
checks should success unless there is a bug or an intermittent failure) 
while providing additional information.



Cheers,


Gilles


On 6/8/2016 6:25 AM, Ralph Castain wrote:

I would agree with all those points

On Jun 7, 2016, at 2:12 PM, Howard Pritchard > wrote:


HI Ralph,

We briefly discussed this some today.  I would like to avoid the 
mini-MTT approach for PR checking.
At the same time, one can also see why it might be useful from time 
to time to make changes to

the script a given jenkins project runs on PRs.

An idea we discussed was to have jenkins folks support a "stable" 
version of their jenkins script.  If they would
like to make changes,  they would create an experimental, temporary 
jenkins project to run the new script.
If the new project's script runs clean against open PRs, the new 
script can be swapped in to the
original jenkins project.  The experimental project could then be 
deactivated.  If the new script showed failures in the
open PRs, or against master or other branch, issues can be opened to 
track the problem(s) found by the
script.  The experimental, temporary jenkins project can continue to 
run, but its  "failure" status can be ignored

until the underlying bug(s) is fixed.

I don't think it makes much sense to run a jenkins script against PRs 
if it fails when run against master.
The purpose of jenkins PR testing is to trap new problems, not to 
keep reminding us there are problems

with the underlying branch which the PR targets.

Howard


2016-06-07 13:33 GMT-06:00 Ralph Castain >:


Hi folks

I’m trying to get a handle on our use of Jenkins testing for PRs
prior to committing them. When we first discussed this, it was my
impression that our objective was to screen PRs to catch any
errors caused by differences in environment and to avoid
regressions. However, it appears that the tests keep changing
without warning, leading to the impression that we are using
Jenkins as a “mini-MTT” testing device.

So I think we need to come to consensus on the purpose of the
Jenkins testing. If it is to screen for regressions, then the
tests need to remain stable. A PR that does not introduce any new
problems might not address old ones, but that is no reason to
flag it as an “error”.

On the other hand, if the objective is to use Jenkins as a
“mini-MTT”, then we need to agree on how/when a PR is ready to be
merged. Insisting that nothing be merged until even a mini-MTT is
perfectly clean is probably excessively prohibitive - it would
require that the entire community (and not just the one proposing
the PR) take responsibility for cleaning up the code base against
any and all imposed tests.

So I would welcome opinions on this: are we using Jenkins as a
screening tool on changes, or as a test for overall correctness
of the code base?

Ralph

___
devel mailing list
de...@open-mpi.org 
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post:
http://www.open-mpi.org/community/lists/devel/2016/06/19087.php


___
devel mailing list
de...@open-mpi.org 
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2016/06/19088.php




___
devel mailing list
de...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2016/06/19089.php