my 0.02 US$
from an implementation point of view, the canonical way of using Jenkins
with Github is
1) receive/poll a new PR
2) create a "check" and mark it pending
3) run a script
4) update the "check" status (OK/Failed) based on the exit status of the
script.
that being said, it is possible to be "creative" in the script and use
the github api.
For example, one script could create several checks, or post comments
and set tags in the PR.
as far as i am concerned, i think (all) the check(s) should be expected
to success.
and i would also consider as a plus if some comments are posted or tags
are set when some non standard tests fail, and/or to remind some issues
still exists.
for example, the check status could be OK if OpenMPI builds
successfully, and a comment could be posted to indicate valgrind test fails.
that would save some resources (only one Jenkins server, OpenMPI is
built once per PR) and make PR statuses easier to interpret (e.g. all
checks should success unless there is a bug or an intermittent failure)
while providing additional information.
Cheers,
Gilles
On 6/8/2016 6:25 AM, Ralph Castain wrote:
I would agree with all those points
On Jun 7, 2016, at 2:12 PM, Howard Pritchard <hpprit...@gmail.com
<mailto:hpprit...@gmail.com>> wrote:
HI Ralph,
We briefly discussed this some today. I would like to avoid the
mini-MTT approach for PR checking.
At the same time, one can also see why it might be useful from time
to time to make changes to
the script a given jenkins project runs on PRs.
An idea we discussed was to have jenkins folks support a "stable"
version of their jenkins script. If they would
like to make changes, they would create an experimental, temporary
jenkins project to run the new script.
If the new project's script runs clean against open PRs, the new
script can be swapped in to the
original jenkins project. The experimental project could then be
deactivated. If the new script showed failures in the
open PRs, or against master or other branch, issues can be opened to
track the problem(s) found by the
script. The experimental, temporary jenkins project can continue to
run, but its "failure" status can be ignored
until the underlying bug(s) is fixed.
I don't think it makes much sense to run a jenkins script against PRs
if it fails when run against master.
The purpose of jenkins PR testing is to trap new problems, not to
keep reminding us there are problems
with the underlying branch which the PR targets.
Howard
2016-06-07 13:33 GMT-06:00 Ralph Castain <r...@open-mpi.org
<mailto:r...@open-mpi.org>>:
Hi folks
I’m trying to get a handle on our use of Jenkins testing for PRs
prior to committing them. When we first discussed this, it was my
impression that our objective was to screen PRs to catch any
errors caused by differences in environment and to avoid
regressions. However, it appears that the tests keep changing
without warning, leading to the impression that we are using
Jenkins as a “mini-MTT” testing device.
So I think we need to come to consensus on the purpose of the
Jenkins testing. If it is to screen for regressions, then the
tests need to remain stable. A PR that does not introduce any new
problems might not address old ones, but that is no reason to
flag it as an “error”.
On the other hand, if the objective is to use Jenkins as a
“mini-MTT”, then we need to agree on how/when a PR is ready to be
merged. Insisting that nothing be merged until even a mini-MTT is
perfectly clean is probably excessively prohibitive - it would
require that the entire community (and not just the one proposing
the PR) take responsibility for cleaning up the code base against
any and all imposed tests.
So I would welcome opinions on this: are we using Jenkins as a
screening tool on changes, or as a test for overall correctness
of the code base?
Ralph
_______________________________________________
devel mailing list
de...@open-mpi.org <mailto:de...@open-mpi.org>
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post:
http://www.open-mpi.org/community/lists/devel/2016/06/19087.php
_______________________________________________
devel mailing list
de...@open-mpi.org <mailto:de...@open-mpi.org>
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post:
http://www.open-mpi.org/community/lists/devel/2016/06/19088.php
_______________________________________________
devel mailing list
de...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post:
http://www.open-mpi.org/community/lists/devel/2016/06/19089.php
