HI Ralph, We briefly discussed this some today. I would like to avoid the mini-MTT approach for PR checking. At the same time, one can also see why it might be useful from time to time to make changes to the script a given jenkins project runs on PRs.
An idea we discussed was to have jenkins folks support a "stable" version of their jenkins script. If they would like to make changes, they would create an experimental, temporary jenkins project to run the new script. If the new project's script runs clean against open PRs, the new script can be swapped in to the original jenkins project. The experimental project could then be deactivated. If the new script showed failures in the open PRs, or against master or other branch, issues can be opened to track the problem(s) found by the script. The experimental, temporary jenkins project can continue to run, but its "failure" status can be ignored until the underlying bug(s) is fixed. I don't think it makes much sense to run a jenkins script against PRs if it fails when run against master. The purpose of jenkins PR testing is to trap new problems, not to keep reminding us there are problems with the underlying branch which the PR targets. Howard 2016-06-07 13:33 GMT-06:00 Ralph Castain <r...@open-mpi.org>: > Hi folks > > I’m trying to get a handle on our use of Jenkins testing for PRs prior to > committing them. When we first discussed this, it was my impression that > our objective was to screen PRs to catch any errors caused by differences > in environment and to avoid regressions. However, it appears that the tests > keep changing without warning, leading to the impression that we are using > Jenkins as a “mini-MTT” testing device. > > So I think we need to come to consensus on the purpose of the Jenkins > testing. If it is to screen for regressions, then the tests need to remain > stable. A PR that does not introduce any new problems might not address old > ones, but that is no reason to flag it as an “error”. > > On the other hand, if the objective is to use Jenkins as a “mini-MTT”, > then we need to agree on how/when a PR is ready to be merged. Insisting > that nothing be merged until even a mini-MTT is perfectly clean is probably > excessively prohibitive - it would require that the entire community (and > not just the one proposing the PR) take responsibility for cleaning up the > code base against any and all imposed tests. > > So I would welcome opinions on this: are we using Jenkins as a screening > tool on changes, or as a test for overall correctness of the code base? > > Ralph > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2016/06/19087.php
