Hi everyone,
I've taken some time recently to look into the current CI for OVS/OVN,
and I think there is room for improvement. Here I will outline a 4 part
plan for improving CI for OVN.
Part 1: Report travis build failures as replies to patch submissions.
Currently, 0-day robot will respond to patches that fail to apply or
fail to pass checkpatch.py. However, builds are handled by travis and
reported out-of-band to the ovs-build list. By having build failures
reported in response to the patch, it makes it much easier to correlate
results to a patch series and does not require signing up for a separate
mailing list.
I suspect this will lead to some noise at first as some tests may need
to be stabilized. If we are diligent about fixing such volatile tests,
then this should reduce the noise after a while. Once we reach a point
where the signal-to-noise ratio is acceptable, we can move on to...
Part 2: Expand testing done by travis.
Travis runs `make distcheck` now. This could be expanded to run the
following:
* system tests
* ovn-kubernetes tests
* Run tests against all supported versions of OVS
CI could also be expanded to run some scale tests, but that likely would
need to be done outside of travis.
Like with part 1, this likely will be noisy as more tests are added.
Again, though, as we fix these problems, it will lead to a more stable
suite of tests. We can then move on to...
Part 3: Correlate tests with patchwork checks
Patchwork has "checks" that can be set on submissions which correspond
to external tests. The tests we add in parts 1 and 2 can correspond to
patchwork checks. The checks can provide an easily-referenced way to
determine the status of tests for a given patch series. Adding checks to
patchwork also lays the groundwork for Part 4.
(Note: It may make sense to switch parts 2 and 3. We may want to
establish patchwork checks prior to adding more tests)
Part 4: Gating
If we have a stable and extensive testsuite, then we can attempt to
enforce gating on patch submissions. In other words, patches that do not
pass all checks cannot be pushed. This can be enforced in a few ways
* Convention: We regulate ourselves. We agree as a community to only
push changes if tests have passed, and we refuse to push patches that
have failed CI.
* Automation: The bot that monitors patchwork for new submissions and
starts CI could potentially merge patches that have received ACKs and
have passed CI. Humans would not push patches themselves.
* Push hook: When pushing patches, if patchwork CI checks have not
passed (and if the patch has not been ACKed), then the hook would reject
the push.
* Pull Requests: Switching from mailing list patches to pull requests
would allow for github to enforce gating.
In any of the above cases, there will be ways for committers to override
the system if absolutely necessary (e.g. CI false positives)
What are your thoughts on this expansion of CI for OVN? Do you have
ideas for other areas where it can be expanded? What are your thoughts
on gating in particular?
Thanks for your feedback,
Mark Michelson
_______________________________________________
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
