Ben Pfaff <b...@ovn.org> writes: > On Thu, Sep 06, 2018 at 04:56:18AM -0400, Aaron Conole wrote: >> As of June, the 0-day robot has tested over 450 patch series. >> Occasionally it spams the list (apologies for that), but for the >> majority of the time it has caught issues before they made it to the >> tree - so it's accomplishing the initial goal just fine. >> >> I see lots of ways it can improve. Currently, the bot runs on a light >> system. It takes ~20 minutes to complete a set of tests, including all >> the checkpatch and rebuild runs. That's not a big issue. BUT, it does >> mean that the machine isn't able to perform all the kinds of regression >> tests that we would want. I want to improve this in a way that various >> contributors can bring their own hardware and regression tests to the >> party. In that way, various projects can detect potential issues before >> they would ever land on the tree and it could flag functional changes >> earlier in the process. >> >> I'm not sure the best way to do that. One thing I'll be doing is >> updating the bot to push a series that successfully builds and passes >> checkpatch to a special branch on a github repository to kick off travis >> builds. That will give us a more complete regression coverage, and we >> could be confident that a series won't break something major. After >> that, I'm not sure how to notify various alternate test infrastructures >> how to kick off their own tests using the patched sources. > > That's pretty exciting. > > Don't forget about appveyor, either. Hardly any of us builds on > Windows, so appveyor is likely to catch things that we won't.
I haven't forgotten about this. Unveiling (after some quick testing... hopefully it continues to work): https://github.com/ovsrobot/ovs https://travis-ci.org/ovsrobot/ovs https://ci.appveyor.com/project/ovsrobot/ovs I don't know if the appveyor stuff is set up correctly, so if anyone who is more knowledgeable in that area wants to assist contact me. I only have one patch from one series built. BUT - every series now is submitted to a github branch, and that kicks off Travis (and hopefully appveyor). They can be found by their series id in patchwork: series_SERIES_ID I don't have any mechanism to clean them up. Presumably I'll need to figure out how to do that at some point... :) >> My goal is to get really early feedback on patch series. I've sent this >> out to the folks I know are involved in testing and test discussions in >> the hopes that we can talk about how best to get more CI happening. The >> open questions: >> >> 1. How can we notify various downstream consumers of OvS of these >> 0-day builds? Should we just rely on people rolling their own? >> Should there be a more formalized framework? How will these other >> test frameworks report any kind of failures? > > Do you mean notify of successes or failures? I assumed that the robot's > email would notify us of that. > > Do you mean actually provide the builds? I don't know a good way to do > that. > >> 2. What kinds of additional testing do we want to see the robot include? >> Should the test results be made available in general on some kind of >> public facing site? Should it just stay as a "bleep bloop - >> failure!" marker? > > It would be super awesome if we could run the various additional > testsuites that we have: check-system-userspace, check-kernel, etc. We > can't run them easily on travis because they require superuser (and bugs > sometimes crash the system). > >> 3. What other concerns should be addressed? _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
