On Fri, 4 Oct 2019 at 14:50, Max Reitz <mre...@redhat.com> wrote: > On 04.10.19 15:16, Peter Maydell wrote: > > 'make check' does have the restriction > > that we don't want the tests to take too long to run, but in > > general the block layer should be running some reasonable subset > > of tests in the project's standard "run the tests please" command. > > (I have no opinion on exactly what that subset ought to be, beyond > > that it would be good if that subset doesn't have known intermittent > > failures in it.) > > Deciding the subset is difficult. My stance was that it’s better to not > choose an arbitrary subset here but ensure that really everything gets > run as part of CI, that is asynchronously so it doesn’t block anyone and > can take as long as it wants.
I wonder if we have a terminology difference confusion here. To me "CI" means "we have tests which we automatically run before pushing commits to master, and if they fail then we don't push". So (a) they have to run synchronously and (b) there is a need for them to run in a reasonable period of time because otherwise it takes too long to run the tests before pushing to master and we get a backlog of unprocessed pullrequests and annoying delays. > If we decide to get pull requests based on that, then that’d bring even > more pain, but at least it’d be honest. But just running half of the > qcow2 tests to me seems only like we want to pretend we ran the iotests. > So for me, iotests still break, and we need to deal with make check > failures on top. I’d at least prefer one or the other. I think the ideal we're aiming for here is: (1) people doing active work in the block subsystem are probably often going to want to run all the iotests, and certainly the subsystem maintainers will want to do that as they put together pull requests. (2) but people who *don't* do active work in the block subsystem still sometimes touch it in passing as part of things like global refactorings or other changes that touch big parts of the tree, or accidentally break it with a change to some other bit of QEMU entirely. These people won't run the full iotests, but it is reasonable to expect them to run 'make check', and it would be good if that caught "whoops you broke the block subsystem". Similarly, "make check" has incredibly spotty coverage of various arm boards and devices, but it does do some basic checks which do catch accidental breakage. thanks -- PMM