Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
Am 10.05.2019 um 17:29 hat Markus Armbruster geschrieben: > Kevin Wolf writes: > > > Am 10.05.2019 um 10:55 hat Thomas Huth geschrieben: > >> On 08/05/2019 07.47, Thomas Huth wrote: > >> > On 07/05/2019 17.50, Eric Blake wrote: > >> >> On 5/7/19 10:22 AM, Thomas Huth wrote: > >> >>> On 07/05/2019 15.22, Markus Armbruster wrote: > >> Thomas Huth writes: > >> > >> > Currently, all tests are in the "auto" group. This is a little bit > >> > pointless. > >> > OTOH, we need a group for the tests that we can automatically run > >> > during > >> > "make check" each time, too. Tests in this new group are supposed to > >> > run > >> > with every possible QEMU configuration, for example they must run > >> > with every > >> > QEMU binary (also non-x86), without failing when an optional > >> > features is > >> > missing (but reporting "skip" is ok), and be able to run on all kind > >> > of host > >> > filesystems and users (i.e. also as "nobody" or "root"). > >> > So let's use the "auto" group for this class of tests now. The > >> > initial > >> > list has been determined by running the iotests with non-x86 QEMU > >> > targets > >> > and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. > >> > including > >> > macOS and FreeBSD). > >> > >> I wonder whether we should additionally limit "make check" to "quick" > >> tests. How slow are the non-quick auto tests for you? > >> >>> > >> >>> I already sorted out some of the tests that run vry long, since the > >> >>> run time on gitlab, cirrus-ci and travis is limited. "make check-block" > >> >>> currently takes 3 minutes on my laptop, I think that's still ok? > >> >>> > >> >>> When I run the tests from the auto group that are not in the quick > >> >>> group, I currently get: > >> >>> > >> >> > >> >> My personal threshold is about 5 seconds for quick, so: > >> >> > >> >>> 003 1s ... > >> >>> 007 2s ... > >> >> > >> >> Should these be moved to quick? > >> > > >> > I'll leave that decision up to the blocklayer folks ... I thought that > >> > there might have been a different reason that these have not been put > >> > into "quick" yet...? > >> > > >> >>> 013 5s ... > >> >> > >> >> this one is borderline > >> >> > >> >>> 014 15s ... > >> >>> 015 9s ... > >> >> > >> >> Definitely not quick, but if you think they are still okay for auto, I > >> >> can live with that. > >> >> > >> >>> 022 1s ... > >> >> > >> >> Another candidate for quick? > >> >> > >> >>> 023 18s ... > >> >> > >> >> Even longer than 14. Okay for auto? > >> > > >> > I think I'd give it a try. If people are complaining later that "make > >> > check" is running now way too long, we still can refine the list later. > >> > >> Thinking about this again, "make check" now runs quite a bit longer > >> indeed. So I now rather tend to remove the tests that run longer than 5s > >> from the auto group instead... I think I'll send a v4 of this patch > >> where I'll remove them from the auto group. > > > > I don't think time is everything. We should also consider how much > > the tests contribute to basic code coverage. There is no point in > > removing a test from the list because it takes 10 seconds, but if I > > split it in two tests taking each 5 seconds, you would include both > > halves. > > > > For example, 030, 040 and 041 are not that quick (14/11/42 seconds, > > respectively), but they are the most important tests for block jobs and > > covering a lot. Sure, 42 seconds is a lot, but I'd keep 030 and 040 at > > least. > > Yes, we want block jobs covered. However, 42 seconds is a lot indeed. > Can you think of ways to get a useful part of the full coverage in five > seconds or less? I'm not even sure which part of the test is taking so long. Maybe it's only one of the 88 test functions that's taking long. Or we're duplicating a lot of tests because we test all combinations of setups where testing only certain combinations would give enough coverage. Answering this will probably take some effort checking what the exact cases tested are and how long each one takes. Kevin
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
Kevin Wolf writes: > Am 10.05.2019 um 10:55 hat Thomas Huth geschrieben: >> On 08/05/2019 07.47, Thomas Huth wrote: >> > On 07/05/2019 17.50, Eric Blake wrote: >> >> On 5/7/19 10:22 AM, Thomas Huth wrote: >> >>> On 07/05/2019 15.22, Markus Armbruster wrote: >> Thomas Huth writes: >> >> > Currently, all tests are in the "auto" group. This is a little bit >> > pointless. >> > OTOH, we need a group for the tests that we can automatically run >> > during >> > "make check" each time, too. Tests in this new group are supposed to >> > run >> > with every possible QEMU configuration, for example they must run with >> > every >> > QEMU binary (also non-x86), without failing when an optional features >> > is >> > missing (but reporting "skip" is ok), and be able to run on all kind >> > of host >> > filesystems and users (i.e. also as "nobody" or "root"). >> > So let's use the "auto" group for this class of tests now. The initial >> > list has been determined by running the iotests with non-x86 QEMU >> > targets >> > and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. >> > including >> > macOS and FreeBSD). >> >> I wonder whether we should additionally limit "make check" to "quick" >> tests. How slow are the non-quick auto tests for you? >> >>> >> >>> I already sorted out some of the tests that run vry long, since the >> >>> run time on gitlab, cirrus-ci and travis is limited. "make check-block" >> >>> currently takes 3 minutes on my laptop, I think that's still ok? >> >>> >> >>> When I run the tests from the auto group that are not in the quick >> >>> group, I currently get: >> >>> >> >> >> >> My personal threshold is about 5 seconds for quick, so: >> >> >> >>> 003 1s ... >> >>> 007 2s ... >> >> >> >> Should these be moved to quick? >> > >> > I'll leave that decision up to the blocklayer folks ... I thought that >> > there might have been a different reason that these have not been put >> > into "quick" yet...? >> > >> >>> 013 5s ... >> >> >> >> this one is borderline >> >> >> >>> 014 15s ... >> >>> 015 9s ... >> >> >> >> Definitely not quick, but if you think they are still okay for auto, I >> >> can live with that. >> >> >> >>> 022 1s ... >> >> >> >> Another candidate for quick? >> >> >> >>> 023 18s ... >> >> >> >> Even longer than 14. Okay for auto? >> > >> > I think I'd give it a try. If people are complaining later that "make >> > check" is running now way too long, we still can refine the list later. >> >> Thinking about this again, "make check" now runs quite a bit longer >> indeed. So I now rather tend to remove the tests that run longer than 5s >> from the auto group instead... I think I'll send a v4 of this patch >> where I'll remove them from the auto group. > > I don't think time is everything. We should also consider how much > the tests contribute to basic code coverage. There is no point in > removing a test from the list because it takes 10 seconds, but if I > split it in two tests taking each 5 seconds, you would include both > halves. > > For example, 030, 040 and 041 are not that quick (14/11/42 seconds, > respectively), but they are the most important tests for block jobs and > covering a lot. Sure, 42 seconds is a lot, but I'd keep 030 and 040 at > least. Yes, we want block jobs covered. However, 42 seconds is a lot indeed. Can you think of ways to get a useful part of the full coverage in five seconds or less?
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
Am 10.05.2019 um 10:55 hat Thomas Huth geschrieben: > On 08/05/2019 07.47, Thomas Huth wrote: > > On 07/05/2019 17.50, Eric Blake wrote: > >> On 5/7/19 10:22 AM, Thomas Huth wrote: > >>> On 07/05/2019 15.22, Markus Armbruster wrote: > Thomas Huth writes: > > > Currently, all tests are in the "auto" group. This is a little bit > > pointless. > > OTOH, we need a group for the tests that we can automatically run during > > "make check" each time, too. Tests in this new group are supposed to run > > with every possible QEMU configuration, for example they must run with > > every > > QEMU binary (also non-x86), without failing when an optional features is > > missing (but reporting "skip" is ok), and be able to run on all kind of > > host > > filesystems and users (i.e. also as "nobody" or "root"). > > So let's use the "auto" group for this class of tests now. The initial > > list has been determined by running the iotests with non-x86 QEMU > > targets > > and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. > > including > > macOS and FreeBSD). > > I wonder whether we should additionally limit "make check" to "quick" > tests. How slow are the non-quick auto tests for you? > >>> > >>> I already sorted out some of the tests that run vry long, since the > >>> run time on gitlab, cirrus-ci and travis is limited. "make check-block" > >>> currently takes 3 minutes on my laptop, I think that's still ok? > >>> > >>> When I run the tests from the auto group that are not in the quick > >>> group, I currently get: > >>> > >> > >> My personal threshold is about 5 seconds for quick, so: > >> > >>> 003 1s ... > >>> 007 2s ... > >> > >> Should these be moved to quick? > > > > I'll leave that decision up to the blocklayer folks ... I thought that > > there might have been a different reason that these have not been put > > into "quick" yet...? > > > >>> 013 5s ... > >> > >> this one is borderline > >> > >>> 014 15s ... > >>> 015 9s ... > >> > >> Definitely not quick, but if you think they are still okay for auto, I > >> can live with that. > >> > >>> 022 1s ... > >> > >> Another candidate for quick? > >> > >>> 023 18s ... > >> > >> Even longer than 14. Okay for auto? > > > > I think I'd give it a try. If people are complaining later that "make > > check" is running now way too long, we still can refine the list later. > > Thinking about this again, "make check" now runs quite a bit longer > indeed. So I now rather tend to remove the tests that run longer than 5s > from the auto group instead... I think I'll send a v4 of this patch > where I'll remove them from the auto group. I don't think time is everything. We should also consider how much the tests contribute to basic code coverage. There is no point in removing a test from the list because it takes 10 seconds, but if I split it in two tests taking each 5 seconds, you would include both halves. For example, 030, 040 and 041 are not that quick (14/11/42 seconds, respectively), but they are the most important tests for block jobs and covering a lot. Sure, 42 seconds is a lot, but I'd keep 030 and 040 at least. Kevin
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
Thomas Huth writes: > Thinking about this again, "make check" now runs quite a bit longer > indeed. So I now rather tend to remove the tests that run longer than 5s > from the auto group instead... I think I'll send a v4 of this patch > where I'll remove them from the auto group. Appreciated! I'm all for automated testing, but "make check" is already slow enough to make certain kinds of work painful. Not quite slow enough to make me renege on commitments and go on a quest to speed it up.
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
On 08/05/2019 07.47, Thomas Huth wrote: > On 07/05/2019 17.50, Eric Blake wrote: >> On 5/7/19 10:22 AM, Thomas Huth wrote: >>> On 07/05/2019 15.22, Markus Armbruster wrote: Thomas Huth writes: > Currently, all tests are in the "auto" group. This is a little bit > pointless. > OTOH, we need a group for the tests that we can automatically run during > "make check" each time, too. Tests in this new group are supposed to run > with every possible QEMU configuration, for example they must run with > every > QEMU binary (also non-x86), without failing when an optional features is > missing (but reporting "skip" is ok), and be able to run on all kind of > host > filesystems and users (i.e. also as "nobody" or "root"). > So let's use the "auto" group for this class of tests now. The initial > list has been determined by running the iotests with non-x86 QEMU targets > and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including > macOS and FreeBSD). I wonder whether we should additionally limit "make check" to "quick" tests. How slow are the non-quick auto tests for you? >>> >>> I already sorted out some of the tests that run vry long, since the >>> run time on gitlab, cirrus-ci and travis is limited. "make check-block" >>> currently takes 3 minutes on my laptop, I think that's still ok? >>> >>> When I run the tests from the auto group that are not in the quick >>> group, I currently get: >>> >> >> My personal threshold is about 5 seconds for quick, so: >> >>> 003 1s ... >>> 007 2s ... >> >> Should these be moved to quick? > > I'll leave that decision up to the blocklayer folks ... I thought that > there might have been a different reason that these have not been put > into "quick" yet...? > >>> 013 5s ... >> >> this one is borderline >> >>> 014 15s ... >>> 015 9s ... >> >> Definitely not quick, but if you think they are still okay for auto, I >> can live with that. >> >>> 022 1s ... >> >> Another candidate for quick? >> >>> 023 18s ... >> >> Even longer than 14. Okay for auto? > > I think I'd give it a try. If people are complaining later that "make > check" is running now way too long, we still can refine the list later. Thinking about this again, "make check" now runs quite a bit longer indeed. So I now rather tend to remove the tests that run longer than 5s from the auto group instead... I think I'll send a v4 of this patch where I'll remove them from the auto group. Thomas
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
Eric Blake writes: > On 5/7/19 10:22 AM, Thomas Huth wrote: >> On 07/05/2019 15.22, Markus Armbruster wrote: >>> Thomas Huth writes: >>> Currently, all tests are in the "auto" group. This is a little bit pointless. OTOH, we need a group for the tests that we can automatically run during "make check" each time, too. Tests in this new group are supposed to run with every possible QEMU configuration, for example they must run with every QEMU binary (also non-x86), without failing when an optional features is missing (but reporting "skip" is ok), and be able to run on all kind of host filesystems and users (i.e. also as "nobody" or "root"). So let's use the "auto" group for this class of tests now. The initial list has been determined by running the iotests with non-x86 QEMU targets and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including macOS and FreeBSD). >>> >>> I wonder whether we should additionally limit "make check" to "quick" >>> tests. How slow are the non-quick auto tests for you? >> >> I already sorted out some of the tests that run vry long, since the >> run time on gitlab, cirrus-ci and travis is limited. "make check-block" >> currently takes 3 minutes on my laptop, I think that's still ok? >> >> When I run the tests from the auto group that are not in the quick >> group, I currently get: >> > > My personal threshold is about 5 seconds for quick, so: > >> 003 1s ... >> 007 2s ... > > Should these be moved to quick? > >> 013 5s ... > > this one is borderline > >> 014 15s ... >> 015 9s ... > > Definitely not quick, but if you think they are still okay for auto, I > can live with that. > >> 022 1s ... > > Another candidate for quick? > >> 023 18s ... > > Even longer than 14. Okay for auto? > > etc. Kevin, Max?
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
On 07/05/2019 17.50, Eric Blake wrote: > On 5/7/19 10:22 AM, Thomas Huth wrote: >> On 07/05/2019 15.22, Markus Armbruster wrote: >>> Thomas Huth writes: >>> Currently, all tests are in the "auto" group. This is a little bit pointless. OTOH, we need a group for the tests that we can automatically run during "make check" each time, too. Tests in this new group are supposed to run with every possible QEMU configuration, for example they must run with every QEMU binary (also non-x86), without failing when an optional features is missing (but reporting "skip" is ok), and be able to run on all kind of host filesystems and users (i.e. also as "nobody" or "root"). So let's use the "auto" group for this class of tests now. The initial list has been determined by running the iotests with non-x86 QEMU targets and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including macOS and FreeBSD). >>> >>> I wonder whether we should additionally limit "make check" to "quick" >>> tests. How slow are the non-quick auto tests for you? >> >> I already sorted out some of the tests that run vry long, since the >> run time on gitlab, cirrus-ci and travis is limited. "make check-block" >> currently takes 3 minutes on my laptop, I think that's still ok? >> >> When I run the tests from the auto group that are not in the quick >> group, I currently get: >> > > My personal threshold is about 5 seconds for quick, so: > >> 003 1s ... >> 007 2s ... > > Should these be moved to quick? I'll leave that decision up to the blocklayer folks ... I thought that there might have been a different reason that these have not been put into "quick" yet...? >> 013 5s ... > > this one is borderline > >> 014 15s ... >> 015 9s ... > > Definitely not quick, but if you think they are still okay for auto, I > can live with that. > >> 022 1s ... > > Another candidate for quick? > >> 023 18s ... > > Even longer than 14. Okay for auto? I think I'd give it a try. If people are complaining later that "make check" is running now way too long, we still can refine the list later. Thomas signature.asc Description: OpenPGP digital signature
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
On 5/7/19 10:22 AM, Thomas Huth wrote: > On 07/05/2019 15.22, Markus Armbruster wrote: >> Thomas Huth writes: >> >>> Currently, all tests are in the "auto" group. This is a little bit >>> pointless. >>> OTOH, we need a group for the tests that we can automatically run during >>> "make check" each time, too. Tests in this new group are supposed to run >>> with every possible QEMU configuration, for example they must run with every >>> QEMU binary (also non-x86), without failing when an optional features is >>> missing (but reporting "skip" is ok), and be able to run on all kind of host >>> filesystems and users (i.e. also as "nobody" or "root"). >>> So let's use the "auto" group for this class of tests now. The initial >>> list has been determined by running the iotests with non-x86 QEMU targets >>> and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including >>> macOS and FreeBSD). >> >> I wonder whether we should additionally limit "make check" to "quick" >> tests. How slow are the non-quick auto tests for you? > > I already sorted out some of the tests that run vry long, since the > run time on gitlab, cirrus-ci and travis is limited. "make check-block" > currently takes 3 minutes on my laptop, I think that's still ok? > > When I run the tests from the auto group that are not in the quick > group, I currently get: > My personal threshold is about 5 seconds for quick, so: > 003 1s ... > 007 2s ... Should these be moved to quick? > 013 5s ... this one is borderline > 014 15s ... > 015 9s ... Definitely not quick, but if you think they are still okay for auto, I can live with that. > 022 1s ... Another candidate for quick? > 023 18s ... Even longer than 14. Okay for auto? etc. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org signature.asc Description: OpenPGP digital signature
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
On 07/05/2019 15.22, Markus Armbruster wrote: > Thomas Huth writes: > >> Currently, all tests are in the "auto" group. This is a little bit pointless. >> OTOH, we need a group for the tests that we can automatically run during >> "make check" each time, too. Tests in this new group are supposed to run >> with every possible QEMU configuration, for example they must run with every >> QEMU binary (also non-x86), without failing when an optional features is >> missing (but reporting "skip" is ok), and be able to run on all kind of host >> filesystems and users (i.e. also as "nobody" or "root"). >> So let's use the "auto" group for this class of tests now. The initial >> list has been determined by running the iotests with non-x86 QEMU targets >> and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including >> macOS and FreeBSD). > > I wonder whether we should additionally limit "make check" to "quick" > tests. How slow are the non-quick auto tests for you? I already sorted out some of the tests that run vry long, since the run time on gitlab, cirrus-ci and travis is limited. "make check-block" currently takes 3 minutes on my laptop, I think that's still ok? When I run the tests from the auto group that are not in the quick group, I currently get: 003 1s ... 007 2s ... 013 5s ... 014 15s ... 015 9s ... 022 1s ... 023 18s ... 043 0s ... 049 3s ... 061 4s ... 079 2s ... 080 4s ... 091 1s ... 097 2s ... 104 0s ... 117 0s ... 122 8s ... 126 1s ... 137 1s ... 142 5s ... 172 3s ... 174 0s ... 176 5s ... 181 2s ... 186 2s ... 187 0s ... 191 5s ... 201 2s ... 214 0s ... That looks reasonable to me. Thomas
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
Thomas Huth writes: > Currently, all tests are in the "auto" group. This is a little bit pointless. > OTOH, we need a group for the tests that we can automatically run during > "make check" each time, too. Tests in this new group are supposed to run > with every possible QEMU configuration, for example they must run with every > QEMU binary (also non-x86), without failing when an optional features is > missing (but reporting "skip" is ok), and be able to run on all kind of host > filesystems and users (i.e. also as "nobody" or "root"). > So let's use the "auto" group for this class of tests now. The initial > list has been determined by running the iotests with non-x86 QEMU targets > and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including > macOS and FreeBSD). I wonder whether we should additionally limit "make check" to "quick" tests. How slow are the non-quick auto tests for you?
