On 07/26/2017 01:58 PM, Stefan Hajnoczi wrote: > On Tue, Jul 25, 2017 at 12:16:13PM -0400, Cleber Rosa wrote: >> On 07/25/2017 11:49 AM, Stefan Hajnoczi wrote: >>> On Fri, Jul 21, 2017 at 10:21:24AM -0400, Cleber Rosa wrote: >>>> On 07/21/2017 10:01 AM, Daniel P. Berrange wrote: >>>>> On Fri, Jul 21, 2017 at 01:33:25PM +0100, Stefan Hajnoczi wrote: >>>>>> On Thu, Jul 20, 2017 at 11:47:27PM -0400, Cleber Rosa wrote: >>>> Without the static capabilities defined, the dynamic check would be >>>> influenced by the run time environment. It would really mean "qemu-io >>>> running on this environment (filesystem?) can do native aio". Again, >>>> that's not the best type of information to depend on when writing tests. >>> >>> Can you explain this more? >>> >>> It seems logical to me that if qemu-io in this environment cannot do >>> aio=native then we must skip those tests. >>> >>> Stefan >>> >> >> OK, let's abstract a bit more. Let's take this part of your statement: >> >> "if qemu-io in this environment cannot do aio=native" >> >> Let's call that a feature check. Depending on how the *feature check* >> is written, a negative result may hide a test failure, because it would >> now be skipped. > > You are saying a pass->skip transition can hide a failure but ./check > tracks skipped tests. See tests/qemu-iotests/check.log for a > pass/fail/skip history. >
You're not focusing on the problem here. The problem is that a test that *was not* supposed to be skipped, would be skipped. Let me reinforce my point, and you can address it directly: feature checks like you proposed can easily produce false negatives. Not something hypothetical or far fetched, I gave very reasonable examples on this same thread. Do you think that's OK because the skip count will get an increment? That's exactly one of the main concerns raised in the original thread ( and break room conversations) that motivated this experiment. > It is the job of the CI system to flag up pass/fail/skip transitions. > You're no worse off using feature tests. > > Stefan > What I'm trying to help us achieve here is a reliable and predictable way for the same test job execution to be comparable across environments. From the individual developer workstation, CI, QA etc. Please let me know If you really believe this should *not* be done here (upstream QEMU). Regards! -- Cleber Rosa [ Sr Software Engineer - Virtualization Team - Red Hat ] [ Avocado Test Framework - avocado-framework.github.io ] [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ]
signature.asc
Description: OpenPGP digital signature
