Wow, interesting indeed. It's a bit of an oversimplification, but generally we know "features" are opposed to "bugs" & problems. We can have features, but they do "write-in" a quantity of bug/problem fix work (which is difficult to measure). So I think these results and bug-fix experience confirms: We can have *magic*, but it demands ongoing maintenance love. Maybe the thing to look at here is the balance of test-writing work and harness-maintenance work. IMHO, we could better serve our larger goals by shifting priorities more to test-writing. Though I'm arguing in this thread, that shift probably requires simplifying the harness stuff first, in order to lower the ongoing maintenance required. If that makes sense.
----- Original Message ----- From: "Lucas Meneghel Rodrigues" <l...@redhat.com> To: "Ademar de Souza Reis Jr." <ar...@redhat.com>, "Chris Evich" <cev...@redhat.com> Cc: "Virt Test Development Mailing List" <virt-test-devel@redhat.com> Sent: Wednesday, December 11, 2013 1:11:05 PM Subject: Re: [Virt-test-devel] [RFC] [Important] Result fidelity vs. Testing time On 12/05/2013 07:22 PM, Ademar de Souza Reis Jr. wrote: > ... > Safe options: > > [lmr@thinkpad-t420s virt-test.git]$ ./run -t qemu --restart-vm > --restore-image --restore-image-between-tests > Running setup. Please wait... > SETUP: PASS (16.73 s) > DATA DIR: /home/lmr/virt_test > DEBUG LOG: /home/lmr/Code/virt-test.git/logs/run-2013-12-11-15.55.43/debug.log > TESTS: 10 > (1/10) type_specific.migrate.default.tcp: PASS (42.05 s) > (2/10) type_specific.migrate.default.unix: PASS (41.07 s) > (3/10) type_specific.migrate.default.exec.default_exec: PASS (40.48 s) > (4/10) type_specific.migrate.default.exec.gzip_exec: PASS (42.92 s) > (5/10) type_specific.migrate.default.fd: PASS (39.93 s) > (6/10) type_specific.migrate.with_set_speed.tcp: PASS (36.98 s) > (7/10) type_specific.migrate.with_set_speed.unix: PASS (37.12 s) > (8/10) type_specific.migrate.with_set_speed.exec.default_exec: PASS (37.57 s) > (9/10) type_specific.migrate.with_set_speed.exec.gzip_exec: PASS (43.40 s) > (10/10) type_specific.migrate.with_set_speed.fd: PASS (36.92 s) > TOTAL TIME: 399.37 s (06:39) > TESTS PASSED: 10 > TESTS FAILED: 0 > SUCCESS RATE: 100.00 % Now 300s vs 400s in a more consistent way [1] makes this a very interesting path to go. Now, with these bugs fixed, we can go further and see how it looks like in the test grid. Consider me excited with the prospect to change to what I'll call to 'safe' mode by default! [1] The previous JeOS caused some hangs that could drag the testing to almost 600s. _______________________________________________ Virt-test-devel mailing list Virt-test-devel@redhat.com https://www.redhat.com/mailman/listinfo/virt-test-devel
