On Sunday 07 March 2010, Søren Sandmann wrote: > Previously this test would try to exhaustively test all combinations > of formats and operators, which meant that it would take years to run. > > Instead, generate random images and test those. The random seed is > based on time(), so we will get different tests every time it is run. > Whenever the test fails, it will print what the value of lcg_seed was > when the test began. > > This patch also adds a table of random seeds that have failed at some > point. These seeds are run first before the random testing begins. > Whenever this tests fails, you are supposed to add the seed that > failed to the table, so that we can track that the bug doesn't get > reintroduced.
I don't quite like any nondeterministic random factor in the standard regression tests. Preferably the results of such tests should be reproducible from run to run, even if they are not perfect and do not provide full coverage. This is quite important for having a clearly defined formal patch submission process (Before submitting a patch, one needs to make sure that the regression tests pass. If they don't pass, the problem has to be investigated and patch fixed or regression tests updated if needed). With the randomness in the tests, patch contributor may end up in different confusing situations: - regression test fails for him, even if his patch is fine (if the problem was introduced by somebody else earlier) - regression test passes for him, but fails for the others later (due to the bug in the patch). In this case it would be hard to say if the contributor did proper job running the regression tests in the first place. So I'm in favor of having a deterministic set of seed values for a standard regression test and a special separate mode for finding new problematic seed values. Also the original test which tries all the possible combinations is a good thing to have. Just to be sure that all the corner cases are checked. While random test asymptotically gets closer to 100% coverage when run longer, it may never reach it. -- Best regards, Siarhei Siamashka
signature.asc
Description: This is a digitally signed message part.
_______________________________________________ Pixman mailing list Pixman@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/pixman
