On Mon, 9 May 2011, Sergei Trofimovich wrote: > Hello, Cocci team! > > Our source based distro as a part of QA package > evaluation attempts to run tests supplied by upstream > packages. It's very handy to make sure package behaves > as expected in various architectures. > > For coccinelle we use the following logic: > > yes | ./spatch -iso_file standard.iso -macro_file_builtins standard.h > -testall || handle_error > ./spatch.opt -iso_file standard.iso -macro_file_builtins standard.h > -testall || handle_error > > So we compare optimized an nonoptimized variant. It works fine until we > disable python support. Moreover results are inconsistent even against > 2 consequent runs of nonoptimized version: > > $ ./configure --without-python > $ make depend > $ make all > $ make opt > $ yes | ./spatch -iso_file standard.iso -macro_file_builtins standard.h > -testall >bench_noopt 2>&1 > $ ./spatch -iso_file standard.iso -macro_file_builtins standard.h > -testall >bench_noopt_rerun 2>&1 > > The second test fails while the first does not. Last lines of files > 'bench_noopt' and > 'bench_noopt_rerun' are the following: > 'bench_noopt': > good = 335/361 > Current score is greater than expected :) > (was expecting 298 but got 335) > Generating new expected score file and saving old one > mv tests/SCORE_expected.sexp tests/SCORE_expected.sexp.save [y/n] ? > mv tests/SCORE_best_of_both.sexp tests/SCORE_expected.sexp [y/n] ? > [pass] > 'bench_noopt_rerun': > good = 335/361 > Current score is lower than expected :( > (was expecting 337 but got 335) > [fail] > > Why it expects 337 passed tests and why it didn't fail in the first run? > It's interesting, that if I enable python things go back to normal. > Looks like a bug.
The first problem is that the SCORE file is not necessarily relative to the distributed version of coccinelle and to the set of tests in that version. This is the reason for the lower number of expected correct results in the first test. Running spatch with the argument testall is not currently part of the distribution build process. But it seems like that should be done to avoid a lot of confusion. The use of yes then updates the expected score to what was obtained on the first run. A program "no" would thus be a better choice than "yes" for this kind of test, if you want to stay with the original results. Then the problem is that there are some tests that include python. Because these tests have succeeded at some time in the past, they are still considered as potential successful tests, even though they did not succeed in the most current run. Thus, the current run has 335 successful tests and 337 expected. To avoid problems, I have either adjusted or removed all of the tests that use scripting languages. Perhaps they can be put back when the testing infrastructure can be made more sensitive to the current configuration. julia _______________________________________________ Cocci mailing list [email protected] http://lists.diku.dk/mailman/listinfo/cocci (Web access from inside DIKUs LAN only)
