On Sun, Dec 17, 2017 at 6:53 PM, Waldek Kozaczuk <jwkozac...@gmail.com>
wrote:
> As I have been working on unit tests around read-only filesystem I noticed
> that some of the unit tests (look for 'debug\(.*PASS' under tests) report
> success even though I know for sure (got it through printf/cout) that many
> assertions fail (as an example look at tst-readdir.cc). It turns out that
> some unit tests use debug method when reporting successes/failures like so
> which does not normally output anything to the console:
>
> static void report(bool ok, const char* msg)
> {
> ++tests;
> fails += !ok;
> debug("%s: %s\n", (ok ? "PASS" : "FAIL"), msg);
> }
>
> Now scripts/tests/testing.py looks for "FAIL" to detect failures (it looks
> like the unit tests system does not look at the exit code of a unit test
> app -> maybe it does not get propagated all the way to QEMU process). So I
> wonder if there is a better way to detect unit tests failures - maybe add
> assert(fails == 0) before unit test exist.
>
Note that the messages you see here are many individual tests. Most tests
then end by printing the final result in of the test, e.g., in
tst-readdir.cc:
debug("SUMMARY: %d tests, %d failures\n", tests, fails);
return fails == 0 ? 0 : 1;
This will again print nothing, but *think* (I don't remember, maybe you can
please verify) that if this returns 1 (failure) a message is printed about
the program failing and test.py recognizes this message too.
But I'm not sure, can you please check?
In any case, I think you're right, all those "debug" commands should be
changed to printf.
Another reason why all tests should use printf is that it allows compiling
them for Linux as well - to confirm this test really checks Linux-like
behavior, not behavior we invented.
Patches doing this will be welcome :-)
Another easier thing we can do is to enable the "verbose" option when
running tests so all debug messages get printed. We even have a verbose
option for test.py - and perhaps we should just make it the default? What
reason do we have to run tests without the verbose option?
> Meanwhile the simple fix is to change debug to prints/cout.
>
> Here is how I run them:
>
> ./scripts/test.py --name tst-readdir.so
>
> Waldek
>
> PS. I am worried that maybe other cases in unit tests where failures are
> masked and maybe lead to the error we see in Jenkins runs sometimes -
> tst-namespace.so.
>
The "make check" runs each test separately as a separate VM, so even if one
test has a problem, it shouldn't cause us to print errors on other tests.
--
You received this message because you are subscribed to the Google Groups "OSv
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
