[Nick Coghlan] > What if we appended unexpected skips to the list of bad tests so that they get > rerun in verbose mode and the return value becomes non-zero? > > print count(len(surprise), "skip"), \ > "unexpected on", plat + ":" > printlist(surprise) > # Add the next line after the previous two in regrtest.py > bad.extend(surprise) > > (This happens after the count of failed tests has been printed, so we don't > affect that output)
While "expected skips" is reliable on Windows, it's just a guess everywhere else. That's because the Windows build is unique in supplying and compiling all the external packages Python ships with on Windows. On non-Windows boxes, which skips "are expected" also depends on which external packages happen to be installed on the box, and how Python was configured, and regrtest.py has no knowledge about those. For example, on the "sparc solaris10 gcc trunk" buildbot, 6 tests show up as "unexpected skips", and there's just no way for a non-platform-expert to guess which of those are and aren't pointing at problems. It's easy enough to guess that test_zipfile is skipped because the box doesn't have an external zlib installed, but why is test_ioctl skipped? I have no idea (and neither does regrtest.py), but it's presumably obvious to a Sun expert. In any case, if "unexpected skips" got appended to `bad`, the non-zero exit code would cause that buildbot slave to fail every time. It's not unique, either; e.g., the "x86 gentoo trunk" slave always skips test_tcl, presumably because Tcl isn't installed on that box. In short, there's simply no way to _guess_ which import errors are and aren't "expected". Improving that requires adding more knowledge to the testing machinery. The most important thing is to add knowledge about which import errors are _never_ expected (perhaps by adding knowledge about which imports _may_ legitimately fail), since we can know that for sure about x-platform modules, and that's exactly the problem that keeps burning us. Confusing ImportError with TestSkipped is an admirably lazy idea that unfortunately didn't turn out to work very well. _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com