On 6 November 2013 10:01, James Hunt <[email protected]> wrote: > Hi Dmitrijs, > > On 06/11/13 04:30, Dmitrijs Ledkovs wrote:> I've committed a few fixes to fix > test-suite errors on PPAs: >> >> * set & use XDG_RUNTIME_DIR, if not set as otherwise spawning session >> init was failing >> >> * which uncovered that test_umask failure, too restrictive umask under >> test (remember session-init needs to create subdirectories) >> >> * and disabled timing tests, since those don't work on virtualised builders >> >> So at the moment this brings all daily builds back to green. Let's try >> to keep those up =) > Thanks! Agreed, but let's not forget that *all* the tests are "green" prior to > any push to lp:upstart (having passed locally). But as we know, new tests > sometimes fail in the slightly odd lp build environment. Maybe using > sbuild-launchpad-chroot would help a little (although that still won't give an > identical environment AIUI). >
Yeah, true. E.g. here it was mostly the case that sbuild doesn't have "user centric post-Precise variables" such as XDG_RUNTIME_DIR and the lack of any recent kernel on virtualised PPAs (long standing known problem, to be resolved) >> >> WTR declaring failing tests, skipped, expected fail, and bailing out. >> >> At the moment, we mostly abort and/or skip executing tests >> with/without telling the user about it. I think it would be best to >> implement TAP[1] output for our tests, such that we can properly skip >> tests, or mark them as expecting to fail. >> >> Announcing that "some tests may or may not fail, because one has >> overlayfs mountpoint" is not helpful. Saying that "test_conf test >> #3-7" are expected to fail with a reason due to overlayfs mountpoint, >> is much better. >> >> In addition it should enable us to write tests, which are known to >> fail at the moment and mark them as TODO for future. >> >> Similarly this will improve ability to report and compare test-suite >> results on different platforms (e.g. how many tests are not run in >> virtualised PPAs vs nonvirt vs overlayfs). >> >> >> [1] http://en.wikipedia.org/wiki/Test_Anything_Protocol >> > Take a close look at https://bugs.launchpad.net/libnih/+bug/933717. I think we > should resurrect this and see if Scott would be happy to merge it. I did > discuss > it a long time ago with Scott and I think his concern was retaining the > exiting > API. If you look at the attachment, you'll see I've proposed defining > NIH_TEST_VERSION_2 to enable the new functionality, but there are other > options > we could explore. > Ah, interesting. I'll have a look. I've started by writting an alternative nih/test_output_tap.h which is decided for inclusion based on a define. I wanted to keep both output formats, and keep the new one backwards compatible. > With minimal changes to the existing Upstart tests (basically we just need to > run sed over the code) we can do everything you suggest and a whole lot more. Will take a look at that one. > That, plus fixing automake to actually show verbose logs by default > ('serial-tests' or add some logic to display the verbose log at completion > time). > Well, I've cheated: In debian/rules packaging I'm now catching error from $ make check and cat "test-suite.log". See for example: https://launchpadlibrarian.net/155928411/buildlog_ubuntu-saucy-amd64.upstart_1.10%2Bdaily%2B1555%2B1520%2B201311060249~ubuntu13.10.1_FAILEDTOBUILD.txt.gz Not ideal, but at least something. And with TAP we will gain _better_ test-suite summary. Regards, Dmitrijs. -- upstart-devel mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/upstart-devel
