The branch, master has been updated via 116a150 knownfail: Mark some ldap.acl search tests as no longer failing. via d9459ed knownfail: Remove working winbind tests. via bf2b801 knownfail: The resolve async tests work now. via b3f81c6 knownfail: remove now succeeding local.iconv test. via a3949f4 knownfail: remove base.maximum_allowed test, as it appears to be passing now. via d9aeb2f knownfail: re-add createx test. via 512d114 knownfail: Exclude specific list of SMB2 oplock tests. via 9968fab knownfail: Remove smb2.compound tests, they pass now. via be37328 Remove samba4.raw.oplock.brl4. Bug 7928 has been fixed. via 0190a55 knownfail: Match on end-of-file, not end-of-file-access. via f186425 knownfail: samba4.base.createx_access.createx_access works. via 850c838 knownfail: LogonControl2Ex works. via 4c3696c knownfail: Ignore failures for drsuapi tests over other transports as well. via 4d551a9 samba4: Remove rpc-handles mixed-shared tests from known failing list. via 4673559 selftest: Re-mark some drsuapi tests as known failing. via 41921d6 wafsamba: Cope with not everything having a final_libs attribute. via f9091d6 selftest: Don't mark drsuapi tests as knownfailing. via bef27a9 wafsamba: Use final_libs list to determine if a target needs private libraries, rather than direct dependencies list. via 8175527 Fix typo. via 4f03164 selftest: 'samba4.rpc.netlogon on ncalrpc with seal,padcheck.netlogon.DsrEnumerateDomainTrusts' now succeeds, remove it from the knownfail list. via f528733 GetForestTrustInformation is implemented, remove it from knownfail. via 112d072 rap netsessiongetinfo apparently *is* provided by samba3, so don't list it in knownfail. via e6b4a74 s3-selftest: Don't mark samba3.blackbox.failure.success test as known failing. via be5d9c8 filter-subunit: Fix default for flapping setting. via 0fee2c0 selftest: Display unexpected successes and expected failures. via e626545 selftest: Add --flapping argument to filter-subunit. via 5643d37 selftest: Split out flapping tests into a separate file. from 5bfd625 s4-drs: do not try to contact for replication servers that are not anymore in reps*
http://gitweb.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit 116a150231d395a10233dcce4af814659322e40c Author: Jelmer Vernooij <jel...@samba.org> Date: Mon Dec 5 23:08:10 2011 +0100 knownfail: Mark some ldap.acl search tests as no longer failing. Autobuild-User: Jelmer Vernooij <jel...@samba.org> Autobuild-Date: Tue Dec 6 00:43:40 CET 2011 on sn-devel-104 commit d9459ed9d358e05bddc912eb087dcad5b9de40f8 Author: Jelmer Vernooij <jel...@samba.org> Date: Mon Dec 5 21:26:28 2011 +0100 knownfail: Remove working winbind tests. commit bf2b801a55041a10d5c5c3e75cde86f6b7f62f75 Author: Jelmer Vernooij <jel...@samba.org> Date: Mon Dec 5 19:11:37 2011 +0100 knownfail: The resolve async tests work now. commit b3f81c671a8db33000b932a782ae18899758f54f Author: Jelmer Vernooij <jel...@samba.org> Date: Mon Dec 5 18:09:05 2011 +0100 knownfail: remove now succeeding local.iconv test. commit a3949f4c866707a1934c72573005d5e728baae3c Author: Jelmer Vernooij <jel...@samba.org> Date: Mon Dec 5 16:54:37 2011 +0100 knownfail: remove base.maximum_allowed test, as it appears to be passing now. commit d9aeb2fc33c396906676a9bf42c34beaca6bd9b8 Author: Jelmer Vernooij <jel...@samba.org> Date: Mon Dec 5 15:29:53 2011 +0100 knownfail: re-add createx test. commit 512d11492fd751ad59b0cf3e37e0e4cf7b9f3530 Author: Jelmer Vernooij <jel...@samba.org> Date: Mon Dec 5 14:27:40 2011 +0100 knownfail: Exclude specific list of SMB2 oplock tests. commit 9968fabde3eecf5778f9494a75ff6183a6b8c463 Author: Jelmer Vernooij <jel...@samba.org> Date: Mon Dec 5 02:59:44 2011 +0100 knownfail: Remove smb2.compound tests, they pass now. commit be373289db1872761d5272e68034215e240dde83 Author: Jelmer Vernooij <jel...@samba.org> Date: Mon Dec 5 02:00:34 2011 +0100 Remove samba4.raw.oplock.brl4. Bug 7928 has been fixed. commit 0190a5589c9032f2f98e468259ac3acf1d881855 Author: Jelmer Vernooij <jel...@samba.org> Date: Mon Dec 5 00:36:29 2011 +0100 knownfail: Match on end-of-file, not end-of-file-access. commit f186425a1be9d5b5fa78e39c99dae52a44d03ed3 Author: Jelmer Vernooij <jel...@samba.org> Date: Sun Dec 4 22:42:22 2011 +0100 knownfail: samba4.base.createx_access.createx_access works. commit 850c83821c2e332c9f8c8992d3a01af23e2a42fc Author: Jelmer Vernooij <jel...@samba.org> Date: Sun Dec 4 21:40:51 2011 +0100 knownfail: LogonControl2Ex works. commit 4c3696c02ca09b444e9b22db6ed2f8d9232dfd0f Author: Jelmer Vernooij <jel...@samba.org> Date: Sun Dec 4 21:00:43 2011 +0100 knownfail: Ignore failures for drsuapi tests over other transports as well. commit 4d551a91a5098a48b7fce7fb7091b4d573436f72 Author: Jelmer Vernooij <jel...@samba.org> Date: Sun Dec 4 20:23:29 2011 +0100 samba4: Remove rpc-handles mixed-shared tests from known failing list. commit 4673559894d155c7525c1cc58c930051185649d3 Author: Jelmer Vernooij <jel...@samba.org> Date: Sun Dec 4 16:45:58 2011 +0100 selftest: Re-mark some drsuapi tests as known failing. commit 41921d66825f9e09168a53b38afa949c684c552b Author: Jelmer Vernooij <jel...@samba.org> Date: Sun Dec 4 15:36:27 2011 +0100 wafsamba: Cope with not everything having a final_libs attribute. commit f9091d6fa048d110666fa04fcc9b70725314f111 Author: Jelmer Vernooij <jel...@samba.org> Date: Sun Dec 4 14:59:07 2011 +0100 selftest: Don't mark drsuapi tests as knownfailing. commit bef27a98900fcef17929587264e3859a0d5c7f05 Author: Jelmer Vernooij <jel...@samba.org> Date: Sun Dec 4 14:58:11 2011 +0100 wafsamba: Use final_libs list to determine if a target needs private libraries, rather than direct dependencies list. commit 817552779d699f1242fea17dc8ae4283949036ed Author: Jelmer Vernooij <jel...@samba.org> Date: Sun Dec 4 14:56:13 2011 +0100 Fix typo. commit 4f031643731b3cc2682516ecf82a6f682dffe8ff Author: Jelmer Vernooij <jel...@samba.org> Date: Sun Dec 4 14:22:40 2011 +0100 selftest: 'samba4.rpc.netlogon on ncalrpc with seal,padcheck.netlogon.DsrEnumerateDomainTrusts' now succeeds, remove it from the knownfail list. commit f528733d7b241d3536abdb7ae9c7aad5e7ea1bc4 Author: Jelmer Vernooij <jel...@samba.org> Date: Sun Dec 4 05:36:08 2011 +0100 GetForestTrustInformation is implemented, remove it from knownfail. commit 112d07256a795f5774d1f1a8138b35269755cd37 Author: Jelmer Vernooij <jel...@samba.org> Date: Sun Dec 4 05:06:15 2011 +0100 rap netsessiongetinfo apparently *is* provided by samba3, so don't list it in knownfail. commit e6b4a7440639df21c7d4762369fab0ee57295c59 Author: Jelmer Vernooij <jel...@samba.org> Date: Sun Dec 4 04:38:07 2011 +0100 s3-selftest: Don't mark samba3.blackbox.failure.success test as known failing. commit be5d9c83e38fdc1d22524551eab4d4c48d7c4872 Author: Jelmer Vernooij <jel...@samba.org> Date: Sun Dec 4 03:55:11 2011 +0100 filter-subunit: Fix default for flapping setting. commit 0fee2c01b8d33710d63bb697351a6a414100b5c2 Author: Jelmer Vernooij <jel...@samba.org> Date: Sun Dec 4 01:55:23 2011 +0100 selftest: Display unexpected successes and expected failures. commit e62654578b3aa1ddafd8faf85b65d9702f49fba3 Author: Jelmer Vernooij <jel...@samba.org> Date: Sun Dec 4 00:23:02 2011 +0100 selftest: Add --flapping argument to filter-subunit. commit 5643d37a000971070b22af7c2710f66b49b57eff Author: Jelmer Vernooij <jel...@samba.org> Date: Sun Dec 4 00:09:04 2011 +0100 selftest: Split out flapping tests into a separate file. ----------------------------------------------------------------------- Summary of changes: buildtools/wafsamba/samba_utils.py | 2 +- selftest/filter-subunit | 34 +++++++++++++++--- selftest/flapping | 19 ++++++++++ selftest/knownfail | 61 +++++++++++++++----------------- selftest/subunithelper.py | 68 ++++++++++++++++++++++++++++++------ selftest/wscript | 2 +- source3/selftest/s3-selftest.sh | 2 +- 7 files changed, 137 insertions(+), 51 deletions(-) create mode 100644 selftest/flapping Changeset truncated at 500 lines: diff --git a/buildtools/wafsamba/samba_utils.py b/buildtools/wafsamba/samba_utils.py index 5e16a5d..71cfbc5 100644 --- a/buildtools/wafsamba/samba_utils.py +++ b/buildtools/wafsamba/samba_utils.py @@ -65,7 +65,7 @@ def ADD_LD_LIBRARY_PATH(path): def needs_private_lib(bld, target): '''return True if a target links to a private library''' - for lib in getattr(target, "uselib_local", []): + for lib in getattr(target, "final_libs", []): t = bld.name_to_obj(lib, bld.env) if t and getattr(t, 'private_library', False): return True diff --git a/selftest/filter-subunit b/selftest/filter-subunit index 5f062d1..bced14b 100755 --- a/selftest/filter-subunit +++ b/selftest/filter-subunit @@ -1,7 +1,22 @@ #!/usr/bin/env python # Filter a subunit stream -# Copyright (C) Jelmer Vernooij <jel...@samba.org> -# Published under the GNU GPL, v3 or later +# Copyright (C) 2009-2011 Jelmer Vernooij <jel...@samba.org> + +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +# NOTE: This script is a hack, meant as a placeholder until we can migrate +# to upstream subunit's filtering tools. import optparse import os @@ -17,9 +32,11 @@ parser = optparse.OptionParser("filter-subunit [options] < instream > outstream" parser.add_option("--expected-failures", type="string", help="File containing list of regexes matching tests to consider known " "failures") -parser.add_option("--strip-passed-output", action="store_true", +parser.add_option("--flapping", type="string", + help="File containing list of flapping tests, of which to ignore results.") +parser.add_option("--strip-passed-output", action="store_true", help="Whether to strip output from tests that passed") -parser.add_option("--fail-immediately", action="store_true", +parser.add_option("--fail-immediately", action="store_true", help="Whether to stop on the first error", default=False) parser.add_option("--prefix", type="string", help="Add prefix to all test names") @@ -42,6 +59,12 @@ if opts.expected_failures: else: expected_failures = {} + +if opts.flapping: + flapping = subunithelper.read_test_regexes(opts.flapping) +else: + flapping = {} + statistics = { 'TESTS_UNEXPECTED_OK': 0, 'TESTS_EXPECTED_OK': 0, @@ -58,7 +81,8 @@ signal.signal(signal.SIGINT, handle_sigint) out = subunithelper.SubunitOps(sys.stdout) msg_ops = subunithelper.FilterOps(out, opts.prefix, expected_failures, opts.strip_passed_output, - fail_immediately=opts.fail_immediately) + fail_immediately=opts.fail_immediately, + flapping=flapping) try: ret = subunithelper.parse_results(msg_ops, statistics, sys.stdin) diff --git a/selftest/flapping b/selftest/flapping new file mode 100644 index 0000000..e8a6b49 --- /dev/null +++ b/selftest/flapping @@ -0,0 +1,19 @@ +# This file contains a list of regular expressions matching the names of +# tests that are flapping. In other words, they sometimes succeed and +# sometimes fail, depending on external factors. +# +# "make test" will not report failures or successes for tests listed here. +# +# DO NOT ADD TESTS HERE UNLESS THEY ARE ACTUALLY FLAPPING +# +# It is much better to add known failing tests to 'knownfail', so the +# test system can warn when they actually start passing. +^samba3.raw.mux.* #This test is flaky on the async lock time +^samba3.smbtorture_s3.*OPLOCK4 # fails sometimes on sn-devel +^samba4.nbt.winsreplication.owned # fails sometimes, timing related +^samba3.posix_s3.rpc.spoolss.*printserver.enum_printers_old # fails on some hosts due to timing issues ? +^samba3.posix_s3.rpc.spoolss.printer.*addprinterex.print_test # another intermittent failure +.*printer.*print_test_extended # fails on some hosts due to timing issues ? +.*printer.*print_test # fails on some hosts due to timing issues ? +^samba3.posix_s3.rap.printing # fails sometimes on sn-devel +^samba3.posix_s3.smb2.lock.*.rw-exclusive # another intermittent failure diff --git a/selftest/knownfail b/selftest/knownfail index 6f83881..754ffb3 100644 --- a/selftest/knownfail +++ b/selftest/knownfail @@ -4,25 +4,14 @@ # "make test" will not report failures for tests listed here and will consider # a successful run for any of these tests an error. -^samba3.blackbox.failure # this is designed to fail, for testing our test infrastructure -.*printer.*print_test_extended # fails on some hosts due to timing issues ? -.*printer.*print_test # fails on some hosts due to timing issues ? -^samba3.posix_s3.rap.printing # fails sometimes on sn-devel -^samba3.posix_s3.rpc.spoolss.*printserver.enum_printers_old # fails on some hosts due to timing issues ? -^samba3.posix_s3.rpc.spoolss.printer.*addprinterex.print_test # another intermittent failure -^samba3.posix_s3.smb2.lock.*.rw-exclusive # another intermittent failure +^samba3.blackbox.failure.failure # this is designed to fail, for testing our test infrastructure .*driver.add_driver_timestamps # we only can store dates, not timestamps -^samba3.raw.mux.* #This test is flaky on the async lock time -^samba3.smbtorture_s3.*OPLOCK4 # fails sometimes on sn-devel ^samba3.smbtorture_s3.LOCAL-MEMCACHE #fails ^samba3.smbtorture_s3.LOCAL-TALLOC-DICT #fails ^samba3.posix_s3.nbt.dgram.*netlogon2 ^samba3.*rap.sam.*.useradd # Not provided by Samba 3 ^samba3.*rap.sam.*.userdelete # Not provided by Samba 3 -^samba3.*rap.basic.*.netsessiongetinfo # Not provided by Samba 3 ^samba3.posix_s3.libsmbclient .opendir # This requires a workgroup called 'WORKGROUP' and for netbios browse lists to have been registered -^samba4.local.resolve.*.async -^samba4.local.iconv.*.next_codepoint() # these show that we still have some differences between our system # with our internal iconv because it passes except when we bypass our # internal iconv modules @@ -38,25 +27,21 @@ ^samba4.local.registry.local.security ^samba4.rpc.wkssvc ^samba4.rpc.handles.*.lsarpc-shared -^samba4.rpc.handles.*.mixed-shared ^samba4.rpc.epmapper -^samba4.rpc.drsuapi.* ^samba4.rpc.lsalookup ^samba4.rpc.cracknames ^samba4.rpc.netlogon.*.LogonUasLogon ^samba4.rpc.netlogon.*.LogonUasLogoff ^samba4.rpc.netlogon.*.DatabaseSync ^samba4.rpc.netlogon.*.DatabaseSync2 -^samba4.rpc.netlogon.*.LogonControl -^samba4.rpc.netlogon.*.LogonControl2 -^samba4.rpc.netlogon.*.DsrEnumerateDomainTrusts +^samba4.rpc.netlogon.*.LogonControl$ +^samba4.rpc.netlogon.*.LogonControl2$ ^samba4.rpc.netlogon.*.NetrEnumerateTrustedDomains ^samba4.rpc.netlogon.*.NetrEnumerateTrustedDomainsEx ^samba4.rpc.netlogon.*.GetPassword ^samba4.rpc.netlogon.*.GetTrustPasswords ^samba4.rpc.netlogon.*.DatabaseRedo ^samba4.rpc.netlogon.*.ServerGetTrustInfo -^samba4.rpc.netlogon.*.GetForestTrustInformation ^samba4.rpc.samr.passwords.badpwdcount # Not provided by Samba 4 yet ^samba4.rpc.samr.passwords.lockout ^samba4.base.charset.*.Testing partial surrogate @@ -67,10 +52,6 @@ ^samba4.rap.*netremotetod ^samba4.smb2.persistent.handles1 ^samba4.winbind.struct.*.show_sequence # Not yet working in winbind -^samba4.winbind.struct.*.getpwent # Not yet working in winbind -^samba4.winbind.struct.*.setpwent # Not yet working in winbind -^samba4.winbind.struct.*.lookup_name_sid # Not yet working in winbind -^samba4.winbind.struct.*.list_groups ^samba4.*base.delaywrite.*update of write time and SMBwrite truncate$ ^samba4.*base.delaywrite.*update of write time and SMBwrite truncate expand$ ^samba4.*base.delaywrite.*delayed update of write time 3a$ @@ -80,14 +61,10 @@ ^samba4.ldap.python \(dc\).Test add_ldif\(\) with BASE64 security descriptor input using WRONG domain SID$ # some operations don't work over the CIFS NTVFS backend yet (eg. root_fid) ^samba4.ntvfs.cifs.*.base.createx_sharemodes_dir -^samba4.ntvfs.cifs.*.base.maximum_allowed -^samba4.*.base.createx_access # this test is broken for non-administrator users -^samba4.smb2.oplock # oplocks in the s4 SMB2 server are a mess ^samba4.raw.lock.*.async # bug 6960 ^samba4.smb2.lock.*.multiple-unlock # bug 6959 -^samba4.raw.sfileinfo.*.end-of-file # bug 6962 +^samba4.raw.sfileinfo.*.end-of-file$ # bug 6962 ^samba4.raw.oplock.*.batch22 # bug 6963 -^samba4.raw.oplock.*.brl4 # bug 7928 ^samba4.raw.lock.*.zerobyteread # bug 6974 ^samba4.smb2.lock.*.zerobyteread # bug 6974 ^samba4.raw.streams.*.delete @@ -99,13 +76,33 @@ ^samba4.smb2.acls.*.generic ^samba4.smb2.acls.*.inheritflags ^samba4.smb2.acls.*.owner -^samba4.smb2.compound.*.related1 -^samba4.smb2.compound.*.related2 -^samba4.smb2.compound.*.invalid2 -^samba4.ldap.acl.*.search.* # ACL search behaviour not enabled by default ^samba4.ldap.acl.*.ntSecurityDescriptor.* # ACL extended checks on search not enabled by default -^samba4.nbt.winsreplication.owned # fails sometimes, timing related ^samba4.ldap.dirsync.python.dc..__main__.ExtendedDirsyncTests.test_dirsync_deleted_items #^samba4.ldap.dirsync.python.dc..__main__.ExtendedDirsyncTests.* ^samba4.drs.fsmo.python ^samba4.libsmbclient.opendir.opendir # This requires netbios browsing +^samba4.rpc.drsuapi .*.drsuapi.DsGetDomainControllerInfo$ +^samba4.rpc.drsuapi .*.drsuapi.DsCrackNames$ +^samba4.smb2.oplock.exclusive2$ # samba 4 oplocks are a mess +^samba4.smb2.oplock.exclusive5$ # samba 4 oplocks are a mess +^samba4.smb2.oplock.exclusive6$ # samba 4 oplocks are a mess +^samba4.smb2.oplock.brl3$ # samba 4 oplocks are a mess +^samba4.smb2.oplock.levelii500$ # samba 4 oplocks are a mess +^samba4.smb2.oplock.brl1$ # samba 4 oplocks are a mess +^samba4.smb2.oplock.batch22$ # samba 4 oplocks are a mess +^samba4.smb2.oplock.batch19$ # samba 4 oplocks are a mess +^samba4.smb2.oplock.batch12$ # samba 4 oplocks are a mess +^samba4.smb2.oplock.batch11$ # samba 4 oplocks are a mess +^samba4.smb2.oplock.batch1$ # samba 4 oplocks are a mess +^samba4.smb2.oplock.batch6$ # samba 4 oplocks are a mess +^samba4.smb2.oplock.batch9$ # samba 4 oplocks are a mess +^samba4.smb2.oplock.batch10$ # samba 4 oplocks are a mess +^samba4.smb2.oplock.batch20$ # samba 4 oplocks are a mess +^samba4.ntvfs.cifs.krb5.base.createx_access.createx_access$ +^samba4.ldap.acl.*.AclSearchTests.test_search_anonymous3$ # ACL search behaviour not enabled by default +^samba4.ldap.acl.*.AclSearchTests.test_search1$ # ACL search behaviour not enabled by default +^samba4.ldap.acl.*.AclSearchTests.test_search2$ # ACL search behaviour not enabled by default +^samba4.ldap.acl.*.AclSearchTests.test_search3$ # ACL search behaviour not enabled by default +^samba4.ldap.acl.*.AclSearchTests.test_search4$ # ACL search behaviour not enabled by default +^samba4.ldap.acl.*.AclSearchTests.test_search5$ # ACL search behaviour not enabled by default +^samba4.ldap.acl.*.AclSearchTests.test_search6$ # ACL search behaviour not enabled by default diff --git a/selftest/subunithelper.py b/selftest/subunithelper.py index c59b6d0..b498878 100644 --- a/selftest/subunithelper.py +++ b/selftest/subunithelper.py @@ -22,8 +22,9 @@ import sys import subunit import subunit.iso8601 import testtools +from testtools import content, content_type -VALID_RESULTS = ['success', 'successful', 'failure', 'fail', 'skip', 'knownfail', 'error', 'xfail', 'skip-testsuite', 'testsuite-failure', 'testsuite-xfail', 'testsuite-success', 'testsuite-error'] +VALID_RESULTS = ['success', 'successful', 'failure', 'fail', 'skip', 'knownfail', 'error', 'xfail', 'skip-testsuite', 'testsuite-failure', 'testsuite-xfail', 'testsuite-success', 'testsuite-error', 'uxsuccess'] class TestsuiteEnabledTestResult(testtools.testresult.TestResult): @@ -33,7 +34,6 @@ class TestsuiteEnabledTestResult(testtools.testresult.TestResult): def parse_results(msg_ops, statistics, fh): exitcode = 0 - expected_fail = 0 open_tests = {} while fh: @@ -111,7 +111,17 @@ def parse_results(msg_ops, statistics, fh): else: statistics['TESTS_EXPECTED_FAIL']+=1 msg_ops.addExpectedFailure(test, remote_error) - expected_fail+=1 + elif result in ("uxsuccess", ): + try: + test = open_tests.pop(testname) + except KeyError: + statistics['TESTS_ERROR']+=1 + exitcode = 1 + msg_ops.addError(subunit.RemotedTestCase(testname), subunit.RemoteError(u"Test was never started")) + else: + statistics['TESTS_UNEXPECTED_OK']+=1 + msg_ops.addUnexpectedSuccess(test, remote_error) + exitcode = 1 elif result in ("failure", "fail"): try: test = open_tests.pop(testname) @@ -285,9 +295,16 @@ class FilterOps(testtools.testresult.TestResult): self._ops.addExpectedFailure(test, details) self.output = None + def addUnexpectedSuccess(self, test, details=None): + test = self._add_prefix(test) + self._ops.addUnexpectedSuccess(test, details) + self.output = None + def addFailure(self, test, details=None): test = self._add_prefix(test) xfail_reason = find_in_list(self.expected_failures, test.id()) + if xfail_reason is None: + xfail_reason = find_in_list(self.flapping, test.id()) if xfail_reason is not None: self.xfail_added+=1 self.total_xfail+=1 @@ -308,7 +325,22 @@ class FilterOps(testtools.testresult.TestResult): def addSuccess(self, test, details=None): test = self._add_prefix(test) - self._ops.addSuccess(test, details) + xfail_reason = find_in_list(self.expected_failures, test.id()) + if xfail_reason is not None: + self.uxsuccess_added += 1 + self.total_uxsuccess += 1 + if details is None: + details = {} + details['reason'] = content.Content( + content_type.ContentType("text", "plain", + {"charset": "utf8"}), lambda: xfail_reason) + self._ops.addUnexpectedSuccess(test, details) + if self.output: + self._ops.output_msg(self.output) + if self.fail_immediately: + raise ImmediateFail() + else: + self._ops.addSuccess(test, details) self.output = None def skip_testsuite(self, name, reason=None): @@ -319,6 +351,7 @@ class FilterOps(testtools.testresult.TestResult): self.error_added = 0 self.fail_added = 0 self.xfail_added = 0 + self.uxsuccess_added = 0 def end_testsuite(self, name, result, reason=None): xfail = False @@ -346,7 +379,8 @@ class FilterOps(testtools.testresult.TestResult): self._ops.end_testsuite(name, result, reason) def __init__(self, out, prefix=None, expected_failures=None, - strip_ok_output=False, fail_immediately=False): + strip_ok_output=False, fail_immediately=False, + flapping=None): self._ops = out self.seen_output = False self.output = None @@ -355,12 +389,18 @@ class FilterOps(testtools.testresult.TestResult): self.expected_failures = expected_failures else: self.expected_failures = {} + if flapping is not None: + self.flapping = flapping + else: + self.flapping = {} self.strip_ok_output = strip_ok_output self.xfail_added = 0 self.fail_added = 0 + self.uxsuccess_added = 0 self.total_xfail = 0 self.total_error = 0 self.total_fail = 0 + self.total_uxsuccess = 0 self.error_added = 0 self.fail_immediately = fail_immediately @@ -484,10 +524,13 @@ class PlainFormatter(TestsuiteEnabledTestResult): def addSkip(self, test, details=None): self.end_test(test.id(), "skip", False, details) - def addExpectedFail(self, test, details=None): + def addExpectedFailure(self, test, details=None): self.end_test(test.id(), "xfail", False, details) - def end_test(self, testname, result, unexpected, reason=None): + def addUnexpectedSuccess(self, test, details=None): + self.end_test(test.id(), "uxsuccess", True, details) + + def end_test(self, testname, result, unexpected, details=None): if not unexpected: self.test_output[self.name] = "" if not self.immediate: @@ -502,17 +545,18 @@ class PlainFormatter(TestsuiteEnabledTestResult): self.test_output[self.name] = "" self.test_output[self.name] += "UNEXPECTED(%s): %s\n" % (result, testname) - if reason is not None: - self.test_output[self.name] += "REASON: %s\n" % (unicode(reason[1]).encode("utf-8").strip(),) + if details is not None: + self.test_output[self.name] += "REASON: %s\n" % (unicode(details[1]).encode("utf-8").strip(),) if self.immediate and not self.verbose: - print self.test_output[self.name] + sys.stdout.write(self.test_output[self.name]) self.test_output[self.name] = "" if not self.immediate: sys.stdout.write({ 'error': 'E', 'failure': 'F', + 'uxsuccess': 'U', 'success': 'S'}.get(result, "?")) def write_summary(self, path): @@ -546,14 +590,16 @@ class PlainFormatter(TestsuiteEnabledTestResult): if (not self.suitesfailed and not self.statistics['TESTS_UNEXPECTED_FAIL'] and + not self.statistics['TESTS_UNEXPECTED_OK'] and not self.statistics['TESTS_ERROR']): ok = (self.statistics['TESTS_EXPECTED_OK'] + self.statistics['TESTS_EXPECTED_FAIL']) print "\nALL OK (%d tests in %d testsuites)" % (ok, self.suites_ok) else: - print "\nFAILED (%d failures and %d errors in %d testsuites)" % ( + print "\nFAILED (%d failures, %d errors and %d unexpected successes in %d testsuites)" % ( self.statistics['TESTS_UNEXPECTED_FAIL'], self.statistics['TESTS_ERROR'], + self.statistics['TESTS_UNEXPECTED_OK'], len(self.suitesfailed)) def skip_testsuite(self, name, reason="UNKNOWN"): diff --git a/selftest/wscript b/selftest/wscript index 5a2cbd1..67ea5a4 100644 --- a/selftest/wscript +++ b/selftest/wscript @@ -88,7 +88,7 @@ def cmd_testonly(opt): env.SUBUNIT_FORMATTER = os.getenv('SUBUNIT_FORMATTER') if not env.SUBUNIT_FORMATTER: env.SUBUNIT_FORMATTER = '${PYTHON} -u ${srcdir}/selftest/format-subunit --prefix=${SELFTEST_PREFIX} --immediate' - env.FILTER_XFAIL = '${PYTHON} -u ${srcdir}/selftest/filter-subunit --expected-failures=${srcdir}/selftest/knownfail' + env.FILTER_XFAIL = '${PYTHON} -u ${srcdir}/selftest/filter-subunit --expected-failures=${srcdir}/selftest/knownfail --flapping=${srcdir}/selftest/flapping' if Options.options.FAIL_IMMEDIATELY: env.FILTER_XFAIL += ' --fail-immediately' diff --git a/source3/selftest/s3-selftest.sh b/source3/selftest/s3-selftest.sh index 4943b1e..0d20324 100755 --- a/source3/selftest/s3-selftest.sh +++ b/source3/selftest/s3-selftest.sh @@ -1,6 +1,6 @@ #!/bin/sh -FILTER_XFAIL="${PYTHON} -u ${SELFTESTDIR}/filter-subunit --expected-failures=${SELFTESTDIR}/knownfail" +FILTER_XFAIL="${PYTHON} -u ${SELFTESTDIR}/filter-subunit --expected-failures=${SELFTESTDIR}/knownfail --flapping=${SELFTESTDIR}/flapping" if [ "x${SUBUNIT_FORMATTER}" = x"" ]; then SUBUNIT_FORMATTER="${PYTHON} -u ${SELFTESTDIR}/format-subunit --prefix=${SELFTESTPREFIX} --immediate" fi -- Samba Shared Repository