The autobuild test system (on sn-devel-144) has detected an intermittent failing test in the current master tree.
The autobuild log of the failure is available here: https://git.samba.org/autobuild.flakey.sn-devel-144/2018-10-30-0735/flakey.log The failure seems to be in the "samba" suite, whose build logs are available here: https://git.samba.org/autobuild.flakey.sn-devel-144/2018-10-30-0735/samba.stderr https://git.samba.org/autobuild.flakey.sn-devel-144/2018-10-30-0735/samba.stdout The top commit at the time of the failure was: commit 500a729a5566a1c8afc36e6f3e73910b13c8b9ad Author: Volker Lendecke <v...@samba.org> Date: Wed Oct 24 14:31:34 2018 +0200 tdb: Make record deletion circular-chain safe Before this patch we had 3 loops walking a hash chain to delete records: tdb_do_delete() to find the predecessor of the record that was to be deleted. tdb_count_dead(), the name says it all and tdb_purge_dead() to give back all dead records from a chain to the freelist. This patch introduces tdb_trim_dead that walks a hash chain just once. While it does so it counts the number of dead records, and all records beyond tdb->max_dead_records are moved to the freelist. Normal record deletion now works by always marking a record as dead in step 1 and then calling tdb_trim_dead. This is made safe against circular chains by doing the slow chain walk only in the case when we did not delete a dead record during our walk. It changes our dynamics a bit: When deleting a record with non-zero max_dead_records, now we always leave that number of records around when deleting, doing a blocking lock on the freelist when we found too many dead records. Previously when exceeding max_dead_records we wiped all dead records to start accumulating them from scratch, assuming we could lock the freelist in a nonblocking fashion. The net effect for an uncontended freelist is the same: In tdb_allocate() we still completely hand over all dead records to the freelist when we could lock it, it just happens later than without this patch. This means for a lightly loaded system we will potentially leave more dead records around in databases like locking.tdb. However, on a heavily loaded system we become more predictable: If the freelist is so heavily contended that across many deletes we can't get hold of it, previously we accumulated more dead records than max_dead_records would allow. This is a really lowlevel tradeoff that is likely hard to measure, but to me becoming more deterministic without sacrificing too much parallelism (we keep more dead records around) is worth trying. Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> Autobuild-User(master): Jeremy Allison <j...@samba.org> Autobuild-Date(master): Tue Oct 30 02:48:38 CET 2018 on sn-devel-144 and the last 50 lines of the stdout log were: [751(4033)/850 at 53m5s] samba.tests.blackbox.traffic_replay(ad_dc_ntvfs) [752(4039)/850 at 53m15s] samba.tests.blackbox.traffic_learner(ad_dc_ntvfs) [753(4041)/850 at 53m15s] samba.tests.blackbox.traffic_summary(ad_dc_ntvfs) [754(4041)/850 at 53m15s] samba.tests.blackbox.smbcontrol(ad_dc_ntvfs:local) [755(4043)/850 at 53m17s] samba.tests.blackbox.smbcontrol(promoted_dc:local) [756(4045)/850 at 53m18s] samba4.ldap.python(ad_dc_ntvfs)(ad_dc_ntvfs) [757(4084)/850 at 53m30s] samba4.tokengroups.krb5.python(ad_dc_ntvfs)(ad_dc_ntvfs:local) [758(4094)/850 at 53m38s] samba4.tokengroups.ntlm.python(ad_dc_ntvfs)(ad_dc_ntvfs:local) [759(4103)/850 at 53m47s] samba4.sam.python(ad_dc_ntvfs)(ad_dc_ntvfs) [760(4120)/850 at 53m50s] samba4.user_account_control.python(ad_dc_ntvfs)(ad_dc_ntvfs) [761(4131)/850 at 54m4s] samba4.schemaInfo.python(ad_dc_ntvfs)(ad_dc_ntvfs) [762(4133)/850 at 54m6s] samba.tests.dsdb_schema_attributes(ad_dc_ntvfs:local) [763(4139)/850 at 54m21s] samba4.urgent_replication.python(ad_dc_ntvfs)(ad_dc_ntvfs:local) [764(4147)/850 at 54m25s] samba4.ldap.dirsync.python(ad_dc_ntvfs)(ad_dc_ntvfs) [765(4171)/850 at 55m55s] samba4.ldap.match_rules.python(ad_dc_ntvfs) [766(4205)/850 at 56m13s] samba4.ldap.notification.python(ad_dc_ntvfs)(ad_dc_ntvfs) [767(4208)/850 at 56m22s] samba4.ldap.sites.python(ad_dc_ntvfs)(ad_dc_ntvfs) [768(4219)/850 at 56m28s] samba4.ldap.sort.python(ad_dc_ntvfs)(ad_dc_ntvfs) [769(4226)/850 at 56m33s] samba4.ldap.vlv.python(ad_dc_ntvfs)(ad_dc_ntvfs) [770(4241)/850 at 1h29s] samba4.ldap.linked_attributes.python(ad_dc_ntvfs)(ad_dc_ntvfs:local) [771(4261)/850 at 1h33s] samba4.ldap.rodc.python(rodc)(rodc) [772(4267)/850 at 1h37s] samba4.ldap.rodc_rwdc.python(rodc)(rodc:local) [773(4287)/850 at 1h8m10s] samba4.drs.replica_sync_rodc.python(rodc)(rodc:local) [774(4289)/850 at 1h8m18s] samba4.ldap.passwordsettings.python(ad_dc_ntvfs) [775(4301)/850 at 1h8m57s] samba4.ldap_schema.python(ad_dc_ntvfs)(ad_dc_ntvfs) GSS client Update(krb5)(1) Update failed: Miscellaneous failure (see text): Clock skew too great UNEXPECTED(error): samba4.ldap_schema.python(ad_dc_ntvfs).__main__.SchemaTests.test_generated_mAPIID(ad_dc_ntvfs) REASON: Exception: Exception: Traceback (most recent call last): File "/memdisk/autobuild/fl/b2733863/samba/source4/dsdb/tests/python/ldap_schema.py", line 1162, in test_generated_mAPIID self.ldb.modify_ldif(ldif) File "bin/python/samba/__init__.py", line 241, in modify_ldif self.modify(msg, controls) LdbError: (3, 'ldb_wait from (null) with LDB_WAIT_ALL: Time limit exceeded (3)') FAILED (0 failures, 1 errors and 0 unexpected successes in 0 testsuites) A summary with detailed information can be found in: ./bin/ab/summary TOP 10 slowest tests samba4.ldap.rodc_rwdc.python(rodc)(rodc:local) -> 453 samba4.ldap.vlv.python(ad_dc_ntvfs)(ad_dc_ntvfs) -> 235 samba4.rpc.samr.passwords.lockout on ncacn_np with (ad_dc_ntvfs) -> 225 samba4.rpc.samr.passwords.pwdlastset on ncacn_np with (ad_dc_ntvfs) -> 200 samba3.smbtorture_s3.vfs_aio_fork(simpleserver).RW2(simpleserver) -> 145 samba4.ldb.ldaps with options -U"$USERNAME%$PASSWORD"(ad_dc_ntvfs)(ad_dc_ntvfs) -> 114 samba4.raw.notify(ad_dc_ntvfs) -> 106 samba4.ldap.dirsync.python(ad_dc_ntvfs)(ad_dc_ntvfs) -> 91 samba.tests.emulate.traffic_packet(ad_dc_ntvfs) -> 67 samba.tests.samba_tool.user(ad_dc_ntvfs:local) -> 65 ERROR: test failed with exit code 1