The branch, master has been updated via 6cf7abb repl_meta_data: Allow delete of an object with dangling backlinks via 40bd7e1 repl_meta_data: Fix removing of backlink on deleted objects via 4815efc selftest: Add more corruption cases for runtime and dbcheck via 70bf809 selftest: add dbcheck tests for duplicate links via 239fbeb dbcheck: detect and fix duplicate links via 9a63156 dbcheck: only calculate linked attribute helper variables once in check_dn() via eb6bd65 dbcheck: remove indentation level via 83aa222 dsdb:extended_dn_store: implement DSDB_CONTROL_DBCHECK_FIX_DUPLICATE_LINKS control via 9a1e23a dsdb:repl_meta_data: implement DSDB_CONTROL_DBCHECK_FIX_DUPLICATE_LINKS control via 1eb8d8e s4:dsdb: allocate DSDB_CONTROL_DBCHECK_FIX_DUPLICATE_LINKS oid via 126d28d s4:schema_samba4: mark DSDB_CONTROL_INVALID_NOT_IMPLEMENTED 1.3.6.1.4.1.7165.4.3.32 as allocated via a784cc3 selftest: Additional check for a backlink pointing at a deleted object via 25ae8d7 selftest: Split out creation of complex (often invalid) links via b99d2ee selftest: Split out dbcheck runs from dangling_multi_valued test via 7be38c6 selftest: add more dbcheck tests via 527f2c9 dbcheck: Use the GUID as the DN to fix replPropertyMetaData via 3b111fb dbcheck: Clarify error count bumping in deleted/gone DN handling from f026314 ctdb-eventd: Simplify eventd code
https://git.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit 6cf7abbcfdad84fee57852862ebe44aa6115ca25 Author: Andrew Bartlett <abart...@samba.org> Date: Wed Nov 1 08:22:22 2017 +1300 repl_meta_data: Allow delete of an object with dangling backlinks This should not happen, but stopping all replication because of it is a pain. BUG: https://bugzilla.samba.org/show_bug.cgi?id=13095 Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> Autobuild-User(master): Stefan Metzmacher <me...@samba.org> Autobuild-Date(master): Fri Nov 24 19:53:50 CET 2017 on sn-devel-144 commit 40bd7e145a68c9a58d6bc3c5526a12fdf0027729 Author: Andrej Gessel <andrej.ges...@janztec.com> Date: Thu Oct 19 17:16:37 2017 +0200 repl_meta_data: Fix removing of backlink on deleted objects USER is memberOf GROUP and they both were deleted on W2K8R2 AD. Domain join ends with error below. Failed to apply records: ../source4/dsdb/samdb/ldb_modules/repl_meta_data.c:421 8: Failed to remove backlink of memberOf when deleting CN=USER\0ADEL:a1f2a2cc-1 179-4734-b753-c121ed02a34c,CN=Deleted Objects,DC=samdom,DC=intern: dsdb_module_ search_dn: did not find base dn CN=GROUP\0ADEL:030d0be1-3ada-4b93-8371-927f2092 3116,CN=Deleted Objects,DC=samdom,DC=intern (0 results): Operations error Failed to commit objects: WERR_GEN_FAILURE/NT_STATUS_INVALID_NETWORK_RESPONSE BUG: https://bugzilla.samba.org/show_bug.cgi?id=13120 Signed-off-by: Andrej Gessel <andrej.ges...@janztec.com> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> commit 4815efc0e3f89079e7c9b868b7514ea7c49a807c Author: Andrew Bartlett <abart...@samba.org> Date: Wed Nov 1 09:02:01 2017 +1300 selftest: Add more corruption cases for runtime and dbcheck These tests now confirm we can handle these issues at runtime as well as at dbcheck Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> commit 70bf809e0cdf84029022ca95fb83d17a0d6e36c0 Author: Stefan Metzmacher <me...@samba.org> Date: Thu Oct 26 14:42:23 2017 +0200 selftest: add dbcheck tests for duplicate links BUG: https://bugzilla.samba.org/show_bug.cgi?id=13095 Pair-Programmed-With: Andrew Bartlett <abart...@samba.org> Signed-off-by: Stefan Metzmacher <me...@samba.org> Signed-off-by: Andrew Bartlett <abart...@samba.org> commit 239fbeb163c24b0f08e1bd9d8f7a9f73443d4b90 Author: Stefan Metzmacher <me...@samba.org> Date: Fri Oct 27 10:21:26 2017 +0200 dbcheck: detect and fix duplicate links Check with git show -w BUG: https://bugzilla.samba.org/show_bug.cgi?id=13095 Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 9a631560c9358e4f28b96fecf23a545e1d04098c Author: Stefan Metzmacher <me...@samba.org> Date: Fri Oct 27 10:21:26 2017 +0200 dbcheck: only calculate linked attribute helper variables once in check_dn() BUG: https://bugzilla.samba.org/show_bug.cgi?id=13095 Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit eb6bd6511a98dafebaa0d3951fe78c77acf13e3a Author: Stefan Metzmacher <me...@samba.org> Date: Thu Oct 26 16:30:28 2017 +0200 dbcheck: remove indentation level Check with git show -w BUG: https://bugzilla.samba.org/show_bug.cgi?id=13095 Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 83aa22260bd39ad0e6aab7764f9a7fc915d02a4b Author: Stefan Metzmacher <me...@samba.org> Date: Thu Oct 26 07:47:48 2017 +0200 dsdb:extended_dn_store: implement DSDB_CONTROL_DBCHECK_FIX_DUPLICATE_LINKS control This will be used by dbcheck to fix duplicate link values. BUG: https://bugzilla.samba.org/show_bug.cgi?id=13095 Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 9a1e23a1f67c38248e41e0d3aa2af8a682477364 Author: Stefan Metzmacher <me...@samba.org> Date: Wed Oct 25 16:48:44 2017 +0200 dsdb:repl_meta_data: implement DSDB_CONTROL_DBCHECK_FIX_DUPLICATE_LINKS control This will be used by dbcheck to fix duplicate link values. BUG: https://bugzilla.samba.org/show_bug.cgi?id=13095 Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 1eb8d8ec5a62baf1b8e3c7cb1856787de4a3ccb2 Author: Stefan Metzmacher <me...@samba.org> Date: Wed Oct 25 16:47:36 2017 +0200 s4:dsdb: allocate DSDB_CONTROL_DBCHECK_FIX_DUPLICATE_LINKS oid BUG: https://bugzilla.samba.org/show_bug.cgi?id=13095 Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 126d28d0b58740a8abd4137562dda685a57449bb Author: Stefan Metzmacher <me...@samba.org> Date: Wed Oct 25 16:26:16 2017 +0200 s4:schema_samba4: mark DSDB_CONTROL_INVALID_NOT_IMPLEMENTED 1.3.6.1.4.1.7165.4.3.32 as allocated Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit a784cc3a7f2043a5762d426e904a90e44b101ecd Author: Andrew Bartlett <abart...@samba.org> Date: Tue Oct 31 11:20:34 2017 +1300 selftest: Additional check for a backlink pointing at a deleted object Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> commit 25ae8d72d66cbe7342b50254ede7e5890bc23b73 Author: Andrew Bartlett <abart...@samba.org> Date: Tue Oct 31 08:23:39 2017 +1300 selftest: Split out creation of complex (often invalid) links This will allow us to test other run-time behaviour with broken databases. Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> commit b99d2ee122991d0bf1742fa5665656bbbba44057 Author: Andrew Bartlett <abart...@samba.org> Date: Tue Oct 31 08:21:15 2017 +1300 selftest: Split out dbcheck runs from dangling_multi_valued test Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> commit 7be38c605468786894a373e15068b8017323da78 Author: Andrew Bartlett <abart...@samba.org> Date: Mon Oct 30 15:29:36 2017 +1300 selftest: add more dbcheck tests This validates some more combinations and ensures that the changes in 962a1b32201fce0a49c6be55943d4fbb57ed781e are tested. Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> commit 527f2c95cfe24d29fa34ed19db3b073781d96d9a Author: Andrew Bartlett <abart...@samba.org> Date: Mon Oct 30 10:51:35 2017 +1300 dbcheck: Use the GUID as the DN to fix replPropertyMetaData This allows this to still work after an object is renamed under the deleted objects container. Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> commit 3b111fbdbed99d5d90c1120243200baae9867534 Author: Andrew Bartlett <abart...@samba.org> Date: Mon Oct 30 09:48:43 2017 +1300 dbcheck: Clarify error count bumping in deleted/gone DN handling Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> ----------------------------------------------------------------------- Summary of changes: python/samba/dbchecker.py | 287 +++++++++++++++------ selftest/knownfail | 2 +- selftest/tests.py | 5 + source4/dsdb/samdb/ldb_modules/extended_dn_store.c | 7 + source4/dsdb/samdb/ldb_modules/repl_meta_data.c | 81 +++++- source4/dsdb/samdb/samdb.h | 3 + .../add-dangling-deleted-link.ldif | 5 + .../add-deleted-backlink-user.ldif | 3 + ...ing-backlink.ldif => add-deleted-backlink.ldif} | 3 +- .../add-deleted-target-backlink-user.ldif | 3 + .../add-deleted-target-backlink.ldif | 4 + ...-one-way-link.ldif => dangling-one-way-dn.ldif} | 0 .../release-4-5-0-pre1/dangling-one-way-link.ldif | 22 +- .../release-4-5-0-pre1/deleted-one-way-dn.ldif | 13 + .../expected-dbcheck-link-output.txt | 58 +++-- ...pected-dbcheck-link-output_duplicate_member.txt | 8 + .../expected-dbcheck-link-output_one_way.txt | 7 + .../expected-duplicates-after-link-dbcheck.ldif | 28 ++ .../revive-backlink-on-deleted-group.ldif | 5 + .../revive-links-on-deleted-group.ldif | 20 ++ source4/setup/schema_samba4.ldif | 2 + testprogs/blackbox/common-links.sh | 215 +++++++++++++++ testprogs/blackbox/dbcheck-links.sh | 210 +++++---------- testprogs/blackbox/runtime-links.sh | 74 ++++++ 24 files changed, 792 insertions(+), 273 deletions(-) create mode 100644 source4/selftest/provisions/release-4-5-0-pre1/add-dangling-deleted-link.ldif create mode 100644 source4/selftest/provisions/release-4-5-0-pre1/add-deleted-backlink-user.ldif copy source4/selftest/provisions/release-4-5-0-pre1/{add-dangling-backlink.ldif => add-deleted-backlink.ldif} (67%) create mode 100644 source4/selftest/provisions/release-4-5-0-pre1/add-deleted-target-backlink-user.ldif create mode 100644 source4/selftest/provisions/release-4-5-0-pre1/add-deleted-target-backlink.ldif copy source4/selftest/provisions/release-4-5-0-pre1/{dangling-one-way-link.ldif => dangling-one-way-dn.ldif} (100%) create mode 100644 source4/selftest/provisions/release-4-5-0-pre1/deleted-one-way-dn.ldif create mode 100644 source4/selftest/provisions/release-4-5-0-pre1/expected-dbcheck-link-output_duplicate_member.txt create mode 100644 source4/selftest/provisions/release-4-5-0-pre1/expected-dbcheck-link-output_one_way.txt create mode 100644 source4/selftest/provisions/release-4-5-0-pre1/expected-duplicates-after-link-dbcheck.ldif create mode 100644 source4/selftest/provisions/release-4-5-0-pre1/revive-backlink-on-deleted-group.ldif create mode 100644 source4/selftest/provisions/release-4-5-0-pre1/revive-links-on-deleted-group.ldif create mode 100644 testprogs/blackbox/common-links.sh create mode 100755 testprogs/blackbox/runtime-links.sh Changeset truncated at 500 lines: diff --git a/python/samba/dbchecker.py b/python/samba/dbchecker.py index 82088a0..1933740 100644 --- a/python/samba/dbchecker.py +++ b/python/samba/dbchecker.py @@ -65,6 +65,7 @@ class dbcheck(object): self.fix_undead_linked_attributes = False self.fix_all_missing_backlinks = False self.fix_all_orphaned_backlinks = False + self.fix_all_duplicate_links = False self.fix_rmd_flags = False self.fix_ntsecuritydescriptor = False self.fix_ntsecuritydescriptor_owner_group = False @@ -498,13 +499,15 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base))) def err_deleted_dn(self, dn, attrname, val, dsdb_dn, correct_dn, remove_plausible=False): """handle a DN pointing to a deleted object""" - self.report("ERROR: target DN is deleted for %s in object %s - %s" % (attrname, dn, val)) - self.report("Target GUID points at deleted DN %r" % str(correct_dn)) if not remove_plausible: + self.report("ERROR: target DN is deleted for %s in object %s - %s" % (attrname, dn, val)) + self.report("Target GUID points at deleted DN %r" % str(correct_dn)) if not self.confirm_all('Remove DN link?', 'remove_implausible_deleted_DN_links'): self.report("Not removing") return else: + self.report("WARNING: target DN is deleted for %s in object %s - %s" % (attrname, dn, val)) + self.report("Target GUID points at deleted DN %r" % str(correct_dn)) if not self.confirm_all('Remove stale DN link?', 'remove_plausible_deleted_DN_links'): self.report("Not removing") return @@ -527,23 +530,45 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base))) linkID, reverse_link_name \ = self.get_attr_linkID_and_reverse_name(attrname) if reverse_link_name is not None: + self.report("WARNING: no target object found for GUID " + "component for one-way forward link " + "%s in object " + "%s - %s" % (attrname, dn, val)) self.report("Not removing dangling forward link") - return + return 0 nc_root = self.samdb.get_nc_root(dn) target_nc_root = self.samdb.get_nc_root(dsdb_dn.dn) if nc_root != target_nc_root: + # We don't bump the error count as Samba produces these + # in normal operation + self.report("WARNING: no target object found for GUID " + "component for cross-partition link " + "%s in object " + "%s - %s" % (attrname, dn, val)) self.report("Not removing dangling one-way " "cross-partition link " "(we might be mid-replication)") - return + return 0 # Due to our link handling one-way links pointing to # missing objects are plausible. + # + # We don't bump the error count as Samba produces these + # in normal operation + self.report("WARNING: no target object found for GUID " + "component for DN value %s in object " + "%s - %s" % (attrname, dn, val)) self.err_deleted_dn(dn, attrname, val, dsdb_dn, dsdb_dn, True) + return 0 + # We bump the error count here, as we should have deleted this + self.report("ERROR: no target object found for GUID " + "component for link %s in object " + "%s - %s" % (attrname, dn, val)) self.err_deleted_dn(dn, attrname, val, dsdb_dn, dsdb_dn, False) + return 1 def err_missing_dn_GUID_component(self, dn, attrname, val, dsdb_dn, errstr): """handle a missing GUID extended DN component""" @@ -696,6 +721,19 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base))) "Failed to fix orphaned backlink %s" % attrname): self.report("Fixed orphaned backlink %s" % (attrname)) + def err_duplicate_links(self, obj, attrname, vals): + '''handle a duplicate links value''' + + if not self.confirm_all("Remove duplicate links in attribute '%s'" % attrname, 'fix_all_duplicate_links'): + self.report("Not removing duplicate links in attribute '%s'" % attrname) + return + m = ldb.Message() + m.dn = obj.dn + m['value'] = ldb.MessageElement(vals, ldb.FLAG_MOD_REPLACE, attrname) + if self.do_modify(m, ["local_oid:1.3.6.1.4.1.7165.4.3.19.2:1"], + "Failed to fix duplicate links in attribute '%s'" % attrname): + self.report("Fixed duplicate links in attribute '%s'" % (attrname)) + def err_no_fsmoRoleOwner(self, obj): '''handle a missing fSMORoleOwner''' self.report("ERROR: fSMORoleOwner not found for role %s" % (obj.dn)) @@ -852,6 +890,81 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base))) error_count = 0 obj_guid = obj['objectGUID'][0] + linkID, reverse_link_name = self.get_attr_linkID_and_reverse_name(attrname) + if reverse_link_name is not None: + reverse_syntax_oid = self.samdb_schema.get_syntax_oid_from_lDAPDisplayName(reverse_link_name) + else: + reverse_syntax_oid = None + + duplicate_dict = dict() + duplicate_list = list() + unique_dict = dict() + unique_list = list() + for val in obj[attrname]: + if linkID & 1: + # + # Only cleanup forward links here, + # back links are handled below. + break + + dsdb_dn = dsdb_Dn(self.samdb, val, syntax_oid) + + # all DNs should have a GUID component + guid = dsdb_dn.dn.get_extended_component("GUID") + if guid is None: + continue + guidstr = str(misc.GUID(guid)) + keystr = guidstr + dsdb_dn.prefix + if keystr not in unique_dict: + unique_dict[keystr] = dsdb_dn + unique_list.append(keystr) + continue + error_count += 1 + if keystr not in duplicate_dict: + duplicate_dict[keystr] = dict() + duplicate_dict[keystr]["keep"] = None + duplicate_dict[keystr]["delete"] = list() + duplicate_list.append(keystr) + + # Now check for the highest RMD_VERSION + v1 = int(unique_dict[keystr].dn.get_extended_component("RMD_VERSION")) + v2 = int(dsdb_dn.dn.get_extended_component("RMD_VERSION")) + if v1 > v2: + duplicate_dict[keystr]["keep"] = unique_dict[keystr] + duplicate_dict[keystr]["delete"].append(dsdb_dn) + continue + if v1 < v2: + duplicate_dict[keystr]["keep"] = dsdb_dn + duplicate_dict[keystr]["delete"].append(unique_dict[keystr]) + unique_dict[keystr] = dsdb_dn + continue + # Fallback to the highest RMD_LOCAL_USN + u1 = int(unique_dict[keystr].dn.get_extended_component("RMD_LOCAL_USN")) + u2 = int(dsdb_dn.dn.get_extended_component("RMD_LOCAL_USN")) + if u1 >= u2: + duplicate_dict[keystr]["keep"] = unique_dict[keystr] + duplicate_dict[keystr]["delete"].append(dsdb_dn) + continue + duplicate_dict[keystr]["keep"] = dsdb_dn + duplicate_dict[keystr]["delete"].append(unique_dict[keystr]) + unique_dict[keystr] = dsdb_dn + + if len(duplicate_list) != 0: + self.report("ERROR: Duplicate link values for attribute '%s' in '%s'" % (attrname, obj.dn)) + for keystr in duplicate_list: + d = duplicate_dict[keystr] + for dd in d["delete"]: + self.report("Duplicate link '%s'" % dd) + self.report("Correct link '%s'" % d["keep"]) + + vals = [] + for keystr in unique_list: + dsdb_dn = unique_dict[keystr] + vals.append(str(dsdb_dn)) + self.err_duplicate_links(obj, attrname, vals) + # We should continue with the fixed values + obj[attrname] = ldb.MessageElement(vals, ldb.FLAG_MOD_REPLACE, attrname) + for val in obj[attrname]: dsdb_dn = dsdb_Dn(self.samdb, val, syntax_oid) @@ -872,7 +985,6 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base))) else: fixing_msDS_HasInstantiatedNCs = False - linkID, reverse_link_name = self.get_attr_linkID_and_reverse_name(attrname) if reverse_link_name is not None: attrs.append(reverse_link_name) @@ -883,12 +995,14 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base))) "reveal_internals:0" ]) except ldb.LdbError, (enum, estr): - error_count += 1 - self.report("ERROR: no target object found for GUID component for %s in object %s - %s" % (attrname, obj.dn, val)) if enum != ldb.ERR_NO_SUCH_OBJECT: raise - self.err_missing_target_dn_or_GUID(obj.dn, attrname, val, dsdb_dn) + # We don't always want to + error_count += self.err_missing_target_dn_or_GUID(obj.dn, + attrname, + val, + dsdb_dn) continue if fixing_msDS_HasInstantiatedNCs: @@ -944,16 +1058,15 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base))) # components on deleted links, as these are allowed to # go stale (we just need the GUID, not the name) rmd_blob = dsdb_dn.dn.get_extended_component("RMD_FLAGS") + rmd_flags = 0 if rmd_blob is not None: rmd_flags = int(rmd_blob) - if rmd_flags & 1: - continue # assert the DN matches in string form, where a reverse # link exists, otherwise (below) offer to fix it as a non-error. # The string form is essentially only kept for forensics, # as we always re-resolve by GUID in normal operations. - if reverse_link_name is not None: + if not rmd_flags & 1 and reverse_link_name is not None: if str(res[0].dn) != str(dsdb_dn.dn): error_count += 1 self.err_dn_component_target_mismatch(obj.dn, attrname, val, dsdb_dn, @@ -982,76 +1095,93 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base))) res[0].dn) continue - else: - # check the reverse_link is correct if there should be one - match_count = 0 - if reverse_link_name in res[0]: - for v in res[0][reverse_link_name]: - v_guid = dsdb_Dn(self.samdb, v).dn.get_extended_component("GUID") - if v_guid == obj_guid: - match_count += 1 - if match_count != 1: - reverse_syntax_oid = self.samdb_schema.get_syntax_oid_from_lDAPDisplayName(reverse_link_name) - if syntax_oid == dsdb.DSDB_SYNTAX_BINARY_DN or reverse_syntax_oid == dsdb.DSDB_SYNTAX_BINARY_DN: - if not linkID & 1: - # Forward binary multi-valued linked attribute - forward_count = 0 - for w in obj[attrname]: - w_guid = dsdb_Dn(self.samdb, w).dn.get_extended_component("GUID") - if w_guid == guid: - forward_count += 1 - - if match_count == forward_count: - continue - - error_count += 1 - - # Add or remove the missing number of backlinks - diff_count = forward_count - match_count - - # Loop until the difference between the forward and - # the backward links is resolved. - while diff_count != 0: - if diff_count > 0: - # self.err_missing_backlink(obj, attrname, - # obj.dn.extended_str(), - # reverse_link_name, - # dsdb_dn.dn) - # diff_count -= 1 - # TODO no method to fix these right now - self.report("ERROR: Can't fix missing " - "multi-valued backlinks on %s" % str(dsdb_dn.dn)) - break - else: - self.err_orphaned_backlink(res[0], reverse_link_name, - obj.dn.extended_str(), attrname, - dsdb_dn.dn) - diff_count += 1 + # check the reverse_link is correct if there should be one + match_count = 0 + if reverse_link_name in res[0]: + for v in res[0][reverse_link_name]: + v_dn = dsdb_Dn(self.samdb, v) + v_guid = v_dn.dn.get_extended_component("GUID") + v_blob = v_dn.dn.get_extended_component("RMD_FLAGS") + v_rmd_flags = 0 + if v_blob is not None: + v_rmd_flags = int(v_blob) + if v_rmd_flags & 1: + continue + if v_guid == obj_guid: + match_count += 1 + + if match_count != 1: + if syntax_oid == dsdb.DSDB_SYNTAX_BINARY_DN or reverse_syntax_oid == dsdb.DSDB_SYNTAX_BINARY_DN: + if not linkID & 1: + # Forward binary multi-valued linked attribute + forward_count = 0 + for w in obj[attrname]: + w_guid = dsdb_Dn(self.samdb, w).dn.get_extended_component("GUID") + if w_guid == guid: + forward_count += 1 + + if match_count == forward_count: + continue + expected_count = 0 + for v in obj[attrname]: + v_dn = dsdb_Dn(self.samdb, v) + v_guid = v_dn.dn.get_extended_component("GUID") + v_blob = v_dn.dn.get_extended_component("RMD_FLAGS") + v_rmd_flags = 0 + if v_blob is not None: + v_rmd_flags = int(v_blob) + if v_rmd_flags & 1: + continue + if v_guid == guid: + expected_count += 1 - else: - # If there's a backward link on binary multi-valued linked attribute, - # let the check on the forward link remedy the value. - # UNLESS, there is no forward link detected. - if match_count == 0: - self.err_orphaned_backlink(obj, attrname, - val, reverse_link_name, - dsdb_dn.dn) + if match_count == expected_count: + continue - continue + diff_count = expected_count - match_count + if linkID & 1: + # If there's a backward link on binary multi-valued linked attribute, + # let the check on the forward link remedy the value. + # UNLESS, there is no forward link detected. + if match_count == 0: error_count += 1 - if linkID & 1: - # Backlink exists, but forward link does not - # Delete the hanging backlink - self.err_orphaned_backlink(obj, attrname, val, reverse_link_name, dsdb_dn.dn) - else: - # Forward link exists, but backlink does not - # Add the missing backlink (if the target object is not Deleted Objects?) - if not target_is_deleted: - self.err_missing_backlink(obj, attrname, obj.dn.extended_str(), reverse_link_name, dsdb_dn.dn) + self.err_orphaned_backlink(obj, attrname, + val, reverse_link_name, + dsdb_dn.dn) continue + # Only warn here and let the forward link logic fix it. + self.report("WARNING: Link (back) mismatch for '%s' (%d) on '%s' to '%s' (%d) on '%s'" % ( + attrname, expected_count, str(obj.dn), + reverse_link_name, match_count, str(dsdb_dn.dn))) + continue + + assert not target_is_deleted + self.report("ERROR: Link (forward) mismatch for '%s' (%d) on '%s' to '%s' (%d) on '%s'" % ( + attrname, expected_count, str(obj.dn), + reverse_link_name, match_count, str(dsdb_dn.dn))) + # Loop until the difference between the forward and + # the backward links is resolved. + while diff_count != 0: + error_count += 1 + if diff_count > 0: + if match_count > 0 or diff_count > 1: + # TODO no method to fix these right now + self.report("ERROR: Can't fix missing " + "multi-valued backlinks on %s" % str(dsdb_dn.dn)) + break + self.err_missing_backlink(obj, attrname, + obj.dn.extended_str(), + reverse_link_name, + dsdb_dn.dn) + diff_count -= 1 + else: + self.err_orphaned_backlink(res[0], reverse_link_name, + obj.dn.extended_str(), attrname, + obj.dn) + diff_count += 1 return error_count @@ -1097,11 +1227,14 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base))) return (set_att, list_attid, wrong_attids) - def fix_metadata(self, dn, attr): + def fix_metadata(self, obj, attr): '''re-write replPropertyMetaData elements for a single attribute for a object. This is used to fix missing replPropertyMetaData elements''' + guid_str = str(ndr_unpack(misc.GUID, obj['objectGUID'][0])) + dn = ldb.Dn(self.samdb, "<GUID=%s>" % guid_str) res = self.samdb.search(base = dn, scope=ldb.SCOPE_BASE, attrs = [attr], - controls = ["search_options:1:2", "show_recycled:1"]) + controls = ["search_options:1:2", + "show_recycled:1"]) msg = res[0] nmsg = ldb.Message() nmsg.dn = dn @@ -1943,7 +2076,7 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base))) if not self.confirm_all("Fix missing replPropertyMetaData element '%s'" % att, 'fix_all_metadata'): self.report("Not fixing missing replPropertyMetaData element '%s'" % att) continue - self.fix_metadata(dn, att) + self.fix_metadata(obj, att) if self.is_fsmo_role(dn): if "fSMORoleOwner" not in obj and ("*" in attrs or "fsmoroleowner" in map(str.lower, attrs)): diff --git a/selftest/knownfail b/selftest/knownfail index 3c910fb..a28329c 100644 --- a/selftest/knownfail +++ b/selftest/knownfail @@ -328,7 +328,7 @@ ^samba3.smb2.credits.session_setup_credits_granted.* ^samba3.smb2.credits.single_req_credits_granted.* ^samba3.smb2.credits.skipped_mid.* -^samba4.blackbox.dbcheck-links.release-4-5-0-pre1.dangling_multi_valued_dbcheck +^samba4.blackbox.dbcheck-links.release-4-5-0-pre1.dbcheck_dangling_multi_valued_clean ^samba4.blackbox.dbcheck-links.release-4-5-0-pre1.dangling_multi_valued_check_missing # # rap password tests don't function in the ad_dc_ntvfs:local environment diff --git a/selftest/tests.py b/selftest/tests.py index 181313e..209800c 100644 --- a/selftest/tests.py +++ b/selftest/tests.py @@ -125,6 +125,11 @@ plantestsuite( ["PYTHON=%s" % python, os.path.join(bbdir, "dbcheck-links.sh"), '$PREFIX_ABS/provision', 'release-4-5-0-pre1', configuration]) +plantestsuite( + "samba4.blackbox.runtime-links.release-4-5-0-pre1", "none", + ["PYTHON=%s" % python, + os.path.join(bbdir, "runtime-links.sh"), + '$PREFIX_ABS/provision', 'release-4-5-0-pre1', configuration]) planpythontestsuite("none", "samba.tests.upgradeprovision") planpythontestsuite("none", "samba.tests.xattr", py3_compatible=True) planpythontestsuite("none", "samba.tests.ntacls") diff --git a/source4/dsdb/samdb/ldb_modules/extended_dn_store.c b/source4/dsdb/samdb/ldb_modules/extended_dn_store.c index 28ad9d0..a32ab8d 100644 --- a/source4/dsdb/samdb/ldb_modules/extended_dn_store.c +++ b/source4/dsdb/samdb/ldb_modules/extended_dn_store.c @@ -375,6 +375,7 @@ static int extended_dn_modify(struct ldb_module *module, struct ldb_request *req unsigned int i, j; struct extended_dn_context *ac; + struct ldb_control *fix_links_control = NULL; int ret; if (ldb_dn_is_special(req->op.mod.message->dn)) { @@ -393,6 +394,12 @@ static int extended_dn_modify(struct ldb_module *module, struct ldb_request *req return ldb_next_request(module, req); } + fix_links_control = ldb_request_get_control(req, + DSDB_CONTROL_DBCHECK_FIX_DUPLICATE_LINKS); + if (fix_links_control != NULL) { + return ldb_next_request(module, req); + } + for (i=0; i < req->op.mod.message->num_elements; i++) { const struct ldb_message_element *el = &req->op.mod.message->elements[i]; const struct dsdb_attribute *schema_attr diff --git a/source4/dsdb/samdb/ldb_modules/repl_meta_data.c b/source4/dsdb/samdb/ldb_modules/repl_meta_data.c index 9a24349..198ac84 100644 --- a/source4/dsdb/samdb/ldb_modules/repl_meta_data.c +++ b/source4/dsdb/samdb/ldb_modules/repl_meta_data.c @@ -3172,9 +3172,22 @@ static int replmd_modify_handle_linked_attribs(struct ldb_module *module, continue; } if ((schema_attr->linkID & 1) == 1) { - if (parent && ldb_request_get_control(parent, DSDB_CONTROL_DBCHECK)) { - continue; + if (parent) { + struct ldb_control *ctrl; + + ctrl = ldb_request_get_control(parent, + DSDB_CONTROL_REPLMD_VANISH_LINKS); + if (ctrl != NULL) { + ctrl->critical = false; + continue; + } + ctrl = ldb_request_get_control(parent, + DSDB_CONTROL_DBCHECK); + if (ctrl != NULL) { + continue; + } } + /* Odd is for the target. Illegal to modify */ ldb_asprintf_errstring(ldb, "attribute %s must not be modified directly, it is a linked attribute", el->name); @@ -3303,6 +3316,7 @@ static int replmd_modify(struct ldb_module *module, struct ldb_request *req) unsigned int functional_level; const struct ldb_message_element *guid_el = NULL; struct ldb_control *sd_propagation_control; + struct ldb_control *fix_links_control = NULL; struct replmd_private *replmd_private = talloc_get_type(ldb_module_get_private(module), struct replmd_private); @@ -3328,6 +3342,39 @@ static int replmd_modify(struct ldb_module *module, struct ldb_request *req) -- Samba Shared Repository