The branch, master has been updated via 74091115f75 tests/ldb_kv: Add another case for completeness via a76d2865372 ldb_kv: Avoid memdup of database records in the case of base searches via d6b1d556593 ldb_mdb: Change function declaration as per README.coding via 65303bf55dc ldb_kv: Remove unnecessary space via 59ee3c864ca selftest: split schemaupgrade testenv out via 5d8895f347c repl: test for schema object and LA repl across chunks via 350fc49e945 selftest: tagging tests for new schemaupgrade_dc target via 4336c058cab selftest: Add new 2-DC testenv for live schema upgrade via 661dc457411 dsdb/repl: we need to replicate the whole schema before we can apply it via b7c1752754d dsdb:samdb: schemainfo update with relax control via 7652439fa1a python/provision: use provision and relax controls for schema provision via 5ea84af2d69 s4:provision: split out provision_self_join_modify_schema.ldif via b5b572d5f71 ldapcmp: ignore 'schemaInfo' if two domains are compared via e34abefb777 samdb: test for schemainfo update with relax control via 140a6733a45 drsuapi.idl: add DRSUAPI_ATTID_schemaInfo from 49b77d8df2d ldb_kv: Skip @ records early in a search full scan
https://git.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit 74091115f7564976c8fc7530fc2cb76c6bac9ce1 Author: Garming Sam <garm...@catalyst.net.nz> Date: Tue Apr 9 13:22:01 2019 +1200 tests/ldb_kv: Add another case for completeness Signed-off-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> Autobuild-User(master): Andrew Bartlett <abart...@samba.org> Autobuild-Date(master): Thu Apr 11 05:25:02 UTC 2019 on sn-devel-144 commit a76d2865372988c29baef42ecc4257e861692e7b Author: Garming Sam <garm...@catalyst.net.nz> Date: Thu Mar 7 16:45:46 2019 +1300 ldb_kv: Avoid memdup of database records in the case of base searches This makes LDAP bind significantly faster in the case of having many members, due to large size of these records (with tens of thousands of member links). During the nested group calculation, you are only interested in memberOf not the member links. (We add a bit-field to determine whether or not the backend actually supports pointing into database memory. For some reason TDB pointers aren't stable, so for now we set this option just on LMDB backends.) Signed-off-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit d6b1d55659348e0347da702794de92b6f789b5cc Author: Garming Sam <garm...@catalyst.net.nz> Date: Thu Apr 11 12:53:45 2019 +1200 ldb_mdb: Change function declaration as per README.coding Signed-off-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 65303bf55dc40fb034b7fca80846b4f21c1149fd Author: Garming Sam <garm...@catalyst.net.nz> Date: Mon Apr 8 14:41:43 2019 +1200 ldb_kv: Remove unnecessary space Signed-off-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 59ee3c864ca360d49196ae4064fe19db01fe6396 Author: Aaron Haslett <aaronhasl...@catalyst.net.nz> Date: Tue Apr 2 10:45:36 2019 +1300 selftest: split schemaupgrade testenv out Schemaupgrade tests are particularly resource intensive and are causing runners to hit their memory and CPU limits, so we need to split them out. Signed-off-by: Aaron Haslett <aaronhasl...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 5d8895f347ca0005240ec166fec4eb875f9cd356 Author: Aaron Haslett <aaronhasl...@catalyst.net.nz> Date: Tue Feb 19 14:33:33 2019 +1300 repl: test for schema object and LA repl across chunks During replication, transmission of objects and linked attributes are split into chunks. These two tests check behavioural consistency across chunks for regular schema objects and linked attributes. Signed-off-by: Aaron Haslett <aaronhasl...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 350fc49e94525232c3556a3ace108f6e1447a490 Author: Aaron Haslett <aaronhasl...@catalyst.net.nz> Date: Tue Feb 12 13:31:57 2019 +1300 selftest: tagging tests for new schemaupgrade_dc target Tagging schema tests against schemaupgrade_dc test target and fixing some DN assertions to be more generic. Signed-off-by: Aaron Haslett <aaronhasl...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 4336c058cabd0a11f48ae76dff4e15e85013e783 Author: Tim Beale <timbe...@catalyst.net.nz> Date: Thu Jan 17 17:18:48 2019 +1300 selftest: Add new 2-DC testenv for live schema upgrade This adds a new 2-DC testenv that: 1. Provisions an AD DC with 2008R2 schema 2. Joins another AD DC with 2008R2 schema 3. Starts Samba 4. Performs a live schema upgrade on the PDC Testenv targetting in tests.py files for this testenv required that we extend the environment dependencies system to include optional post-startup dependencies specified in ENV_DEPS_POST maps. Signed-off-by: Tim Beale <timbe...@catalyst.net.nz> Signed-off-by: Aaron Haslett <aaronhasl...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 661dc4574110b0f13c127c8aa6ed0c385292b77c Author: Stefan Metzmacher <me...@samba.org> Date: Fri Feb 8 14:49:55 2019 +0000 dsdb/repl: we need to replicate the whole schema before we can apply it Otherwise we may not be able to construct a working schema that's required to apply the changes. BUG: https://bugzilla.samba.org/show_bug.cgi?id=12204 Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit b7c1752754da1e8a83a53670cf4a410ec6e9d7b7 Author: Aaron Haslett <aaronhasl...@catalyst.net.nz> Date: Wed Apr 3 16:34:42 2019 +1300 dsdb:samdb: schemainfo update with relax control Currently schema info's revision field isn't incremented if relax control is present. This is so that no increment is done during provision, but we need the relax control in other situations where the increment is desired, so we should use the provision control instead to disable schema info update. BUG: https://bugzilla.samba.org/show_bug.cgi?id=13799 Signed-off-by: Aaron Haslett <aaronhasl...@catalyst.net.nz> Reviewed-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 7652439fa1aab92945f5540a43fc49568d446917 Author: Stefan Metzmacher <me...@samba.org> Date: Fri Mar 8 11:28:42 2019 +0100 python/provision: use provision and relax controls for schema provision BUG: https://bugzilla.samba.org/show_bug.cgi?id=13799 Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 5ea84af2d69e0b3a2a801ea0cc3f4ffc66bf1764 Author: Stefan Metzmacher <me...@samba.org> Date: Fri Mar 8 11:27:14 2019 +0100 s4:provision: split out provision_self_join_modify_schema.ldif BUG: https://bugzilla.samba.org/show_bug.cgi?id=13799 Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit b5b572d5f71e2b9783ddb25c21ac32904fbfd661 Author: Stefan Metzmacher <me...@samba.org> Date: Thu Feb 21 09:20:48 2019 +0100 ldapcmp: ignore 'schemaInfo' if two domains are compared BUG: https://bugzilla.samba.org/show_bug.cgi?id=13799 Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit e34abefb77729330cd48bc039c82b03fe545f8a9 Author: Aaron Haslett <aaronhasl...@catalyst.net.nz> Date: Thu Apr 4 14:39:41 2019 +1300 samdb: test for schemainfo update with relax control Currently schema info's revision field isn't incremented if relax control is present. This is so that no increment is done during provision, but we need the relax control in other situations where the increment is desired. This patch adds a failing test to expose the problem. BUG: https://bugzilla.samba.org/show_bug.cgi?id=13799 Signed-off-by: Aaron Haslett <aaronhasl...@catalyst.net.nz> Reviewed-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 140a6733a458d0afa20237a09ef4ee2546a83a8f Author: Stefan Metzmacher <me...@samba.org> Date: Sat Feb 23 00:14:31 2019 +0100 drsuapi.idl: add DRSUAPI_ATTID_schemaInfo BUG: https://bugzilla.samba.org/show_bug.cgi?id=13799 Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> ----------------------------------------------------------------------- Summary of changes: .gitlab-ci.yml | 3 + lib/ldb/include/ldb_module.h | 1 + lib/ldb/ldb_key_value/ldb_kv.c | 2 +- lib/ldb/ldb_key_value/ldb_kv.h | 9 + lib/ldb/ldb_key_value/ldb_kv_index.c | 16 +- lib/ldb/ldb_key_value/ldb_kv_search.c | 58 ++-- lib/ldb/ldb_mdb/ldb_mdb.c | 7 +- lib/ldb/ldb_tdb/ldb_tdb.c | 39 +-- lib/ldb/tests/ldb_kv_ops_test.c | 1 + librpc/idl/drsuapi.idl | 1 + python/samba/netcmd/ldapcmp.py | 2 +- python/samba/provision/__init__.py | 25 +- python/samba/tests/samba_tool/schema.py | 10 +- script/autobuild.py | 14 + selftest/target/README | 7 + selftest/target/Samba.pm | 17 ++ selftest/target/Samba3.pm | 2 + selftest/target/Samba4.pm | 102 +++++++ source4/dsdb/repl/drepl_out_helpers.c | 174 +++++++++++- source4/dsdb/samdb/ldb_modules/samldb.c | 2 +- source4/dsdb/tests/python/dsdb_schema_info.py | 15 +- source4/selftest/tests.py | 30 +- .../setup/provision_self_join_modify_config.ldif | 5 - .../setup/provision_self_join_modify_schema.ldif | 4 + source4/setup/schema_samba4.ldif | 1 + source4/torture/drs/python/drs_base.py | 20 +- source4/torture/drs/python/getnc_schema.py | 308 +++++++++++++++++++++ 27 files changed, 785 insertions(+), 90 deletions(-) create mode 100644 source4/setup/provision_self_join_modify_schema.ldif create mode 100644 source4/torture/drs/python/getnc_schema.py Changeset truncated at 500 lines: diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 1e836c6f28d..fcd456f59af 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -91,6 +91,9 @@ samba-ad-dc-5: samba-ad-dc-6: extends: .shared_template +samba-schemaupgrade: + extends: .shared_template + samba-ad-dc-backup: extends: .shared_template diff --git a/lib/ldb/include/ldb_module.h b/lib/ldb/include/ldb_module.h index 6ba2a49300a..b45142abe5c 100644 --- a/lib/ldb/include/ldb_module.h +++ b/lib/ldb/include/ldb_module.h @@ -560,6 +560,7 @@ int ldb_unpack_data_only_attr_list_flags(struct ldb_context *ldb, #define LDB_UNPACK_DATA_FLAG_NO_DN 0x0002 #define LDB_UNPACK_DATA_FLAG_NO_VALUES_ALLOC 0x0004 #define LDB_UNPACK_DATA_FLAG_NO_ATTRS 0x0008 +#define LDB_UNPACK_DATA_FLAG_READ_LOCKED 0x0010 /** Forces a specific ldb handle to use the global event context. diff --git a/lib/ldb/ldb_key_value/ldb_kv.c b/lib/ldb/ldb_key_value/ldb_kv.c index 2fc3aa7d030..0a81524d87f 100644 --- a/lib/ldb/ldb_key_value/ldb_kv.c +++ b/lib/ldb/ldb_key_value/ldb_kv.c @@ -622,7 +622,7 @@ int ldb_kv_delete_noindex(struct ldb_module *module, return LDB_ERR_OTHER; } - ret = ldb_kv->kv_ops->delete (ldb_kv, key); + ret = ldb_kv->kv_ops->delete(ldb_kv, key); TALLOC_FREE(tdb_key_ctx); if (ret != 0) { diff --git a/lib/ldb/ldb_key_value/ldb_kv.h b/lib/ldb/ldb_key_value/ldb_kv.h index 92106caae85..778a8991bb8 100644 --- a/lib/ldb/ldb_key_value/ldb_kv.h +++ b/lib/ldb/ldb_key_value/ldb_kv.h @@ -13,6 +13,8 @@ typedef int (*ldb_kv_traverse_fn)(struct ldb_kv_private *ldb_kv, void *ctx); struct kv_db_ops { + uint32_t options; + int (*store)(struct ldb_kv_private *ldb_kv, struct ldb_val key, struct ldb_val data, @@ -175,6 +177,13 @@ struct ldb_kv_reindex_context { #define LDB_KV_GUID_SIZE 16 #define LDB_KV_GUID_KEY_SIZE (LDB_KV_GUID_SIZE + sizeof(LDB_KV_GUID_KEY_PREFIX) - 1) +/* LDB KV options */ +/* + * This allows pointers to be referenced after the callback to any variant of + * iterate or fetch_and_parse -- as long as an overall read lock is held. + */ +#define LDB_KV_OPTION_STABLE_READ_LOCK 0x00000001 + /* * The following definitions come from lib/ldb/ldb_key_value/ldb_kv_cache.c */ diff --git a/lib/ldb/ldb_key_value/ldb_kv_index.c b/lib/ldb/ldb_key_value/ldb_kv_index.c index 350289a78e3..9f0de7d260e 100644 --- a/lib/ldb/ldb_key_value/ldb_kv_index.c +++ b/lib/ldb/ldb_key_value/ldb_kv_index.c @@ -406,7 +406,13 @@ normal_index: dn, msg, LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC | - LDB_UNPACK_DATA_FLAG_NO_DN); + LDB_UNPACK_DATA_FLAG_NO_DN | + /* + * The entry point ldb_kv_search_indexed is + * only called from the read-locked + * ldb_kv_search. + */ + LDB_UNPACK_DATA_FLAG_READ_LOCKED); if (ret != LDB_SUCCESS) { talloc_free(msg); return ret; @@ -2222,7 +2228,13 @@ static int ldb_kv_index_filter(struct ldb_kv_private *ldb_kv, keys[i], msg, LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC | - LDB_UNPACK_DATA_FLAG_NO_VALUES_ALLOC); + LDB_UNPACK_DATA_FLAG_NO_VALUES_ALLOC | + /* + * The entry point ldb_kv_search_indexed is + * only called from the read-locked + * ldb_kv_search. + */ + LDB_UNPACK_DATA_FLAG_READ_LOCKED); if (ret == LDB_ERR_NO_SUCH_OBJECT) { /* * the record has disappeared? yes, this can diff --git a/lib/ldb/ldb_key_value/ldb_kv_search.c b/lib/ldb/ldb_key_value/ldb_kv_search.c index aa086a88d9a..a2946d6506b 100644 --- a/lib/ldb/ldb_key_value/ldb_kv_search.c +++ b/lib/ldb/ldb_key_value/ldb_kv_search.c @@ -190,22 +190,45 @@ static int ldb_kv_parse_data_unpack(struct ldb_val key, struct ldb_context *ldb = ldb_module_get_ctx(ctx->module); struct ldb_val data_parse = data; - if (ctx->unpack_flags & LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC) { - /* - * If we got LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC - * we need at least do a memdup on the whole - * data buffer as that may change later - * and the caller needs a stable result. - */ - data_parse.data = talloc_memdup(ctx->msg, - data.data, - data.length); - if (data_parse.data == NULL) { - ldb_debug(ldb, LDB_DEBUG_ERROR, - "Unable to allocate data(%d) for %*.*s\n", - (int)data.length, - (int)key.length, (int)key.length, key.data); - return LDB_ERR_OPERATIONS_ERROR; + struct ldb_kv_private *ldb_kv = + talloc_get_type(ldb_module_get_private(ctx->module), struct ldb_kv_private); + + if ((ctx->unpack_flags & LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC)) { + if ((ldb_kv->kv_ops->options & LDB_KV_OPTION_STABLE_READ_LOCK) && + (ctx->unpack_flags & LDB_UNPACK_DATA_FLAG_READ_LOCKED) && + !ldb_kv->kv_ops->transaction_active(ldb_kv)) { + /* + * In the case where no transactions are active and + * we're in a read-lock, we can point directly into + * database memory. + * + * The database can't be changed underneath us and we + * will duplicate this data in the call to filter. + * + * This is seen in: + * - ldb_kv_index_filter + * - ldb_kv_search_and_return_base + */ + } else { + /* + * In every other case, if we got + * LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC we need at least + * do a memdup on the whole data buffer as that may + * change later and the caller needs a stable result. + * + * During transactions, pointers could change and in + * TDB, there just aren't the same guarantees. + */ + data_parse.data = talloc_memdup(ctx->msg, + data.data, + data.length); + if (data_parse.data == NULL) { + ldb_debug(ldb, LDB_DEBUG_ERROR, + "Unable to allocate data(%d) for %*.*s\n", + (int)data.length, + (int)key.length, (int)key.length, key.data); + return LDB_ERR_OPERATIONS_ERROR; + } } } @@ -635,7 +658,8 @@ static int ldb_kv_search_and_return_base(struct ldb_kv_private *ldb_kv, ctx->base, msg, LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC | - LDB_UNPACK_DATA_FLAG_NO_VALUES_ALLOC); + LDB_UNPACK_DATA_FLAG_NO_VALUES_ALLOC | + LDB_UNPACK_DATA_FLAG_READ_LOCKED); if (ret == LDB_ERR_NO_SUCH_OBJECT) { if (ldb_kv->check_base == false) { diff --git a/lib/ldb/ldb_mdb/ldb_mdb.c b/lib/ldb/ldb_mdb/ldb_mdb.c index b57998c1e31..68ee97acb64 100644 --- a/lib/ldb/ldb_mdb/ldb_mdb.c +++ b/lib/ldb/ldb_mdb/ldb_mdb.c @@ -725,7 +725,8 @@ static bool lmdb_changed(struct ldb_kv_private *ldb_kv) * The mdb_env_stat call returns an accurate count, so we return the actual * number of records in the database rather than an estimate. */ -static size_t lmdb_get_size(struct ldb_kv_private *ldb_kv) { +static size_t lmdb_get_size(struct ldb_kv_private *ldb_kv) +{ struct MDB_stat stats = {0}; struct lmdb_private *lmdb = ldb_kv->lmdb_private; @@ -738,9 +739,9 @@ static size_t lmdb_get_size(struct ldb_kv_private *ldb_kv) { return stats.ms_entries; } - - static struct kv_db_ops lmdb_key_value_ops = { + .options = LDB_KV_OPTION_STABLE_READ_LOCK, + .store = lmdb_store, .delete = lmdb_delete, .iterate = lmdb_traverse_fn, diff --git a/lib/ldb/ldb_tdb/ldb_tdb.c b/lib/ldb/ldb_tdb/ldb_tdb.c index 3dcc158729a..ae0001e8084 100644 --- a/lib/ldb/ldb_tdb/ldb_tdb.c +++ b/lib/ldb/ldb_tdb/ldb_tdb.c @@ -434,24 +434,27 @@ static size_t ltdb_get_size(struct ldb_kv_private *ldb_kv) } static const struct kv_db_ops key_value_ops = { - .store = ltdb_store, - .delete = ltdb_delete, - .iterate = ltdb_traverse_fn, - .update_in_iterate = ltdb_update_in_iterate, - .fetch_and_parse = ltdb_parse_record, - .iterate_range = ltdb_iterate_range, - .lock_read = ltdb_lock_read, - .unlock_read = ltdb_unlock_read, - .begin_write = ltdb_transaction_start, - .prepare_write = ltdb_transaction_prepare_commit, - .finish_write = ltdb_transaction_commit, - .abort_write = ltdb_transaction_cancel, - .error = ltdb_error, - .errorstr = ltdb_errorstr, - .name = ltdb_name, - .has_changed = ltdb_changed, - .transaction_active = ltdb_transaction_active, - .get_size = ltdb_get_size, + /* No support for any additional features */ + .options = 0, + + .store = ltdb_store, + .delete = ltdb_delete, + .iterate = ltdb_traverse_fn, + .update_in_iterate = ltdb_update_in_iterate, + .fetch_and_parse = ltdb_parse_record, + .iterate_range = ltdb_iterate_range, + .lock_read = ltdb_lock_read, + .unlock_read = ltdb_unlock_read, + .begin_write = ltdb_transaction_start, + .prepare_write = ltdb_transaction_prepare_commit, + .finish_write = ltdb_transaction_commit, + .abort_write = ltdb_transaction_cancel, + .error = ltdb_error, + .errorstr = ltdb_errorstr, + .name = ltdb_name, + .has_changed = ltdb_changed, + .transaction_active = ltdb_transaction_active, + .get_size = ltdb_get_size, }; /* diff --git a/lib/ldb/tests/ldb_kv_ops_test.c b/lib/ldb/tests/ldb_kv_ops_test.c index c2538fc1e18..57a4711706c 100644 --- a/lib/ldb/tests/ldb_kv_ops_test.c +++ b/lib/ldb/tests/ldb_kv_ops_test.c @@ -908,6 +908,7 @@ static void test_iterate_range(void **state) */ do_iterate_range_test(state, 0, 20, false); do_iterate_range_test(state, 1010, 1030, false); + do_iterate_range_test(state, 0, 1030, false); } struct update_context { diff --git a/librpc/idl/drsuapi.idl b/librpc/idl/drsuapi.idl index cd90500faf5..448a58bcd1f 100644 --- a/librpc/idl/drsuapi.idl +++ b/librpc/idl/drsuapi.idl @@ -548,6 +548,7 @@ interface drsuapi DRSUAPI_ATTID_objectCategory = 0x0009030e, DRSUAPI_ATTID_gPLink = 0x0009037b, DRSUAPI_ATTID_transportAddressAttribute = 0x0009037f, + DRSUAPI_ATTID_schemaInfo = 0x0009054e, DRSUAPI_ATTID_msDS_Behavior_Version = 0x000905b3, DRSUAPI_ATTID_msDS_KeyVersionNumber = 0x000906f6, DRSUAPI_ATTID_msDS_NonMembers = 0x00090701, diff --git a/python/samba/netcmd/ldapcmp.py b/python/samba/netcmd/ldapcmp.py index 17c62928a55..6051b55b31a 100644 --- a/python/samba/netcmd/ldapcmp.py +++ b/python/samba/netcmd/ldapcmp.py @@ -460,7 +460,7 @@ class LDAPObject(object): "msDs-masteredBy", "lastSetTime", "ipsecNegotiationPolicyReference", "subRefs", "gPCFileSysPath", "accountExpires", "invocationId", "operatingSystemVersion", - "oEMInformation", + "oEMInformation", "schemaInfo", # After Exchange preps "targetAddress", "msExchMailboxGuid", "siteFolderGUID"] # diff --git a/python/samba/provision/__init__.py b/python/samba/provision/__init__.py index 0a3a7b89cb7..2bb2614e629 100644 --- a/python/samba/provision/__init__.py +++ b/python/samba/provision/__init__.py @@ -1193,11 +1193,16 @@ def setup_self_join(samdb, admin_session_info, names, fill, machinepass, "DOMAIN_CONTROLLER_FUNCTIONALITY": str( domainControllerFunctionality)}) - # Setup fSMORoleOwner entries to point at the newly created DC entry + # Setup fSMORoleOwner entries to point at the newly created DC entry + setup_modify_ldif(samdb, + setup_path("provision_self_join_modify_schema.ldif"), { + "SCHEMADN": names.schemadn, + "SERVERDN": names.serverdn, + }, + controls=["provision:0", "relax:0"]) setup_modify_ldif(samdb, setup_path("provision_self_join_modify_config.ldif"), { "CONFIGDN": names.configdn, - "SCHEMADN": names.schemadn, "DEFAULTSITE": names.sitename, "NETBIOSNAME": names.netbiosname, "SERVERDN": names.serverdn, @@ -1412,16 +1417,20 @@ def fill_samdb(samdb, lp, names, logger, policyguid, # The LDIF here was created when the Schema object was constructed ignore_checks_oid = "local_oid:%s:0" % samba.dsdb.DSDB_CONTROL_SKIP_DUPLICATES_CHECK_OID + schema_controls = [ + "provision:0", + "relax:0", + ignore_checks_oid + ] + logger.info("Setting up sam.ldb schema") - samdb.add_ldif(schema.schema_dn_add, - controls=["relax:0", ignore_checks_oid]) - samdb.modify_ldif(schema.schema_dn_modify, - controls=[ignore_checks_oid]) + samdb.add_ldif(schema.schema_dn_add, controls=schema_controls) + samdb.modify_ldif(schema.schema_dn_modify, controls=schema_controls) samdb.write_prefixes_from_schema() - samdb.add_ldif(schema.schema_data, controls=["relax:0", ignore_checks_oid]) + samdb.add_ldif(schema.schema_data, controls=schema_controls) setup_add_ldif(samdb, setup_path("aggregate_schema.ldif"), {"SCHEMADN": names.schemadn}, - controls=["relax:0", ignore_checks_oid]) + controls=schema_controls) # Now register this container in the root of the forest msg = ldb.Message(ldb.Dn(samdb, names.domaindn)) diff --git a/python/samba/tests/samba_tool/schema.py b/python/samba/tests/samba_tool/schema.py index 6d502ef4e17..7a80b33cd26 100644 --- a/python/samba/tests/samba_tool/schema.py +++ b/python/samba/tests/samba_tool/schema.py @@ -42,7 +42,7 @@ class SchemaCmdTestCase(SambaToolCmdTest): self.assertCmdSuccess(result, out, err) self.assertEquals(err, "", "Shouldn't be any error messages") - self.assertIn("dn: CN=uid,CN=Schema,CN=Configuration,DC=samba,DC=example,DC=com", out) + self.assertIn("dn: CN=uid,CN=Schema,CN=Configuration,", out) def test_modify_attribute_searchflags(self): """Tests that we can modify searchFlags of an attribute""" @@ -62,7 +62,7 @@ class SchemaCmdTestCase(SambaToolCmdTest): self.assertCmdSuccess(result, out, err) self.assertEquals(err, "", "Shouldn't be any error messages") - self.assertIn("modified cn=uid,CN=Schema,CN=Configuration,DC=samba,DC=example,DC=com", out) + self.assertIn("modified cn=uid,CN=Schema,CN=Configuration,", out) (result, out, err) = self.runsublevelcmd("schema", ("attribute", "modify"), "uid", @@ -73,7 +73,7 @@ class SchemaCmdTestCase(SambaToolCmdTest): self.assertCmdSuccess(result, out, err) self.assertEquals(err, "", "Shouldn't be any error messages") - self.assertIn("modified cn=uid,CN=Schema,CN=Configuration,DC=samba,DC=example,DC=com", out) + self.assertIn("modified cn=uid,CN=Schema,CN=Configuration,", out) (result, out, err) = self.runsublevelcmd("schema", ("attribute", "modify"), "uid", @@ -84,7 +84,7 @@ class SchemaCmdTestCase(SambaToolCmdTest): self.assertCmdSuccess(result, out, err) self.assertEquals(err, "", "Shouldn't be any error messages") - self.assertIn("modified cn=uid,CN=Schema,CN=Configuration,DC=samba,DC=example,DC=com", out) + self.assertIn("modified cn=uid,CN=Schema,CN=Configuration,", out) def test_show_oc_attribute(self): """Tests that we can modify searchFlags of an attribute""" @@ -109,4 +109,4 @@ class SchemaCmdTestCase(SambaToolCmdTest): self.assertCmdSuccess(result, out, err) self.assertEquals(err, "", "Shouldn't be any error messages") - self.assertIn("dn: CN=Person,CN=Schema,CN=Configuration,DC=samba,DC=example,DC=com", out) + self.assertIn("dn: CN=Person,CN=Schema,CN=Configuration,", out) diff --git a/script/autobuild.py b/script/autobuild.py index 0f95cc50bc8..13ea568237e 100755 --- a/script/autobuild.py +++ b/script/autobuild.py @@ -56,6 +56,7 @@ builddirs = { "samba-systemkrb5": ".", "samba-nopython": ".", "samba-nopython-py2": ".", + "samba-schemaupgrade": ".", "ldb": "lib/ldb", "tdb": "lib/tdb", "talloc": "lib/talloc", @@ -131,6 +132,8 @@ tasks = { "--exclude-env=rodc " "--exclude-env=ad_dc_default " "--exclude-env=ad_dc_slowtests " + "--exclude-env=schemaupgrade_pair_dc " + "--exclude-env=schemaupgrade_dc " "'", "text/plain"), ("install", "make install", "text/plain"), @@ -236,6 +239,17 @@ tasks = { "'", "text/plain"), ("check-clean-tree", "script/clean-source-tree.sh", "text/plain")], + + "samba-schemaupgrade": [("random-sleep", "script/random-sleep.sh 1 1", "text/plain"), + ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"), + ("make", "make -j", "text/plain"), + ("test", "make test FAIL_IMMEDIATELY=1 " + "TESTS='" + "--include-env=schemaupgrade_dc " + "--include-env=schemaupgrade_pair_dc " + "'", "text/plain"), + ("check-clean-tree", "script/clean-source-tree.sh", "text/plain")], + # We split out the ad_dc_ntvfs tests (which are long) so other test do not wait # This is currently the longest task, so we don't randomly delay it. "samba-ad-dc-ntvfs": [ diff --git a/selftest/target/README b/selftest/target/README index 69d70217bdf..acdbb421d27 100644 --- a/selftest/target/README +++ b/selftest/target/README @@ -128,3 +128,10 @@ Used to test process limits on the standard model. It sets the number of allowed processes artificially low, to test that new connections are refused correctly. Due to the limited number of connections accepted, it's not recommended to use this testenv for anything else. + +schemaupgrade_dc +---------------- +This is a 2-DC testenv setup (schemaupgrade_dc and schemaupgrade_pair_dc). +We provision the first DC, and join the second, using an older version of the +schema (2008R2), then start-up Samba. Then, we run a schema upgrade (i.e. +'samba-tool domain schemaupgrade') on the PDC. diff --git a/selftest/target/Samba.pm b/selftest/target/Samba.pm index 6f28a017b17..38b38669dac 100644 --- a/selftest/target/Samba.pm +++ b/selftest/target/Samba.pm @@ -25,6 +25,9 @@ sub new($$$$$) { %Samba::ENV_DEPS = (%Samba3::ENV_DEPS, %Samba4::ENV_DEPS); our %ENV_DEPS; +%Samba::ENV_DEPS_POST = (%Samba3::ENV_DEPS_POST, %Samba4::ENV_DEPS_POST); +our %ENV_DEPS_POST; + %Samba::ENV_TARGETS = ( (map { $_ => "Samba3" } keys %Samba3::ENV_DEPS), (map { $_ => "Samba4" } keys %Samba4::ENV_DEPS), @@ -59,6 +62,8 @@ sub setup_env($$$) return $target->{vars}->{$envname}; } + $target->{vars}->{$envname} = ""; + my @dep_vars; foreach(@{$ENV_DEPS{$envname}}) { my $vars = $self->setup_env($_, $path); @@ -87,6 +92,13 @@ sub setup_env($$$) $target->{vars}->{$envname} = $env; $target->{vars}->{$envname}->{target} = $target; + foreach(@{$ENV_DEPS_POST{$envname}}) { + my $vars = $self->setup_env($_, $path); + if (not defined($vars)) { + return undef; + } + } + return $env; } @@ -401,6 +413,7 @@ sub realm_to_ip_mappings 'backupdom.samba.example.com' => 'backupfromdc', 'renamedom.samba.example.com' => 'renamedc', 'labdom.samba.example.com' => 'labdc', + 'schema.samba.example.com' => 'liveupgrade1dc', 'samba.example.com' => 'localdc', ); @@ -473,9 +486,13 @@ sub get_interface($) customdc => 45, prockilldc => 46, proclimitdc => 47, + liveupgrade1dc => 48, + liveupgrade2dc => 49, rootdnsforwarder => 64, + # Note: that you also need to update dns_hub.py when adding a new + # multi-DC testenv # update lib/socket_wrapper/socket_wrapper.c # #define MAX_WRAPPED_INTERFACES 64 # if you wish to have more than 64 interfaces diff --git a/selftest/target/Samba3.pm b/selftest/target/Samba3.pm index 647a3b50dfc..b8a734adbe8 100755 -- Samba Shared Repository