The branch, master has been updated via 9aa03be9464 torture: Test g_lock deadlock detection via d734547488c lib: Change the g_lock data model via effad89f622 lib: Add g_lock_ctx_init_backend() via 859148479b9 torture: g_lock will allow duplicate READ locks soon via 97ba1ee2f94 lib: Make struct g_lock_rec private to g_lock.c via c9b34cd2057 g_lock: Change prototype of g_lock_dump via 06b0bcdf8d0 smbd: Remove share_mode_data->record via 0a66c973545 smbd: Pass the db_record down to share_mode_data_store() via 24f56aefb48 smbd: Use share_mode_watch_send() via 2379dc29bc7 smbd: Add share_mode_watch_send/recv via 341223a0059 dbwrap_watch: Don't store in-RAM caches via e5c28fec6f9 dbwrap_watch: Simplify struct dbwrap_watched_watch_state via a793cc2276e dbwrap_watch: Simplify the wakeup messages via 7349612c59a dbwrap_watch: Add a uin64_t instance to watchers via 053405719a5 dbwrap_watch: Encapsulate watchers in "struct dbwrap_watcher" via 7cea5fec29a dbwrap_watch: Prevent two watchers per fetch_locked sequence via 79dcabfbabb torture3: Test the next patch: No two waiters in one do_locked() via e341911ddad torture3: Consolidate dbwrap_watch test initialization via ca8c23bb42e dbwrap: Don't set rec->value in dbwrap_do_locked() via ffe4933c776 locking: Remove a call to dbwrap_record_get_value() via 1dc3c168245 dbwrap: Protect against invalid db_record->value via 5291208836e vfs: Use dbwrap_do_locked() in vfs_offload_token_db_store_fsp() via dbf24bcb1ba vfs: Use dbwrap_parse_record() in vfs_offload_token_db_fetch_fsp() via 78c96f6995a g_lock: Pass "data" down to g_lock_trylock() via caef82b1fe6 dbwrap: Pass "value" to dbwrap_do_locked() callback via a58633f64ea dbwrap_tdb: Return correct error from db_tdb_storev() via bc2fc51abef dbwrap_tdb: Avoid includes.h from e7f0e858e00 s3:printing: Use httpConnect2 from CUPS
https://git.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit 9aa03be946467698ac7509251f366e165672feaa Author: Volker Lendecke <v...@samba.org> Date: Wed Nov 20 16:03:37 2019 +0100 torture: Test g_lock deadlock detection Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> Autobuild-User(master): Jeremy Allison <j...@samba.org> Autobuild-Date(master): Sat Nov 23 01:25:12 UTC 2019 on sn-devel-184 commit d734547488c6fab8d136239a95db640df57dbb97 Author: Volker Lendecke <v...@samba.org> Date: Tue Nov 19 17:29:18 2019 +0100 lib: Change the g_lock data model Now we have one fixed field for the exclusive lock holder and an array of shared locks. This way we now prioritize writers over readers: If a pending write comes in while readers are active, it will put itself into the exclusive slot. Then it waits for the readers to vanish. Only when all readers are gone the exclusive lock request is granted. New readers will just look at the exclusive slot and see it's taken. They will then line up as watchers, retrying whenever things change. This also means that it will be cheaper to support many shared locks: Granting a shared lock just means to extend the array. We don't have to walk the array for possible conflicts. This also adds explicit UPGRADE and DOWNGRADE operations for better error checking. Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit effad89f6224fbaeed5ef67016bd23e645763bc9 Author: Volker Lendecke <v...@samba.org> Date: Tue Nov 5 16:36:44 2019 +0100 lib: Add g_lock_ctx_init_backend() This will allow using the g_lock.c logic on other databases as well Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 859148479b98678ee4bfa8084a6dc56bcc10a4c5 Author: Volker Lendecke <v...@samba.org> Date: Mon Oct 28 13:50:09 2019 +0100 torture: g_lock will allow duplicate READ locks soon In the future, g_lock will not check for duplicate READ locks anymore, as there might be many of them and the check might be expensive Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 97ba1ee2f94e5529839a1c0852f50544dbbb83cf Author: Volker Lendecke <v...@samba.org> Date: Tue Nov 19 15:31:57 2019 +0100 lib: Make struct g_lock_rec private to g_lock.c This is no longer exposed in the API Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit c9b34cd2057c05e9965464ca5e9e28b423cb418b Author: Volker Lendecke <v...@samba.org> Date: Fri Oct 25 13:35:39 2019 +0200 g_lock: Change prototype of g_lock_dump Soon the g_lock database format will change. There will be one exclusive entry and an array of shared entries. In that format, there's no need to attach a lock_type to each entry in the g_lock database. Reflect this change in the g_lock_dump API Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 06b0bcdf8d0abba2f0625ff458adb56bf471b9e0 Author: Volker Lendecke <v...@samba.org> Date: Mon Nov 4 13:13:02 2019 +0100 smbd: Remove share_mode_data->record In the future, there won't be a record anymore Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 0a66c9735456dbcee4b59e2bf8cd91402884b894 Author: Volker Lendecke <v...@samba.org> Date: Mon Nov 4 13:10:54 2019 +0100 smbd: Pass the db_record down to share_mode_data_store() Remove a dependency on share_mode_data->record Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 24f56aefb487a6b269baf55b6840670666fbf0f3 Author: Volker Lendecke <v...@samba.org> Date: Mon Nov 4 13:06:20 2019 +0100 smbd: Use share_mode_watch_send() This removes some dependencies on share_mode_data->record Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 2379dc29bc73bdd1598a4ddaf47240cea5fd0697 Author: Volker Lendecke <v...@samba.org> Date: Mon Nov 4 12:57:35 2019 +0100 smbd: Add share_mode_watch_send/recv For now this is a simple wrapper around dbwrap_watched_watch_send() that will make the direct db_record reference in struct share_mode_data unnecessary. Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 341223a00596ef1cdc63d9eb0a0b2e496bd6f0f4 Author: Volker Lendecke <v...@samba.org> Date: Mon Nov 18 21:46:55 2019 +0100 dbwrap_watch: Don't store in-RAM caches The history of this file is a mess with lots of bugs. Most of the bugs I believe are based on the cache of database contents we maintain in struct dbwrap_watch_rec. This patch removes that cache and does all modifications directly in the backend database. This means we have to mess with the database format in a few more places, but I think the format is simple enough that this does not really hurt. I tried for a few days to split this up into small pieces that are easier to understand, but every time I separated out individual chunks I found difficult to track down bugs that are all resolved in the final code presented here. It's more lines of code, but I hope it's more robust. Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit e5c28fec6f900dd1d8f1ee81d2cd008a93207a1a Author: Volker Lendecke <v...@samba.org> Date: Mon Nov 18 13:43:13 2019 +0100 dbwrap_watch: Simplify struct dbwrap_watched_watch_state With the wakeup messages changed, we don't need to store what used to be the old message in the struct dbwrap_watched_watch_state anymore. We still need to store the key that is watched for removal of our watch entry in the request destructor Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit a793cc2276ef2123bfec1bc1828224372fcd3d93 Author: Volker Lendecke <v...@samba.org> Date: Mon Nov 18 13:36:58 2019 +0100 dbwrap_watch: Simplify the wakeup messages The instance ID per process globally identifies a watch instance across databases. It's not necessary to send the database ID and the watched key across, the uin64_t instance ID is sufficient. Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 7349612c59a518a2858a4a7bb268fa1ad14016d3 Author: Volker Lendecke <v...@samba.org> Date: Mon Nov 18 13:23:52 2019 +0100 dbwrap_watch: Add a uin64_t instance to watchers Initially used for debugging purposes only Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 053405719a5bd4121a2ca1f756dc7859ebb91faf Author: Volker Lendecke <v...@samba.org> Date: Mon Nov 18 12:37:21 2019 +0100 dbwrap_watch: Encapsulate watchers in "struct dbwrap_watcher" Next patch will extend this structure Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 7cea5fec29a0d425263a7ecb79d89a4f13de3f7e Author: Volker Lendecke <v...@samba.org> Date: Sun Nov 17 16:45:45 2019 +0100 dbwrap_watch: Prevent two watchers per fetch_locked sequence This can lead to very confusing bugs, and the code right now does not deal with it well. Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 79dcabfbabb9503eff9efe7cd32b1cb431d3ee69 Author: Volker Lendecke <v...@samba.org> Date: Sat Nov 16 13:20:14 2019 +0100 torture3: Test the next patch: No two waiters in one do_locked() Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit e341911ddad078c3c0c95f7e0ec16701510cb273 Author: Volker Lendecke <v...@samba.org> Date: Sat Nov 16 12:41:13 2019 +0100 torture3: Consolidate dbwrap_watch test initialization More lines, but less error-prone copy&paste Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit ca8c23bb42e501d5fe7f47f6c59feb5e7524a0cf Author: Volker Lendecke <v...@samba.org> Date: Wed Oct 23 12:19:38 2019 +0200 dbwrap: Don't set rec->value in dbwrap_do_locked() We pass that via the callback now Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit ffe4933c776bc89ab8a6bb09f74288b8bf7b1310 Author: Volker Lendecke <v...@samba.org> Date: Sun Nov 17 14:50:51 2019 +0100 locking: Remove a call to dbwrap_record_get_value() This makes a copy of the record values TDB_DATA. In this case, it is okay: We only do a dbwrap_record_storev modifying the value that we retrieved from the share_mode_lock destructor. This patch is necessary because the next commit will make the "value" argument to the dbwrap_do_locked() callback the only source of the record value. Thus for a nested share_mode_do_locked() call we have to maintain the record value explicitly. Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 1dc3c168245366d219fc8c3b116f4f5904a9cb07 Author: Volker Lendecke <v...@samba.org> Date: Thu Oct 24 16:41:47 2019 +0200 dbwrap: Protect against invalid db_record->value After dbwrap_record_storev()/delete(), dbwrap_record_get_value() information is stale. Assert on the attempt to re-fetch data after it became stale. This can't protect against someone copying the result from dbwrap_record_get_value() somewhere else, but it's better than nothing. Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 5291208836ea85b1b02f5b31b577fcc9050fa8b9 Author: Volker Lendecke <v...@samba.org> Date: Wed Oct 9 16:02:32 2019 +0200 vfs: Use dbwrap_do_locked() in vfs_offload_token_db_store_fsp() Less malloc Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit dbf24bcb1baa588295313e4d37ce84ee6db80168 Author: Volker Lendecke <v...@samba.org> Date: Wed Oct 9 15:13:05 2019 +0200 vfs: Use dbwrap_parse_record() in vfs_offload_token_db_fetch_fsp() Less malloc Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 78c96f6995aa0cb8e0ec0872bdc00bb452bc2d36 Author: Volker Lendecke <v...@samba.org> Date: Wed Oct 23 11:21:16 2019 +0200 g_lock: Pass "data" down to g_lock_trylock() Avoid a call to dbwrap_record_get_value(), dbwrap_do_locked() already gave us the value. Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit caef82b1fe6ae7732abcd04a3696756274c935ea Author: Volker Lendecke <v...@samba.org> Date: Wed Oct 23 11:34:47 2019 +0200 dbwrap: Pass "value" to dbwrap_do_locked() callback I want to reduce dbwrap_record_get_value(). It makes the caller believe it can make a copy of the TDB_DATA returned and that the value remains constant. It's not, as you can always do a dbwrap_record_store(). This patch removes one requirement for getting the value out of a db_record via dbwrap_record_get_value(). You can still make a copy, but from an API perspective to me it's more obvious that "value" as a parameter to the callback has a limited lifetime. Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit a58633f64ea4e43dd3f860b4176ebae080a0a4f1 Author: Volker Lendecke <v...@samba.org> Date: Wed Oct 9 17:17:07 2019 +0200 dbwrap_tdb: Return correct error from db_tdb_storev() Don't lose information to NT_STATUS_UNSUCCESSFUL Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit bc2fc51abef823090f178de7e39890bdd91f9067 Author: Volker Lendecke <v...@samba.org> Date: Wed Oct 16 15:29:28 2019 +0200 dbwrap_tdb: Avoid includes.h Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> ----------------------------------------------------------------------- Summary of changes: lib/dbwrap/dbwrap.c | 32 +- lib/dbwrap/dbwrap.h | 1 + lib/dbwrap/dbwrap_private.h | 2 + lib/dbwrap/dbwrap_rbt.c | 2 + lib/dbwrap/dbwrap_tdb.c | 25 +- source3/include/g_lock.h | 23 +- source3/lib/dbwrap/dbwrap_ctdb.c | 2 + source3/lib/dbwrap/dbwrap_watch.c | 805 +++++++++++++++++--------------- source3/lib/g_lock.c | 705 ++++++++++++++++++---------- source3/librpc/idl/open_files.idl | 1 - source3/locking/leases_db.c | 6 +- source3/locking/posix.c | 11 +- source3/locking/proto.h | 8 + source3/locking/share_mode_lock.c | 137 +++++- source3/modules/offload_token.c | 167 ++++--- source3/selftest/tests.py | 2 + source3/smbd/blocking.c | 8 +- source3/smbd/open.c | 19 +- source3/smbd/server.c | 7 +- source3/smbd/smb2_lock.c | 6 +- source3/smbd/smb2_setinfo.c | 6 +- source3/torture/proto.h | 2 + source3/torture/test_dbwrap_do_locked.c | 10 +- source3/torture/test_dbwrap_watch.c | 296 +++++++++--- source3/torture/test_g_lock.c | 250 ++++++++-- source3/torture/torture.c | 8 + source3/utils/net_g_lock.c | 29 +- 27 files changed, 1699 insertions(+), 871 deletions(-) Changeset truncated at 500 lines: diff --git a/lib/dbwrap/dbwrap.c b/lib/dbwrap/dbwrap.c index 3d8d2ae78d7..baab0338088 100644 --- a/lib/dbwrap/dbwrap.c +++ b/lib/dbwrap/dbwrap.c @@ -79,6 +79,7 @@ TDB_DATA dbwrap_record_get_key(const struct db_record *rec) TDB_DATA dbwrap_record_get_value(const struct db_record *rec) { + SMB_ASSERT(rec->value_valid); return rec->value; } @@ -87,6 +88,12 @@ NTSTATUS dbwrap_record_storev(struct db_record *rec, { NTSTATUS status; + /* + * Invalidate before rec->storev() is called, give + * rec->storev() the chance to re-validate rec->value. + */ + rec->value_valid = false; + status = rec->storev(rec, dbufs, num_dbufs, flags); if (!NT_STATUS_IS_OK(status)) { return status; @@ -103,6 +110,12 @@ NTSTATUS dbwrap_record_delete(struct db_record *rec) { NTSTATUS status; + /* + * Invalidate before rec->delete_rec() is called, give + * rec->delete_rec() the chance to re-validate rec->value. + */ + rec->value_valid = false; + status = rec->delete_rec(rec); if (!NT_STATUS_IS_OK(status)) { return status; @@ -307,7 +320,10 @@ struct dbwrap_store_state { NTSTATUS status; }; -static void dbwrap_store_fn(struct db_record *rec, void *private_data) +static void dbwrap_store_fn( + struct db_record *rec, + TDB_DATA value, + void *private_data) { struct dbwrap_store_state *state = private_data; state->status = dbwrap_record_store(rec, state->data, state->flags); @@ -331,7 +347,10 @@ struct dbwrap_delete_state { NTSTATUS status; }; -static void dbwrap_delete_fn(struct db_record *rec, void *private_data) +static void dbwrap_delete_fn( + struct db_record *rec, + TDB_DATA value, + void *private_data) { struct dbwrap_delete_state *state = private_data; state->status = dbwrap_record_delete(rec); @@ -514,6 +533,7 @@ NTSTATUS dbwrap_parse_record_recv(struct tevent_req *req) NTSTATUS dbwrap_do_locked(struct db_context *db, TDB_DATA key, void (*fn)(struct db_record *rec, + TDB_DATA value, void *private_data), void *private_data) { @@ -542,7 +562,13 @@ NTSTATUS dbwrap_do_locked(struct db_context *db, TDB_DATA key, return NT_STATUS_NO_MEMORY; } - fn(rec, private_data); + /* + * Invalidate rec->value, nobody shall assume it's set from + * within dbwrap_do_locked(). + */ + rec->value_valid = false; + + fn(rec, rec->value, private_data); TALLOC_FREE(rec); diff --git a/lib/dbwrap/dbwrap.h b/lib/dbwrap/dbwrap.h index 591eac5c63a..b5a7fb315c7 100644 --- a/lib/dbwrap/dbwrap.h +++ b/lib/dbwrap/dbwrap.h @@ -81,6 +81,7 @@ struct db_context *dbwrap_record_get_db(struct db_record *rec); NTSTATUS dbwrap_do_locked(struct db_context *db, TDB_DATA key, void (*fn)(struct db_record *rec, + TDB_DATA value, void *private_data), void *private_data); diff --git a/lib/dbwrap/dbwrap_private.h b/lib/dbwrap/dbwrap_private.h index b548168b9a9..8a1f03c7bec 100644 --- a/lib/dbwrap/dbwrap_private.h +++ b/lib/dbwrap/dbwrap_private.h @@ -29,6 +29,7 @@ struct tevent_req; struct db_record { struct db_context *db; TDB_DATA key, value; + bool value_valid; NTSTATUS (*storev)(struct db_record *rec, const TDB_DATA *dbufs, int num_dbufs, int flag); NTSTATUS (*delete_rec)(struct db_record *rec); @@ -70,6 +71,7 @@ struct db_context { NTSTATUS (*parse_record_recv)(struct tevent_req *req); NTSTATUS (*do_locked)(struct db_context *db, TDB_DATA key, void (*fn)(struct db_record *rec, + TDB_DATA value, void *private_data), void *private_data); int (*exists)(struct db_context *db,TDB_DATA key); diff --git a/lib/dbwrap/dbwrap_rbt.c b/lib/dbwrap/dbwrap_rbt.c index 145cfccf082..d988ca776de 100644 --- a/lib/dbwrap/dbwrap_rbt.c +++ b/lib/dbwrap/dbwrap_rbt.c @@ -375,6 +375,7 @@ static struct db_record *db_rbt_fetch_locked(struct db_context *db_ctx, rec_priv->node = res.node; result->value = res.val; + result->value_valid = true; if (found) { result->key = res.key; @@ -447,6 +448,7 @@ static int db_rbt_traverse_internal(struct db_context *db, rec.storev = db_rbt_storev; rec.delete_rec = db_rbt_delete; db_rbt_parse_node(rec_priv.node, &rec.key, &rec.value); + rec.value_valid = true; if (rw) { ctx->traverse_nextp = &next; diff --git a/lib/dbwrap/dbwrap_tdb.c b/lib/dbwrap/dbwrap_tdb.c index 1ac7dc9ed7a..74b895a7736 100644 --- a/lib/dbwrap/dbwrap_tdb.c +++ b/lib/dbwrap/dbwrap_tdb.c @@ -17,12 +17,14 @@ along with this program. If not, see <http://www.gnu.org/licenses/>. */ -#include "includes.h" +#include "replace.h" #include "dbwrap/dbwrap.h" #include "dbwrap/dbwrap_private.h" #include "dbwrap/dbwrap_tdb.h" #include "lib/tdb_wrap/tdb_wrap.h" #include "lib/util/util_tdb.h" +#include "lib/util/debug.h" +#include "lib/util/samba_util.h" #include "system/filesys.h" #include "lib/param/param.h" #include "libcli/util/error.h" @@ -106,6 +108,7 @@ static int db_tdb_fetchlock_parse(TDB_DATA key, TDB_DATA data, else { result->value.dptr = NULL; } + result->value_valid = true; return 0; } @@ -182,6 +185,7 @@ static struct db_record *db_tdb_try_fetch_locked( static NTSTATUS db_tdb_do_locked(struct db_context *db, TDB_DATA key, void (*fn)(struct db_record *rec, + TDB_DATA value, void *private_data), void *private_data) { @@ -210,13 +214,14 @@ static NTSTATUS db_tdb_do_locked(struct db_context *db, TDB_DATA key, rec = (struct db_record) { .db = db, .key = key, - .value = (struct TDB_DATA) { .dptr = buf, - .dsize = talloc_get_size(buf) }, + .value_valid = false, .storev = db_tdb_storev, .delete_rec = db_tdb_delete, .private_data = ctx }; - fn(&rec, private_data); + fn(&rec, + (TDB_DATA) { .dptr = buf, .dsize = talloc_get_size(buf) }, + private_data); tdb_chainunlock(ctx->wtdb->tdb, key); @@ -292,6 +297,8 @@ static NTSTATUS db_tdb_storev(struct db_record *rec, { struct db_tdb_ctx *ctx = talloc_get_type_abort(rec->private_data, struct db_tdb_ctx); + struct tdb_context *tdb = ctx->wtdb->tdb; + NTSTATUS status = NT_STATUS_OK; int ret; /* @@ -300,8 +307,12 @@ static NTSTATUS db_tdb_storev(struct db_record *rec, * anymore after it was stored. */ - ret = tdb_storev(ctx->wtdb->tdb, rec->key, dbufs, num_dbufs, flag); - return (ret == 0) ? NT_STATUS_OK : NT_STATUS_UNSUCCESSFUL; + ret = tdb_storev(tdb, rec->key, dbufs, num_dbufs, flag); + if (ret == -1) { + enum TDB_ERROR err = tdb_error(tdb); + status = map_nt_error_from_tdb(err); + } + return status; } static NTSTATUS db_tdb_delete(struct db_record *rec) @@ -335,6 +346,7 @@ static int db_tdb_traverse_func(TDB_CONTEXT *tdb, TDB_DATA kbuf, TDB_DATA dbuf, rec.key = kbuf; rec.value = dbuf; + rec.value_valid = true; rec.storev = db_tdb_storev; rec.delete_rec = db_tdb_delete; rec.private_data = ctx->db->private_data; @@ -378,6 +390,7 @@ static int db_tdb_traverse_read_func(TDB_CONTEXT *tdb, TDB_DATA kbuf, TDB_DATA d rec.key = kbuf; rec.value = dbuf; + rec.value_valid = true; rec.storev = db_tdb_storev_deny; rec.delete_rec = db_tdb_delete_deny; rec.private_data = ctx->db->private_data; diff --git a/source3/include/g_lock.h b/source3/include/g_lock.h index 782a2e9d160..54f60e410ba 100644 --- a/source3/include/g_lock.h +++ b/source3/include/g_lock.h @@ -26,15 +26,16 @@ struct g_lock_ctx; struct messaging_context; enum g_lock_type { - G_LOCK_READ = 0, - G_LOCK_WRITE = 1, -}; - -struct g_lock_rec { - enum g_lock_type lock_type; - struct server_id pid; + G_LOCK_READ, + G_LOCK_WRITE, + G_LOCK_UPGRADE, + G_LOCK_DOWNGRADE, }; +struct g_lock_ctx *g_lock_ctx_init_backend( + TALLOC_CTX *mem_ctx, + struct messaging_context *msg, + struct db_context **backend); struct g_lock_ctx *g_lock_ctx_init(TALLOC_CTX *mem_ctx, struct messaging_context *msg); @@ -54,9 +55,11 @@ NTSTATUS g_lock_write_data(struct g_lock_ctx *ctx, TDB_DATA key, int g_lock_locks(struct g_lock_ctx *ctx, int (*fn)(TDB_DATA key, void *private_data), void *private_data); -NTSTATUS g_lock_dump(struct g_lock_ctx *ctx, TDB_DATA key, - void (*fn)(const struct g_lock_rec *locks, - size_t num_locks, +NTSTATUS g_lock_dump(struct g_lock_ctx *ctx, + TDB_DATA key, + void (*fn)(struct server_id exclusive, + size_t num_shared, + struct server_id *shared, const uint8_t *data, size_t datalen, void *private_data), diff --git a/source3/lib/dbwrap/dbwrap_ctdb.c b/source3/lib/dbwrap/dbwrap_ctdb.c index 0e108920f58..434a356feef 100644 --- a/source3/lib/dbwrap/dbwrap_ctdb.c +++ b/source3/lib/dbwrap/dbwrap_ctdb.c @@ -554,6 +554,7 @@ static struct db_record *db_ctdb_fetch_locked_transaction(struct db_ctdb_ctx *ct DEBUG(0, ("talloc failed\n")); TALLOC_FREE(result); } + result->value_valid = true; SAFE_FREE(ctdb_data.dptr); @@ -1246,6 +1247,7 @@ again: TALLOC_FREE(result); } } + result->value_valid = true; SAFE_FREE(ctdb_data.dptr); diff --git a/source3/lib/dbwrap/dbwrap_watch.c b/source3/lib/dbwrap/dbwrap_watch.c index c5d55a3c93d..a7a98cfaa29 100644 --- a/source3/lib/dbwrap/dbwrap_watch.c +++ b/source3/lib/dbwrap/dbwrap_watch.c @@ -28,164 +28,109 @@ #include "server_id_watch.h" #include "lib/dbwrap/dbwrap_private.h" -static ssize_t dbwrap_record_watchers_key(struct db_context *db, - struct db_record *rec, - uint8_t *wkey, size_t wkey_len) -{ - size_t db_id_len = dbwrap_db_id(db, NULL, 0); - uint8_t db_id[db_id_len]; - size_t needed; - TDB_DATA key; - - dbwrap_db_id(db, db_id, db_id_len); - - key = dbwrap_record_get_key(rec); - - needed = sizeof(uint32_t) + db_id_len; - if (needed < sizeof(uint32_t)) { - return -1; - } - - needed += key.dsize; - if (needed < key.dsize) { - return -1; - } - - if (wkey != NULL && wkey_len >= needed) { - SIVAL(wkey, 0, db_id_len); - memcpy(wkey + sizeof(uint32_t), db_id, db_id_len); - memcpy(wkey + sizeof(uint32_t) + db_id_len, - key.dptr, key.dsize); - } - - return needed; -} - -static bool dbwrap_record_watchers_key_parse( - TDB_DATA wkey, uint8_t **p_db_id, size_t *p_db_id_len, TDB_DATA *key) -{ - size_t db_id_len; +struct dbwrap_watcher { + /* + * Process watching this record + */ + struct server_id pid; + /* + * Individual instance inside the waiter, incremented each + * time a watcher is created + */ + uint64_t instance; +}; - if (wkey.dsize < sizeof(uint32_t)) { - DBG_WARNING("Invalid watchers key, dsize=%zu\n", wkey.dsize); - return false; - } - db_id_len = IVAL(wkey.dptr, 0); - if (db_id_len > (wkey.dsize - sizeof(uint32_t))) { - DBG_WARNING("Invalid watchers key, wkey.dsize=%zu, " - "db_id_len=%zu\n", wkey.dsize, db_id_len); - return false; - } - if (p_db_id != NULL) { - *p_db_id = wkey.dptr + sizeof(uint32_t); - } - if (p_db_id_len != NULL) { - *p_db_id_len = db_id_len; - } - if (key != NULL) { - key->dptr = wkey.dptr + sizeof(uint32_t) + db_id_len; - key->dsize = wkey.dsize - sizeof(uint32_t) - db_id_len; - } - return true; -} +#define DBWRAP_WATCHER_BUF_LENGTH (SERVER_ID_BUF_LENGTH + sizeof(uint64_t)) /* * Watched records contain a header of: * * [uint32] num_records | deleted bit - * 0 [SERVER_ID_BUF_LENGTH] \ - * 1 [SERVER_ID_BUF_LENGTH] | + * 0 [DBWRAP_WATCHER_BUF_LENGTH] \ + * 1 [DBWRAP_WATCHER_BUF_LENGTH] | * .. |- Array of watchers - * (num_records-1)[SERVER_ID_BUF_LENGTH] / + * (num_records-1)[DBWRAP_WATCHER_BUF_LENGTH] / * * [Remainder of record....] * * If this header is absent then this is a * fresh record of length zero (no watchers). - * - * Note that a record can be deleted with - * watchers present. If so the deleted bit - * is set and the watcher server_id's are - * woken to allow them to remove themselves - * from the watcher array. The record is left - * present marked with the deleted bit until all - * watchers are removed, then the record itself - * is deleted. */ -#define NUM_WATCHERS_DELETED_BIT (1UL<<31) -#define NUM_WATCHERS_MASK (NUM_WATCHERS_DELETED_BIT-1) - -struct dbwrap_watch_rec { - uint8_t *watchers; - size_t num_watchers; - bool deleted; - TDB_DATA data; -}; - -static bool dbwrap_watch_rec_parse(TDB_DATA data, - struct dbwrap_watch_rec *wrec) +static bool dbwrap_watch_rec_parse( + TDB_DATA data, + uint8_t **pwatchers, + size_t *pnum_watchers, + TDB_DATA *pdata) { size_t num_watchers; - bool deleted; - TDB_DATA userdata = { 0 }; + + if (data.dsize == 0) { + /* Fresh record */ + if (pwatchers != NULL) { + *pwatchers = NULL; + } + if (pnum_watchers != NULL) { + *pnum_watchers = 0; + } + if (pdata != NULL) { + *pdata = (TDB_DATA) { .dptr = NULL }; + } + return true; + } if (data.dsize < sizeof(uint32_t)) { - /* Fresh or invalid record */ + /* Invalid record */ return false; } num_watchers = IVAL(data.dptr, 0); - deleted = num_watchers & NUM_WATCHERS_DELETED_BIT; - num_watchers &= NUM_WATCHERS_MASK; - data.dptr += sizeof(uint32_t); data.dsize -= sizeof(uint32_t); - if (num_watchers > data.dsize/SERVER_ID_BUF_LENGTH) { + if (num_watchers > data.dsize/DBWRAP_WATCHER_BUF_LENGTH) { /* Invalid record */ return false; } - if (!deleted) { - size_t watchers_len = num_watchers * SERVER_ID_BUF_LENGTH; - userdata = (TDB_DATA) { + if (pwatchers != NULL) { + *pwatchers = data.dptr; + } + if (pnum_watchers != NULL) { + *pnum_watchers = num_watchers; + } + if (pdata != NULL) { + size_t watchers_len = num_watchers * DBWRAP_WATCHER_BUF_LENGTH; + *pdata = (TDB_DATA) { .dptr = data.dptr + watchers_len, .dsize = data.dsize - watchers_len }; } - *wrec = (struct dbwrap_watch_rec) { - .watchers = data.dptr, .num_watchers = num_watchers, - .deleted = deleted, .data = userdata - }; - return true; } -static void dbwrap_watch_rec_get_watcher( - struct dbwrap_watch_rec *wrec, size_t i, struct server_id *watcher) +static void dbwrap_watcher_get(struct dbwrap_watcher *w, + const uint8_t buf[DBWRAP_WATCHER_BUF_LENGTH]) { - if (i >= wrec->num_watchers) { - abort(); - } - server_id_get(watcher, wrec->watchers + i * SERVER_ID_BUF_LENGTH); + server_id_get(&w->pid, buf); + w->instance = BVAL(buf, SERVER_ID_BUF_LENGTH); } -static void dbwrap_watch_rec_del_watcher(struct dbwrap_watch_rec *wrec, - size_t i) +static void dbwrap_watcher_put(uint8_t buf[DBWRAP_WATCHER_BUF_LENGTH], + const struct dbwrap_watcher *w) -- Samba Shared Repository