The branch, v4-2-test has been updated via c6dc67a nss_wrapper: check for nss.h via 2201a3c ctdb-daemon: Use correct tdb flags when enabling robust mutex support via 2887007 tdb: version 1.3.4 via 4a52345 tdb/toos: allow transactions with TDB_MUTEX_LOCKING via 9ec5518 tdb/test: add tdb1-run-mutex-transaction1 test via 953d373 tdb: allow transactions on on tdb's with TDB_MUTEX_LOCKING from 93b73bf VERSION: Bump version up to 4.2.0rc4 and...
https://git.samba.org/?p=samba.git;a=shortlog;h=v4-2-test - Log ----------------------------------------------------------------- commit c6dc67a10b2d728b44acde82b78fe0b659a52518 Author: Andreas Schneider <a...@samba.org> Date: Wed Oct 15 11:29:12 2014 +0200 nss_wrapper: check for nss.h Bug: https://bugzilla.samba.org/show_bug.cgi?id=11026 Signed-off-by: Andreas Schneider <a...@samba.org> Reviewed-by: Ralph Boehme <s...@samba.org> Autobuild-User(master): Ralph Böhme <s...@samba.org> Autobuild-Date(master): Wed Oct 15 14:06:11 CEST 2014 on sn-devel-104 (cherry picked from commit ee9d075ae44ca8fdfa16afe878038711641f9a0c) Autobuild-User(v4-2-test): Karolin Seeger <ksee...@samba.org> Autobuild-Date(v4-2-test): Fri Jan 2 23:52:50 CET 2015 on sn-devel-104 commit 2201a3c2eb46185fb01269d2e663c6be476e5045 Author: Amitay Isaacs <ami...@gmail.com> Date: Thu Dec 11 13:16:47 2014 +1100 ctdb-daemon: Use correct tdb flags when enabling robust mutex support BUG: https://bugzilla.samba.org/show_bug.cgi?id=11000 Signed-off-by: Amitay Isaacs <ami...@gmail.com> Reviewed-by: Stefan Metzmacher <me...@samba.org> (cherry picked from commit e0bf5dd4566785b41ad1fa0492a9f215639f1685) commit 2887007a53a933a71930c6278948de53c9905ab1 Author: Stefan Metzmacher <me...@samba.org> Date: Fri Dec 12 12:28:47 2014 +0100 tdb: version 1.3.4 Transactions are supported with TDB_MUTEX_LOCKING. This fixes https://bugzilla.samba.org/show_bug.cgi?id=11004 Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Amitay Isaacs <ami...@gmail.com> Autobuild-User(master): Amitay Isaacs <ami...@samba.org> Autobuild-Date(master): Fri Dec 19 11:41:26 CET 2014 on sn-devel-104 (cherry picked from commit a1a90f74eb10a10ac9d508028ed998f8c843f88a) commit 4a52345c2fc6cd8a23906dbbd20c3dbe40f1c951 Author: Stefan Metzmacher <me...@samba.org> Date: Fri Dec 12 12:53:37 2014 +0100 tdb/toos: allow transactions with TDB_MUTEX_LOCKING Bug: https://bugzilla.samba.org/show_bug.cgi?id=11004 Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Amitay Isaacs <ami...@gmail.com> (cherry picked from commit f32747ddd06445b5c51aa3f1b56fead48285a735) commit 9ec55189f70f50fe96960237b6cfdb64b8a645f5 Author: Stefan Metzmacher <me...@samba.org> Date: Fri Dec 12 12:24:50 2014 +0100 tdb/test: add tdb1-run-mutex-transaction1 test Bug: https://bugzilla.samba.org/show_bug.cgi?id=11004 Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Amitay Isaacs <ami...@gmail.com> (cherry picked from commit 5d5d91fae55a806261375b87bf7109a980471799) commit 953d37311aba4e99cabed707344be5ca95548d24 Author: Stefan Metzmacher <me...@samba.org> Date: Fri Dec 12 11:22:47 2014 +0100 tdb: allow transactions on on tdb's with TDB_MUTEX_LOCKING There's no real reason to disallow transactions as the allrecord lock is also available with mutexes enabled. E.g. ctdbd requires transactions also on non-persistent databases opened with TDB_CLEAR_IF_FIRST and TDB_MUTEX_LOCKING. Bug: https://bugzilla.samba.org/show_bug.cgi?id=11004 Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Amitay Isaacs <ami...@gmail.com> (cherry picked from commit d0839af9d60a6fdf0720cd0b1a859e0e2241903f) ----------------------------------------------------------------------- Summary of changes: ctdb/client/ctdb_client.c | 30 +++- ctdb/server/ctdb_lock.c | 30 +++- ctdb/server/ctdb_lock_helper.c | 28 +-- ctdb/server/ctdb_ltdb_server.c | 4 +- lib/nss_wrapper/wscript | 4 +- lib/tdb/ABI/{tdb-1.3.0.sigs => tdb-1.3.4.sigs} | 0 lib/tdb/common/transaction.c | 2 +- lib/tdb/test/run-mutex-transaction1.c | 236 +++++++++++++++++++++++++ lib/tdb/tools/tdbtorture.c | 1 - lib/tdb/wscript | 3 +- 10 files changed, 312 insertions(+), 26 deletions(-) copy lib/tdb/ABI/{tdb-1.3.0.sigs => tdb-1.3.4.sigs} (100%) create mode 100644 lib/tdb/test/run-mutex-transaction1.c Changeset truncated at 500 lines: diff --git a/ctdb/client/ctdb_client.c b/ctdb/client/ctdb_client.c index 07b17d0..da18826 100644 --- a/ctdb/client/ctdb_client.c +++ b/ctdb/client/ctdb_client.c @@ -1928,7 +1928,7 @@ int ctdb_ctrl_createdb(struct ctdb_context *ctdb, struct timeval timeout, uint32 #ifdef TDB_MUTEX_LOCKING if (!persistent && ctdb->tunable.mutex_enabled == 1) { - tdb_flags |= TDB_MUTEX_LOCKING; + tdb_flags |= (TDB_MUTEX_LOCKING | TDB_CLEAR_IF_FIRST); } #endif @@ -2055,6 +2055,9 @@ struct ctdb_db_context *ctdb_attach(struct ctdb_context *ctdb, TDB_DATA data; int ret; int32_t res; +#ifdef TDB_MUTEX_LOCKING + uint32_t mutex_enabled = 0; +#endif ctdb_db = ctdb_db_handle(ctdb, name); if (ctdb_db) { @@ -2080,8 +2083,18 @@ struct ctdb_db_context *ctdb_attach(struct ctdb_context *ctdb, } #ifdef TDB_MUTEX_LOCKING - if (!persistent && ctdb->tunable.mutex_enabled == 1) { - tdb_flags |= TDB_MUTEX_LOCKING; + if (!persistent) { + ret = ctdb_ctrl_get_tunable(ctdb, timeval_current_ofs(3,0), + CTDB_CURRENT_NODE, + "TDBMutexEnabled", + &mutex_enabled); + if (ret != 0) { + DEBUG(DEBUG_WARNING, ("Assuming no mutex support.\n")); + } + + if (mutex_enabled == 1) { + tdb_flags |= (TDB_MUTEX_LOCKING | TDB_CLEAR_IF_FIRST); + } } #endif @@ -2105,7 +2118,16 @@ struct ctdb_db_context *ctdb_attach(struct ctdb_context *ctdb, return NULL; } - tdb_flags = persistent?TDB_DEFAULT:TDB_NOSYNC; + if (persistent) { + tdb_flags = TDB_DEFAULT; + } else { + tdb_flags = TDB_NOSYNC; +#ifdef TDB_MUTEX_LOCKING + if (mutex_enabled) { + tdb_flags |= (TDB_MUTEX_LOCKING | TDB_CLEAR_IF_FIRST); + } +#endif + } if (ctdb->valgrinding) { tdb_flags |= TDB_NOMMAP; } diff --git a/ctdb/server/ctdb_lock.c b/ctdb/server/ctdb_lock.c index 22a88b3..7959d40 100644 --- a/ctdb/server/ctdb_lock.c +++ b/ctdb/server/ctdb_lock.c @@ -544,11 +544,23 @@ static int db_count_handler(struct ctdb_db_context *ctdb_db, uint32_t priority, { int *count = (int *)private_data; - (*count)++; + (*count) += 2; return 0; } +static int db_flags(struct ctdb_db_context *ctdb_db) +{ + int tdb_flags = TDB_DEFAULT; + +#ifdef TDB_MUTEX_LOCKING + if (!ctdb_db->persistent && ctdb_db->ctdb->tunable.mutex_enabled) { + tdb_flags = (TDB_MUTEX_LOCKING | TDB_CLEAR_IF_FIRST); + } +#endif + return tdb_flags; +} + struct db_namelist { const char **names; int n; @@ -560,7 +572,9 @@ static int db_name_handler(struct ctdb_db_context *ctdb_db, uint32_t priority, struct db_namelist *list = (struct db_namelist *)private_data; list->names[list->n] = talloc_strdup(list->names, ctdb_db->db_path); - list->n++; + list->names[list->n+1] = talloc_asprintf(list->names, "0x%x", + db_flags(ctdb_db)); + list->n += 2; return 0; } @@ -577,11 +591,11 @@ static bool lock_helper_args(TALLOC_CTX *mem_ctx, switch (lock_ctx->type) { case LOCK_RECORD: - nargs = 5; + nargs = 6; break; case LOCK_DB: - nargs = 4; + nargs = 5; break; case LOCK_ALLDB_PRIO: @@ -612,16 +626,20 @@ static bool lock_helper_args(TALLOC_CTX *mem_ctx, case LOCK_RECORD: args[2] = talloc_strdup(args, "RECORD"); args[3] = talloc_strdup(args, lock_ctx->ctdb_db->db_path); + args[4] = talloc_asprintf(args, "0x%x", + db_flags(lock_ctx->ctdb_db)); if (lock_ctx->key.dsize == 0) { - args[4] = talloc_strdup(args, "NULL"); + args[5] = talloc_strdup(args, "NULL"); } else { - args[4] = hex_encode_talloc(args, lock_ctx->key.dptr, lock_ctx->key.dsize); + args[5] = hex_encode_talloc(args, lock_ctx->key.dptr, lock_ctx->key.dsize); } break; case LOCK_DB: args[2] = talloc_strdup(args, "DB"); args[3] = talloc_strdup(args, lock_ctx->ctdb_db->db_path); + args[4] = talloc_asprintf(args, "0x%x", + db_flags(lock_ctx->ctdb_db)); break; case LOCK_ALLDB_PRIO: diff --git a/ctdb/server/ctdb_lock_helper.c b/ctdb/server/ctdb_lock_helper.c index 2161a9a..7a09ecf 100644 --- a/ctdb/server/ctdb_lock_helper.c +++ b/ctdb/server/ctdb_lock_helper.c @@ -36,9 +36,9 @@ static void send_result(int fd, char result) static void usage(void) { fprintf(stderr, "\n"); - fprintf(stderr, "Usage: %s <log-fd> <ctdbd-pid> <output-fd> RECORD <db-path> <db-key>\n", + fprintf(stderr, "Usage: %s <log-fd> <ctdbd-pid> <output-fd> RECORD <db-path> <db-flags> <db-key>\n", progname); - fprintf(stderr, " %s <log-fd> <ctdbd-pid> <output-fd> DB <db1-path> [<db2-path> ...]\n", + fprintf(stderr, " %s <log-fd> <ctdbd-pid> <output-fd> DB <db1-path> <db1-flags> [<db2-path> <db2-flags>...]\n", progname); } @@ -59,10 +59,14 @@ static uint8_t *hex_decode_talloc(TALLOC_CTX *mem_ctx, return buffer; } -static int lock_record(const char *dbpath, const char *dbkey) +static int lock_record(const char *dbpath, const char *dbflags, const char *dbkey) { TDB_DATA key; struct tdb_context *tdb; + int tdb_flags; + + /* No error checking since CTDB always passes sane values */ + tdb_flags = strtol(dbflags, NULL, 0); /* Convert hex key to key */ if (strcmp(dbkey, "NULL") == 0) { @@ -72,7 +76,7 @@ static int lock_record(const char *dbpath, const char *dbkey) key.dptr = hex_decode_talloc(NULL, dbkey, &key.dsize); } - tdb = tdb_open(dbpath, 0, TDB_DEFAULT, O_RDWR, 0600); + tdb = tdb_open(dbpath, 0, tdb_flags, O_RDWR, 0600); if (tdb == NULL) { fprintf(stderr, "%s: Error opening database %s\n", progname, dbpath); return 1; @@ -89,11 +93,15 @@ static int lock_record(const char *dbpath, const char *dbkey) } -static int lock_db(const char *dbpath) +static int lock_db(const char *dbpath, const char *dbflags) { struct tdb_context *tdb; + int tdb_flags; + + /* No error checking since CTDB always passes sane values */ + tdb_flags = strtol(dbflags, NULL, 0); - tdb = tdb_open(dbpath, 0, TDB_DEFAULT, O_RDWR, 0600); + tdb = tdb_open(dbpath, 0, tdb_flags, O_RDWR, 0600); if (tdb == NULL) { fprintf(stderr, "%s: Error opening database %s\n", progname, dbpath); return 1; @@ -140,21 +148,21 @@ int main(int argc, char *argv[]) lock_type = argv[4]; if (strcmp(lock_type, "RECORD") == 0) { - if (argc != 7) { + if (argc != 8) { fprintf(stderr, "%s: Invalid number of arguments (%d)\n", progname, argc); usage(); exit(1); } - result = lock_record(argv[5], argv[6]); + result = lock_record(argv[5], argv[6], argv[7]); } else if (strcmp(lock_type, "DB") == 0) { int n; /* If there are no databases specified, no need for lock */ if (argc > 5) { - for (n=5; n<argc; n++) { - result = lock_db(argv[n]); + for (n=5; n+1<argc; n+=2) { + result = lock_db(argv[n], argv[n+1]); if (result != 0) { break; } diff --git a/ctdb/server/ctdb_ltdb_server.c b/ctdb/server/ctdb_ltdb_server.c index 9ac2217..174a460 100644 --- a/ctdb/server/ctdb_ltdb_server.c +++ b/ctdb/server/ctdb_ltdb_server.c @@ -844,7 +844,7 @@ static int ctdb_local_attach(struct ctdb_context *ctdb, const char *db_name, #ifdef TDB_MUTEX_LOCKING if (ctdb->tunable.mutex_enabled && mutexes && tdb_runtime_check_for_robust_mutexes()) { - tdb_flags |= TDB_MUTEX_LOCKING; + tdb_flags |= (TDB_MUTEX_LOCKING | TDB_CLEAR_IF_FIRST); } #endif @@ -1138,7 +1138,7 @@ int32_t ctdb_control_db_attach(struct ctdb_context *ctdb, TDB_DATA indata, that tdb_flags is passed in via the (otherwise unused) srvid to the attach control */ #ifdef TDB_MUTEX_LOCKING - tdb_flags &= (TDB_NOSYNC|TDB_INCOMPATIBLE_HASH|TDB_MUTEX_LOCKING); + tdb_flags &= (TDB_NOSYNC|TDB_INCOMPATIBLE_HASH|TDB_MUTEX_LOCKING|TDB_CLEAR_IF_FIRST); #else tdb_flags &= (TDB_NOSYNC|TDB_INCOMPATIBLE_HASH); #endif diff --git a/lib/nss_wrapper/wscript b/lib/nss_wrapper/wscript index 34026c0..78fe4cd 100644 --- a/lib/nss_wrapper/wscript +++ b/lib/nss_wrapper/wscript @@ -9,6 +9,8 @@ def configure(conf): conf.DEFINE('USING_SYSTEM_NSS_WRAPPER', 1) libnss_wrapper_so_path = 'libnss_wrapper.so' else: + conf.CHECK_HEADERS('nss.h') + # check HAVE_GCC_THREAD_LOCAL_STORAGE conf.CHECK_CODE(''' __thread int tls; @@ -94,7 +96,7 @@ def configure(conf): conf.DEFINE('NSS_WRAPPER', 1) def build(bld): - if not bld.CONFIG_SET("USING_SYSTEM_NSS_WRAPPER"): + if bld.CONFIG_SET("HAVE_NSS_H") and not bld.CONFIG_SET("USING_SYSTEM_NSS_WRAPPER"): # We need to do it this way or the library wont work. # Using private_library=True will add symbol version which # breaks preloading! diff --git a/lib/tdb/ABI/tdb-1.3.0.sigs b/lib/tdb/ABI/tdb-1.3.4.sigs similarity index 100% copy from lib/tdb/ABI/tdb-1.3.0.sigs copy to lib/tdb/ABI/tdb-1.3.4.sigs diff --git a/lib/tdb/common/transaction.c b/lib/tdb/common/transaction.c index caef0be..0dd057b 100644 --- a/lib/tdb/common/transaction.c +++ b/lib/tdb/common/transaction.c @@ -421,7 +421,7 @@ static int _tdb_transaction_start(struct tdb_context *tdb, enum tdb_lock_flags lockflags) { /* some sanity checks */ - if (tdb->read_only || (tdb->flags & (TDB_INTERNAL|TDB_MUTEX_LOCKING)) + if (tdb->read_only || (tdb->flags & TDB_INTERNAL) || tdb->traverse_read) { TDB_LOG((tdb, TDB_DEBUG_ERROR, "tdb_transaction_start: cannot start a transaction on a read-only or internal db\n")); tdb->ecode = TDB_ERR_EINVAL; diff --git a/lib/tdb/test/run-mutex-transaction1.c b/lib/tdb/test/run-mutex-transaction1.c new file mode 100644 index 0000000..7b9f7b1 --- /dev/null +++ b/lib/tdb/test/run-mutex-transaction1.c @@ -0,0 +1,236 @@ +#include "../common/tdb_private.h" +#include "../common/io.c" +#include "../common/tdb.c" +#include "../common/lock.c" +#include "../common/freelist.c" +#include "../common/traverse.c" +#include "../common/transaction.c" +#include "../common/error.c" +#include "../common/open.c" +#include "../common/check.c" +#include "../common/hash.c" +#include "../common/mutex.c" +#include "tap-interface.h" +#include <stdlib.h> +#include <sys/types.h> +#include <sys/wait.h> +#include <stdarg.h> + +static TDB_DATA key, data; + +static void log_fn(struct tdb_context *tdb, enum tdb_debug_level level, + const char *fmt, ...) +{ + va_list ap; + va_start(ap, fmt); + vfprintf(stderr, fmt, ap); + va_end(ap); +} + +static int do_child(int tdb_flags, int to, int from) +{ + struct tdb_context *tdb; + unsigned int log_count; + struct tdb_logging_context log_ctx = { log_fn, &log_count }; + int ret; + char c = 0; + + tdb = tdb_open_ex("mutex-transaction1.tdb", 3, tdb_flags, + O_RDWR|O_CREAT, 0755, &log_ctx, NULL); + ok(tdb, "tdb_open_ex should succeed"); + + ret = tdb_transaction_start(tdb); + ok(ret == 0, "tdb_transaction_start should succeed"); + + ret = tdb_store(tdb, key, data, TDB_INSERT); + ok(ret == 0, "tdb_store(tdb, key, data, TDB_INSERT) should succeed"); + + write(to, &c, sizeof(c)); + read(from, &c, sizeof(c)); + + ret = tdb_transaction_cancel(tdb); + ok(ret == 0, "tdb_transaction_cancel should succeed"); + + write(to, &c, sizeof(c)); + read(from, &c, sizeof(c)); + + ret = tdb_transaction_start(tdb); + ok(ret == 0, "tdb_transaction_start should succeed"); + + ret = tdb_store(tdb, key, data, TDB_INSERT); + ok(ret == 0, "tdb_store(tdb, key, data, TDB_INSERT) should succeed"); + + write(to, &c, sizeof(c)); + read(from, &c, sizeof(c)); + + ret = tdb_transaction_commit(tdb); + ok(ret == 0, "tdb_transaction_commit should succeed"); + + write(to, &c, sizeof(c)); + read(from, &c, sizeof(c)); + + ret = tdb_transaction_start(tdb); + ok(ret == 0, "tdb_transaction_start should succeed"); + + ret = tdb_store(tdb, key, key, TDB_REPLACE); + ok(ret == 0, "tdb_store(tdb, key, data, TDB_REPLACE) should succeed"); + + write(to, &c, sizeof(c)); + read(from, &c, sizeof(c)); + + ret = tdb_transaction_commit(tdb); + ok(ret == 0, "tdb_transaction_commit should succeed"); + + write(to, &c, sizeof(c)); + read(from, &c, sizeof(c)); + + return 0; +} + +/* The code should barf on TDBs created with rwlocks. */ +int main(int argc, char *argv[]) +{ + struct tdb_context *tdb; + unsigned int log_count; + struct tdb_logging_context log_ctx = { log_fn, &log_count }; + int ret, status; + pid_t child, wait_ret; + int fromchild[2]; + int tochild[2]; + TDB_DATA val; + char c; + int tdb_flags; + bool runtime_support; + + runtime_support = tdb_runtime_check_for_robust_mutexes(); + + if (!runtime_support) { + skip(1, "No robust mutex support"); + return exit_status(); + } + + key.dsize = strlen("hi"); + key.dptr = discard_const_p(uint8_t, "hi"); + data.dsize = strlen("world"); + data.dptr = discard_const_p(uint8_t, "world"); + + pipe(fromchild); + pipe(tochild); + + tdb_flags = TDB_INCOMPATIBLE_HASH| + TDB_MUTEX_LOCKING| + TDB_CLEAR_IF_FIRST; + + child = fork(); + if (child == 0) { + close(fromchild[0]); + close(tochild[1]); + return do_child(tdb_flags, fromchild[1], tochild[0]); + } + close(fromchild[1]); + close(tochild[0]); + + read(fromchild[0], &c, sizeof(c)); + + tdb = tdb_open_ex("mutex-transaction1.tdb", 0, + tdb_flags, O_RDWR|O_CREAT, 0755, + &log_ctx, NULL); + ok(tdb, "tdb_open_ex should succeed"); + + /* + * The child has the transaction running + */ + ret = tdb_transaction_start_nonblock(tdb); + ok(ret == -1, "tdb_transaction_start_nonblock not succeed"); + + ret = tdb_chainlock_nonblock(tdb, key); + ok(ret == -1, "tdb_chainlock_nonblock should not succeed"); + + /* + * We can still read + */ + ret = tdb_exists(tdb, key); + ok(ret == 0, "tdb_exists(tdb, key) should return 0"); + + val = tdb_fetch(tdb, key); + ok(val.dsize == 0, "tdb_fetch(tdb, key) should return an empty value"); + + write(tochild[1], &c, sizeof(c)); + + /* + * When the child canceled we can start... + */ + ret = tdb_transaction_start(tdb); + ok(ret == 0, "tdb_transaction_start should succeed"); + + read(fromchild[0], &c, sizeof(c)); + write(tochild[1], &c, sizeof(c)); + + ret = tdb_transaction_cancel(tdb); + ok(ret == 0, "tdb_transaction_cancel should succeed"); + + /* + * When we canceled the child can start and store... + */ + read(fromchild[0], &c, sizeof(c)); + + /* + * We still see the old values before the child commits... + */ + ret = tdb_exists(tdb, key); + ok(ret == 0, "tdb_exists(tdb, key) should return 0"); + + val = tdb_fetch(tdb, key); + ok(val.dsize == 0, "tdb_fetch(tdb, key) should return an empty value"); + + write(tochild[1], &c, sizeof(c)); + read(fromchild[0], &c, sizeof(c)); + + /* + * We see the new values after the commit... + */ + ret = tdb_exists(tdb, key); + ok(ret == 1, "tdb_exists(tdb, key) should return 1"); + + val = tdb_fetch(tdb, key); + ok(val.dsize != 0, "tdb_fetch(tdb, key) should return a value"); + ok(val.dsize == data.dsize, "tdb_fetch(tdb, key) should return a value"); + ok(memcmp(val.dptr, data.dptr, data.dsize) == 0, "tdb_fetch(tdb, key) should return a value"); + + write(tochild[1], &c, sizeof(c)); + read(fromchild[0], &c, sizeof(c)); + + /* + * The child started a new transaction and replaces the value, + * but we still see the old values before the child commits... + */ + ret = tdb_exists(tdb, key); + ok(ret == 1, "tdb_exists(tdb, key) should return 1"); + -- Samba Shared Repository