The branch, master has been updated via 944c92a15db ctdb-daemon: Modernise debug during record deletion for vacuuming via cdca0d7e78a ctdb-daemon Add extra debug during record deletion for vacuuming via 2e3ad8c20d2 ctdb-tests: Minimise chances of test interfering with itself via f1b594dce1c ctdb-daemon: Do not force full vacuum on first vacuuming run via 9bdd6814e4c ctdb-packaging: Update library versions to upstream versions via 59e244c9d04 ctdb-packaging: Match configure command as per spec file via 4443124fe8d ctdb-packaging: Call waf with python wrapper via 9912709eca2 ctdb-build: Use open() instead of file() for python3 via 1e061ff1e3f ctdb-tool: Avoid data uninitialized warnings from 1ed91f0e102 ctdb-tests: Do not force TEST_VAR_DIR to be absolute
https://git.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit 944c92a15dbbcaaea1cd4e63bf2a109a33126437 Author: Martin Schwenke <mar...@meltin.net> Date: Wed Oct 24 12:29:54 2018 +1100 ctdb-daemon: Modernise debug during record deletion for vacuuming Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> Autobuild-User(master): Amitay Isaacs <ami...@samba.org> Autobuild-Date(master): Tue Dec 18 10:13:50 CET 2018 on sn-devel-144 commit cdca0d7e78a4cad797ff457f860a37de78bcdb44 Author: Martin Schwenke <mar...@meltin.net> Date: Mon Oct 15 21:21:25 2018 +1100 ctdb-daemon Add extra debug during record deletion for vacuuming It isn't currently possible to distinguish these 2 cases. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 2e3ad8c20d2964fba14d0d5b0b36c334ac22caa5 Author: Martin Schwenke <mar...@meltin.net> Date: Tue Dec 18 14:31:24 2018 +1100 ctdb-tests: Minimise chances of test interfering with itself Checking that the database contains 0 records cause a traverse. This may take a lock and cause vacuuming to fail (or be deferred for a particular record/chain). Minimise the chance of this happening by only checking for 0 records every 10 seconds. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit f1b594dce1c081924b16c52c8c50c984a16df098 Author: Martin Schwenke <mar...@meltin.net> Date: Mon Oct 22 21:40:22 2018 +1100 ctdb-daemon: Do not force full vacuum on first vacuuming run When the number of fast path vacuuming runs is 0 then a full vacuuming run is done. This means the first one is a full run, which is almost certainly not what is intended. Combine the 2 conditionals to only flag a full vacuuming run when the count exceeds the configured limit. This means that the full_vacuum_run flag is set in both parent and child, but this is harmless... and is better than getting it wrong. Also tweak the comparison to be less-than-or-equal, since the zeroth run needs to be counted. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 9bdd6814e4c64f3be04daa7e8739e141d2d664b7 Author: Amitay Isaacs <ami...@gmail.com> Date: Tue Dec 18 13:37:40 2018 +1100 ctdb-packaging: Update library versions to upstream versions Signed-off-by: Amitay Isaacs <ami...@gmail.com> Reviewed-by: Martin Schwenke <mar...@meltin.net> commit 59e244c9d0402062ae77c882e2af28d00da44449 Author: Amitay Isaacs <ami...@gmail.com> Date: Tue Dec 18 11:01:35 2018 +1100 ctdb-packaging: Match configure command as per spec file Signed-off-by: Amitay Isaacs <ami...@gmail.com> Reviewed-by: Martin Schwenke <mar...@meltin.net> commit 4443124fe8d7c12dfab62f9962939d14cbed1291 Author: Amitay Isaacs <ami...@gmail.com> Date: Tue Dec 18 11:03:51 2018 +1100 ctdb-packaging: Call waf with python wrapper This allows to build packages even when python3 is not available by setting PYTHON variable. Signed-off-by: Amitay Isaacs <ami...@gmail.com> Reviewed-by: Martin Schwenke <mar...@meltin.net> commit 9912709eca29d21cf56f342079db6b5cc02bbea2 Author: Amitay Isaacs <ami...@gmail.com> Date: Tue Dec 18 10:54:50 2018 +1100 ctdb-build: Use open() instead of file() for python3 Signed-off-by: Amitay Isaacs <ami...@gmail.com> Reviewed-by: Martin Schwenke <mar...@meltin.net> commit 1e061ff1e3fcfcd020df1a35c1c0697c114cb209 Author: Amitay Isaacs <ami...@gmail.com> Date: Tue Dec 18 11:33:30 2018 +1100 ctdb-tool: Avoid data uninitialized warnings ../../tools/ctdb.c: In function 'str_to_data': ../../tools/ctdb.c:624: warning: 'data.dsize' may be used uninitialized in this function ../../tools/ctdb.c:624: warning: 'data.dptr' may be used uninitialized in this function Signed-off-by: Amitay Isaacs <ami...@gmail.com> Reviewed-by: Martin Schwenke <mar...@meltin.net> ----------------------------------------------------------------------- Summary of changes: ctdb/configure.rpm | 4 +++- ctdb/packaging/RPM/ctdb.spec.in | 7 ++++--- ctdb/server/ctdb_recover.c | 23 ++++++++++++---------- ctdb/server/ctdb_vacuum.c | 13 ++++++------ ctdb/tests/simple/69_recovery_resurrect_deleted.sh | 5 +++-- ctdb/tools/ctdb.c | 5 ++++- ctdb/wscript | 2 +- 7 files changed, 34 insertions(+), 25 deletions(-) Changeset truncated at 500 lines: diff --git a/ctdb/configure.rpm b/ctdb/configure.rpm index 32e6673d310..e8ec3b24459 100755 --- a/ctdb/configure.rpm +++ b/ctdb/configure.rpm @@ -9,9 +9,11 @@ fi CFLAGS="-Wall -g -D_GNU_SOURCE" ./configure \ --builtin-libraries=replace,popt \ --bundled-libraries=!talloc,!tevent,!tdb \ - --minimum-library-version=talloc:2.0.8,tdb:1.2.11,tevent:0.9.16 \ + --minimum-library-version=talloc:2.1.14,tdb:1.3.17,tevent:0.9.37 \ --prefix=/usr \ + --includedir=/usr/include/ctdb \ --libdir=${_libdir} \ + --libexecdir=/usr/libexec \ --sysconfdir=/etc \ --mandir=/usr/man \ --localstatedir=/var \ diff --git a/ctdb/packaging/RPM/ctdb.spec.in b/ctdb/packaging/RPM/ctdb.spec.in index 537f158b2d1..d25735e06e0 100644 --- a/ctdb/packaging/RPM/ctdb.spec.in +++ b/ctdb/packaging/RPM/ctdb.spec.in @@ -87,7 +87,8 @@ fi export CC -CFLAGS="$RPM_OPT_FLAGS $EXTRA -D_GNU_SOURCE" ./buildtools/bin/waf configure \ +CFLAGS="$RPM_OPT_FLAGS $EXTRA -D_GNU_SOURCE" \ +$PYTHON ./buildtools/bin/waf configure \ --builtin-libraries=replace,popt \ --bundled-libraries=!talloc,!tevent,!tdb \ --minimum-library-version=talloc:%libtalloc_version,tdb:%libtdb_version,tevent:%libtevent_version \ @@ -102,7 +103,7 @@ CFLAGS="$RPM_OPT_FLAGS $EXTRA -D_GNU_SOURCE" ./buildtools/bin/waf configure \ --mandir=%{_mandir} \ --localstatedir=%{_localstatedir} -./buildtools/bin/waf build +$PYTHON ./buildtools/bin/waf build %install # Clean up in case there is trash left from a previous build @@ -111,7 +112,7 @@ rm -rf $RPM_BUILD_ROOT # Create the target build directory hierarchy mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/sudoers.d -DESTDIR=$RPM_BUILD_ROOT ./buildtools/bin/waf install +DESTDIR=$RPM_BUILD_ROOT $PYTHON ./buildtools/bin/waf install install -m644 config/ctdb.conf $RPM_BUILD_ROOT%{_sysconfdir}/ctdb install -m644 config/ctdb.tunables $RPM_BUILD_ROOT%{_sysconfdir}/ctdb diff --git a/ctdb/server/ctdb_recover.c b/ctdb/server/ctdb_recover.c index f05052e8466..cfe77f643a6 100644 --- a/ctdb/server/ctdb_recover.c +++ b/ctdb/server/ctdb_recover.c @@ -956,12 +956,12 @@ static int delete_tdb_record(struct ctdb_context *ctdb, struct ctdb_db_context * data.dptr = &rec->data[rec->keylen]; if (ctdb_lmaster(ctdb, &key) == ctdb->pnn) { - DEBUG(DEBUG_INFO,(__location__ " Called delete on record where we are lmaster\n")); + DBG_INFO("Called delete on record where we are lmaster\n"); return -1; } if (data.dsize != sizeof(struct ctdb_ltdb_header)) { - DEBUG(DEBUG_ERR,(__location__ " Bad record size\n")); + DBG_ERR("Bad record size\n"); return -1; } @@ -969,6 +969,7 @@ static int delete_tdb_record(struct ctdb_context *ctdb, struct ctdb_db_context * /* use a non-blocking lock */ if (tdb_chainlock_nonblock(ctdb_db->ltdb->tdb, key) != 0) { + DBG_INFO("Failed to get non-blocking chain lock\n"); return -1; } @@ -981,10 +982,10 @@ static int delete_tdb_record(struct ctdb_context *ctdb, struct ctdb_db_context * if (data2.dsize < sizeof(struct ctdb_ltdb_header)) { if (tdb_lock_nonblock(ctdb_db->ltdb->tdb, -1, F_WRLCK) == 0) { if (tdb_delete(ctdb_db->ltdb->tdb, key) != 0) { - DEBUG(DEBUG_CRIT,(__location__ " Failed to delete corrupt record\n")); + DBG_ERR("Failed to delete corrupt record\n"); } tdb_unlock(ctdb_db->ltdb->tdb, -1, F_WRLCK); - DEBUG(DEBUG_CRIT,(__location__ " Deleted corrupt record\n")); + DBG_ERR("Deleted corrupt record\n"); } tdb_chainunlock(ctdb_db->ltdb->tdb, key); free(data2.dptr); @@ -995,8 +996,9 @@ static int delete_tdb_record(struct ctdb_context *ctdb, struct ctdb_db_context * if (hdr2->rsn > hdr->rsn) { tdb_chainunlock(ctdb_db->ltdb->tdb, key); - DEBUG(DEBUG_INFO,(__location__ " Skipping record with rsn=%llu - called with rsn=%llu\n", - (unsigned long long)hdr2->rsn, (unsigned long long)hdr->rsn)); + DBG_INFO("Skipping record with rsn=%llu - called with rsn=%llu\n", + (unsigned long long)hdr2->rsn, + (unsigned long long)hdr->rsn); free(data2.dptr); return -1; } @@ -1004,26 +1006,27 @@ static int delete_tdb_record(struct ctdb_context *ctdb, struct ctdb_db_context * /* do not allow deleting record that have readonly flags set. */ if (hdr->flags & CTDB_REC_RO_FLAGS) { tdb_chainunlock(ctdb_db->ltdb->tdb, key); - DEBUG(DEBUG_INFO,(__location__ " Skipping record with readonly flags set\n")); + DBG_INFO("Skipping record with readonly flags set\n"); free(data2.dptr); return -1; } if (hdr2->flags & CTDB_REC_RO_FLAGS) { tdb_chainunlock(ctdb_db->ltdb->tdb, key); - DEBUG(DEBUG_INFO,(__location__ " Skipping record with readonly flags set\n")); + DBG_INFO("Skipping record with readonly flags set locally\n"); free(data2.dptr); return -1; } if (hdr2->dmaster == ctdb->pnn) { tdb_chainunlock(ctdb_db->ltdb->tdb, key); - DEBUG(DEBUG_INFO,(__location__ " Attempted delete record where we are the dmaster\n")); + DBG_INFO("Attempted delete record where we are the dmaster\n"); free(data2.dptr); return -1; } if (tdb_lock_nonblock(ctdb_db->ltdb->tdb, -1, F_WRLCK) != 0) { tdb_chainunlock(ctdb_db->ltdb->tdb, key); + DBG_INFO("Failed to get non-blocking freelist lock\n"); free(data2.dptr); return -1; } @@ -1031,7 +1034,7 @@ static int delete_tdb_record(struct ctdb_context *ctdb, struct ctdb_db_context * if (tdb_delete(ctdb_db->ltdb->tdb, key) != 0) { tdb_unlock(ctdb_db->ltdb->tdb, -1, F_WRLCK); tdb_chainunlock(ctdb_db->ltdb->tdb, key); - DEBUG(DEBUG_INFO,(__location__ " Failed to delete record\n")); + DBG_INFO("Failed to delete record\n"); free(data2.dptr); return -1; } diff --git a/ctdb/server/ctdb_vacuum.c b/ctdb/server/ctdb_vacuum.c index 2194b7f4da7..9d086917f3c 100644 --- a/ctdb/server/ctdb_vacuum.c +++ b/ctdb/server/ctdb_vacuum.c @@ -1205,6 +1205,7 @@ static void ctdb_vacuum_event(struct tevent_context *ev, struct ctdb_context *ctdb = ctdb_db->ctdb; struct ctdb_vacuum_child_context *child_ctx; struct tevent_fd *fde; + bool full_vacuum_run = false; int ret; /* we don't vacuum if we are in recovery mode, or db frozen */ @@ -1247,7 +1248,11 @@ static void ctdb_vacuum_event(struct tevent_context *ev, return; } - if (vacuum_handle->fast_path_count > ctdb->tunable.vacuum_fast_path_count) { + if (vacuum_handle->fast_path_count >= + ctdb->tunable.vacuum_fast_path_count) { + if (ctdb->tunable.vacuum_fast_path_count > 0) { + full_vacuum_run = true; + } vacuum_handle->fast_path_count = 0; } @@ -1266,7 +1271,6 @@ static void ctdb_vacuum_event(struct tevent_context *ev, if (child_ctx->child_pid == 0) { char cc = 0; - bool full_vacuum_run = false; close(child_ctx->fd[0]); DEBUG(DEBUG_INFO,("Vacuuming child process %d for db %s started\n", getpid(), ctdb_db->db_name)); @@ -1276,11 +1280,6 @@ static void ctdb_vacuum_event(struct tevent_context *ev, _exit(1); } - if ((ctdb->tunable.vacuum_fast_path_count > 0) && - (vacuum_handle->fast_path_count == 0)) - { - full_vacuum_run = true; - } cc = ctdb_vacuum_and_repack_db(ctdb_db, full_vacuum_run); sys_write(child_ctx->fd[1], &cc, 1); diff --git a/ctdb/tests/simple/69_recovery_resurrect_deleted.sh b/ctdb/tests/simple/69_recovery_resurrect_deleted.sh index b69bc5d2672..2c8fc12edef 100755 --- a/ctdb/tests/simple/69_recovery_resurrect_deleted.sh +++ b/ctdb/tests/simple/69_recovery_resurrect_deleted.sh @@ -63,8 +63,9 @@ echo "Wait until vacuuming deletes the record on active nodes" # 1. Original node processes delete queue, asks lmaster to fetch # 2. lmaster recoverd fetches # 3. lmaster processes delete queue -# If vacuuming is just missed then need an extra interval. -wait_until $((vacuum_interval * 4)) database_has_zero_records +# If vacuuming is just missed then need an extra interval +t=$((vacuum_interval * 4)) +wait_until "${t}/10" database_has_zero_records echo "Continue node ${first}" try_command_on_node $first $CTDB continue diff --git a/ctdb/tools/ctdb.c b/ctdb/tools/ctdb.c index b598a866c9b..eb4c684e8b0 100644 --- a/ctdb/tools/ctdb.c +++ b/ctdb/tools/ctdb.c @@ -626,6 +626,9 @@ static int str_to_data(const char *str, size_t len, TALLOC_CTX *mem_ctx, if (strncmp(str, "0x", 2) == 0) { ret = hex_to_data(str+2, len-2, mem_ctx, &data); + if (ret != 0) { + return ret; + } } else { data.dptr = talloc_memdup(mem_ctx, str, len); if (data.dptr == NULL) { @@ -635,7 +638,7 @@ static int str_to_data(const char *str, size_t len, TALLOC_CTX *mem_ctx, } *out = data; - return ret; + return 0; } static int run_helper(TALLOC_CTX *mem_ctx, const char *command, diff --git a/ctdb/wscript b/ctdb/wscript index 44a9d08acf9..30b09d6dc16 100644 --- a/ctdb/wscript +++ b/ctdb/wscript @@ -1177,7 +1177,7 @@ def distonly(ctx): version = get_version() - distfile = file('.distversion', 'w') + distfile = open('.distversion', 'w') for field in version.vcs_fields: distfile.write('%s=%s\n' % (field, str(version.vcs_fields[field]))) distfile.close() -- Samba Shared Repository