The branch, master has been updated via 7c6713e tdb2: make --enable-tdb2 the default. via e1665c9 tdb2: add -1 and -2 options to tdbtorture via 1023082 tdb2:tdbtorture: use TEST_DATA_PREFIX for files via 45ae436 tdb2: name tools the same as TDB1 tools. via 23f1f5e tdb2: tools/tdb2backup via 641beb3 samdb: use compat wrappers for tdb_fetch(). via dd1d573 tdb_compat: only use hashsize attribute when O_CREAT via ae62d46 ldb_wrap.c: fix TDB2-incompatible API usage. via efbf52b tdb2: copy tdb1's changed expansion logic. via 205e198 tdb2: careful on wrap. from 697a6e9 auth: provide private pointer and do not return original PAC signatures
http://gitweb.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit 7c6713e78ff22ebf0aa1caa10697bad9d4cc885e Author: Rusty Russell <ru...@rustcorp.com.au> Date: Mon Jan 30 15:59:18 2012 +1030 tdb2: make --enable-tdb2 the default. We still use the tdb1 on-disk format, but we do so via the tdb2 library. Signed-off-by: Rusty Russell <ru...@rustcorp.com.au> Autobuild-User: Rusty Russell <ru...@rustcorp.com.au> Autobuild-Date: Mon Jan 30 08:02:43 CET 2012 on sn-devel-104 commit e1665c94ac290e030321f7d243e3e70661874778 Author: Rusty Russell <ru...@rustcorp.com.au> Date: Mon Jan 30 15:59:16 2012 +1030 tdb2: add -1 and -2 options to tdbtorture (For now, -1 is the default). commit 10230829df5c5624edaa5fabaf84c9ac5bc31285 Author: Rusty Russell <ru...@rustcorp.com.au> Date: Mon Jan 30 15:31:05 2012 +1030 tdb2:tdbtorture: use TEST_DATA_PREFIX for files TDB2 version of commit b83672b36c1ea8c35833c40c3919b63809f16624. Signed-off-by: Rusty Russell <ru...@rustcorp.com.au> commit 45ae436b19925930b4a60fd8078edcec9ef45e96 Author: Rusty Russell <ru...@rustcorp.com.au> Date: Mon Jan 30 10:07:18 2012 +1030 tdb2: name tools the same as TDB1 tools. Otherwise, when we switch everyone's scripts will break (including our own tests!). Signed-off-by: Rusty Russell <ru...@rustcorp.com.au> commit 23f1f5e0e3516092c47e942c03cef42f4256bf2a Author: Rusty Russell <ru...@rustcorp.com.au> Date: Mon Jan 30 10:07:17 2012 +1030 tdb2: tools/tdb2backup Minor changes from tdb/tools/tdbbackup.c. Signed-off-by: Rusty Russell <ru...@rustcorp.com.au> commit 641beb35bff243bab5651a1cab8aa4b305ecefa4 Author: Rusty Russell <ru...@rustcorp.com.au> Date: Mon Jan 30 10:07:17 2012 +1030 samdb: use compat wrappers for tdb_fetch(). TDB2's tdb_fetch() returns an error code; use tdb_fetch_compat() for now. Similarly, tdb_errorstr() -> tdb_errorstr_compat(). Signed-off-by: Rusty Russell <ru...@rustcorp.com.au> commit dd1d57370d6d83473a7ada3ceb8d250d357ff429 Author: Rusty Russell <ru...@rustcorp.com.au> Date: Mon Jan 30 10:06:55 2012 +1030 tdb_compat: only use hashsize attribute when O_CREAT tdb2 complains if you specify a tdb1 hashsize, and you're not actually trying to create a new database. Signed-off-by: Rusty Russell <ru...@rustcorp.com.au> commit ae62d46a9fcfee9db177ddb3d0f1f2c28c889a35 Author: Rusty Russell <ru...@rustcorp.com.au> Date: Mon Jan 30 09:25:50 2012 +1030 ldb_wrap.c: fix TDB2-incompatible API usage. Auditing revealed one place still expecting a -1 return on failure: tdb2 returns the (negative) errcode directly, so the portable way to do this is to check for != 0. Signed-off-by: Rusty Russell <ru...@rustcorp.com.au> commit efbf52b4fe62eeed085961d7e2689b869bae63dc Author: Rusty Russell <ru...@rustcorp.com.au> Date: Mon Jan 30 09:24:50 2012 +1030 tdb2: copy tdb1's changed expansion logic. TDB2 uses the same expansion logic as TDB1, which got factored out recently. So update TDB2 to match. Signed-off-by: Rusty Russell <ru...@rustcorp.com.au> (Imported from CCAN commit c438ec17d7b2efe76e56e5fc5ab88bd4a02735e8) commit 205e198471a481b849d05b5756261f1739c0c8b2 Author: Rusty Russell <ru...@rustcorp.com.au> Date: Mon Jan 30 09:23:50 2012 +1030 tdb2: careful on wrap. It's much harder to wrap a 64-bit tdb2 than a 32-bit tdb1, but we should still take care against bugs. Also, we should *not* cast the length to a size_t when comparing it to the stat result, in case size_t is 32 bit. Signed-off-by: Rusty Russell <ru...@rustcorp.com.au> (Imported from CCAN commit 6f7cb26e589cea081e71c59801eae87178967861) ----------------------------------------------------------------------- Summary of changes: lib/ldb-samba/ldb_wrap.c | 2 +- lib/tdb2/check.c | 4 +- lib/tdb2/free.c | 57 +++++---- lib/tdb2/io.c | 29 +++-- lib/tdb2/open.c | 2 +- lib/tdb2/private.h | 5 +- .../tools/tdbbackup.c => tdb2/tools/tdb2backup.c} | 136 ++++++++++++-------- lib/tdb2/tools/tdb2torture.c | 62 ++++++++-- lib/tdb2/transaction.c | 19 ++- lib/tdb2/wscript | 16 ++- lib/tdb_compat/tdb_compat.c | 2 +- .../dsdb/samdb/ldb_modules/partition_metadata.c | 12 +- 12 files changed, 221 insertions(+), 125 deletions(-) copy lib/{tdb/tools/tdbbackup.c => tdb2/tools/tdb2backup.c} (70%) Changeset truncated at 500 lines: diff --git a/lib/ldb-samba/ldb_wrap.c b/lib/ldb-samba/ldb_wrap.c index 66213bf..b8af8c1 100644 --- a/lib/ldb-samba/ldb_wrap.c +++ b/lib/ldb-samba/ldb_wrap.c @@ -332,7 +332,7 @@ int samba_ldb_connect(struct ldb_context *ldb, struct loadparm_context *lp_ctx, } } - if (tdb_reopen_all(1) == -1) { + if (tdb_reopen_all(1) != 0) { smb_panic("tdb_reopen_all failed\n"); } } diff --git a/lib/tdb2/check.c b/lib/tdb2/check.c index 238a5b3..ecd6c13 100644 --- a/lib/tdb2/check.c +++ b/lib/tdb2/check.c @@ -497,8 +497,8 @@ static enum TDB_ERROR check_free(struct tdb_context *tdb, } - ecode = tdb->tdb2.io->oob(tdb, off - + frec_len(frec) + ecode = tdb->tdb2.io->oob(tdb, off, + frec_len(frec) + sizeof(struct tdb_used_record), false); if (ecode != TDB_SUCCESS) { diff --git a/lib/tdb2/free.c b/lib/tdb2/free.c index bb819a2..a6d4c7a 100644 --- a/lib/tdb2/free.c +++ b/lib/tdb2/free.c @@ -876,10 +876,38 @@ enum TDB_ERROR set_header(struct tdb_context *tdb, return TDB_SUCCESS; } +/* You need 'size', this tells you how much you should expand by. */ +tdb_off_t tdb_expand_adjust(tdb_off_t map_size, tdb_off_t size) +{ + tdb_off_t new_size, top_size; + + /* limit size in order to avoid using up huge amounts of memory for + * in memory tdbs if an oddball huge record creeps in */ + if (size > 100 * 1024) { + top_size = map_size + size * 2; + } else { + top_size = map_size + size * 100; + } + + /* always make room for at least top_size more records, and at + least 25% more space. if the DB is smaller than 100MiB, + otherwise grow it by 10% only. */ + if (map_size > 100 * 1024 * 1024) { + new_size = map_size * 1.10; + } else { + new_size = map_size * 1.25; + } + + /* Round the database up to a multiple of the page size */ + if (new_size < top_size) + new_size = top_size; + return new_size - map_size; +} + /* Expand the database. */ static enum TDB_ERROR tdb_expand(struct tdb_context *tdb, tdb_len_t size) { - uint64_t old_size, rec_size, map_size; + uint64_t old_size; tdb_len_t wanted; enum TDB_ERROR ecode; @@ -898,35 +926,14 @@ static enum TDB_ERROR tdb_expand(struct tdb_context *tdb, tdb_len_t size) /* Someone else may have expanded the file, so retry. */ old_size = tdb->file->map_size; - tdb->tdb2.io->oob(tdb, tdb->file->map_size + 1, true); + tdb->tdb2.io->oob(tdb, tdb->file->map_size, 1, true); if (tdb->file->map_size != old_size) { tdb_unlock_expand(tdb, F_WRLCK); return TDB_SUCCESS; } - /* limit size in order to avoid using up huge amounts of memory for - * in memory tdbs if an oddball huge record creeps in */ - if (size > 100 * 1024) { - rec_size = size * 2; - } else { - rec_size = size * 100; - } - - /* always make room for at least rec_size more records, and at - least 25% more space. if the DB is smaller than 100MiB, - otherwise grow it by 10% only. */ - if (old_size > 100 * 1024 * 1024) { - map_size = old_size / 10; - } else { - map_size = old_size / 4; - } - - if (map_size > rec_size) { - wanted = map_size; - } else { - wanted = rec_size; - } - + /* Overallocate. */ + wanted = tdb_expand_adjust(old_size, size); /* We need room for the record header too. */ wanted = adjust_size(0, sizeof(struct tdb_used_record) + wanted); diff --git a/lib/tdb2/io.c b/lib/tdb2/io.c index afab0c1..b4a6f0b 100644 --- a/lib/tdb2/io.c +++ b/lib/tdb2/io.c @@ -81,8 +81,8 @@ void tdb_mmap(struct tdb_context *tdb) If probe is true, len being too large isn't a failure. */ -static enum TDB_ERROR tdb_oob(struct tdb_context *tdb, tdb_off_t len, - bool probe) +static enum TDB_ERROR tdb_oob(struct tdb_context *tdb, + tdb_off_t off, tdb_len_t len, bool probe) { struct stat st; enum TDB_ERROR ecode; @@ -92,7 +92,16 @@ static enum TDB_ERROR tdb_oob(struct tdb_context *tdb, tdb_off_t len, || (tdb->flags & TDB_NOLOCK) || tdb_has_expansion_lock(tdb)); - if (len <= tdb->file->map_size) + if (len + off < len) { + if (probe) + return TDB_SUCCESS; + + return tdb_logerr(tdb, TDB_ERR_IO, TDB_LOG_ERROR, + "tdb_oob off %llu len %llu wrap\n", + (long long)off, (long long)len); + } + + if (len + off <= tdb->file->map_size) return TDB_SUCCESS; if (tdb->flags & TDB_INTERNAL) { if (probe) @@ -101,7 +110,7 @@ static enum TDB_ERROR tdb_oob(struct tdb_context *tdb, tdb_off_t len, tdb_logerr(tdb, TDB_ERR_IO, TDB_LOG_ERROR, "tdb_oob len %lld beyond internal" " malloc size %lld", - (long long)len, + (long long)(off + len), (long long)tdb->file->map_size); return TDB_ERR_IO; } @@ -120,13 +129,13 @@ static enum TDB_ERROR tdb_oob(struct tdb_context *tdb, tdb_off_t len, tdb_unlock_expand(tdb, F_RDLCK); - if (st.st_size < (size_t)len) { + if (st.st_size < off + len) { if (probe) return TDB_SUCCESS; tdb_logerr(tdb, TDB_ERR_IO, TDB_LOG_ERROR, - "tdb_oob len %zu beyond eof at %zu", - (size_t)len, st.st_size); + "tdb_oob len %llu beyond eof at %zu", + (long long)(off + len), st.st_size); return TDB_ERR_IO; } @@ -253,7 +262,7 @@ static enum TDB_ERROR tdb_write(struct tdb_context *tdb, tdb_off_t off, "Write to read-only database"); } - ecode = tdb->tdb2.io->oob(tdb, off + len, false); + ecode = tdb->tdb2.io->oob(tdb, off, len, false); if (ecode != TDB_SUCCESS) { return ecode; } @@ -283,7 +292,7 @@ static enum TDB_ERROR tdb_read(struct tdb_context *tdb, tdb_off_t off, { enum TDB_ERROR ecode; - ecode = tdb->tdb2.io->oob(tdb, off + len, false); + ecode = tdb->tdb2.io->oob(tdb, off, len, false); if (ecode != TDB_SUCCESS) { return ecode; } @@ -574,7 +583,7 @@ static void *tdb_direct(struct tdb_context *tdb, tdb_off_t off, size_t len, if (unlikely(!tdb->file->map_ptr)) return NULL; - ecode = tdb_oob(tdb, off + len, false); + ecode = tdb_oob(tdb, off, len, false); if (unlikely(ecode != TDB_SUCCESS)) return TDB_ERR_PTR(ecode); return (char *)tdb->file->map_ptr + off; diff --git a/lib/tdb2/open.c b/lib/tdb2/open.c index 4965d86..2730f21 100644 --- a/lib/tdb2/open.c +++ b/lib/tdb2/open.c @@ -747,7 +747,7 @@ finished: if (tdb->flags & TDB_VERSION1) { ecode = tdb1_probe_length(tdb); } else { - ecode = tdb->tdb2.io->oob(tdb, tdb->file->map_size + 1, true); + ecode = tdb->tdb2.io->oob(tdb, tdb->file->map_size, 1, true); } if (unlikely(ecode != TDB_SUCCESS)) goto fail; diff --git a/lib/tdb2/private.h b/lib/tdb2/private.h index 85cd15a..91d3ca0 100644 --- a/lib/tdb2/private.h +++ b/lib/tdb2/private.h @@ -359,7 +359,7 @@ struct tdb_methods { tdb_len_t); enum TDB_ERROR (*twrite)(struct tdb_context *, tdb_off_t, const void *, tdb_len_t); - enum TDB_ERROR (*oob)(struct tdb_context *, tdb_off_t, bool); + enum TDB_ERROR (*oob)(struct tdb_context *, tdb_off_t, tdb_len_t, bool); enum TDB_ERROR (*expand_file)(struct tdb_context *, tdb_len_t); void *(*direct)(struct tdb_context *, tdb_off_t, size_t, bool); }; @@ -437,6 +437,9 @@ tdb_off_t bucket_off(tdb_off_t ftable_off, unsigned bucket); /* Used by tdb_summary */ tdb_off_t dead_space(struct tdb_context *tdb, tdb_off_t off); +/* Adjust expansion, used by create_recovery_area */ +tdb_off_t tdb_expand_adjust(tdb_off_t map_size, tdb_off_t size); + /* io.c: */ /* Initialize tdb->methods. */ void tdb_io_init(struct tdb_context *tdb); diff --git a/lib/tdb/tools/tdbbackup.c b/lib/tdb2/tools/tdb2backup.c similarity index 70% copy from lib/tdb/tools/tdbbackup.c copy to lib/tdb2/tools/tdb2backup.c index 11ecaa0..531bf29 100644 --- a/lib/tdb/tools/tdbbackup.c +++ b/lib/tdb2/tools/tdb2backup.c @@ -1,4 +1,4 @@ -/* +/* Unix SMB/CIFS implementation. low level tdb backup and restore utility Copyright (C) Andrew Tridgell 2002 @@ -7,12 +7,12 @@ it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version. - + This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - + You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>. */ @@ -40,12 +40,11 @@ */ -#include "replace.h" -#include "system/locale.h" -#include "system/time.h" +#include "tdb2.h" #include "system/filesys.h" -#include "system/wait.h" -#include "tdb.h" + +/* Currently we default to creating a tdb1. This will change! */ +#define TDB2_IS_DEFAULT false #ifdef HAVE_GETOPT_H #include <getopt.h> @@ -53,19 +52,13 @@ static int failed; -static struct tdb_logging_context log_ctx; - -#ifdef PRINTF_ATTRIBUTE -static void tdb_log(struct tdb_context *tdb, enum tdb_debug_level level, const char *format, ...) PRINTF_ATTRIBUTE(3,4); -#endif -static void tdb_log(struct tdb_context *tdb, enum tdb_debug_level level, const char *format, ...) +static void tdb_log(struct tdb_context *tdb, + enum tdb_log_level level, + enum TDB_ERROR ecode, + const char *message, + void *data) { - va_list ap; - - va_start(ap, format); - vfprintf(stdout, format, ap); - va_end(ap); - fflush(stdout); + fprintf(stderr, "%s:%s\n", tdb_errorstr(ecode), message); } static char *add_suffix(const char *name, const char *suffix) @@ -81,12 +74,15 @@ static char *add_suffix(const char *name, const char *suffix) return ret; } -static int copy_fn(TDB_CONTEXT *tdb, TDB_DATA key, TDB_DATA dbuf, void *state) +static int copy_fn(struct tdb_context *tdb, TDB_DATA key, TDB_DATA dbuf, void *state) { - TDB_CONTEXT *tdb_new = (TDB_CONTEXT *)state; + struct tdb_context *tdb_new = (struct tdb_context *)state; + enum TDB_ERROR err; - if (tdb_store(tdb_new, key, dbuf, TDB_INSERT) != 0) { - fprintf(stderr,"Failed to insert into %s\n", tdb_name(tdb_new)); + err = tdb_store(tdb_new, key, dbuf, TDB_INSERT); + if (err) { + fprintf(stderr,"Failed to insert into %s: %s\n", + tdb_name(tdb_new), tdb_errorstr(err)); failed = 1; return 1; } @@ -94,7 +90,7 @@ static int copy_fn(TDB_CONTEXT *tdb, TDB_DATA key, TDB_DATA dbuf, void *state) } -static int test_fn(TDB_CONTEXT *tdb, TDB_DATA key, TDB_DATA dbuf, void *state) +static int test_fn(struct tdb_context *tdb, TDB_DATA key, TDB_DATA dbuf, void *state) { return 0; } @@ -104,13 +100,21 @@ static int test_fn(TDB_CONTEXT *tdb, TDB_DATA key, TDB_DATA dbuf, void *state) only doing the backup if its OK this function is also used for restore */ -static int backup_tdb(const char *old_name, const char *new_name, int hash_size) +static int backup_tdb(const char *old_name, const char *new_name, + bool tdb2, int hash_size) { - TDB_CONTEXT *tdb; - TDB_CONTEXT *tdb_new; + struct tdb_context *tdb; + struct tdb_context *tdb_new; char *tmp_name; struct stat st; int count1, count2; + enum TDB_ERROR err; + union tdb_attribute log_attr, hsize_attr; + int tdb_flags = TDB_DEFAULT; + + if (!tdb2) { + tdb_flags |= TDB_VERSION1; + } tmp_name = add_suffix(new_name, ".tmp"); @@ -121,9 +125,12 @@ static int backup_tdb(const char *old_name, const char *new_name, int hash_size) return 1; } + log_attr.base.attr = TDB_ATTRIBUTE_LOG; + log_attr.base.next = NULL; + log_attr.log.fn = tdb_log; + /* open the old tdb */ - tdb = tdb_open_ex(old_name, 0, 0, - O_RDWR, 0, &log_ctx, NULL); + tdb = tdb_open(old_name, TDB_DEFAULT, O_RDWR, 0, &log_attr); if (!tdb) { printf("Failed to open %s\n", old_name); free(tmp_name); @@ -131,20 +138,27 @@ static int backup_tdb(const char *old_name, const char *new_name, int hash_size) } /* create the new tdb */ + if (!tdb2 && hash_size) { + hsize_attr.base.attr = TDB_ATTRIBUTE_TDB1_HASHSIZE; + hsize_attr.base.next = NULL; + hsize_attr.tdb1_hashsize.hsize = hash_size; + log_attr.base.next = &hsize_attr; + } + unlink(tmp_name); - tdb_new = tdb_open_ex(tmp_name, - hash_size ? hash_size : tdb_hash_size(tdb), - TDB_DEFAULT, - O_RDWR|O_CREAT|O_EXCL, st.st_mode & 0777, - &log_ctx, NULL); + tdb_new = tdb_open(tmp_name, tdb_flags, + O_RDWR|O_CREAT|O_EXCL, st.st_mode & 0777, + &log_attr); if (!tdb_new) { perror(tmp_name); free(tmp_name); return 1; } - if (tdb_transaction_start(tdb) != 0) { - printf("Failed to start transaction on old tdb\n"); + err = tdb_transaction_start(tdb); + if (err) { + fprintf(stderr, "Failed to start transaction on old tdb: %s\n", + tdb_errorstr(err)); tdb_close(tdb); tdb_close(tdb_new); unlink(tmp_name); @@ -153,8 +167,10 @@ static int backup_tdb(const char *old_name, const char *new_name, int hash_size) } /* lock the backup tdb so that nobody else can change it */ - if (tdb_lockall(tdb_new) != 0) { - printf("Failed to lock backup tdb\n"); + err = tdb_lockall(tdb_new); + if (err) { + fprintf(stderr, "Failed to lock backup tdb: %s\n", + tdb_errorstr(err)); tdb_close(tdb); tdb_close(tdb_new); unlink(tmp_name); @@ -192,11 +208,11 @@ static int backup_tdb(const char *old_name, const char *new_name, int hash_size) /* close the new tdb and re-open read-only */ tdb_close(tdb_new); - tdb_new = tdb_open_ex(tmp_name, - 0, - TDB_DEFAULT, - O_RDONLY, 0, - &log_ctx, NULL); + + /* we don't need the hash attr any more */ + log_attr.base.next = NULL; + + tdb_new = tdb_open(tmp_name, TDB_DEFAULT, O_RDONLY, 0, &log_attr); if (!tdb_new) { fprintf(stderr,"failed to reopen %s\n", tmp_name); unlink(tmp_name); @@ -204,7 +220,7 @@ static int backup_tdb(const char *old_name, const char *new_name, int hash_size) free(tmp_name); return 1; } - + /* traverse the new tdb to confirm */ count2 = tdb_traverse(tdb_new, test_fn, NULL); if (count2 != count1) { @@ -233,12 +249,16 @@ static int backup_tdb(const char *old_name, const char *new_name, int hash_size) */ static int verify_tdb(const char *fname, const char *bak_name) { - TDB_CONTEXT *tdb; + struct tdb_context *tdb; int count = -1; + union tdb_attribute log_attr; + + log_attr.base.attr = TDB_ATTRIBUTE_LOG; + log_attr.base.next = NULL; + log_attr.log.fn = tdb_log; /* open the tdb */ - tdb = tdb_open_ex(fname, 0, 0, - O_RDONLY, 0, &log_ctx, NULL); + tdb = tdb_open(fname, TDB_DEFAULT, O_RDONLY, 0, &log_attr); /* traverse the tdb, then close it */ if (tdb) { @@ -249,7 +269,7 @@ static int verify_tdb(const char *fname, const char *bak_name) /* count is < 0 means an error */ if (count < 0) { printf("restoring %s\n", fname); - return backup_tdb(bak_name, fname, 0); + return backup_tdb(bak_name, fname, TDB2_IS_DEFAULT, 0); } printf("%s : %d records\n", fname, count); @@ -276,11 +296,14 @@ static void usage(void) { printf("Usage: tdbbackup [options] <fname...>\n\n"); printf(" -h this help message\n"); + printf(" -1 make the backup a TDB1 file\n"); + printf(" -2 make the backup a TDB2 file\n"); + printf(" -v verify mode (restore if corrupt)\n"); printf(" -s suffix set the backup suffix\n"); printf(" -v verify mode (restore if corrupt)\n"); printf(" -n hashsize set the new hash size for the backup\n"); } - + int main(int argc, char *argv[]) { @@ -289,11 +312,10 @@ static void usage(void) int c; int verify = 0; int hashsize = 0; + bool tdb2 = TDB2_IS_DEFAULT; const char *suffix = ".bak"; - log_ctx.log_fn = tdb_log; -- Samba Shared Repository