The branch, master has been updated via 0958010 ldb: Avoid multiple tiny allocations during full DB scan via 198471f ldb: Avoid cost of talloc_free() for unmatched messages via 45373b9 docs: Explain that "reset on zero vc" is an SMB1 feature via af79484 autobuild: disable abi check on O3 build via 1d0dd5b selftest/wscript: format perftest as json via dfac53c selftest: add an option to specify the test list via e908873 make perftest: for performance testing via 288efc5 Add AD DC performance tests via 85b4a3e filter-subunit: default to empty affixes, saving verbose checks via 809f4c7 subunithelper: use set for efficient inclusion test via 6c391e1 selftest: allow tests.py scripts to run independently via dda1b74 blackbox tests: add timestamps for subunit tests via db85692 autobuild --restrict-tests to test a restricted set via 25efdf3 autobuild: remove unused argument from 91a2eb0 s3: modules: vfs_acl_common - Add Ralph's copyright.
https://git.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit 09580103fe30896c61e173c836aa4ca525cc6243 Author: Andrew Bartlett <abart...@samba.org> Date: Tue Aug 23 11:38:26 2016 +1200 ldb: Avoid multiple tiny allocations during full DB scan LDB_UNPACK_DATA_FLAG_NO_VALUES_ALLOC allows us to consolidate some of these allocations Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Autobuild-User(master): Douglas Bagnall <dbagn...@samba.org> Autobuild-Date(master): Wed Aug 31 10:53:09 CEST 2016 on sn-devel-144 commit 198471f9edfb9da2ee5b54e60a46d208f58ca2e4 Author: Andrew Bartlett <abart...@samba.org> Date: Fri Aug 26 09:58:38 2016 +1200 ldb: Avoid cost of talloc_free() for unmatched messages Instead, we pay the cost of allocating a copy of the whole message once and we pay the cost of allocating a "struct ldb_val" that will not be used for each element in that message. This differes from the approach of ldb_unpack_data_only_attr_list() in that we need not allocate each value for a message that we do not return, so is more efficient for large multi-valued attributes and un-indexed or poorly indexed searches Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> commit 45373b988d65c31e74af67a40888553c41d15532 Author: Andrew Bartlett <abart...@samba.org> Date: Tue Aug 16 12:21:59 2016 +1200 docs: Explain that "reset on zero vc" is an SMB1 feature Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> commit af7948495f31199e560f8fc49b5fc508564a0e09 Author: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Date: Fri Aug 26 12:07:34 2016 +1200 autobuild: disable abi check on O3 build Signed-off-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 1d0dd5bf5eef4ceedb4c99bd1de8042f26201bb8 Author: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Date: Thu Aug 4 15:35:46 2016 +1200 selftest/wscript: format perftest as json This makes it easier to use with common web-based graphing systems. Signed-off-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit dfac53cd40d8f44cebcf15d9aceba5b4a8106b2a Author: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Date: Wed Aug 31 14:56:25 2016 +1200 selftest: add an option to specify the test list This can be used to override the default test lists used by `make test` and `make perftest`. This tests can either be programmatically generated (as is done for the defaults -- see selftest/tests.py for an example), or from a static list. For the generated lists, append a pipe symbol: make test TEST_LIST='/bin/sh /tmp/tests.sh|' and omit the pipe for a static list: make test TEST_LIST='/tmp/tests.txt' There are likely other useful modes of operation -- see `perldoc open` for the wondrous details. Signed-off-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit e908873757171db5b65296c5c3cdefe7d0fb0c01 Author: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Date: Wed Aug 17 10:56:50 2016 +1200 make perftest: for performance testing This runs a selection of subunit tests and reduces the output to only the time it takes to run each test. The tests are listed in selftest/perf_tests.py. Signed-off-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 288efc55608e7d465f68d782790cf485defb79b6 Author: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Date: Wed Aug 17 10:56:39 2016 +1200 Add AD DC performance tests These test a variety of simple AD DC operations. These tests are NOT independent of each other and must be run in the right order (alphabetically, which is guaranteed by Python's unittest module) -- the running of each test is part of the set-up for later modules. This means we have to subvert unittest a bit, but it saves hours of repeated set-up. These tests are not intended to push edge cases, but to hammer common operations that should work on all versions of Samba. The tests have been tested back to Samba 4.0.26. Signed-off-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 85b4a3ea613b2f2b531551787f764c940bcf3f83 Author: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Date: Tue Aug 2 11:00:27 2016 +1200 filter-subunit: default to empty affixes, saving verbose checks Signed-off-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 809f4c718af00909bbce46ddde6a2688a88e73fc Author: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Date: Tue Aug 2 10:27:05 2016 +1200 subunithelper: use set for efficient inclusion test Signed-off-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 6c391e18a47782a78e035f3e35a41710d8424105 Author: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Date: Fri Jul 29 10:57:52 2016 +1200 selftest: allow tests.py scripts to run independently These generate lists of test commands. Usually they are run in special environments, but they should work from the command line. This restores the intended behaviour. Signed-off-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit dda1b74ea10a45d9a810b2a982e93cfb13dfeaba Author: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Date: Wed Aug 3 16:03:57 2016 +1200 blackbox tests: add timestamps for subunit tests There is the icky thing with sed because some kinds of `date` don't have sub-second resolution, which we really want. Another way to do it would be: python -c "import datetime; print datetime.datetime.utcnow().strftime('time: %Y-%m-%d %H:%M:%S.%fZ')" which should be universal, but is a little slower. Signed-off-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit db85692e4cd41499b5c4ec3220ae36da77afa2ef Author: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Date: Wed Jul 27 14:28:04 2016 +1200 autobuild --restrict-tests to test a restricted set This allows autobuild to be used as a test framework for running particular Samba tests in a cloud environment. Signed-off-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 25efdf330bf91e7995c4d53a25713f2eaa364c0b Author: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Date: Wed Jul 27 14:09:08 2016 +1200 autobuild: remove unused argument The "tasklist" parameter is the same as the global "tasks" variable, but only the latter is used. Signed-off-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> ----------------------------------------------------------------------- Summary of changes: Makefile | 3 + docs-xml/smbdotconf/misc/resetonzerovc.xml | 7 +- lib/ldb/common/ldb_pack.c | 25 +- lib/ldb/include/ldb_module.h | 16 +- lib/ldb/ldb_tdb/ldb_index.c | 12 +- lib/ldb/ldb_tdb/ldb_search.c | 157 ++++++++---- lib/ldb/ldb_tdb/ldb_tdb.h | 4 +- script/autobuild.py | 27 +- selftest/filter-subunit | 39 ++- selftest/format-subunit-json | 54 ++++ selftest/perf_tests.py | 26 ++ selftest/subunithelper.py | 83 +++++- selftest/tests.py | 1 + selftest/wscript | 28 +- source3/selftest/tests.py | 1 + source4/dsdb/tests/python/ad_dc_performance.py | 338 +++++++++++++++++++++++++ source4/torture/ldb/ldb.c | 18 ++ testprogs/blackbox/subunit.sh | 11 + 18 files changed, 762 insertions(+), 88 deletions(-) create mode 100644 selftest/format-subunit-json create mode 100644 selftest/perf_tests.py create mode 100644 source4/dsdb/tests/python/ad_dc_performance.py Changeset truncated at 500 lines: diff --git a/Makefile b/Makefile index 95681ae..5cc9077 100644 --- a/Makefile +++ b/Makefile @@ -16,6 +16,9 @@ uninstall: test: $(WAF) test $(TEST_OPTIONS) +perftest: + $(WAF) test --perf-test $(TEST_OPTIONS) + help: @echo NOTE: to run extended waf options use $(WAF_BINARY) or modify your PATH $(WAF) --help diff --git a/docs-xml/smbdotconf/misc/resetonzerovc.xml b/docs-xml/smbdotconf/misc/resetonzerovc.xml index f2ad85d..a991733 100644 --- a/docs-xml/smbdotconf/misc/resetonzerovc.xml +++ b/docs-xml/smbdotconf/misc/resetonzerovc.xml @@ -4,7 +4,7 @@ xmlns:samba="http://www.samba.org/samba/DTD/samba-doc"> <description> <para> - This boolean option controls whether an incoming session setup + This boolean option controls whether an incoming SMB1 session setup should kill other connections coming from the same IP. This matches the default Windows 2003 behaviour. @@ -17,8 +17,9 @@ kills all other connections coming from the same IP. This way the locked files are accessible again. - Please be aware that enabling this option will kill connections behind - a masquerading router. + Please be aware that enabling this option will kill + connections behind a masquerading router, and will not trigger + for clients that only use SMB2 or SMB3. </para> </description> diff --git a/lib/ldb/common/ldb_pack.c b/lib/ldb/common/ldb_pack.c index 7e6dd2d..a63dd58 100644 --- a/lib/ldb/common/ldb_pack.c +++ b/lib/ldb/common/ldb_pack.c @@ -233,6 +233,7 @@ int ldb_unpack_data_only_attr_list_flags(struct ldb_context *ldb, unsigned int nelem = 0; size_t len; unsigned int found = 0; + struct ldb_val *ldb_val_single_array = NULL; if (list == NULL) { list_size = 0; @@ -313,6 +314,26 @@ int ldb_unpack_data_only_attr_list_flags(struct ldb_context *ldb, goto failed; } + /* + * In typical use, most values are single-valued. This makes + * it quite expensive to allocate an array of ldb_val for each + * of these, just to then hold the pointer to the data buffer + * (in the LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC we don't + * allocate the data). So with + * LDB_UNPACK_DATA_FLAG_NO_VALUES_ALLOC we allocate this ahead + * of time and use it for the single values where possible. + * (This is used the the normal search case, but not in the + * index case because of caller requirements). + */ + if (flags & LDB_UNPACK_DATA_FLAG_NO_VALUES_ALLOC) { + ldb_val_single_array = talloc_array(message->elements, struct ldb_val, + message->num_elements); + if (ldb_val_single_array == NULL) { + errno = ENOMEM; + goto failed; + } + } + for (i=0;i<message->num_elements;i++) { const char *attr = NULL; size_t attr_len; @@ -396,7 +417,9 @@ int ldb_unpack_data_only_attr_list_flags(struct ldb_context *ldb, p += attr_len + 1; element->num_values = pull_uint32(p, 0); element->values = NULL; - if (element->num_values != 0) { + if ((flags & LDB_UNPACK_DATA_FLAG_NO_VALUES_ALLOC) && element->num_values == 1) { + element->values = &ldb_val_single_array[nelem]; + } else if (element->num_values != 0) { element->values = talloc_array(message->elements, struct ldb_val, element->num_values); diff --git a/lib/ldb/include/ldb_module.h b/lib/ldb/include/ldb_module.h index 1c48590..833d5a8 100644 --- a/lib/ldb/include/ldb_module.h +++ b/lib/ldb/include/ldb_module.h @@ -412,8 +412,15 @@ int ldb_unpack_data(struct ldb_context *ldb, * Giving a NULL list (or a list_size of 0) unpacks all the attributes. * * Flags allow control of allocation, so that if - * LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC is specified, then values are - * not allocate, instead they point into the supplier constant buffer. + * LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC is specified, then data in values are + * not allocated, instead they point into the supplier constant buffer. + * + * If LDB_UNPACK_DATA_FLAG_NO_VALUES_ALLOC is specified, then values + * array are not allocated individually (for single-valued + * attributes), instead they point into a single buffer per message. + * + * LDB_UNPACK_DATA_FLAG_NO_VALUES_ALLOC is only valid when + * LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC is also specified. * * Likewise if LDB_UNPACK_DATA_FLAG_NO_DN is specified, the DN is omitted. */ @@ -425,7 +432,8 @@ int ldb_unpack_data_only_attr_list_flags(struct ldb_context *ldb, unsigned int flags, unsigned int *nb_elements_in_db); -#define LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC 0x0001 -#define LDB_UNPACK_DATA_FLAG_NO_DN 0x0002 +#define LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC 0x0001 +#define LDB_UNPACK_DATA_FLAG_NO_DN 0x0002 +#define LDB_UNPACK_DATA_FLAG_NO_VALUES_ALLOC 0x0004 #endif diff --git a/lib/ldb/ldb_tdb/ldb_index.c b/lib/ldb/ldb_tdb/ldb_index.c index 392b4a7..7924172 100644 --- a/lib/ldb/ldb_tdb/ldb_index.c +++ b/lib/ldb/ldb_tdb/ldb_index.c @@ -931,6 +931,7 @@ static int ltdb_index_filter(const struct dn_list *dn_list, { struct ldb_context *ldb; struct ldb_message *msg; + struct ldb_message *filtered_msg; unsigned int i; ldb = ldb_module_get_ctx(ac->module); @@ -951,7 +952,9 @@ static int ltdb_index_filter(const struct dn_list *dn_list, return LDB_ERR_OPERATIONS_ERROR; } - ret = ltdb_search_dn1(ac->module, dn, msg, 0); + ret = ltdb_search_dn1(ac->module, dn, msg, + LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC| + LDB_UNPACK_DATA_FLAG_NO_VALUES_ALLOC); talloc_free(dn); if (ret == LDB_ERR_NO_SUCH_OBJECT) { /* the record has disappeared? yes, this can happen */ @@ -977,14 +980,15 @@ static int ltdb_index_filter(const struct dn_list *dn_list, } /* filter the attributes that the user wants */ - ret = ltdb_filter_attrs(msg, ac->attrs); + ret = ltdb_filter_attrs(ac, msg, ac->attrs, &filtered_msg); + + talloc_free(msg); if (ret == -1) { - talloc_free(msg); return LDB_ERR_OPERATIONS_ERROR; } - ret = ldb_module_send_entry(ac->req, msg, NULL); + ret = ldb_module_send_entry(ac->req, filtered_msg, NULL); if (ret != LDB_SUCCESS) { /* Regardless of success or failure, the msg * is the callbacks responsiblity, and should diff --git a/lib/ldb/ldb_tdb/ldb_search.c b/lib/ldb/ldb_tdb/ldb_search.c index 133e5d4..373855f 100644 --- a/lib/ldb/ldb_tdb/ldb_search.c +++ b/lib/ldb/ldb_tdb/ldb_search.c @@ -385,81 +385,144 @@ int ltdb_add_attr_results(struct ldb_module *module, /* filter the specified list of attributes from a message - removing not requested attrs. + removing not requested attrs from the new message constructed. + + The reason this makes a new message is that the old one may not be + individually allocated, which is what our callers expect. + */ -int ltdb_filter_attrs(struct ldb_message *msg, const char * const *attrs) +int ltdb_filter_attrs(TALLOC_CTX *mem_ctx, + const struct ldb_message *msg, const char * const *attrs, + struct ldb_message **filtered_msg) { unsigned int i; - int keep_all = 0; - struct ldb_message_element *el2; + bool keep_all = false; + bool add_dn = false; uint32_t num_elements; + uint32_t elements_size; + struct ldb_message *msg2; + + msg2 = ldb_msg_new(mem_ctx); + if (msg2 == NULL) { + goto failed; + } + + msg2->dn = ldb_dn_copy(msg2, msg->dn); + if (msg2->dn == NULL) { + goto failed; + } if (attrs) { /* check for special attrs */ for (i = 0; attrs[i]; i++) { - if (strcmp(attrs[i], "*") == 0) { - keep_all = 1; + int cmp = strcmp(attrs[i], "*"); + if (cmp == 0) { + keep_all = true; break; } - - if (ldb_attr_cmp(attrs[i], "distinguishedName") == 0) { - if (msg_add_distinguished_name(msg) != 0) { - return -1; - } + cmp = ldb_attr_cmp(attrs[i], "distinguishedName"); + if (cmp == 0) { + add_dn = true; } } } else { - keep_all = 1; + keep_all = true; } - + if (keep_all) { - if (msg_add_distinguished_name(msg) != 0) { + add_dn = true; + elements_size = msg->num_elements + 1; + + /* Shortcuts for the simple cases */ + } else if (add_dn && i == 1) { + if (msg_add_distinguished_name(msg2) != 0) { return -1; } + *filtered_msg = msg2; + return 0; + } else if (i == 0) { + *filtered_msg = msg2; return 0; - } - el2 = talloc_array(msg, struct ldb_message_element, msg->num_elements); - if (el2 == NULL) { - return -1; + /* Otherwise we are copying at most as many element as we have attributes */ + } else { + elements_size = i; } + + msg2->elements = talloc_array(msg2, struct ldb_message_element, + elements_size); + if (msg2->elements == NULL) goto failed; + num_elements = 0; for (i = 0; i < msg->num_elements; i++) { + struct ldb_message_element *el = &msg->elements[i]; + struct ldb_message_element *el2 = &msg2->elements[num_elements]; unsigned int j; - int found = 0; - - for (j = 0; attrs[j]; j++) { - if (ldb_attr_cmp(msg->elements[i].name, attrs[j]) == 0) { - found = 1; - break; + + if (keep_all == false) { + bool found = false; + for (j = 0; attrs[j]; j++) { + int cmp = ldb_attr_cmp(el->name, attrs[j]); + if (cmp == 0) { + found = true; + break; + } + } + if (found == false) { + continue; + } + } + *el2 = *el; + el2->name = talloc_strdup(msg2->elements, el->name); + if (el2->name == NULL) { + goto failed; + } + el2->values = talloc_array(msg2->elements, struct ldb_val, el->num_values); + if (el2->values == NULL) { + goto failed; + } + for (j=0;j<el->num_values;j++) { + el2->values[j] = ldb_val_dup(el2->values, &el->values[j]); + if (el2->values[j].data == NULL && el->values[j].length != 0) { + goto failed; } } + num_elements++; - if (found) { - el2[num_elements] = msg->elements[i]; - talloc_steal(el2, el2[num_elements].name); - talloc_steal(el2, el2[num_elements].values); - num_elements++; + /* Pidginhole principle: we can't have more elements + * than the number of attributes if they are unique in + * the DB */ + if (num_elements > elements_size) { + goto failed; } } - talloc_free(msg->elements); + msg2->num_elements = num_elements; - if (num_elements > 0) { - msg->elements = talloc_realloc(msg, el2, struct ldb_message_element, - num_elements); - } else { - msg->elements = talloc_array(msg, struct ldb_message_element, 0); - talloc_free(el2); + if (add_dn) { + if (msg_add_distinguished_name(msg2) != 0) { + return -1; + } } - if (msg->elements == NULL) { - return -1; + + if (msg2->num_elements > 0) { + msg2->elements = talloc_realloc(msg2, msg2->elements, + struct ldb_message_element, + msg2->num_elements); + if (msg2->elements == NULL) { + return -1; + } + } else { + talloc_free(msg2->elements); + msg2->elements = NULL; } - msg->num_elements = num_elements; + *filtered_msg = msg2; return 0; +failed: + return -1; } /* @@ -469,13 +532,14 @@ static int search_func(struct tdb_context *tdb, TDB_DATA key, TDB_DATA data, voi { struct ldb_context *ldb; struct ltdb_context *ac; - struct ldb_message *msg; + struct ldb_message *msg, *filtered_msg; const struct ldb_val val = { .data = data.dptr, .length = data.dsize, }; int ret; bool matched; + unsigned int nb_elements_in_db; ac = talloc_get_type(state, struct ltdb_context); ldb = ldb_module_get_ctx(ac->module); @@ -492,7 +556,12 @@ static int search_func(struct tdb_context *tdb, TDB_DATA key, TDB_DATA data, voi } /* unpack the record */ - ret = ldb_unpack_data(ldb, &val, msg); + ret = ldb_unpack_data_only_attr_list_flags(ldb, &val, + msg, + NULL, 0, + LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC| + LDB_UNPACK_DATA_FLAG_NO_VALUES_ALLOC, + &nb_elements_in_db); if (ret == -1) { talloc_free(msg); ac->error = LDB_ERR_OPERATIONS_ERROR; @@ -523,15 +592,15 @@ static int search_func(struct tdb_context *tdb, TDB_DATA key, TDB_DATA data, voi } /* filter the attributes that the user wants */ - ret = ltdb_filter_attrs(msg, ac->attrs); + ret = ltdb_filter_attrs(ac, msg, ac->attrs, &filtered_msg); + talloc_free(msg); if (ret == -1) { - talloc_free(msg); ac->error = LDB_ERR_OPERATIONS_ERROR; return -1; } - ret = ldb_module_send_entry(ac->req, msg, NULL); + ret = ldb_module_send_entry(ac->req, filtered_msg, NULL); if (ret != LDB_SUCCESS) { ac->request_terminated = true; /* the callback failed, abort the operation */ diff --git a/lib/ldb/ldb_tdb/ldb_tdb.h b/lib/ldb/ldb_tdb/ldb_tdb.h index 7390a04..b5b78a9 100644 --- a/lib/ldb/ldb_tdb/ldb_tdb.h +++ b/lib/ldb/ldb_tdb/ldb_tdb.h @@ -109,7 +109,9 @@ int ltdb_add_attr_results(struct ldb_module *module, const char * const attrs[], unsigned int *count, struct ldb_message ***res); -int ltdb_filter_attrs(struct ldb_message *msg, const char * const *attrs); +int ltdb_filter_attrs(TALLOC_CTX *mem_ctx, + const struct ldb_message *msg, const char * const *attrs, + struct ldb_message **filtered_msg); int ltdb_search(struct ltdb_context *ctx); /* The following definitions come from lib/ldb/ldb_tdb/ldb_tdb.c */ diff --git a/script/autobuild.py b/script/autobuild.py index 419fe1a..b435790 100755 --- a/script/autobuild.py +++ b/script/autobuild.py @@ -28,6 +28,7 @@ builddirs = { "samba-ctdb" : ".", "samba-libs" : ".", "samba-static" : ".", + "samba-test-only" : ".", "ldb" : "lib/ldb", "tdb" : "lib/tdb", "talloc" : "lib/talloc", @@ -75,6 +76,10 @@ tasks = { ("check-clean-tree", "script/clean-source-tree.sh", "text/plain"), ("clean", "make clean", "text/plain") ], + "samba-test-only" : [ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab --abi-check-disable" + samba_configure_params, "text/plain"), + ("make", "make -j", "text/plain"), + ("test", "make test FAIL_IMMEDIATELY=1 TESTS=${TESTS}", "text/plain") ], + # Test cross-compile infrastructure "samba-xc" : [ ("configure-native", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"), ("configure-cross-execute", "./configure.developer -b ./bin-xe --cross-compile --cross-execute=script/identity_cc.sh" \ @@ -85,7 +90,7 @@ tasks = { # test build with -O3 -- catches extra warnings and bugs "samba-o3" : [ ("random-sleep", "../script/random-sleep.sh 60 600", "text/plain"), - ("configure", "ADDITIONAL_CFLAGS='-O3' ./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"), + ("configure", "ADDITIONAL_CFLAGS='-O3' ./configure.developer --with-selftest-prefix=./bin/ab --abi-check-disable" + samba_configure_params, "text/plain"), ("make", "make -j", "text/plain"), ("test", "make quicktest FAIL_IMMEDIATELY=1 TESTS='\(ad_dc\)'", "text/plain"), ("install", "make install", "text/plain"), @@ -278,6 +283,7 @@ class builder(object): self.cmd = self.cmd.replace("${PREFIX}", "--prefix=%s" % self.prefix) self.cmd = self.cmd.replace("${EXTRA_PYTHON}", "%s" % extra_python) self.cmd = self.cmd.replace("${PREFIX_DIR}", "%s" % self.prefix) + self.cmd = self.cmd.replace("${TESTS}", options.restrict_tests) # if self.output_mime_type == "text/x-subunit": # self.cmd += " | %s --immediate" % (os.path.join(os.path.dirname(__file__), "selftest/format-subunit")) print '%s: [%s] Running %s' % (self.name, self.stage, self.cmd) @@ -292,13 +298,16 @@ class builder(object): class buildlist(object): '''handle build of multiple directories''' - def __init__(self, tasklist, tasknames, rebase_url, rebase_branch="master"): + def __init__(self, tasknames, rebase_url, rebase_branch="master"): global tasks self.tlist = [] self.tail_proc = None self.retry = None if tasknames == []: - tasknames = defaulttasks + if options.restrict_tests: + tasknames = ["samba-test-only"] + else: + tasknames = defaulttasks else: # If we are only running one test, # do not sleep randomly to wait for it to start @@ -555,6 +564,8 @@ parser.add_option("", "--log-base", help="location where the logs can be found ( default=gitroot, type='str') parser.add_option("", "--attach-logs", help="Attach logs to mails sent on success/failure?", default=False, action="store_true") +parser.add_option("", "--restrict-tests", help="run as make test with this TESTS= regex", + default='') def send_email(subject, text, log_tar): outer = MIMEMultipart() @@ -599,6 +610,10 @@ A summary of the autobuild process is here: %s/autobuild.log ''' % (platform.node(), elapsed_minutes, failed_task, errstr, log_base) + if options.restrict_tests: + text += """ +The build was restricted to tests matching %s\n""" % options.restrict_tests + if failed_task != 'rebase': text += ''' You can see logs of the failed task here: @@ -650,6 +665,10 @@ Your autobuild on %s has succeeded after %.1f minutes. ''' % (platform.node(), elapsed_time / 60.) + if options.restrict_tests: -- Samba Shared Repository