The branch, master has been updated via c2e83ebe726 mdssvc: fix returning file modification date for older Mac releases via 620ca1e68d0 mdssvc: fix date marshalling via 9dc66fecf7c mdssvc: prepare for returning timestamps with sub-seconds granularity via 724a0518c90 mdssvc: reduce pagesize to 50 via 7f5e4edf64f tests/mdssvc: match hits:total:value to be the actual amount of entries in hits via d8fa5c8e2a1 mdssvc: fix enforcement of "elasticsearch:max results" via 086c2602d07 mdssvc: add and use SL_PAGESIZE via 925fefae20e mdssvc: fix long running backend queries via 4149ef97e59 mdssvc: set query state for continued queries to SLQ_STATE_RUNNING from e86e0da9de6 WHATSNEW: Add TLS cert reload feature
https://git.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit c2e83ebe726b7bc42b329198214c784936f19888 Author: Ralph Boehme <s...@samba.org> Date: Thu May 18 18:12:19 2023 +0200 mdssvc: fix returning file modification date for older Mac releases Mac 10.10 uses kMDItemContentModificationDate instead of kMDItemFSContentChangeDate. BUG: https://bugzilla.samba.org/show_bug.cgi?id=15427 Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> Autobuild-User(master): Jeremy Allison <j...@samba.org> Autobuild-Date(master): Wed Jul 26 23:42:44 UTC 2023 on atb-devel-224 commit 620ca1e68d02be45a94aa41217a141d211fceb1f Author: Ralph Boehme <s...@samba.org> Date: Wed May 17 16:38:39 2023 +0200 mdssvc: fix date marshalling Did this ever work? Possible just copied over from Netatalk and was always broken... The Mac client expects the timevalue as seconds relative to 2001-01-01 00:00:00 UTC, packed as IEEE float. BUG: https://bugzilla.samba.org/show_bug.cgi?id=15427 Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 9dc66fecf7c1743d264c5c4f8978b77bab75ed86 Author: Ralph Boehme <s...@samba.org> Date: Wed May 17 16:37:36 2023 +0200 mdssvc: prepare for returning timestamps with sub-seconds granularity BUG: https://bugzilla.samba.org/show_bug.cgi?id=15427 Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 724a0518c901589fe1171d94648391832e056f4d Author: Ralph Boehme <s...@samba.org> Date: Thu Apr 20 17:27:20 2023 +0200 mdssvc: reduce pagesize to 50 Lastest macOS queries additional file metadata per search result, which causes the mashalled paged result set including metadata to exceed the 64 KB result fragment buffer. Lacking fragementation support in mdssvc (it's supported by the protocol), for now just reduce the maximum number of results per search page. BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342 Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 7f5e4edf64f7e4175f652bf8762d4edc110ad6b1 Author: Ralph Boehme <s...@samba.org> Date: Fri Apr 21 07:07:13 2023 +0200 tests/mdssvc: match hits:total:value to be the actual amount of entries in hits BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342 Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit d8fa5c8e2a1794ea8dc663485315ebd9401b2628 Author: Ralph Boehme <s...@samba.org> Date: Thu Apr 20 17:58:38 2023 +0200 mdssvc: fix enforcement of "elasticsearch:max results" This wasn't enforced at all thus a query would return all available matches without limit. BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342 Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 086c2602d074d4dc0d44f5534857e5f59a8690b2 Author: Ralph Boehme <s...@samba.org> Date: Thu Apr 20 17:24:30 2023 +0200 mdssvc: add and use SL_PAGESIZE SL_PAGESIZE is the number of entries we want to process per paged search result set. This is different from MAX_SL_RESULTS which ought to be a default maximum value for total number of results returned for a search query. BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342 Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 925fefae20e52a3c89a56bdd0cd5b98cc951db5f Author: Ralph Boehme <s...@samba.org> Date: Thu Mar 23 16:39:11 2023 +0100 mdssvc: fix long running backend queries If a query is still running in the backend and we have no results yet, returning 0 triggers a search termination by the client in latest macOS releases. macOS returns 0x23 in this case. BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342 Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 4149ef97e5906604be1587622f390f121db183e2 Author: Ralph Boehme <s...@samba.org> Date: Wed Apr 19 14:38:45 2023 +0200 mdssvc: set query state for continued queries to SLQ_STATE_RUNNING SLQ_STATE_RESULTS implies that there are already results attached to the slq which is not the case. Instead the backend will start processing from where it left off when it hits the maximum result limit and had set the state to SLQ_STATE_FULL. BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342 Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> ----------------------------------------------------------------------- Summary of changes: python/samba/tests/blackbox/mdsearch.py | 2 +- python/samba/tests/dcerpc/mdssvc.py | 6 +++--- source3/rpc_server/mdssvc/marshalling.c | 29 +++++++++++++++++++++++------ source3/rpc_server/mdssvc/mdssvc.c | 21 +++++++++++++++++---- source3/rpc_server/mdssvc/mdssvc.h | 1 + source3/rpc_server/mdssvc/mdssvc_es.c | 8 ++++---- 6 files changed, 49 insertions(+), 18 deletions(-) Changeset truncated at 500 lines: diff --git a/python/samba/tests/blackbox/mdsearch.py b/python/samba/tests/blackbox/mdsearch.py index c8e75661f15..46af6f94624 100644 --- a/python/samba/tests/blackbox/mdsearch.py +++ b/python/samba/tests/blackbox/mdsearch.py @@ -100,7 +100,7 @@ class MdfindBlackboxTests(BlackboxTestCase): config = os.environ["SMB_CONF_PATH"] json_in = r'''{ - "from": 0, "size": 100, "_source": ["path.real"], + "from": 0, "size": 50, "_source": ["path.real"], "query": { "query_string": { "query": "(samba*) AND path.real.fulltext:\"%BASEPATH%\"" diff --git a/python/samba/tests/dcerpc/mdssvc.py b/python/samba/tests/dcerpc/mdssvc.py index 5002e5d26d6..7d09a555936 100644 --- a/python/samba/tests/dcerpc/mdssvc.py +++ b/python/samba/tests/dcerpc/mdssvc.py @@ -125,7 +125,7 @@ class MdssvcTests(RpcInterfaceTestCase): def test_mdscli_search(self): exp_json_query = r'''{ - "from": 0, "size": 100, "_source": ["path.real"], + "from": 0, "size": 50, "_source": ["path.real"], "query": { "query_string": { "query": "(samba*) AND path.real.fulltext:\"%BASEPATH%\"" @@ -157,7 +157,7 @@ class MdssvcTests(RpcInterfaceTestCase): r'kMDItemFSName=="x\\x"' ) exp_json_query = r'''{ - "from": 0, "size": 100, "_source": ["path.real"], + "from": 0, "size": 50, "_source": ["path.real"], "query": { "query_string": { "query": "(file.filename:x\\+x OR file.filename:x\\*x OR file.filename:x=x OR file.filename:x'x OR file.filename:x\\?x OR file.filename:x\\ x OR file.filename:x\\(x OR file.filename:x\\\"x OR file.filename:x\\\\x) AND path.real.fulltext:\"%BASEPATH%\"" @@ -166,7 +166,7 @@ class MdssvcTests(RpcInterfaceTestCase): }''' fake_json_response = r'''{ "hits" : { - "total" : {"value" : 2}, + "total" : {"value" : 9}, "hits" : [ {"_source" : {"path" : {"real" : "%BASEPATH%/x+x"}}}, {"_source" : {"path" : {"real" : "%BASEPATH%/x*x"}}}, diff --git a/source3/rpc_server/mdssvc/marshalling.c b/source3/rpc_server/mdssvc/marshalling.c index 3bbd5889aaa..0e323fa979e 100644 --- a/source3/rpc_server/mdssvc/marshalling.c +++ b/source3/rpc_server/mdssvc/marshalling.c @@ -43,8 +43,8 @@ * RPC data marshalling and unmarshalling ******************************************************************************/ -/* Spotlight epoch is UNIX epoch minus SPOTLIGHT_TIME_DELTA */ -#define SPOTLIGHT_TIME_DELTA 280878921600ULL +/* Spotlight epoch is 1.1.2001 00:00 UTC */ +#define SPOTLIGHT_TIME_DELTA 978307200 /* Diff from UNIX epoch to Spotlight epoch */ #define SQ_TYPE_NULL 0x0000 #define SQ_TYPE_COMPLEX 0x0200 @@ -253,6 +253,10 @@ static ssize_t sl_pack_date(sl_time_t t, char *buf, ssize_t offset, size_t bufsi { uint64_t data; uint64_t tag; + union { + double d; + uint64_t w; + } ieee_fp_union; tag = sl_pack_tag(SQ_TYPE_DATE, 2, 1); offset = sl_push_uint64_val(buf, offset, bufsize, tag); @@ -260,7 +264,10 @@ static ssize_t sl_pack_date(sl_time_t t, char *buf, ssize_t offset, size_t bufsi return -1; } - data = (t.tv_sec + SPOTLIGHT_TIME_DELTA) << 24; + ieee_fp_union.d = (double)(t.tv_sec - SPOTLIGHT_TIME_DELTA); + ieee_fp_union.d += (double)t.tv_usec / 1000000; + + data = ieee_fp_union.w; offset = sl_push_uint64_val(buf, offset, bufsize, data); if (offset == -1) { return -1; @@ -723,6 +730,11 @@ static int sl_unpack_date(DALLOC_CTX *query, int i, result; struct sl_tag tag; uint64_t query_data64; + union { + double d; + uint64_t w; + } ieee_fp_union; + double fraction; sl_time_t t; offset = sl_unpack_tag(buf, offset, bufsize, encoding, &tag); @@ -735,9 +747,14 @@ static int sl_unpack_date(DALLOC_CTX *query, if (offset == -1) { return -1; } - query_data64 = query_data64 >> 24; - t.tv_sec = query_data64 - SPOTLIGHT_TIME_DELTA; - t.tv_usec = 0; + ieee_fp_union.w = query_data64; + fraction = ieee_fp_union.d - (uint64_t)ieee_fp_union.d; + + t = (sl_time_t) { + .tv_sec = ieee_fp_union.d + SPOTLIGHT_TIME_DELTA, + .tv_usec = fraction * 1000000 + }; + result = dalloc_add_copy(query, &t, sl_time_t); if (result != 0) { return -1; diff --git a/source3/rpc_server/mdssvc/mdssvc.c b/source3/rpc_server/mdssvc/mdssvc.c index 335b267c0e5..abeaac8263c 100644 --- a/source3/rpc_server/mdssvc/mdssvc.c +++ b/source3/rpc_server/mdssvc/mdssvc.c @@ -188,8 +188,10 @@ static bool add_filemeta(struct mds_ctx *mds_ctx, if (result != 0) { return false; } - } else if (strcmp(attribute, "kMDItemFSContentChangeDate") == 0) { - sl_time.tv_sec = sp->st_ex_mtime.tv_sec; + } else if (strcmp(attribute, "kMDItemFSContentChangeDate") == 0 || + strcmp(attribute, "kMDItemContentModificationDate") == 0) + { + sl_time = convert_timespec_to_timeval(sp->st_ex_mtime); result = dalloc_add_copy(meta, &sl_time, sl_time_t); if (result != 0) { return false; @@ -306,10 +308,21 @@ static bool create_result_handle(struct sl_query *slq) static bool add_results(sl_array_t *array, struct sl_query *slq) { sl_filemeta_t *fm; - uint64_t status = 0; + uint64_t status; int result; bool ok; + /* + * Taken from a network trace against a macOS SMB Spotlight server. If + * the first fetch-query-results has no results yet because the search + * is still running, macOS returns 0x23, otherwise 0x0. + */ + if (slq->state >= SLQ_STATE_RESULTS ) { + status = 0; + } else { + status = 0x23; + } + /* FileMeta */ fm = dalloc_zero(array, sl_filemeta_t); if (fm == NULL) { @@ -1126,7 +1139,7 @@ static bool slrpc_fetch_query_results(struct mds_ctx *mds_ctx, goto error; } if (slq->state == SLQ_STATE_FULL) { - slq->state = SLQ_STATE_RESULTS; + slq->state = SLQ_STATE_RUNNING; slq->mds_ctx->backend->search_cont(slq); } break; diff --git a/source3/rpc_server/mdssvc/mdssvc.h b/source3/rpc_server/mdssvc/mdssvc.h index 5836caff740..6d4e684ae8d 100644 --- a/source3/rpc_server/mdssvc/mdssvc.h +++ b/source3/rpc_server/mdssvc/mdssvc.h @@ -35,6 +35,7 @@ #undef FALSE #define MAX_SL_RESULTS 100 +#define SL_PAGESIZE 50 #define MAX_SL_RUNTIME 30 #define MDS_TRACKER_ASYNC_TIMEOUT_MS 250 diff --git a/source3/rpc_server/mdssvc/mdssvc_es.c b/source3/rpc_server/mdssvc/mdssvc_es.c index dafb42610fa..8460b48b80a 100644 --- a/source3/rpc_server/mdssvc/mdssvc_es.c +++ b/source3/rpc_server/mdssvc/mdssvc_es.c @@ -398,7 +398,7 @@ static bool mds_es_search(struct sl_query *slq) .ev = mds_es_ctx->mdssvc_es_ctx->mdssvc_ctx->ev_ctx, .mds_es_ctx = mds_es_ctx, .slq = slq, - .size = MAX_SL_RESULTS, + .size = SL_PAGESIZE, }; /* 0 would mean no limit */ @@ -502,7 +502,7 @@ static void mds_es_search_done(struct tevent_req *subreq) goto trigger; } - if (slq->query_results->num_results >= MAX_SL_RESULTS) { + if (slq->query_results->num_results >= SL_PAGESIZE) { slq->state = SLQ_STATE_FULL; goto trigger; } @@ -693,7 +693,7 @@ static void mds_es_search_http_send_done(struct tevent_req *subreq) subreq = http_read_response_send(state, state->ev, state->s->mds_es_ctx->http_conn, - MAX_SL_RESULTS * 8192); + SL_PAGESIZE * 8192); if (tevent_req_nomem(subreq, req)) { return; } @@ -800,7 +800,7 @@ static void mds_es_search_http_read_done(struct tevent_req *subreq) } DBG_DEBUG("Hits: %zu\n", hits); - for (i = 0; i < hits; i++) { + for (i = 0; i < hits && s->from + i < s->max; i++) { const char *path = NULL; match = json_array_get(matches, i); -- Samba Shared Repository