The branch, v4-18-test has been updated via 35573c04465 mdssvc: fix returning file modification date for older Mac releases via f4a3373585f mdssvc: fix date marshalling via aa4db8a3cf8 mdssvc: prepare for returning timestamps with sub-seconds granularity via 590c91640bf mdssvc: reduce pagesize to 50 via 3036f3cb96f tests/mdssvc: match hits:total:value to be the actual amount of entries in hits via 39c2b556456 mdssvc: fix enforcement of "elasticsearch:max results" via 60fdb3adbee mdssvc: add and use SL_PAGESIZE via 7d5e9f5fc23 mdssvc: fix long running backend queries via 160b7dc05dc mdssvc: set query state for continued queries to SLQ_STATE_RUNNING from c1c2a0ec80d s3:winbindd: Fix double close(fd)
https://git.samba.org/?p=samba.git;a=shortlog;h=v4-18-test - Log ----------------------------------------------------------------- commit 35573c04465820f78642a22231a4040ca855e1e1 Author: Ralph Boehme <s...@samba.org> Date: Thu May 18 18:12:19 2023 +0200 mdssvc: fix returning file modification date for older Mac releases Mac 10.10 uses kMDItemContentModificationDate instead of kMDItemFSContentChangeDate. BUG: https://bugzilla.samba.org/show_bug.cgi?id=15427 Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> Autobuild-User(master): Jeremy Allison <j...@samba.org> Autobuild-Date(master): Wed Jul 26 23:42:44 UTC 2023 on atb-devel-224 (cherry picked from commit c2e83ebe726b7bc42b329198214c784936f19888) Autobuild-User(v4-18-test): Jule Anger <jan...@samba.org> Autobuild-Date(v4-18-test): Mon Jul 31 09:11:21 UTC 2023 on atb-devel-224 commit f4a3373585f0ba0b87763d35bb90054a983109c6 Author: Ralph Boehme <s...@samba.org> Date: Wed May 17 16:38:39 2023 +0200 mdssvc: fix date marshalling Did this ever work? Possible just copied over from Netatalk and was always broken... The Mac client expects the timevalue as seconds relative to 2001-01-01 00:00:00 UTC, packed as IEEE float. BUG: https://bugzilla.samba.org/show_bug.cgi?id=15427 Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> (cherry picked from commit 620ca1e68d02be45a94aa41217a141d211fceb1f) commit aa4db8a3cf82c278380600fa6ce293ac7a7379c3 Author: Ralph Boehme <s...@samba.org> Date: Wed May 17 16:37:36 2023 +0200 mdssvc: prepare for returning timestamps with sub-seconds granularity BUG: https://bugzilla.samba.org/show_bug.cgi?id=15427 Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> (cherry picked from commit 9dc66fecf7c1743d264c5c4f8978b77bab75ed86) commit 590c91640bfec9909a543eafeef8f52d5a42a89e Author: Ralph Boehme <s...@samba.org> Date: Thu Apr 20 17:27:20 2023 +0200 mdssvc: reduce pagesize to 50 Lastest macOS queries additional file metadata per search result, which causes the mashalled paged result set including metadata to exceed the 64 KB result fragment buffer. Lacking fragementation support in mdssvc (it's supported by the protocol), for now just reduce the maximum number of results per search page. BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342 Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> (cherry picked from commit 724a0518c901589fe1171d94648391832e056f4d) commit 3036f3cb96f1923949c7c73d2d3e9273539d0978 Author: Ralph Boehme <s...@samba.org> Date: Fri Apr 21 07:07:13 2023 +0200 tests/mdssvc: match hits:total:value to be the actual amount of entries in hits BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342 Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> (cherry picked from commit 7f5e4edf64f7e4175f652bf8762d4edc110ad6b1) commit 39c2b556456bd0a348148df72cafdecf8e3c4a6f Author: Ralph Boehme <s...@samba.org> Date: Thu Apr 20 17:58:38 2023 +0200 mdssvc: fix enforcement of "elasticsearch:max results" This wasn't enforced at all thus a query would return all available matches without limit. BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342 Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> (cherry picked from commit d8fa5c8e2a1794ea8dc663485315ebd9401b2628) commit 60fdb3adbee0130ecc993741688979d0a1104ee5 Author: Ralph Boehme <s...@samba.org> Date: Thu Apr 20 17:24:30 2023 +0200 mdssvc: add and use SL_PAGESIZE SL_PAGESIZE is the number of entries we want to process per paged search result set. This is different from MAX_SL_RESULTS which ought to be a default maximum value for total number of results returned for a search query. BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342 Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> (cherry picked from commit 086c2602d074d4dc0d44f5534857e5f59a8690b2) commit 7d5e9f5fc23d5be2a6b38ee04548378ff8a25ea0 Author: Ralph Boehme <s...@samba.org> Date: Thu Mar 23 16:39:11 2023 +0100 mdssvc: fix long running backend queries If a query is still running in the backend and we have no results yet, returning 0 triggers a search termination by the client in latest macOS releases. macOS returns 0x23 in this case. BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342 Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> (cherry picked from commit 925fefae20e52a3c89a56bdd0cd5b98cc951db5f) commit 160b7dc05dc666dc642bbb17d7a338db17a49168 Author: Ralph Boehme <s...@samba.org> Date: Wed Apr 19 14:38:45 2023 +0200 mdssvc: set query state for continued queries to SLQ_STATE_RUNNING SLQ_STATE_RESULTS implies that there are already results attached to the slq which is not the case. Instead the backend will start processing from where it left off when it hits the maximum result limit and had set the state to SLQ_STATE_FULL. BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342 Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> (cherry picked from commit 4149ef97e5906604be1587622f390f121db183e2) ----------------------------------------------------------------------- Summary of changes: python/samba/tests/blackbox/mdsearch.py | 2 +- python/samba/tests/dcerpc/mdssvc.py | 6 +++--- source3/rpc_server/mdssvc/marshalling.c | 29 +++++++++++++++++++++++------ source3/rpc_server/mdssvc/mdssvc.c | 21 +++++++++++++++++---- source3/rpc_server/mdssvc/mdssvc.h | 1 + source3/rpc_server/mdssvc/mdssvc_es.c | 8 ++++---- 6 files changed, 49 insertions(+), 18 deletions(-) Changeset truncated at 500 lines: diff --git a/python/samba/tests/blackbox/mdsearch.py b/python/samba/tests/blackbox/mdsearch.py index c8e75661f15..46af6f94624 100644 --- a/python/samba/tests/blackbox/mdsearch.py +++ b/python/samba/tests/blackbox/mdsearch.py @@ -100,7 +100,7 @@ class MdfindBlackboxTests(BlackboxTestCase): config = os.environ["SMB_CONF_PATH"] json_in = r'''{ - "from": 0, "size": 100, "_source": ["path.real"], + "from": 0, "size": 50, "_source": ["path.real"], "query": { "query_string": { "query": "(samba*) AND path.real.fulltext:\"%BASEPATH%\"" diff --git a/python/samba/tests/dcerpc/mdssvc.py b/python/samba/tests/dcerpc/mdssvc.py index 5002e5d26d6..7d09a555936 100644 --- a/python/samba/tests/dcerpc/mdssvc.py +++ b/python/samba/tests/dcerpc/mdssvc.py @@ -125,7 +125,7 @@ class MdssvcTests(RpcInterfaceTestCase): def test_mdscli_search(self): exp_json_query = r'''{ - "from": 0, "size": 100, "_source": ["path.real"], + "from": 0, "size": 50, "_source": ["path.real"], "query": { "query_string": { "query": "(samba*) AND path.real.fulltext:\"%BASEPATH%\"" @@ -157,7 +157,7 @@ class MdssvcTests(RpcInterfaceTestCase): r'kMDItemFSName=="x\\x"' ) exp_json_query = r'''{ - "from": 0, "size": 100, "_source": ["path.real"], + "from": 0, "size": 50, "_source": ["path.real"], "query": { "query_string": { "query": "(file.filename:x\\+x OR file.filename:x\\*x OR file.filename:x=x OR file.filename:x'x OR file.filename:x\\?x OR file.filename:x\\ x OR file.filename:x\\(x OR file.filename:x\\\"x OR file.filename:x\\\\x) AND path.real.fulltext:\"%BASEPATH%\"" @@ -166,7 +166,7 @@ class MdssvcTests(RpcInterfaceTestCase): }''' fake_json_response = r'''{ "hits" : { - "total" : {"value" : 2}, + "total" : {"value" : 9}, "hits" : [ {"_source" : {"path" : {"real" : "%BASEPATH%/x+x"}}}, {"_source" : {"path" : {"real" : "%BASEPATH%/x*x"}}}, diff --git a/source3/rpc_server/mdssvc/marshalling.c b/source3/rpc_server/mdssvc/marshalling.c index 5f866d7fb6e..b3e16d9c18f 100644 --- a/source3/rpc_server/mdssvc/marshalling.c +++ b/source3/rpc_server/mdssvc/marshalling.c @@ -43,8 +43,8 @@ * RPC data marshalling and unmarshalling ******************************************************************************/ -/* Spotlight epoch is UNIX epoch minus SPOTLIGHT_TIME_DELTA */ -#define SPOTLIGHT_TIME_DELTA 280878921600ULL +/* Spotlight epoch is 1.1.2001 00:00 UTC */ +#define SPOTLIGHT_TIME_DELTA 978307200 /* Diff from UNIX epoch to Spotlight epoch */ #define SQ_TYPE_NULL 0x0000 #define SQ_TYPE_COMPLEX 0x0200 @@ -253,6 +253,10 @@ static ssize_t sl_pack_date(sl_time_t t, char *buf, ssize_t offset, size_t bufsi { uint64_t data; uint64_t tag; + union { + double d; + uint64_t w; + } ieee_fp_union; tag = sl_pack_tag(SQ_TYPE_DATE, 2, 1); offset = sl_push_uint64_val(buf, offset, bufsize, tag); @@ -260,7 +264,10 @@ static ssize_t sl_pack_date(sl_time_t t, char *buf, ssize_t offset, size_t bufsi return -1; } - data = (t.tv_sec + SPOTLIGHT_TIME_DELTA) << 24; + ieee_fp_union.d = (double)(t.tv_sec - SPOTLIGHT_TIME_DELTA); + ieee_fp_union.d += (double)t.tv_usec / 1000000; + + data = ieee_fp_union.w; offset = sl_push_uint64_val(buf, offset, bufsize, data); if (offset == -1) { return -1; @@ -723,6 +730,11 @@ static int sl_unpack_date(DALLOC_CTX *query, int i, result; struct sl_tag tag; uint64_t query_data64; + union { + double d; + uint64_t w; + } ieee_fp_union; + double fraction; sl_time_t t; offset = sl_unpack_tag(buf, offset, bufsize, encoding, &tag); @@ -735,9 +747,14 @@ static int sl_unpack_date(DALLOC_CTX *query, if (offset == -1) { return -1; } - query_data64 = query_data64 >> 24; - t.tv_sec = query_data64 - SPOTLIGHT_TIME_DELTA; - t.tv_usec = 0; + ieee_fp_union.w = query_data64; + fraction = ieee_fp_union.d - (uint64_t)ieee_fp_union.d; + + t = (sl_time_t) { + .tv_sec = ieee_fp_union.d + SPOTLIGHT_TIME_DELTA, + .tv_usec = fraction * 1000000 + }; + result = dalloc_add_copy(query, &t, sl_time_t); if (result != 0) { return -1; diff --git a/source3/rpc_server/mdssvc/mdssvc.c b/source3/rpc_server/mdssvc/mdssvc.c index 22cc7abd153..a7d5d7afc02 100644 --- a/source3/rpc_server/mdssvc/mdssvc.c +++ b/source3/rpc_server/mdssvc/mdssvc.c @@ -188,8 +188,10 @@ static bool add_filemeta(struct mds_ctx *mds_ctx, if (result != 0) { return false; } - } else if (strcmp(attribute, "kMDItemFSContentChangeDate") == 0) { - sl_time.tv_sec = sp->st_ex_mtime.tv_sec; + } else if (strcmp(attribute, "kMDItemFSContentChangeDate") == 0 || + strcmp(attribute, "kMDItemContentModificationDate") == 0) + { + sl_time = convert_timespec_to_timeval(sp->st_ex_mtime); result = dalloc_add_copy(meta, &sl_time, sl_time_t); if (result != 0) { return false; @@ -306,10 +308,21 @@ static bool create_result_handle(struct sl_query *slq) static bool add_results(sl_array_t *array, struct sl_query *slq) { sl_filemeta_t *fm; - uint64_t status = 0; + uint64_t status; int result; bool ok; + /* + * Taken from a network trace against a macOS SMB Spotlight server. If + * the first fetch-query-results has no results yet because the search + * is still running, macOS returns 0x23, otherwise 0x0. + */ + if (slq->state >= SLQ_STATE_RESULTS ) { + status = 0; + } else { + status = 0x23; + } + /* FileMeta */ fm = dalloc_zero(array, sl_filemeta_t); if (fm == NULL) { @@ -1126,7 +1139,7 @@ static bool slrpc_fetch_query_results(struct mds_ctx *mds_ctx, goto error; } if (slq->state == SLQ_STATE_FULL) { - slq->state = SLQ_STATE_RESULTS; + slq->state = SLQ_STATE_RUNNING; slq->mds_ctx->backend->search_cont(slq); } break; diff --git a/source3/rpc_server/mdssvc/mdssvc.h b/source3/rpc_server/mdssvc/mdssvc.h index 3b2ce250f1f..8aabf5b86e5 100644 --- a/source3/rpc_server/mdssvc/mdssvc.h +++ b/source3/rpc_server/mdssvc/mdssvc.h @@ -36,6 +36,7 @@ #define MAX_SL_FRAGMENT_SIZE 0xFFFFF #define MAX_SL_RESULTS 100 +#define SL_PAGESIZE 50 #define MAX_SL_RUNTIME 30 #define MDS_TRACKER_ASYNC_TIMEOUT_MS 250 diff --git a/source3/rpc_server/mdssvc/mdssvc_es.c b/source3/rpc_server/mdssvc/mdssvc_es.c index dafb42610fa..8460b48b80a 100644 --- a/source3/rpc_server/mdssvc/mdssvc_es.c +++ b/source3/rpc_server/mdssvc/mdssvc_es.c @@ -398,7 +398,7 @@ static bool mds_es_search(struct sl_query *slq) .ev = mds_es_ctx->mdssvc_es_ctx->mdssvc_ctx->ev_ctx, .mds_es_ctx = mds_es_ctx, .slq = slq, - .size = MAX_SL_RESULTS, + .size = SL_PAGESIZE, }; /* 0 would mean no limit */ @@ -502,7 +502,7 @@ static void mds_es_search_done(struct tevent_req *subreq) goto trigger; } - if (slq->query_results->num_results >= MAX_SL_RESULTS) { + if (slq->query_results->num_results >= SL_PAGESIZE) { slq->state = SLQ_STATE_FULL; goto trigger; } @@ -693,7 +693,7 @@ static void mds_es_search_http_send_done(struct tevent_req *subreq) subreq = http_read_response_send(state, state->ev, state->s->mds_es_ctx->http_conn, - MAX_SL_RESULTS * 8192); + SL_PAGESIZE * 8192); if (tevent_req_nomem(subreq, req)) { return; } @@ -800,7 +800,7 @@ static void mds_es_search_http_read_done(struct tevent_req *subreq) } DBG_DEBUG("Hits: %zu\n", hits); - for (i = 0; i < hits; i++) { + for (i = 0; i < hits && s->from + i < s->max; i++) { const char *path = NULL; match = json_array_get(matches, i); -- Samba Shared Repository