The branch, master has been updated via 64f182412a5 s3:mdssvc: fix slrpc_fetch_attributes() when CNID is not known via b2bf13ecf7b s3:mdssvc: close mdssvc rpc command must return in handle via a5e705504bf s3:mdssvc: failing the RPC request if the mdssvc policy handle is not found via 63366996879 s3:mdssvc: the open command must work on shares with Spotlight disabled via 520175e2275 s3:mdssvc: use an early return if spotlight is not enabled. via 68a9d1205e5 s3:mdssvc: update a few debug macros to modern style via 940c3b31dd1 s3:mdssvv: don't fail the RPC request if the share name is unknown via fef3a4c75f9 s3:mdssvc: initialize the returned share_path with the empty string via c4b2dc95d5d s3:mdssvc: macOS returns the client values in the open command via a3cb4fca864 s3:mdssvc: macOS returns UINT64_MAX, not 0 for closeQueryForContext mdscmd via 5750f38ca78 s3:mdssvc: fix error handling of mdssvc RPC requests via 017af5d5831 torture: beginning of a mdssvc RPC service test-suite via b45a09f0e1a rpcclient: add some simple commands for Spotlight RPC via 2cb978b6498 rpcclient: domain_sid is only needed for SAMR via 4162ba78f81 lib:torture: add torture_assert_u32_[not_]equal[_goto] macros via b34fd5b9978 s3:mdssvc: fix unmarshalling of empty CNID array via 29ef0306241 s3:mdssvc: marshalling: fix unpacking empty CNID structure via d67c98184a1 s3:mdssvc: marshalling: fix unpacking empty filemeta structure via ed37f2d3c47 s3:mdssvc: marshalling: add a newline to a DEBUG message via c282d76d556 torture: start of a mdssvc packet (un)marshalling testsuite via 9d6794df728 librpc: add RPC_NDR_MDSSVC via 12ef7b38437 s3:wscript: enable Spotlight by default via 25c5012c53f s3:mdssvc: avoid strncpy when marshalling strings via 9e706ee5737 unittest: workaround dependency problem in test_lib_util_modules via 0b1d9d40c24 s3:mdssvc: supposed status field is in fact a fragment indicator via c742ab7a4c7 s3:mdssvc: add noindex backend via e5a4114bb56 s3:mdssvc: log inode number in ino_path_map_destr_cb() via 2d98fb57e05 s3:mdssvc: move some code to a subsystem via f58026b30b2 s3:mdssvc: mds_dalloc_dump() -> dalloc_dump() via b5731a94bbc s3:mdssvc: add Unicode normalisation via 1b897df4f74 s3:mdssvc: new option "spotlight backend" via 66b8d86d715 s3:mdssvc: pass down sharename and SNUM via 1ef96e09071 s3-mdssvc: factor out Tracker backend logic via f80f8c5f99f s3:mdssvc: move debug macro to header file via 17f29fa8677 s3:mdssvc: README.Coding fixes, long lines, modernize debug macros via 9e0b6f667f0 s3:rpc_server: factor out rpc_mdssvc_sources from 3bf915846f7 tfork: Free memory in torture test to satisfy sanitizer
https://git.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit 64f182412a5b7bdb1f8eccffaee296e7c8554a95 Author: Ralph Boehme <s...@samba.org> Date: Mon May 6 14:19:31 2019 +0200 s3:mdssvc: fix slrpc_fetch_attributes() when CNID is not known Samba currenlty fails the whole RPC request, macOS returns returns a nil entry for the requested CNID: DALLOC_CTX(#1): { sl_array_t(#3): { uint64_t: 0x0000 CNIDs: unkn1: 0xfec, unkn2: 0x6b000020 DALLOC_CTX(#1): { uint64_t: 0xe4bbf314c03b1e } sl_filemeta_t(#1): { sl_array_t(#2): { nil nil } } } } Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> Autobuild-User(master): Jeremy Allison <j...@samba.org> Autobuild-Date(master): Thu Aug 8 21:43:14 UTC 2019 on sn-devel-184 commit b2bf13ecf7b5406665f27487cb892e4f15fb28c4 Author: Ralph Boehme <s...@samba.org> Date: Mon May 6 14:18:27 2019 +0200 s3:mdssvc: close mdssvc rpc command must return in handle Checked against macOS mdssvc. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit a5e705504bf5a9cade2e57a18b068f654fa27a32 Author: Ralph Boehme <s...@samba.org> Date: Mon May 6 14:14:26 2019 +0200 s3:mdssvc: failing the RPC request if the mdssvc policy handle is not found Turns out macOS mdssvc doesn't fail the RPC request if the policy handle is all zero. Also, if it fails with a non-all-zero handle, it returns a different RPC error, namely DCERPC_NCA_S_PROTO_ERROR, not DCERPC_FAULT_CONTEXT_MISMATCH (or rather their mapped NT_STATUS codes). Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 63366996879fab5b1729de6d6224ca167bdb05b0 Author: Ralph Boehme <s...@samba.org> Date: Mon May 6 14:11:31 2019 +0200 s3:mdssvc: the open command must work on shares with Spotlight disabled Move the implementation of this setting down to the actual search query processing. macOS has no notion of "spotlight = false" at the DCERPC layer and the open request will always succeed even on all shares. When later the client issues search requests on such shares, we ensure we use the noindex backend. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 520175e2275abfabfdb5a62e192c4936a6c74d35 Author: Ralph Boehme <s...@samba.org> Date: Tue Apr 16 19:32:52 2019 +0200 s3:mdssvc: use an early return if spotlight is not enabled. No change in behaviour, best viewed with git show -w. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 68a9d1205e54d2f447baad5481ff20dfb184dfe6 Author: Ralph Boehme <s...@samba.org> Date: Wed Apr 17 07:38:53 2019 +0200 s3:mdssvc: update a few debug macros to modern style Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 940c3b31dd1fd68c9d7c9f4e9989d725dc77af4a Author: Ralph Boehme <s...@samba.org> Date: Mon May 6 14:07:53 2019 +0200 s3:mdssvv: don't fail the RPC request if the share name is unknown Taken from macOS. We have to return an empty share_path and an empty policy handle, but not fail the RPC request. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit fef3a4c75f9cad1a81201257f0208b241a93f6c9 Author: Ralph Boehme <s...@samba.org> Date: Tue Apr 16 19:25:09 2019 +0200 s3:mdssvc: initialize the returned share_path with the empty string macOS returns the empty path for an unknown share. This paves the way for that change. Currently we still fail the RPC request if the share is not known with DCERPC_FAULT_CANT_PERFORM, but this is wrong and is going to be changed in the next commit. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit c4b2dc95d5dfb4e875ea58ff6568b89dacf8b916 Author: Ralph Boehme <s...@samba.org> Date: Tue Apr 16 19:23:25 2019 +0200 s3:mdssvc: macOS returns the client values in the open command Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit a3cb4fca864b6e872f453d60b5e58591da3624b1 Author: Ralph Boehme <s...@samba.org> Date: Tue Apr 16 19:14:39 2019 +0200 s3:mdssvc: macOS returns UINT64_MAX, not 0 for closeQueryForContext mdscmd Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 5750f38ca7834254268a93193d72184d03990c1b Author: Ralph Boehme <s...@samba.org> Date: Tue Apr 16 14:17:11 2019 +0200 s3:mdssvc: fix error handling of mdssvc RPC requests It seems for certain error cases macOS just sends an empty response blob. So if our mdssvc request processing fails, we should just return an empty response blob, but not fail the mdssvc request at the DCERPC layer. Example, passing "xxx" as sharename which does not exist at the server: $ bin/rpcclient -U slow%pass macmini -c "fetch_attributes xxx /foo/bar 123" -d 10 .... Got pdu len 56, data_len 32 rpc_api_pipe: got frag len of 56 at offset 0: NT_STATUS_OK rpc_api_pipe: host macmini returned 32 bytes. mdssvc_cmd: struct mdssvc_cmd out: struct mdssvc_cmd fragment : * fragment : 0x00000000 (0) response_blob : * response_blob: struct mdssvc_blob length : 0x00000000 (0) size : 0x00010000 (65536) spotlight_blob : * spotlight_blob: ARRAY(0) unkn9 : * unkn9 : 0x00000000 (0) ... Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 017af5d583175ff9f15c64fa7cc631eb4512367b Author: Ralph Boehme <s...@samba.org> Date: Tue Apr 16 16:54:21 2019 +0200 torture: beginning of a mdssvc RPC service test-suite Yikes! Most tests fail atm. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit b45a09f0e1ada1b5fa4a16fb652871f9ae973e2e Author: Ralph Boehme <s...@samba.org> Date: Sun Mar 8 07:57:57 2015 +0100 rpcclient: add some simple commands for Spotlight RPC Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 2cb978b6498c42947377a50fa8a5748f596d9114 Author: Ralph Boehme <s...@samba.org> Date: Fri Nov 16 16:07:13 2018 +0100 rpcclient: domain_sid is only needed for SAMR This allows using rpcclient commands with servers that don't implement LSA QueryInfoPolicy. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 4162ba78f8146ad9b39d0749a36fab674197c78e Author: Ralph Boehme <s...@samba.org> Date: Tue Apr 16 16:46:43 2019 +0200 lib:torture: add torture_assert_u32_[not_]equal[_goto] macros Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit b34fd5b9978904ef0b316ba11564691fdac5316c Author: Ralph Boehme <s...@samba.org> Date: Wed Aug 7 14:02:12 2019 +0200 s3:mdssvc: fix unmarshalling of empty CNID array len=0 is invalid, len=8 is an empty array, len>8 is an array with members, so for the len=8 case we must add the empty cnid array. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 29ef0306241da38356ffa1ee927e1b343564426e Author: Ralph Boehme <s...@samba.org> Date: Fri May 3 22:52:33 2019 +0200 s3:mdssvc: marshalling: fix unpacking empty CNID structure Pass the correct tag member tag.size to sl_unpack_CNID(), not tag.length. tag.size is the size actually used in a buffer of size tag.length. Cf other users of tag.size that already do this correctly, this was only wrong in this place. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit d67c98184a176805c0406ea4d6d56bcde78bb26d Author: Ralph Boehme <s...@samba.org> Date: Fri May 3 22:52:56 2019 +0200 s3:mdssvc: marshalling: fix unpacking empty filemeta structure This is how a correct dump of an empty sl_filemeta_t container should look like: DALLOC_CTX(#1): { sl_array_t(#3): { uint64_t: 0x0023 CNIDs: unkn1: 0x0, unkn2: 0x0 DALLOC_CTX(#0): { } sl_filemeta_t(#0): { } } } This is basically the response from macOS mdssvc for a query that yields no results: sl_filemeta_t is empty, the CNIDs array as well. Looking at the raw packet data, the empty sl_filemeta_t container as a size of 8 bytes which fails the following check in sl_unpack_cpx(): case SQ_CPX_TYPE_FILEMETA: ... if (tag.size < 16) { *boom* } Only tag.size=0 is invalid, tag.size=8 denotes an empty container and tag.size>=16 denotes a sl_filemeta_t container with actual content must be unpacked by calling sl_unpack(). Note that size is always a muliple of 8. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit ed37f2d3c478da34dd9b80171ec603be3c23e37b Author: Ralph Boehme <s...@samba.org> Date: Sat May 4 11:10:37 2019 +0200 s3:mdssvc: marshalling: add a newline to a DEBUG message Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit c282d76d556ae3744cae3f113afeda746b93949f Author: Ralph Boehme <s...@samba.org> Date: Sat May 4 08:53:48 2019 +0200 torture: start of a mdssvc packet (un)marshalling testsuite Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 9d6794df728d772e6bae611f782d70ed97e5c5aa Author: Ralph Boehme <s...@samba.org> Date: Mon Jul 15 16:44:08 2019 +0200 librpc: add RPC_NDR_MDSSVC Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 12ef7b38437f4a4211fa3dce3e3c1e76a3d3df78 Author: Ralph Boehme <s...@samba.org> Date: Wed Apr 17 16:42:20 2019 +0200 s3:wscript: enable Spotlight by default Now that we have a no-op backend that is always available, we can compile mdssvc by default. The new behaviour is: option not used Default: build mdsvc with available backends from autodetection --disable-spotlight Do not build mdssvc --enable-spotlight Build mdssvc and require a real backend (currently Tracker, in the future also Elasticsearch) Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 25c5012c53f4d8efbe04ee8d3c8af256fec592a7 Author: Ralph Boehme <s...@samba.org> Date: Tue Jul 30 15:39:38 2019 +0200 s3:mdssvc: avoid strncpy when marshalling strings Avoids failure when at O3 level: [2082/4232] Compiling source3/rpc_server/mdssvc/marshalling.c ==> /builds/samba-team/devel/samba/samba-o3.stderr <== In file included from /usr/include/string.h:494, from /usr/include/bsd/string.h:30, from ../../lib/tevent/../replace/replace.h:164, from ../../source3/include/includes.h:23, from ../../source3/rpc_server/mdssvc/marshalling.c:21: In function ‘strncpy’, inlined from ‘sl_pack_string’ at ../../source3/rpc_server/mdssvc/marshalling.c:493:2, inlined from ‘sl_pack_loop’ at ../../source3/rpc_server/mdssvc/marshalling.c:607:13: /usr/include/bits/string_fortified.h:106:10: error: ‘__builtin_strncpy’ output truncated before terminating nul copying as many bytes from a string as its length [-Werror=stringop-truncation] 106 | return __builtin___strncpy_chk (__dest, __src, __len, __bos (__dest)); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ../../source3/rpc_server/mdssvc/marshalling.c: In function ‘sl_pack_loop’: ../../source3/rpc_server/mdssvc/marshalling.c:458:8: note: length computed here 458 | len = strlen(s); | ^~~~~~~~~ cc1: all warnings being treated as errors Marshalled strings are not 0 terminated. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 9e706ee5737bffd6f0a699ce68a75bbd0f7fd7dd Author: Ralph Boehme <s...@samba.org> Date: Sun Apr 21 08:38:23 2019 +0200 unittest: workaround dependency problem in test_lib_util_modules waf somehow screws the dependencies and the module ends up with a bunch of missing RPC related symbols once an RPC service has special dependencies like the mdssvc RPC service. $ bin/test_lib_util_modules test: test_samba_module_probe success: test_samba_module_probe test: test_samba_module_probe_dummy Error loading module '/home/samba/src/bin/modules/rpc/test_dummy_module.so': /home/samba/src/bin/modules/rpc/test_dummy_module.so: undefined symbol: rpc_service_mode failure: test_samba_module_probe_dummy [ NT_STATUS_IS_OK(status) ../../testsuite/unittests/test_lib_util_modules.c:39: error: Failure! ] test: test_samba_module_probe_slash success: test_samba_module_probe_slash This is currently not noticed as mdssvc is not enabled in selftest, but that is going to change with a subsequent commit. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 0b1d9d40c249ca7f893cbcc54587ae8fc93f15ee Author: Ralph Boehme <s...@samba.org> Date: Tue Apr 16 14:04:16 2019 +0200 s3:mdssvc: supposed status field is in fact a fragment indicator Spotted this in mdssvc response that containied many results for a search request: if the mdssvc response blob is larger then ~32k, the server fragments the response in 32k fragments and sets the "fragment" field to 1. Note that mdssvc implemenets result set "fragmentation" at the result set layer, not at the marshalled response buffer layer. Therefor mdssvc always sets this field to 0. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit c742ab7a4c78db4101499d15ada1a6635d5cc35b Author: Ralph Boehme <s...@samba.org> Date: Wed Apr 17 11:00:52 2019 +0200 s3:mdssvc: add noindex backend Add a new default backend that, while allowing mdsvc RPC and search queries from clients, always returns no results. Shares using this backend will behave the same way as shares on a macOS SMB server where indexing is disabled. This change will later also allow us to compile the Spotlight RPC service by default which is a big step in the direction of adding tests to CI. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit e5a4114bb563b1ebf8ce7cad2a335bf24dbbdd32 Author: Ralph Boehme <s...@samba.org> Date: Tue Apr 16 11:24:02 2019 +0200 s3:mdssvc: log inode number in ino_path_map_destr_cb() Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 2d98fb57e05ebe16c3a3d8256281830435320607 Author: Ralph Boehme <s...@samba.org> Date: Sun Mar 8 08:01:06 2015 +0100 s3:mdssvc: move some code to a subsystem We need this in a later patchset when adding a mdssvc client library and for unit tests. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit f58026b30b2bf9d60222d39947d07ee858ee5756 Author: Ralph Boehme <s...@samba.org> Date: Fri Nov 16 14:24:35 2018 +0100 s3:mdssvc: mds_dalloc_dump() -> dalloc_dump() Move mds_dalloc_dump() to dalloc and rename it to dalloc_dump(). Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit b5731a94bbc95fa001059d488a2c16a988127619 Author: Ralph Boehme <s...@samba.org> Date: Tue Apr 9 11:19:43 2019 +0200 s3:mdssvc: add Unicode normalisation Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 1b897df4f7489c6ac7663ca4830dc4c129dd2457 Author: Ralph Boehme <s...@samba.org> Date: Mon May 6 15:03:02 2019 +0200 s3:mdssvc: new option "spotlight backend" Currently there's only the tracker backend, but subsequent commits will add other backends. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 66b8d86d71551ab8c615428211294ca110608645 Author: Ralph Boehme <s...@samba.org> Date: Fri Apr 12 12:15:56 2019 +0200 s3:mdssvc: pass down sharename and SNUM Not used for now, will be needed in the upcoming Elasticsearch backend. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 1ef96e09071458322845b4323fa59cf4e7475783 Author: Ralph Boehme <s...@samba.org> Date: Thu Mar 14 07:38:20 2019 +0100 s3-mdssvc: factor out Tracker backend logic This moves all Tracker backend logic into a modularized component. This should not result in any change in behaviour, it just paves the way for adding additional backends. Currently the only available backend is Gnome Tracker. slq_destroy_send/recv is not needed anymore as the problem is solved now by correctly checking if an async Tracker request was cancelled and we got G_IO_ERROR_CANCELLED in tracker_con_cb() or tracker_query_cb() and avoid using user_data in that the case. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit f80f8c5f99f6b5949d14ee62261a7dd448e9b0fe Author: Ralph Boehme <s...@samba.org> Date: Wed Aug 7 11:59:39 2019 +0200 s3:mdssvc: move debug macro to header file This is going to be used from other parts of the code soon. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 17f29fa86772a3fe9c03ecee6813c377d85dde6c Author: Ralph Boehme <s...@samba.org> Date: Wed Aug 7 11:55:44 2019 +0200 s3:mdssvc: README.Coding fixes, long lines, modernize debug macros Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 9e0b6f667f055ab74389759471c0b7181aaf4973 Author: Ralph Boehme <s...@samba.org> Date: Wed Mar 13 15:42:51 2019 +0100 s3:rpc_server: factor out rpc_mdssvc_sources Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> ----------------------------------------------------------------------- Summary of changes: docs-xml/smbdotconf/misc/spotlight_backend.xml | 26 + lib/param/loadparm.c | 2 + lib/param/loadparm.h | 6 + lib/param/param_table.c | 6 + lib/torture/torture.h | 48 ++ librpc/idl/mdssvc.idl | 2 +- librpc/wscript_build | 5 + selftest/target/Samba3.pm | 9 + source3/param/loadparm.c | 1 + source3/rpc_server/mdssvc/dalloc.c | 171 +++++ source3/rpc_server/mdssvc/dalloc.h | 2 + source3/rpc_server/mdssvc/marshalling.c | 32 +- source3/rpc_server/mdssvc/mdssvc.c | 765 +++++-------------- source3/rpc_server/mdssvc/mdssvc.h | 64 +- source3/rpc_server/mdssvc/mdssvc_noindex.c | 57 ++ .../mdssvc/mdssvc_noindex.h} | 13 +- source3/rpc_server/mdssvc/mdssvc_tracker.c | 491 ++++++++++++ source3/rpc_server/mdssvc/mdssvc_tracker.h | 62 ++ source3/rpc_server/mdssvc/sparql_parser.y | 9 +- source3/rpc_server/mdssvc/sparql_parser_test.c | 11 +- source3/rpc_server/mdssvc/srv_mdssvc_nt.c | 100 ++- source3/rpc_server/wscript_build | 34 +- source3/rpcclient/cmd_spotlight.c | 460 +++++++++++ source3/rpcclient/rpcclient.c | 15 +- source3/rpcclient/wscript_build | 3 + source3/selftest/tests.py | 3 + source3/wscript | 38 +- source3/wscript_build | 2 +- source4/selftest/tests.py | 7 +- source4/torture/local/local.c | 1 + source4/torture/local/mdspkt.c | 104 +++ source4/torture/local/wscript_build | 1 + source4/torture/rpc/mdssvc.c | 845 +++++++++++++++++++++ source4/torture/rpc/rpc.c | 81 ++ source4/torture/rpc/torture_rpc.h | 24 +- source4/torture/wscript_build | 3 + testsuite/unittests/wscript | 2 +- 37 files changed, 2841 insertions(+), 664 deletions(-) create mode 100644 docs-xml/smbdotconf/misc/spotlight_backend.xml create mode 100644 source3/rpc_server/mdssvc/mdssvc_noindex.c copy source3/{lib/util_macstreams.h => rpc_server/mdssvc/mdssvc_noindex.h} (76%) create mode 100644 source3/rpc_server/mdssvc/mdssvc_tracker.c create mode 100644 source3/rpc_server/mdssvc/mdssvc_tracker.h create mode 100644 source3/rpcclient/cmd_spotlight.c create mode 100644 source4/torture/local/mdspkt.c create mode 100644 source4/torture/rpc/mdssvc.c Changeset truncated at 500 lines: diff --git a/docs-xml/smbdotconf/misc/spotlight_backend.xml b/docs-xml/smbdotconf/misc/spotlight_backend.xml new file mode 100644 index 00000000000..6d224b81e5f --- /dev/null +++ b/docs-xml/smbdotconf/misc/spotlight_backend.xml @@ -0,0 +1,26 @@ +<samba:parameter name="spotlight backend" + context="S" + type="enum" + enumlist="enum_spotlight_backend" + xmlns:samba="http://www.samba.org/samba/DTD/samba-doc"> +<description> + <para> + Spotlight search backend. Available backends: + + <itemizedlist> + <listitem><para><constant>noindex</constant> - + a backend that returns no results. + </para></listitem> + + </itemizedlist> + + <itemizedlist> + <listitem><para><constant>tracker</constant> - + Gnome Tracker. + </para></listitem> + + </itemizedlist> + </para> +</description> +<value type="default">noindex</value> +</samba:parameter> diff --git a/lib/param/loadparm.c b/lib/param/loadparm.c index 883d4167bf4..13e8b677e30 100644 --- a/lib/param/loadparm.c +++ b/lib/param/loadparm.c @@ -3025,6 +3025,8 @@ struct loadparm_context *loadparm_init(TALLOC_CTX *mem_ctx) lpcfg_do_global_parameter(lp_ctx, "debug encryption", "no"); + lpcfg_do_global_parameter(lp_ctx, "spotlight backend", "noindex"); + for (i = 0; parm_table[i].label; i++) { if (!(lp_ctx->flags[i] & FLAG_CMDLINE)) { lp_ctx->flags[i] |= FLAG_DEFAULT; diff --git a/lib/param/loadparm.h b/lib/param/loadparm.h index 0b2e302d2a9..5c5b1cd7cd4 100644 --- a/lib/param/loadparm.h +++ b/lib/param/loadparm.h @@ -248,6 +248,12 @@ enum inheritowner_options { /* mangled names options */ enum mangled_names_options {MANGLED_NAMES_NO, MANGLED_NAMES_YES, MANGLED_NAMES_ILLEGAL}; +/* Spotlight backend options */ +enum spotlight_backend_options { + SPOTLIGHT_BACKEND_NOINDEX, + SPOTLIGHT_BACKEND_TRACKER, +}; + /* * Default passwd chat script. */ diff --git a/lib/param/param_table.c b/lib/param/param_table.c index f9d3b55adf2..82dc5cd6cda 100644 --- a/lib/param/param_table.c +++ b/lib/param/param_table.c @@ -351,6 +351,12 @@ static const struct enum_list enum_ntlm_auth[] = { {-1, NULL} }; +static const struct enum_list enum_spotlight_backend[] = { + {SPOTLIGHT_BACKEND_NOINDEX, "noindex"}, + {SPOTLIGHT_BACKEND_TRACKER, "tracker"}, + {-1, NULL} +}; + /* Note: We do not initialise the defaults union - it is not allowed in ANSI C * * NOTE: Handling of duplicated (synonym) parameters: diff --git a/lib/torture/torture.h b/lib/torture/torture.h index 9f55c164ac1..3507740967c 100644 --- a/lib/torture/torture.h +++ b/lib/torture/torture.h @@ -520,6 +520,54 @@ static inline void torture_dump_data_str_cb(const char *buf, void *private_data) } \ } while(0) +#define torture_assert_u32_equal(torture_ctx,got,expected,cmt)\ + do { uint32_t __got = (got), __expected = (expected); \ + if (__got != __expected) { \ + torture_result(torture_ctx, TORTURE_FAIL, \ + __location__": "#got" was %ju (0x%jX), expected %ju (0x%jX): %s", \ + (uintmax_t)__got, (uintmax_t)__got, \ + (uintmax_t)__expected, (uintmax_t)__expected, \ + cmt); \ + return false; \ + } \ + } while(0) + +#define torture_assert_u32_equal_goto(torture_ctx,got,expected,ret,label,cmt)\ + do { uint32_t __got = (got), __expected = (expected); \ + if (__got != __expected) { \ + torture_result(torture_ctx, TORTURE_FAIL, \ + __location__": "#got" was %ju (0x%jX), expected %ju (0x%jX): %s", \ + (uintmax_t)__got, (uintmax_t)__got, \ + (uintmax_t)__expected, (uintmax_t)__expected, \ + cmt); \ + ret = false; \ + goto label; \ + } \ + } while(0) + +#define torture_assert_u32_not_equal(torture_ctx,got,not_expected,cmt)\ + do { uint32_t __got = (got), __not_expected = (not_expected); \ + if (__got == __not_expected) { \ + torture_result(torture_ctx, TORTURE_FAIL, \ + __location__": "#got" was %ju (0x%jX), expected a different number: %s", \ + (uintmax_t)__got, (uintmax_t)__got, \ + cmt); \ + return false; \ + } \ + } while(0) + +#define torture_assert_u32_not_equal_goto(torture_ctx,got,not_expected,ret,label,cmt)\ + do { uint32_t __got = (got), __not_expected = (not_expected); \ + if (__got == __not_expected) { \ + torture_result(torture_ctx, TORTURE_FAIL, \ + __location__": "#got" was %ju (0x%jX), expected a different number: %s", \ + (uintmax_t)__got, (uintmax_t)__got, \ + cmt); \ + ret = false; \ + goto label; \ + } \ + } while(0) + #define torture_assert_u64_equal(torture_ctx,got,expected,cmt)\ do { uint64_t __got = (got), __expected = (expected); \ if (__got != __expected) { \ diff --git a/librpc/idl/mdssvc.idl b/librpc/idl/mdssvc.idl index bfb27c4a795..43447961b0d 100644 --- a/librpc/idl/mdssvc.idl +++ b/librpc/idl/mdssvc.idl @@ -51,7 +51,7 @@ interface mdssvc [in] uint32 max_fragment_size2, [in] uint32 unkn7, /* always 0 ? */ [in] uint32 unkn8, /* always 0 ? */ - [out,ref] uint32 *status, + [out,ref] uint32 *fragment, [out,ref] mdssvc_blob *response_blob, [out,ref] uint32 *unkn9 /* always 0 ? */ ); diff --git a/librpc/wscript_build b/librpc/wscript_build index 70fe8c2f7fe..d72592b52cc 100644 --- a/librpc/wscript_build +++ b/librpc/wscript_build @@ -597,6 +597,11 @@ bld.SAMBA_SUBSYSTEM('RPC_NDR_CLUSAPI', public_deps='dcerpc-binding NDR_CLUSAPI' ) +bld.SAMBA_SUBSYSTEM('RPC_NDR_MDSSVC', + source='gen_ndr/ndr_mdssvc_c.c', + public_deps='dcerpc-binding NDR_MDSSVC' + ) + # a grouping library for NDR subsystems that may be used by more than one target bld.SAMBA_LIBRARY('ndr-samba', source=[], diff --git a/selftest/target/Samba3.pm b/selftest/target/Samba3.pm index 5c327cab543..2587ea31253 100755 --- a/selftest/target/Samba3.pm +++ b/selftest/target/Samba3.pm @@ -939,6 +939,7 @@ sub setup_fileserver my $fileserver_options = " kernel change notify = yes + rpc_server:mdssvc = embedded usershare path = $usershare_dir usershare max shares = 10 @@ -947,6 +948,14 @@ sub setup_fileserver get quota command = $prefix_abs/getset_quota.py set quota command = $prefix_abs/getset_quota.py +[spotlight] + path = $share_dir + spotlight = yes + read only = no +[no_spotlight] + path = $share_dir + spotlight = no + read only = no [lowercase] path = $lower_case_share_dir comment = smb username is [%U] diff --git a/source3/param/loadparm.c b/source3/param/loadparm.c index 9e97d719f4e..0daac0e44cb 100644 --- a/source3/param/loadparm.c +++ b/source3/param/loadparm.c @@ -247,6 +247,7 @@ static const struct loadparm_service _sDefault = .param_opt = NULL, .smbd_search_ask_sharemode = true, .smbd_getinfo_ask_sharemode = true, + .spotlight_backend = SPOTLIGHT_BACKEND_NOINDEX, .dummy = "" }; diff --git a/source3/rpc_server/mdssvc/dalloc.c b/source3/rpc_server/mdssvc/dalloc.c index 28944b809f1..23193fee70a 100644 --- a/source3/rpc_server/mdssvc/dalloc.c +++ b/source3/rpc_server/mdssvc/dalloc.c @@ -18,6 +18,10 @@ #include "replace.h" #include <talloc.h> #include "dalloc.h" +#include "marshalling.h" +#include "lib/util/charset/charset.h" +#include "lib/util/talloc_stack.h" +#include "system/time.h" /** * Dynamic Datastore @@ -230,3 +234,170 @@ int dalloc_stradd(DALLOC_CTX *d, const char *string) return 0; } + +static char *tab_level(TALLOC_CTX *mem_ctx, int level) +{ + int i; + char *string = talloc_array(mem_ctx, char, level + 1); + + for (i = 0; i < level; i++) { + string[i] = '\t'; + } + + string[i] = '\0'; + return string; +} + +char *dalloc_dump(DALLOC_CTX *dd, int nestinglevel) +{ + const char *type; + int n, result; + uint64_t i; + sl_bool_t bl; + sl_time_t t; + struct tm *tm; + char datestring[256]; + sl_cnids_t cnids; + char *logstring, *nested_logstring; + char *tab_string1, *tab_string2; + void *p; + bool ok; + char *utf8string; + size_t utf8len; + + tab_string1 = tab_level(dd, nestinglevel); + if (tab_string1 == NULL) { + return NULL; + } + tab_string2 = tab_level(dd, nestinglevel + 1); + if (tab_string2 == NULL) { + return NULL; + } + + logstring = talloc_asprintf(dd, + "%s%s(#%lu): {\n", + tab_string1, + talloc_get_name(dd), + dalloc_size(dd)); + if (logstring == NULL) { + return NULL; + } + + for (n = 0; n < dalloc_size(dd); n++) { + type = dalloc_get_name(dd, n); + if (type == NULL) { + return NULL; + } + p = dalloc_get_object(dd, n); + if (p == NULL) { + return NULL; + } + if (strcmp(type, "DALLOC_CTX") == 0 + || strcmp(type, "sl_array_t") == 0 + || strcmp(type, "sl_filemeta_t") == 0 + || strcmp(type, "sl_dict_t") == 0) { + nested_logstring = dalloc_dump(p, nestinglevel + 1); + if (nested_logstring == NULL) { + return NULL; + } + logstring = talloc_strdup_append(logstring, + nested_logstring); + } else if (strcmp(type, "uint64_t") == 0) { + memcpy(&i, p, sizeof(uint64_t)); + logstring = talloc_asprintf_append( + logstring, + "%suint64_t: 0x%04jx\n", + tab_string2, (uintmax_t)i); + } else if (strcmp(type, "char *") == 0) { + logstring = talloc_asprintf_append( + logstring, + "%sstring: %s\n", + tab_string2, + (char *)p); + } else if (strcmp(type, "smb_ucs2_t *") == 0) { + ok = convert_string_talloc(talloc_tos(), + CH_UTF16LE, + CH_UTF8, + p, + talloc_get_size(p), + &utf8string, + &utf8len); + if (!ok) { + return NULL; + } + logstring = talloc_asprintf_append( + logstring, + "%sUTF16-string: %s\n", + tab_string2, + utf8string); + TALLOC_FREE(utf8string); + } else if (strcmp(type, "sl_bool_t") == 0) { + memcpy(&bl, p, sizeof(sl_bool_t)); + logstring = talloc_asprintf_append( + logstring, + "%sbool: %s\n", + tab_string2, + bl ? "true" : "false"); + } else if (strcmp(type, "sl_nil_t") == 0) { + logstring = talloc_asprintf_append( + logstring, + "%snil\n", + tab_string2); + } else if (strcmp(type, "sl_time_t") == 0) { + memcpy(&t, p, sizeof(sl_time_t)); + tm = localtime(&t.tv_sec); + if (tm == NULL) { + return NULL; + } + result = strftime(datestring, + sizeof(datestring), + "%Y-%m-%d %H:%M:%S", tm); + if (result == 0) { + return NULL; + } + logstring = talloc_asprintf_append( + logstring, + "%ssl_time_t: %s.%06lu\n", + tab_string2, + datestring, + (unsigned long)t.tv_usec); + } else if (strcmp(type, "sl_cnids_t") == 0) { + memcpy(&cnids, p, sizeof(sl_cnids_t)); + logstring = talloc_asprintf_append( + logstring, + "%sCNIDs: unkn1: 0x%" PRIx16 ", unkn2: 0x%" PRIx32 "\n", + tab_string2, + cnids.ca_unkn1, + cnids.ca_context); + if (logstring == NULL) { + return NULL; + } + if (cnids.ca_cnids) { + nested_logstring = dalloc_dump( + cnids.ca_cnids, + nestinglevel + 2); + if (!nested_logstring) { + return NULL; + } + logstring = talloc_strdup_append(logstring, + nested_logstring); + } + } else { + logstring = talloc_asprintf_append( + logstring, + "%stype: %s\n", + tab_string2, + type); + } + if (logstring == NULL) { + return NULL; + } + } + logstring = talloc_asprintf_append(logstring, + "%s}\n", + tab_string1); + if (logstring == NULL) { + return NULL; + } + return logstring; +} diff --git a/source3/rpc_server/mdssvc/dalloc.h b/source3/rpc_server/mdssvc/dalloc.h index b268893160b..69650b8a7a0 100644 --- a/source3/rpc_server/mdssvc/dalloc.h +++ b/source3/rpc_server/mdssvc/dalloc.h @@ -160,4 +160,6 @@ extern int dalloc_stradd(DALLOC_CTX *d, const char *string); extern void *_dalloc_new(TALLOC_CTX *mem_ctx, const char *type); extern int _dalloc_add_talloc_chunk(DALLOC_CTX *d, void *obj, const char *type, size_t size); +extern char *dalloc_dump(DALLOC_CTX *dd, int nestinglevel); + #endif /* DALLOC_H */ diff --git a/source3/rpc_server/mdssvc/marshalling.c b/source3/rpc_server/mdssvc/marshalling.c index b5931c7f060..1aa750413cd 100644 --- a/source3/rpc_server/mdssvc/marshalling.c +++ b/source3/rpc_server/mdssvc/marshalling.c @@ -490,7 +490,7 @@ static ssize_t sl_pack_string(char *s, char *buf, ssize_t offset, size_t bufsize } memset(buf + offset, 0, octets * 8); - strncpy(buf + offset, s, len); + memcpy(buf + offset, s, len); offset += octets * 8; return offset; @@ -847,12 +847,17 @@ static int sl_unpack_CNID(DALLOC_CTX *query, return -1; } - if (length <= 16) { + if (length < 8) { + return -1; + } + if (length == 8) { /* - * That's permitted, iirc length = 16 is an empty - * array, so anything lesser then 16 should probably - * be treated as an error, but I'm not quite sure. + * That's permitted, length=8 is an empty CNID array. */ + result = dalloc_add(query, cnids, sl_cnids_t); + if (result != 0) { + return -1; + } return 0; } @@ -1005,8 +1010,8 @@ static ssize_t sl_unpack_cpx(DALLOC_CTX *query, if (offset == -1) { return -1; } - if (tag.size < 16) { - DEBUG(1, ("%s: size too mall: %zu", __func__, tag.size)); + if (tag.size < 8) { + DBG_WARNING("size too mall: %zu\n", tag.size); return -1; } @@ -1014,9 +1019,14 @@ static ssize_t sl_unpack_cpx(DALLOC_CTX *query, if (sl_fm == NULL) { return -1; } - result = sl_unpack(sl_fm, buf + offset, bufsize - offset ); - if (result == -1) { - return -1; + + if (tag.size >= 16) { + result = sl_unpack(sl_fm, + buf + offset, + bufsize - offset ); + if (result == -1) { + return -1; + } } result = dalloc_add(query, sl_fm, sl_filemeta_t); if (result != 0) { @@ -1032,7 +1042,7 @@ static ssize_t sl_unpack_cpx(DALLOC_CTX *query, } result = sl_unpack_CNID(query, buf, offset, bufsize, - tag.length, encoding); + tag.size, encoding); if (result == -1) { return -1; } diff --git a/source3/rpc_server/mdssvc/mdssvc.c b/source3/rpc_server/mdssvc/mdssvc.c index 58a219bedb0..24183329301 100644 --- a/source3/rpc_server/mdssvc/mdssvc.c +++ b/source3/rpc_server/mdssvc/mdssvc.c @@ -27,32 +27,14 @@ #include "lib/dbwrap/dbwrap_rbt.h" #include "libcli/security/dom_sid.h" #include "mdssvc.h" -#include "rpc_server/mdssvc/sparql_parser.tab.h" -#include "lib/tevent_glib_glue.h" +#include "mdssvc_noindex.h" +#ifdef HAVE_SPOTLIGHT_BACKEND_TRACKER +#include "mdssvc_tracker.h" +#endif #undef DBGC_CLASS #define DBGC_CLASS DBGC_RPC_SRV -#define SLQ_DEBUG(lvl, _slq, state) do { if (CHECK_DEBUGLVL(lvl)) { \ - const struct sl_query *__slq = _slq; \ - struct timeval_buf start_buf; \ - const char *start; \ - struct timeval_buf last_used_buf; \ - const char *last_used; \ -- Samba Shared Repository