Hi Jack, In general, it should always be possible to get ft:mark working somehow; it’s just difficult to give general advice how to do it ;)
If you like, you can provide us with a stripped-down version of your code that you can’t get to work. Best, Christian On Wed, Mar 6, 2024 at 5:25 AM Jack Steyn <steynj...@gmail.com> wrote: > Hi Christian, > > Thank you very much for your explanation and variant example. > > In my use case, the local:search function is itself being called (as a > named function reference) from within another function that is the endpoint > of a RESTXQ API. This containing function handles a number of things e.g. > pagination, deduplication, transformation of XML into HTML. > > Even when I rewrite this local:search to match your variant example, > incorrect results are still returned. But when I then add %rest:GET > annotations to turn the local:search function into its own endpoint, the > correct results are returned only when I use that endpoint directly. > > Thus I assume the containing function makes things again too complicated > for metadata to be propagated. > > Does that sound plausible to you? And can you suggest any simple ways > around it? I'm afraid applying %basex:inline hasn't helped. > > Very best, > > Jack > > On Fri, 1 Mar 2024, 8:12 pm Christian Grün, <christian.gr...@gmail.com> > wrote: > >> Hi Jack, >> >> > When you say you can't reproduce it, do you mean you get 14 results >> from running this script? >> >> Yes, that’s what I meant. >> >> The upcoming information will be very technical and specific. You are >> welcome to focus on the examples. >> >> Your updated example was helpful, and I noticed it’s a bunch of issues >> that lead to the unexpected results. The core challenge is that ft:mark and >> ft:extract only yield expected results if the internally collected >> full-text metadata is not lost at some stage during the internal processing >> – which can happen at many places hidden to the writer of the query. >> >> In your specific example, the full-text information gets lost because the >> local:search >> function is too complex to be inlined by the compiler (which enables >> further optimizations that eventually allow metadata propagation). You can >> tackle this by forcing the compiler to inline your function: >> >> declare %basex:inline function local:search(...) >> >> Using '(ethnicgroups, languages)' instead of 'name() = (...)' is another >> practical advice; it helps the optimizer to detect at compile time that >> metadata will be available at runtime. Another solution is to use >> 'local-name()' instead of 'name()' (local-name does not rely on namespace >> that may possibly occur in a database, which also affects the way how >> full-text queries are evaluated). >> >> Here’s a variant that should work: >> >> declare function local:search( >> $database as xs:string, >> $query as xs:string >> ) { >> let $country := ft:search($database, $query)/ancestor::country >> let $search := function($node) { $node/text() contains text { $query } } >> return ( >> ft:mark($country[.//name[$search(.)]]), >> ft:mark($country[.//city[$search(.)]]), >> ft:mark($country[.//(ethnicgroups, languages)[$search(.)]]) >> ) >> }; >> local:search('factbook', 'German') >> >> …or… >> >> let $search := function($nodes) { $nodes[text() contains text { $query >> }] } >> return (ft:mark($country[$search(.//name)]), ... >> >> From today’s perspective, we would certainly design ft:mark and >> ft:extract in a way that the results are always correct. The consequences, >> however, would be a much more restricted syntax. >> >> Hope this helps, >> Christian >> >> >> On Thu, Feb 29, 2024 at 12:13 AM Jack Steyn <steynj...@gmail.com> wrote: >> >>> Hi Christian, >>> >>> When I run your script, I do get 14 elements. >>> >>> When I run the following script I just get 12. >>> >>> <commands> >>> <set option='ftindex'>true</set> >>> <create-db name='factbook'>https://files.basex.org/xml/factbook.xml >>> </create-db> >>> <xquery><![CDATA[ >>> declare function local:search( >>> $database as xs:string, >>> $query as xs:string >>> ) { >>> let $country-search := ft:search($database, $query)/ancestor::country >>> let $city-search := ft:search($database, >>> $query)/ancestor::city/ancestor::country >>> let $other-search := ft:search($database, $query)/parent::*[name() = >>> ('ethnicgroups', 'languages')]/ancestor::country >>> let $country-mark := $country-search[.//name[text() contains text { >>> $query }]] => ft:mark() >>> let $city-mark := $city-search[.//city[text() contains text { $query >>> }]] => ft:mark() >>> let $other-mark := $other-search[.//*[name() = ('ethnicgroups', >>> 'languages')][text() contains text { $query }]] => ft:mark() >>> return ( >>> $country-mark, >>> $city-mark, >>> $other-mark >>> ) >>> }; >>> >>> local:search('factbook', 'German')//mark >>> ]]></xquery> >>> </commands> >>> >>> When you say you can't reproduce it, do you mean you get 14 results from >>> running this script? >>> >>> Cheers, >>> >>> Jack >>> >>> On Thu, 29 Feb 2024, 1:02 am Christian Grün, <christian.gr...@gmail.com> >>> wrote: >>> >>>> Hi Jack, >>>> >>>> Thanks for your observation. >>>> >>>> >>>>> The first result of this query is the entry for Austria. I would >>>>> expect both of the instances of the word 'German' in that entry to be >>>>> surrounded by <mark> tags. However only the first instance is. >>>>> >>>> >>>> I couldn’t reproduce this yet. Here’s a command script that returns 14 >>>> <mark>German</mark> elements: >>>> >>>> <commands> >>>> <set option='ftindex'>true</set> >>>> <create-db name='factbook'>https://files.basex.org/xml/factbook.xml >>>> </create-db> >>>> <xquery><![CDATA[ >>>> let $groups := ('ethnicgroups', 'languages') >>>> let $database := 'factbook' >>>> let $query := 'German' >>>> >>>> let $search := ft:search($database, $query)/parent::* >>>> [name() = $groups]/ancestor::country >>>> let $marked := ft:mark( >>>> $search[.//*[name() = $groups][text() contains text { $query }]] >>>> ) >>>> return $marked//*[text() = 'German'] >>>> ]]></xquery> >>>> </commands> >>>> >>>> Could you check if you get the same result? >>>> >>>> Thanks in advance >>>> Christian >>>> >>>>