Danny - Yes, a good alternative to using replace().
I'm not sure if this sounds like a reasonable feature request, but I have some
beef with the way search:parse behaves. First, I didn't realize that an empty
word query - cts:word-query("") - would return zero results; I assumed it would
return everything. But given that, if you search:parse with the
case-insensitive option, then any string input with floating punctuation will
return a cts:word-query("&punctuation;",("case-insensitive")), equivalent to
the empty word query and and-ed with the rest of the parsed query will always
return nothing. It's an edge case, but seems undesirable in any scenario.
-Will
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Danny Sokolsky
Sent: Friday, January 27, 2012 10:25 AM
To: General MarkLogic Developer Discussion
Subject: Re: [MarkLogic Dev General] en/em dashes punctuation?
Just a thought here: Since those values are coming out of a lexicon (I am
assuming), maybe your javascript code that displays it in the browser can
remove the unwanted characters (and maybe lower-case them?) before it gives the
suggestion to the ui? That way people can still search for those characters by
typing them in.
-Danny
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Will Thompson
Sent: Thursday, January 26, 2012 7:20 PM
To: General MarkLogic Developer Discussion
Subject: Re: [MarkLogic Dev General] en/em dashes punctuation?
Okay, very cool. Thanks.
-Will
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Michael Blakeley
Sent: Thursday, January 26, 2012 7:17 PM
To: General MarkLogic Developer Discussion
Subject: Re: [MarkLogic Dev General] en/em dashes punctuation?
The initial release did not include query-eval for direct cts:query output. I
added that a bit later, to make it easier to get started.
But the default query-eval module is still meant to be modified and customized
for your application. It takes some shortcuts: for example, it assumes that
foo=bar maps to cts:element-value-query(xs:QName('foo'), 'bar'). A
sophisticated application might use a lookup table to map codes to QNames, and
possibly mix in other options like collation, language, case-sensitivity, etc.
If you customize query-eval in interesting or useful ways, I am open to adding
more sample evaluators to github.
-- Mike
On 26 Jan 2012, at 19:04 , Will Thompson wrote:
> Thanks Mike - Your parser was on my radar, but I did not realize it returns
> ML query syntax (I thought you had to DIY to get it from your AST to ML).
>
> As a quick fix I may have to just
> replace($querystring,"&endash;|&emdash;,""), but I will definitely give xqysp
> a second look.
>
> -Will
>
>
> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]] On Behalf Of Michael Blakeley
> Sent: Thursday, January 26, 2012 6:46 PM
> To: General MarkLogic Developer Discussion
> Subject: Re: [MarkLogic Dev General] en/em dashes punctuation?
>
> Will, you could also try https://github.com/mblakele/xqysp
>
> Here are some tests comparing search:parse output with the query-eval module
> that is included as an example in xqysp. Since you have access to the
> abstract syntax tree from the parser, you can also customize the cts:query
> output to suit your needs. Integration with the search API is easy: instead
> of calling search:search, pass the cts:query to search:resolve.
>
> import module namespace qe="com.blakeley.xqysp.query-eval" at
> "query-eval.xqy";
> import module namespace search = "http://marklogic.com/appservices/search";
> at "/MarkLogic/appservices/search/search.xqy";
>
> for $q in ('foo-bar', 'foo - bar', 'foo–bar', 'foo – bar')
> return element test {
> attribute query { $q },
> search:parse($q),
> qe:parse($q) }
> =>
> <test query="foo-bar">
> <cts:word-query qtextref="cts:text" xmlns:cts="http://marklogic.com/cts";>
> <cts:text>foo-bar</cts:text>
> </cts:word-query>
> <cts:word-query xmlns:cts="http://marklogic.com/cts";>
> <cts:text xml:lang="en">foo-bar</cts:text>
> </cts:word-query>
> </test>
> <test query="foo - bar">
> <cts:and-query strength="20" qtextjoin="" qtextgroup="( )"
> xmlns:cts="http://marklogic.com/cts";>
> <cts:word-query qtextref="cts:text">
> <cts:text>foo</cts:text>
> </cts:word-query>
> <cts:not-query qtextstart="-" strength="40">
> <cts:word-query qtextref="cts:text">
> <cts:text>bar</cts:text>
> </cts:word-query>
> </cts:not-query>
> </cts:and-query>
> <cts:and-query xmlns:cts="http://marklogic.com/cts";>
> <cts:word-query>
> <cts:text xml:lang="en">foo</cts:text>
> </cts:word-query>
> <cts:not-query>
> <cts:word-query>
> <cts:text xml:lang="en">bar</cts:text>
> </cts:word-query>
> </cts:not-query>
> </cts:and-query>
> </test>
> <test query="foo–bar">
> <cts:word-query qtextref="cts:text" xmlns:cts="http://marklogic.com/cts";>
> <cts:text>foo–bar</cts:text>
> </cts:word-query>
> <cts:and-query xmlns:cts="http://marklogic.com/cts";>
> <cts:word-query>
> <cts:text xml:lang="en">foo</cts:text>
> </cts:word-query>
> <cts:word-query>
> <cts:text xml:lang="en">bar</cts:text>
> </cts:word-query>
> </cts:and-query>
> </test>
> <test query="foo – bar">
> <cts:and-query strength="20" qtextjoin="" qtextgroup="( )"
> xmlns:cts="http://marklogic.com/cts";>
> <cts:word-query qtextref="cts:text">
> <cts:text>foo</cts:text>
> </cts:word-query>
> <cts:and-query strength="20" qtextjoin="" qtextgroup="( )">
> <cts:word-query qtextref="cts:text">
> <cts:text>–</cts:text>
> </cts:word-query>
> <cts:word-query qtextref="cts:text">
> <cts:text>bar</cts:text>
> </cts:word-query>
> </cts:and-query>
> </cts:and-query>
> <cts:and-query xmlns:cts="http://marklogic.com/cts";>
> <cts:word-query>
> <cts:text xml:lang="en">foo</cts:text>
> </cts:word-query>
> <cts:word-query>
> <cts:text xml:lang="en">bar</cts:text>
> </cts:word-query>
> </cts:and-query>
> </test>
>
> -- Mike
>
> On 26 Jan 2012, at 18:13 , Danny Sokolsky wrote:
>
>> Sorry Will, I misunderstood (thought you meant the - was being treated as
>> negation).
>>
>> Since you are pulling those from data that you know is in your database, how
>> about if you make the whole thing a phrase put surrounding it with quotes.
>> Here is an example of what I mean:
>>
>> xquery version "1.0-ml";
>>
>> import module namespace search =
>> "http://marklogic.com/appservices/search";
>> at "/MarkLogic/appservices/search/search.xqy";
>>
>> search:parse('"Venue ― Motion to Transfer"')
>>
>> -Danny
>>
>> -----Original Message-----
>> From: [email protected]
>> [mailto:[email protected]] On Behalf Of Will Thompson
>> Sent: Thursday, January 26, 2012 6:01 PM
>> To: General MarkLogic Developer Discussion
>> Subject: Re: [MarkLogic Dev General] en/em dashes punctuation?
>>
>> Thanks Danny, but I'm not sure I follow. Maybe that was not the best
>> explanation. Rather than use dashes like hyphens, I just want a search for
>> something like "Venue ― Motion to Transfer" to ignore the dash when
>> parsed. It appears to be treating it like a word instead and is not ignored:
>>
>> cts:and-query(
>> (cts:word-query("Venue", ("case-insensitive", "punctuation-insensitive",
>> "lang=en"), 1),
>> cts:word-query("―", ("case-insensitive", "punctuation-insensitive",
>> "lang=en"), 1),
>> cts:word-query("Motion", ("case-insensitive", "punctuation-insensitive",
>> "lang=en"), 1),
>> cts:word-query("to", ("case-insensitive", "punctuation-insensitive",
>> "lang=en"), 1),
>> cts:word-query("Transfer", ("case-insensitive", "punctuation-insensitive",
>> "lang=en"), 1)),
>> ())
>>
>> -Will
>>
>> -----Original Message-----
>> From: [email protected]
>> [mailto:[email protected]] On Behalf Of Danny Sokolsky
>> Sent: Thursday, January 26, 2012 5:35 PM
>> To: General MarkLogic Developer Discussion
>> Subject: Re: [MarkLogic Dev General] en/em dashes punctuation?
>>
>> Hi Will,
>>
>> One thing you can do is change your search grammar to use a joiner other
>> than the negative sign.
>>
>> Here is the default grammar:
>>
>> http://docs.marklogic.com/5.0doc/docapp.xqy#display.xqy?fname=http://pubs/5.0doc/xml/search-dev-guide/search-api.xml%2344520
>>
>> -Danny
>>
>> -----Original Message-----
>> From: [email protected]
>> [mailto:[email protected]] On Behalf Of Will Thompson
>> Sent: Thursday, January 26, 2012 4:34 PM
>> To: General MarkLogic Developer Discussion
>> Subject: [MarkLogic Dev General] en/em dashes punctuation?
>>
>> Our search autocomplete pulls from doc titles, some of which contain en or
>> em dashes. However, if the dash is "floating"- i.e.: "Venue - Motion to
>> Transfer" - search:parse parses it into the query, even though
>> <term-option>punctuation-insensitive</term-option> is included in the <term>
>> section of the search options node. I thought it may just be getting ignored
>> when it's evaluated but it's definitely limiting the query.
>>
>> I can confirm they are punctuation: cts:tokenize("hyphen-en-em-bar―")[.
>> instance of cts:punctuation] => "- - - ―"
>>
>> But is there an exception here (the same way hyphens are always parsed to
>> negate)? Do I just need to remove these from the query string before calling
>> search:parse? If there is a cleaner way, that would be great.
>>
>>
>> Best,
>>
>> Will
>> _______________________________________________
>> General mailing list
>> [email protected]
>> http://developer.marklogic.com/mailman/listinfo/general
>> _______________________________________________
>> General mailing list
>> [email protected]
>> http://developer.marklogic.com/mailman/listinfo/general
>> _______________________________________________
>> General mailing list
>> [email protected]
>> http://developer.marklogic.com/mailman/listinfo/general
>> _______________________________________________
>> General mailing list
>> [email protected]
>> http://developer.marklogic.com/mailman/listinfo/general
>
> _______________________________________________
> General mailing list
> [email protected]
> http://developer.marklogic.com/mailman/listinfo/general
> _______________________________________________
> General mailing list
> [email protected]
> http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________
General mailing list
[email protected]
http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________
General mailing list
[email protected]
http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________
General mailing list
[email protected]
http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________
General mailing list
[email protected]
http://developer.marklogic.com/mailman/listinfo/general
