Hi Lisa, None of the settings affect the way tokenization is done; I will file the tokenization issue as a bug. As a work-around in the short term, you can substitute alternate snippeting code when using search:search(); check out section 2.2.6.2 of the Search Developer's Guide for an extension pattern. http://developer.marklogic.com/pubs/4.1/books/search-dev-guide.pdf
--Colleen Whitney ________________________________________ From: [email protected] [[email protected]] On Behalf Of Lisa Liddle [[email protected]] Sent: Monday, August 03, 2009 1:34 PM To: [email protected] Subject: [MarkLogic Dev General] search:snippet doing unusual tokenizing Hi, I’m using search:snippet to get back a snippet to display on a search results page and I’m finding that sometimes it returns characters I wouldn’t expect at the beginning or end of the snippet. I’ve seen several instances where it ends or begins with a paren: As I read this letter, I realized my mother’s greatest <search:highlight>hope</search:highlight> was that I remain pure and virtuous. Virtue “is a pattern of thought and behavior based on high moral standards” ( I’ve also seen where it breaks in the middle of a contraction: ...’t predict all the struggles and storms in life, not even the ones just around the next corner, but as persons of faith and <search:highlight>hope</search:highlight>... And sometimes the “…” isn’t used when it makes sense to use it, as in this example: ...life to end in tragedy. But all too often, like the pilots and passengers of the sightseeing flight, we set out on what we <search:highlight>hope</search:highlight> will be an exciting journey only I’ve changed the per-match-tokens and the max-snippet-chars settings in transform-results, but it doesn’t seem to have any affect on the tokenizing. Is there anything that can be done about this? Thanks, Lisa NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. _______________________________________________ General mailing list [email protected] http://xqzone.com/mailman/listinfo/general
