[
https://issues.apache.org/jira/browse/LUCENE-5734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14017831#comment-14017831
]
David Smiley commented on LUCENE-5734:
--------------------------------------
[~romseygeek] ok then it should be configurable, and _consistent_ too. *if*
the user wants a closing element offset to be included with the token (as it
currently does) then then an adjacent opening element should mark the start of
the token too. IMO it shouldn't work this way by default, though.
> HTMLStripCharFilter end offset should be left of closing tags
> -------------------------------------------------------------
>
> Key: LUCENE-5734
> URL: https://issues.apache.org/jira/browse/LUCENE-5734
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/analysis
> Reporter: David Smiley
> Priority: Minor
>
> Consider this simple input:
> {noformat}
> <em>hello</em>
> {noformat}
> to be analyzed by HTMLStripCharFilter and WhitespaceTokenizer.
> You get back one token for "hello". Good. The start offset of this token is
> at the position of 'h' -- good. But the end offset is surprisingly plus one
> to the adjacent </em>. I argue that it should be plus one to the last
> character of the token (following 'o').
> FYI it behaves as I expect if after hello is an XML entity such as in this
> example: {noformat}hello {noformat} The end offset immediately follows
> the 'o'.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
