[
https://issues.apache.org/jira/browse/LUCENE-5734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14025659#comment-14025659
]
David Smiley commented on LUCENE-5734:
--------------------------------------
I had a realization this weekend. I don't actually think there's a problem
with HTMLStripCharFilter. It just strips. The non-stripped characters should
map to the same location as the input (and they do). Naively, correctOffset()
shouldn't be called on the filter with a stripped offset provided because the
filter shouldn't be seeing those stripped characters in the first place. Well
actually, every Tokenizer calls {{correctOffset(endOffset)}} wherein endOffset
points to one character _beyond_ the last character. I don't think someone
thought of the ramifications of this; at least it rarely matters. What all
Tokenizers could do to correct the problem is {{correctOffset(endOffset-1)+1}}.
That's a little awkward so perhaps a simple convenience method could be added
such as {{CharFilter.correctEndOffset()}}. It's also a single spot to
potentially vary behavior between 5x/4x, or by looking at the lucene version,
or even making this a configurable option on the CharFilter.
I tried another solution idea which is to modify
{{BaseCharFilter.correct(off)}} so that it does not "correct" the first
character of an elided span, but it didn't seem to work for non-elided cases so
I dropped that idea.
Here is a test method I added to TestMappingCharFilter locally, for convenience
of experimentation, that fails now. I think it should pass:
{code:java}
public void testEndOffset() throws IOException {
NormalizeCharMap.Builder builder = new NormalizeCharMap.Builder();
builder.add( "elided", "" );
builder.add( "t", "TOKEN" );//expanded
final NormalizeCharMap normMap = builder.build();
String testString = "telided";
CharFilter cs = new MappingCharFilter( normMap,
new StringReader( testString ) );
TokenStream ts = whitespaceMockTokenizer(cs);
assertTokenStreamContents(ts,
new String[]{"TOKEN"},
new int[]{0},
new int[]{1},
testString.length()
);
}
{code}
It's failing because it expects endOffset to be 7 (the length of the whole
string), not 1.
Opinions?
> HTMLStripCharFilter end offset should be left of closing tags
> -------------------------------------------------------------
>
> Key: LUCENE-5734
> URL: https://issues.apache.org/jira/browse/LUCENE-5734
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/analysis
> Reporter: David Smiley
> Priority: Minor
>
> Consider this simple input:
> {noformat}
> <em>hello</em>
> {noformat}
> to be analyzed by HTMLStripCharFilter and WhitespaceTokenizer.
> You get back one token for "hello". Good. The start offset of this token is
> at the position of 'h' -- good. But the end offset is surprisingly plus one
> to the adjacent </em>. I argue that it should be plus one to the last
> character of the token (following 'o').
> FYI it behaves as I expect if after hello is an XML entity such as in this
> example: {noformat}hello {noformat} The end offset immediately follows
> the 'o'.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
