Do stopfilters create non-contiguous token positions?
I was interested in experimenting with the highlighter and using the
TokenSources.getTokenStream(TermPositionVector
<file:///C:\mysvn\lucene\build\docs\api\org\apache\lucene\index\TermPosition
Vector.html> tpv, boolean
tokenPositionsGuaranteedContiguous) method
The javadocs for this method note that:
tokenPositionsGuaranteedContiguous - true if the token position numbers have
no overlaps or gaps.
The example used for comparison to re-Analyzing the the text includes
stopwords ("timings above were using a stemmer/lowercaser/stopword combo").
I was curious if a stopwords, by definition meant that tokens were not
contiguous? Is this still true if the the query uses the same analyzer and
filters out the same stopwords?
Thanks,
Dan
