> A question before I dive into coding a fix: can I assume (for
> all analyzers) that the tokens produced by the tokenStream
> have the following property:
>currentToken.startOffset() >= lastToken.startOffset()
>
> The analyzers I have tested the highlighter with so far have
> the property:
>
A question before I dive into coding a fix: can I assume (for all analyzers) that the
tokens produced by the tokenStream
have the following property:
currentToken.startOffset() >= lastToken.startOffset()
The analyzers I have tested the highlighter with so far have the property:
currentTok
Erik Hatcher wrote:
On Jun 19, 2004, at 2:29 AM, David Spencer wrote:
A naive analyzer would turn something like "SyncThreadPool" into one
token. Mine uses the great Lucene capability of Tokens being able to
have a "0" position increment to turn it into the token stream:
Sync (incr = 0)
Thread
[EMAIL PROTECTED] wrote:
Yes, this issue has come up before with other choices of analyzers.
I think it should be fixable without changing any of the highlighter APIs
- can you email me or post here the source to your analyzer?
Code attached - don't make fun of it please :) - very prelim. I thi
On Jun 19, 2004, at 2:29 AM, David Spencer wrote:
A naive analyzer would turn something like "SyncThreadPool" into one
token. Mine uses the great Lucene capability of Tokens being able to
have a "0" position increment to turn it into the token stream:
Sync (incr = 0)
Thread (incr = 0)
Pool (in
I've run across an amusing interaction between advanced
Analyzers/TokenStreams and the very useful "term highlighter":
http://cvs.apache.org/viewcvs/jakarta-lucene-sandbox/contributions/highlighter/
I have a custom Analyzer I'm using to index javadoc-generated web pages.
The Analyzer in turn has
