On Oct 13, 2008, at 3:34 PM, Doug Cutting wrote:
Michael Busch wrote:
public abstract boolean nextToken() throws IOException;
What's the point of a separate Token and TokenStream if there's only
a single Token per TokenStream? If that's really the direction
we'll go, then all of the Token methods should be on TokenStream,
and Token should disappear. Are there cases where a stream might
switch token classes midstream? If not, then a single, combined API
should suffice.
There are several streams that analyze the input and output several
tokens for each one in the stream. For example, synonyms, shingles,
stemming.
There are also some caching TokenStreams that can be reset to replay
their stream.
-- DM
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
