[
https://issues.apache.org/jira/browse/LUCENE-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DM Smith updated LUCENE-1333:
-----------------------------
Attachment: LUCENE-1333.patch
* Added reinit(Token ...) methods to initialize one token from another.
* Improved hashCode.
* Made Token next(Token) have a final argument everywhere it is implemented to
clarify the "best practice" for reuse and did the same for helper methods in
various classes. As part of this, I renamed the token retrieved by next(Token)
to be nextToken.
The typical pattern I used is for TokenFilters:
{code}
public Token next(final Token reusableToken) {
assert reusableToken != null;
Token nextToken = input.next(reusableToken);
if (nextToken != null)
return null;
.... Do something with nextToken ....
return nextToken;
}
{code}
and for other TokenStreams:
{code}
public Token next(final Token reusableToken) {
assert reusableToken != null;
.... Do something with reusableToken ....
return reusableToken;
}
{code}
and for looping over a TokenStream:
{code}
final Token reusableToken = new Token();
for (Token nextToken = stream.next(reusableToken); nextToken != null; nextToken
= stream.next(reusableToken)) {
.... Do something with nextToken ....
}
{code}
* Improved Payload.clone() to avoid new if possible.
* Removed last remaining calls to termText() in a *.jj file.
> Token implementation needs improvements
> ---------------------------------------
>
> Key: LUCENE-1333
> URL: https://issues.apache.org/jira/browse/LUCENE-1333
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 2.3.1
> Environment: All
> Reporter: DM Smith
> Priority: Minor
> Fix For: 2.4
>
> Attachments: LUCENE-1333-analysis.patch, LUCENE-1333-analyzers.patch,
> LUCENE-1333-core.patch, LUCENE-1333-highlighter.patch,
> LUCENE-1333-instantiated.patch, LUCENE-1333-lucli.patch,
> LUCENE-1333-memory.patch, LUCENE-1333-miscellaneous.patch,
> LUCENE-1333-queries.patch, LUCENE-1333-snowball.patch,
> LUCENE-1333-wikipedia.patch, LUCENE-1333-wordnet.patch,
> LUCENE-1333-xml-query-parser.patch, LUCENE-1333.patch, LUCENE-1333.patch,
> LUCENE-1333.patch, LUCENE-1333.patch, LUCENE-1333.patch, LUCENE-1333.patch,
> LUCENE-1333.patch, LUCENE-1333.patch, LUCENE-1333a.txt
>
>
> This was discussed in the thread (not sure which place is best to reference
> so here are two):
> http://mail-archives.apache.org/mod_mbox/lucene-java-dev/200805.mbox/[EMAIL
> PROTECTED]
> or to see it all at once:
> http://www.gossamer-threads.com/lists/lucene/java-dev/62851
> Issues:
> 1. JavaDoc is insufficient, leading one to read the code to figure out how to
> use the class.
> 2. Deprecations are incomplete. The constructors that take String as an
> argument and the methods that take and/or return String should *all* be
> deprecated.
> 3. The allocation policy is too aggressive. With large tokens the resulting
> buffer can be over-allocated. A less aggressive algorithm would be better. In
> the thread, the Python example is good as it is computationally simple.
> 4. The parts of the code that currently use Token's deprecated methods can be
> upgraded now rather than waiting for 3.0. As it stands, filter chains that
> alternate between char[] and String are sub-optimal. Currently, it is used in
> core by Query classes. The rest are in contrib, mostly in analyzers.
> 5. Some internal optimizations can be done with regard to char[] allocation.
> 6. TokenStream has next() and next(Token), next() should be deprecated, so
> that reuse is maximized and descendant classes should be rewritten to
> over-ride next(Token)
> 7. Tokens are often stored as a String in a Term. It would be good to add
> constructors that took a Token. This would simplify the use of the two
> together.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
