[
https://issues.apache.org/jira/browse/LUCENE-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12647346#action_12647346
]
Michael Busch commented on LUCENE-1422:
---------------------------------------
{quote}
Looks good!
{quote}
Thanks for reviewing ... again!
{quote}
I'm a little confused by 'start' and 'initialize' in TokenStream.
{quote}
Good question. The only reason why I added two separate methods here was to
enforce that TokenFilter#start() always calls input.start() first. I think
otherwise this will be a pitfall for people who want to implement their own
filters and override start(). (happened to me when I modified the tests a
couple of times)
On the other hand I agree that Three methods (start(), initialize(), reset())
are confusing. I guess we can deprecate reset() in the future and have start()
rewind the stream if supported by the stream. (I think you mentioned a boolean
method could return true if rewind is supported?)
So I'd be ok with collapsing start/initialize into one method if you prefer
that. I'd just have to add an explicit warning to the javadocs in TokenFilter.
{quote}
I'm seeing this failure (in test I just committed this AM). I think
it's OK, because the new API is enabled for all tests and I'm using
the old API with that analyzer?
{quote}
Yeah, I've to modify the new test to use the new API.
{quote}
- BackCompatTokenStream is calling attributes.put directly but all
others use super's addAttribute.
- Why is BackCompatTokenStream overriding so many methods? EG
has/get/addAttribute - won't super do the same thing?
{quote}
I had a different implementation of BCTS, so this is left-over. Of course
you're right, I can simplify that class.
{quote}
Maybe add reasons to some of the asserts, eg StopFilter has
"assert termAtt != null", so maybe append to that ": initialize()
wasn't called".
{quote}
Yeah good idea, will do.
> New TokenStream API
> -------------------
>
> Key: LUCENE-1422
> URL: https://issues.apache.org/jira/browse/LUCENE-1422
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Analysis
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 2.9
>
> Attachments: lucene-1422-take4.patch, lucene-1422-take5.patch,
> lucene-1422.patch, lucene-1422.take2.patch, lucene-1422.take3.patch,
> lucene-1422.take3.patch
>
>
> This is a very early version of the new TokenStream API that
> we started to discuss here:
> http://www.gossamer-threads.com/lists/lucene/java-dev/66227
> This implementation is a bit different from what I initially
> proposed in the thread above. I introduced a new class called
> AttributedToken, which contains the same termBuffer logic
> from Token. In addition it has a lazily-initialized map of
> Class<? extends Attribute> -> Attribute. Attribute is also a
> new class in a new package, plus several implementations like
> PositionIncrementAttribute, PayloadAttribute, etc.
> Similar to my initial proposal is the prototypeToken() method
> which the consumer (e. g. DocumentsWriter) needs to call.
> The token is created by the tokenizer at the end of the chain
> and pushed through all filters to the end consumer. The
> tokenizer and also all filters can add Attributes to the
> token and can keep references to the actual types of the
> attributes that they need to read of modify. This way, when
> boolean nextToken() is called, no casting is necessary.
> I added a class called TestNewTokenStreamAPI which is not
> really a test case yet, but has a static demo() method, which
> demonstrates how to use the new API.
> The reason to not merge Token and TokenStream into one class
> is that we might have caching (or tee/sink) filters in the
> chain that might want to store cloned copies of the tokens
> in a cache. I added a new class NewCachingTokenStream that
> shows how such a class could work. I also implemented a deep
> clone method in AttributedToken and a
> copyFrom(AttributedToken) method, which is needed for the
> caching. Both methods have to iterate over the list of
> attributes. The Attribute subclasses itself also have a
> copyFrom(Attribute) method, which unfortunately has to down-
> cast to the actual type. I first thought that might be very
> inefficient, but it's not so bad. Well, if you add all
> Attributes to the AttributedToken that our old Token class
> had (like offsets, payload, posIncr), then the performance
> of the caching is somewhat slower (~40%). However, if you
> add less attributes, because not all might be needed, then
> the performance is even slightly faster than with the old API.
> Also the new API is flexible enough so that someone could
> implement a custom caching filter that knows all attributes
> the token can have, then the caching should be just as
> fast as with the old API.
> This patch is not nearly ready, there are lot's of things
> missing:
> - unit tests
> - change DocumentsWriter to use new API
> (in backwards-compatible fashion)
> - patch is currently java 1.5; need to change before
> commiting to 2.9
> - all TokenStreams and -Filters should be changed to use
> new API
> - javadocs incorrect or missing
> - hashcode and equals methods missing in Attributes and
> AttributedToken
>
> I wanted to submit it already for brave people to give me
> early feedback before I spend more time working on this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
