[
https://issues.apache.org/jira/browse/LUCENE-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844450#action_12844450
]
Michael McCandless commented on LUCENE-2309:
--------------------------------------------
bq. The IndexWriter or rather DocInverterPerField are simply an attribute
consumer. None of them needs to know about Analyzer or TokenStream at all.
Neither needs the analyzer to iterate over tokens.
[Carrying over discussions on IRC with Chris Male & Uwe...]
Actually, TokenStream is already AttrSource + incrementing, so it
seems like the right start...
However, the .reset() method is redundant from indexer's standpoint --
ie when indexer calls Field.getTokenStream (say) whatever init'ing /
reset'ing should already have be done by that method by the time it
returns the TokenStream.
Also, .close and .end are redundant -- seems like we should only have
.end (few token streams do anything in .close...). But coalescing
those two would be a good chunk of work at this point :) Or maybe we
make a .finish that simply both by default ;)
Finally, indexer doesn't really need a Document; it just needs
something abstract that's provides an iterator over all fields that
need indexing (and separately, storing).
> Fully decouple IndexWriter from analyzers
> -----------------------------------------
>
> Key: LUCENE-2309
> URL: https://issues.apache.org/jira/browse/LUCENE-2309
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael McCandless
>
> IndexWriter only needs an AttributeSource to do indexing.
> Yet, today, it interacts with Field instances, holds a private
> analyzers, invokes analyzer.reusableTokenStream, has to deal with a
> wide variety (it's not analyzed; it is analyzed but it's a Reader,
> String; it's pre-analyzed).
> I'd like to have IW only interact with attr sources that already
> arrived with the fields. This would be a powerful decoupling -- it
> means others are free to make their own attr sources.
> They need not even use any of Lucene's analysis impls; eg they can
> integrate to other things like [OpenPipeline|http://www.openpipeline.org].
> Or make something completely custom.
> LUCENE-2302 is already a big step towards this: it makes IW agnostic
> about which attr is "the term", and only requires that it provide a
> BytesRef (for flex).
> Then I think LUCENE-2308 would get us most of the remaining way -- ie, if the
> FieldType knows the analyzer to use, then we could simply create a
> getAttrSource() method (say) on it and move all the logic IW has today
> onto there. (We'd still need existing IW code for back-compat).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
