[
https://issues.apache.org/jira/browse/LUCENE-2295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841877#action_12841877
]
Uwe Schindler edited comment on LUCENE-2295 at 3/5/10 2:25 PM:
---------------------------------------------------------------
-In the indexer, the backwards code is hairy: If a max limit is set, the
indexer just plugs our Filter around the TokenStream. This would limit nr. of
tokens for all cases:-
# -An ANALYZED field (wrapping the TokenStream got from Analyzer)-
# -A Field with ctor taking TokenStream: wrap the Field's TokenStream-
-We need some cache here, too :( -- The analyzer cannot be used because of (2).-
For backwards we leave the code as it is and just remove the limiting code in
4.0. We just add coments, what needs to be removed.
was (Author: thetaphi):
In the indexer, the backwards code is hairy: If a max limit is set, the
indexer just plugs our Filter around the TokenStream. This would limit nr. of
tokens for all cases:
# An ANALYZED field (wrapping the TokenStream got from Analyzer)
# A Field with ctor taking TokenStream: wrap the Field's TokenStream
We need some cache here, too :( -- The analyzer cannot be used because of (2).
> Create a MaxFieldLengthAnalyzer to wrap any other Analyzer and provide the
> same functionality as MaxFieldLength provided on IndexWriter
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2295
> URL: https://issues.apache.org/jira/browse/LUCENE-2295
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Shai Erera
> Fix For: 3.1
>
>
> A spinoff from LUCENE-2294. Instead of asking the user to specify on
> IndexWriter his requested MFL limit, we can get rid of this setting entirely
> by providing an Analyzer which will wrap any other Analyzer and its
> TokenStream with a TokenFilter that keeps track of the number of tokens
> produced and stop when the limit has reached.
> This will remove any count tracking in IW's indexing, which is done even if I
> specified UNLIMITED for MFL.
> Let's try to do it for 3.1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
