[
https://issues.apache.org/jira/browse/LUCENE-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Smiley updated LUCENE-6121:
---------------------------------
Attachment:
LUCENE-6121_CachingTokenFilter_reset_propagates_reset_if_not_cached.patch
bq. How is that? MockTokenizer should be throwing an exception here. its not ok
to reset twice.
To be clear, this worked and still works (though we might not want to advertise
that it does): _this is what I meant by it'll all work out fine_
{code}
input.reset();
buffer = new CachingTokenFilter(input);
buffer.incrementToken() ...
{code}
This used to work (even though it was bad) and doesn't work any longer --
MockTokenizer will see the second reset() and throw:
{code}
input.reset();
buffer = new CachingTokenFilter(input);
buffer.reset(); // CachingTokenizer used to swallow this but now won't. This
is what was indirectly happening in my story about Highlighter/WSTE/MemoryIndex
but I fixed Highlighter & WSTE.
buffer.incrementToken() ...
{code}
MockTokenizer never 'sees' any reset past the first because if you c
bq. Please change the text to "you must".
Ok.
bq. > In fact there's one query builder ('flexible' AnalyzerQueryNodeProcessor)
that I didn't change to move what it calls reset() on and it still works.
bq. Then it should be fixed, ideally to use MockAnalyzer so the double-reset
causes a test failure first.
Of course, I left it temporarily to show this isn't going to be the
backwards-break we thought it would be. I fixed it in this patch. I also added
a double-reset test to CachingTokenFilter.
RE close(), I could add a boolean closed flag so we only propagate once and
only if cache != null. I'd rather not bother. It's seems you're fine either
way so I'll let it be.
Do you think this patch is ready to be committed? I'll do a "precommit" before
literally doing it.
> Fix CachingTokenFilter to propagate reset() the first time
> ----------------------------------------------------------
>
> Key: LUCENE-6121
> URL: https://issues.apache.org/jira/browse/LUCENE-6121
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: David Smiley
> Assignee: David Smiley
> Fix For: 5.0, Trunk
>
> Attachments:
> LUCENE-6121_CachingTokenFilter_reset_propagates_reset_if_not_cached.patch,
> LUCENE-6121_CachingTokenFilter_reset_propagates_reset_if_not_cached.patch
>
>
> CachingTokenFilter should have been propagating reset() _but only the first
> time_ and thus you would then use CachingTokenFilter in a more normal way –
> wrap it and call reset() then increment in a loop, etc., instead of knowing
> you need to reset() on what it wraps but not this token filter itself. That's
> weird. It's ab-normal for a TokenFilter to never propagate reset, so every
> user of CachingTokenFilter to date has worked around this by calling reset()
> on the underlying input instead of the final wrapping token filter
> (CachingTokenFilter in this case).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
