[
https://issues.apache.org/jira/browse/LUCENE-3848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Muir updated LUCENE-3848:
--------------------------------
Attachment: LUCENE-3848.patch
patch fixing the bug in WikipediaTokenizer.
But i think we just dont have good tests for the removers.
Ideally for tests i think we should have a simple 'MockSynonymsFilter' that is
juts stupid and slow and makes certain synonyms (maybe some multi-word) to use
in testing.
Then we can write tests to find and fix the bugs in the removingfilter.
> basetokenstreamtestcase should fail if tokenstream starts with posinc=0
> -----------------------------------------------------------------------
>
> Key: LUCENE-3848
> URL: https://issues.apache.org/jira/browse/LUCENE-3848
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3848.patch
>
>
> This is meaningless for a tokenstream to start with posinc=0,
> Its also caused problems and hairiness in the indexer (LUCENE-1255,
> LUCENE-1542),
> and it makes senseless tokenstreams. We should add a check and fix any that
> do this.
> Furthermore the same bug can exist in removing-filters if they have
> enablePositionIncrements=false.
> I think this option is useful: but it shouldnt mean 'allow broken
> tokenstream', it just means we
> don't add gaps.
> If you remove tokens with enablePositionIncrements=false it should not cause
> the TS to start with
> positionincrement=0, and it shouldnt 'restructure' the tokenstream (e.g.
> moving synonyms on top of a different word).
> It should just not add any 'holes'.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
