CachingTokenFilter tests fail when using MockTokenizer

2015-03-23 Thread Spyros Kapnissis
Hello,  We have a couple of custom token filters that use CachingTokenFilter internally. However, when we try to test them with MockTokenizer so that we can have these nice TokenStream API checks that it provides, the tests fail with: java.lang.AssertionError: end() called before

How to merge several Taxonomy indexes

2015-03-23 Thread Gimantha Bandara
Hi all, Can anyone point me how to merge several taxonomy indexes? My requirement is as follows. I have several taxonomy indexes and normal document indexes. I want to merge taxonomy indexes together and other document indexes together and perform search on them. One part I have figured out. It

Re: How to merge several Taxonomy indexes

2015-03-23 Thread Christoph Kaser
Hi Gimantha, have a look at the class org.apache.lucene.facet.taxonomy.TaxonomyMergeUtils, which does exactly what you need. Best regards, Christoph Am 23.03.2015 um 15:44 schrieb Gimantha Bandara: Hi all, Can anyone point me how to merge several taxonomy indexes? My requirement is as

Re: CachingTokenFilter tests fail when using MockTokenizer

2015-03-23 Thread Ahmet Arslan
Hi Spyros, Not 100% sure but I think you should override reset method. @Override public void reset() throws IOException { super.reset(); cachedInput = null; } Ahmet On Monday, March 23, 2015 1:29 PM, Spyros Kapnissis ska...@yahoo.com.INVALID wrote: Hello, We have a couple of custom token

Re: How to merge several Taxonomy indexes

2015-03-23 Thread Gimantha Bandara
Hi Christoph, I think TaxonomyMergeUtils is to merge a taxonomy directory and an index together (Correct me if I am wrong). Can it be used to merge several taxonomyDirectories together and create one taxonomy index? On Mon, Mar 23, 2015 at 9:19 PM, Christoph Kaser lucene_l...@iconparc.de wrote:

Re: CachingTokenFilter tests fail when using MockTokenizer

2015-03-23 Thread Spyros Kapnissis
Hello Ahmet,  Unfortunately the test still fails with the same error: end() called before incrementToken() returned false!. I am not sure if I am misusing CachingTokenFilter, or if it cannot be used with MockTokenizer, since it always calls end() before incrementToken() returns false. Spyros

RE: CachingTokenFilter tests fail when using MockTokenizer

2015-03-23 Thread Uwe Schindler
Hi, One of the problems is CachingTokenFilter not 100% conformant to the TokenStream/TokenFilter specs. It is mainly used in Lucene internally for stuff like the highlighter, who needs to consume the same TokenStream multiple times. But when doing this, the code knows how to handle that. One