Re: custom token filter generates empty tokens

2014-10-10 Thread G.Long
Thanks Ahmet and Jose for you help :) Regards, Le 09/10/2014 18:29, Jose Fernandez a écrit : When you return true from incrementToken() you tell Lucene to add your token to the token stream. At the end of incrementToken() check for an empty token. If it's empty then return incrementToken()

RE: custom token filter generates empty tokens

2014-10-09 Thread Jose Fernandez
When you return true from incrementToken() you tell Lucene to add your token to the token stream. At the end of incrementToken() check for an empty token. If it's empty then return incrementToken() to process the next token. This will affect your positions so if you're doing phrase search you

Re: custom token filter generates empty tokens

2014-10-09 Thread Ahmet Arslan
Hi G.Long, You can use TrimFilter+LengthFilter to remove empty/whitespace tokens. Ahmet On Thursday, October 9, 2014 5:54 PM, G.Long jde...@gmail.com wrote: Hi :) I wrote a custom token filter which removes special characters. Sometimes, all characters of the token are removed so the filter