Re: tokenizer's tokens

2012-11-01 Thread Igal @ getRailo.org
thank you :) On 11/1/2012 4:45 PM, Robert Muir wrote: this is intentional (since you have a bug in your code). you need to call reset(): see the tokenstream contract, step 2: http://lucene.apache.org/core/4_0_0/core/org/apache/lucene/analysis/TokenStream.html On Thu, Nov 1, 2012 at 7:31 PM, I

Re: tokenizer's tokens

2012-11-01 Thread Robert Muir
this is intentional (since you have a bug in your code). you need to call reset(): see the tokenstream contract, step 2: http://lucene.apache.org/core/4_0_0/core/org/apache/lucene/analysis/TokenStream.html On Thu, Nov 1, 2012 at 7:31 PM, Igal @ getRailo.org wrote: > I'm trying to write a very si

tokenizer's tokens

2012-11-01 Thread Igal @ getRailo.org
I'm trying to write a very simple method to show the different tokens that come out of a tokenizer. when I call WhitespaceTokenizer's (or LetterTokenizer's) incrementToken() method though I get an ArrayIndexOutOfBoundsException (see below) any ideas? p.s. if I use StandardTokenizer it works