I didn't read your code, but do you have the "reset" that is now mandatory
and throws AIOOBE if not present?
-- Jack Krupansky
-----Original Message-----
From: andi rexha
Sent: Monday, April 15, 2013 10:21 AM
To: java-user@lucene.apache.org
Subject: WhitespaceTokenizer, incrementToke() ArrayOutOfBoundException
Hi,
I have tryed to get all the tokens from a TokenStream in the same way as I
was doing in the 3.x version of Lucene, but now (at least with
WhitespaceTokenizer) I get an exception:
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: -1
at java.lang.Character.codePointAtImpl(Character.java:2405)
at java.lang.Character.codePointAt(Character.java:2369)
at
org.apache.lucene.analysis.util.CharacterUtils$Java5CharacterUtils.codePointAt(CharacterUtils.java:164)
at
org.apache.lucene.analysis.util.CharTokenizer.incrementToken(CharTokenizer.java:166)
The code is quite simple, and I thought that it could have worked, but
obviously it doesn't (unless I have made some mistakes).
Here is the code, in case you spot some bugs on it (although it is trivial):
String str = "this is a test";
Reader reader = new StringReader(str);
TokenStream tokenStream = new WhitespaceTokenizer(Version.LUCENE_42,
reader); //tokenStreamAnalyzer.tokenStream("test", reader);
CharTermAttribute attribute =
tokenStream.getAttribute(CharTermAttribute.class);
while (tokenStream.incrementToken()) {
System.out.println(new String(attribute.buffer(), 0,
attribute.length()));
}
Hope you have any idea of why it is happening.
Regards,
Andi
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
