iterate through tokens in standardtokenizer

[Karthik K]

Hi,
I am trying to use standardTokenizer in a non-lucene project to generate
tokens. The previous versions i used supported token.next , getToken to
iterate over and retrieve the tokens continuously. 3.0.2 doesnt have that
and i cant figure out how to iterate. Can get number of tokens with
increment token, but cant get the exact terms. Is this the right place to
ask?

Thanks