Try change TokenGroup.isDistinct();
Maybe the offset test code should be >= rather than >
ie
boolean isDistinct(Token token)
{
return token.startOffset()>=endOffset;
}
I've just tried the change with the Junit test and all
seems well still with the non CJK tests.
___________________________________________________________
To help you stay safe and secure online, we've developed the all new Yahoo!
Security Centre. http://uk.security.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
