StandardTokenizer Javadoc does not correctly describe tokenization around punctuation characters ------------------------------------------------------------------------------------------------
Key: LUCENE-2167 URL: https://issues.apache.org/jira/browse/LUCENE-2167 Project: Lucene - Java Issue Type: Bug Affects Versions: 3.0, 2.9.1, 2.9, 2.4.1 Reporter: Shyamal Prasad Priority: Minor The Javadoc for StandardTokenization states: {bq} Splits words at punctuation characters, removing punctuation. However, a dot that's not followed by whitespace is considered part of a token. Splits words at hyphens, unless there's a number in the token, in which case the whole token is interpreted as a product number and is not split. {bq} This is not accurate. The actual JFlex implementation treats hyphens interchangeably with punctuation. So, for example "video,mp4,test" results in a *single* token and not three tokens as the documentation would suggest. Additionally, the documentation suggests that "video-mp4-test-again" would become a single token, but in reality it results in two tokens: "video-mp4-test" and "again". IMHO the parser implementation is fine as is since it is hard to keep everyone happy, but it is probably worth cleaning up the documentation string. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org