StandardTokenizer Javadoc does not correctly describe tokenization around
punctuation characters
------------------------------------------------------------------------------------------------
Key: LUCENE-2167
URL: https://issues.apache.org/jira/browse/LUCENE-2167
Project: Lucene - Java
Issue Type: Bug
Affects Versions: 3.0, 2.9.1, 2.9, 2.4.1
Reporter: Shyamal Prasad
Priority: Minor
The Javadoc for StandardTokenization states:
{bq}
Splits words at punctuation characters, removing punctuation.
However, a dot that's not followed by whitespace is considered part of a token.
Splits words at hyphens, unless there's a number in the token, in which case
the whole
token is interpreted as a product number and is not split.
{bq}
This is not accurate. The actual JFlex implementation treats hyphens
interchangeably with
punctuation. So, for example "video,mp4,test" results in a *single* token and
not three tokens
as the documentation would suggest.
Additionally, the documentation suggests that "video-mp4-test-again" would
become a single
token, but in reality it results in two tokens: "video-mp4-test" and "again".
IMHO the parser implementation is fine as is since it is hard to keep everyone
happy, but it is probably
worth cleaning up the documentation string.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
