[
https://issues.apache.org/jira/browse/LUCENE-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838094#action_12838094
]
Robert Muir commented on LUCENE-2167:
-------------------------------------
Steven, thanks for providing the link.
I guess this is the point where I also say, I think it would be really nice for
StandardTokenizer to adhere straight to the standard as much as we can with
jflex (I realize in 1.5, we won't have > 0xffff support). Then its name would
actually make sense.
In my opinion, such a transition would involve something like renaming the old
StandardTokenizer to EuropeanTokenizer, as its javadoc claims:
{code}
This should be a good tokenizer for most European-language documents
{code}
The new StandardTokenizer could then say
{code}
This should be a good tokenizer for most languages.
{code}
All the english/euro-centric stuff like the acronym/company/apostrophe stuff
could stay with that "EuropeanTokenizer" or whatever its called, and it could
be used by the european analyzers.
but if we implement the Unicode rules, I think we should drop all this
english/euro-centric stuff for StandardTokenizer. Otherwise it should be called
*StandardishTokenizer*.
we can obviously preserve the backwards compat with Version, as Uwe has created
a way to use a different grammar for a different Version.
I expect some -1 to this, waiting comments :)
> StandardTokenizer Javadoc does not correctly describe tokenization around
> punctuation characters
> ------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2167
> URL: https://issues.apache.org/jira/browse/LUCENE-2167
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.4.1, 2.9, 2.9.1, 3.0
> Reporter: Shyamal Prasad
> Priority: Minor
> Attachments: LUCENE-2167.patch, LUCENE-2167.patch
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> The Javadoc for StandardTokenizer states:
> {quote}
> Splits words at punctuation characters, removing punctuation.
> However, a dot that's not followed by whitespace is considered part of a
> token.
> Splits words at hyphens, unless there's a number in the token, in which case
> the whole
> token is interpreted as a product number and is not split.
> {quote}
> This is not accurate. The actual JFlex implementation treats hyphens
> interchangeably with
> punctuation. So, for example "video,mp4,test" results in a *single* token and
> not three tokens
> as the documentation would suggest.
> Additionally, the documentation suggests that "video-mp4-test-again" would
> become a single
> token, but in reality it results in two tokens: "video-mp4-test" and "again".
> IMHO the parser implementation is fine as is since it is hard to keep
> everyone happy, but it is probably
> worth cleaning up the documentation string.
> The patch included here updates the documentation string and adds a few test
> cases to confirm the cases described above.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
