[
https://issues.apache.org/jira/browse/LUCENE-966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12517368
]
Mark Miller commented on LUCENE-966:
------------------------------------
>When digging deeper into the issues of compatibility with the original
>StandardAnalyzer, I stumbled upon something strange. Take the following >text:
>
>78academyawards/rules/rule02.html,7194,7227,type
>
>which was tokenized by the original StandardAnalyzer as one <NUM>. If you look
>at the definition of the <NUM> token:
>
>// every other segment must have at least one digit
><NUM: (<ALPHANUM> <P> <HAS_DIGIT>
> | <HAS_DIGIT> <P> <ALPHANUM>
> | <ALPHANUM> (<P> <HAS_DIGIT> <P> <ALPHANUM>)+
> | <HAS_DIGIT> (<P> <ALPHANUM> <P> <HAS_DIGIT>)+
> | <ALPHANUM> <P> <HAS_DIGIT> (<P> <ALPHANUM> <P> <HAS_DIGIT>)+
> | <HAS_DIGIT> <P> <ALPHANUM> (<P> <HAS_DIGIT> <P> <ALPHANUM>)+
> )
>
>you'll see that, as explained in the comment, every other segment must have at
>least one digit. But actually, according to my understanding, this >rule
>should not match the above text as a whole (and with JFlex it doesn't ,
>actually). Below is the text split by punctuation characters, and it looks
>>like there is no way of splitting this text into alternating segments, every
>second of which must have a digit (A = ALPHANUM, H = HAS_DIGIT):
>
>78academyawards / rules / rule02 . html , 7194 , 7227 , type
> H P A P H P A P H P A P H?* (starting from the beginning)
>
> H?* P A P H P A (starting from the end)
>
>* (would have to be H, but no digits in substring "type" or "html")
>
>I have no idea why JavaCC matched the whole text as a <NUM>, JFlex behaved
>"more correctly" here.
I think that JavaCC is correct here. Every other segment must have a digit:
78academyawards / rules / rule02 . html
It looks to me like every other segment does have a digit. First segment has a
digit, second does not, third does, fourth does not. Matches NUM correctly.
> A faster JFlex-based replacement for StandardAnalyzer
> -----------------------------------------------------
>
> Key: LUCENE-966
> URL: https://issues.apache.org/jira/browse/LUCENE-966
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Reporter: Stanislaw Osinski
> Fix For: 2.3
>
> Attachments: AnalyzerBenchmark.java, jflex-analyzer-patch.txt,
> jflex-analyzer-r560135-patch.txt, jflex-analyzer-r561292-patch.txt,
> jflex-analyzer-r561693-compatibility.txt
>
>
> JFlex (http://www.jflex.de/) can be used to generate a faster (up to several
> times) replacement for StandardAnalyzer. Will add a patch and a simple
> benchmark code in a while.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
