[
https://issues.apache.org/jira/browse/LUCENE-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12871642#action_12871642
]
Robert Muir commented on LUCENE-2458:
-------------------------------------
bq. I've got to -1 this commit.
As mentioned on apache's website:
{code}
To prevent vetos from being used capriciously, they must be accompanied by a
technical justification showing why the change is bad (opens a security
exposure, negatively affects performance, etc.). A veto without a justification
is invalid and has no weight.
{code}
No one has been able to provide any technical justifications, only subjective
opinions.
When standard test collections were used, it was shown that this behavior
significant hurts CJK and delivers only 10% of standard IR techniques (not
generating phrases but using boolean word/bigram queries). See Ivan's results
above. This isn't surprising since CJK IR has been pretty well studied, there
is nothing new here.
At the same time, when english test collections were used, there was no
difference, on the contrary, it only tended to slightly improve relevance for
english, too.
Why do we even bother trying to start an openrelevance project if people do not
want to go with the scientific method but prefer subjective opinion?
> queryparser makes all CJK queries phrase queries regardless of analyzer
> -----------------------------------------------------------------------
>
> Key: LUCENE-2458
> URL: https://issues.apache.org/jira/browse/LUCENE-2458
> Project: Lucene - Java
> Issue Type: Bug
> Components: QueryParser
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Blocker
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2458.patch, LUCENE-2458.patch, LUCENE-2458.patch
>
>
> The queryparser automatically makes *ALL* CJK, Thai, Lao, Myanmar, Tibetan,
> ... queries into phrase queries, even though you didn't ask for one, and
> there isn't a way to turn this off.
> This completely breaks lucene for these languages, as it treats all queries
> like 'grep'.
> Example: if you query for f:abcd with standardanalyzer, where a,b,c,d are
> chinese characters, you get a phrasequery of "a b c d". if you use cjk
> analyzer, its no better, its a phrasequery of "ab bc cd", and if you use
> smartchinese analyzer, you get a phrasequery like "ab cd". But the user
> didn't ask for one, and they cannot turn it off.
> The reason is that the code to form phrase queries is not internationally
> appropriate and assumes whitespace tokenization. If more than one token comes
> out of whitespace delimited text, its automatically a phrase query no matter
> what.
> The proposed patch fixes the core queryparser (with all backwards compat
> kept) to only form phrase queries when the double quote operator is used.
> Implementing subclasses can always extend the QP and auto-generate whatever
> kind of queries they want that might completely break search for languages
> they don't care about, but core general-purpose QPs should be language
> independent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
