[jira] [Comment Edited] (SOLR-2649) MM ignored in edismax queries with operators
[ https://issues.apache.org/jira/browse/SOLR-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15039772#comment-15039772 ] Erick Erickson edited comment on SOLR-2649 at 12/4/15 5:29 AM: --- [~gpendleb] OK, we're using the same patch which was what I wanted to be sure of. Sorry for the mis-attribution. Regardless of what we call it, my questions are: 1> Are we going to include this change in 5.4.x? I have no intention of doing so, that ship has already sailed. 2> Does this change behavior that we've supported in the past? 2a> If "yes", then we need to concern ourselves with back-compat and it's arguable whether this should be put in 5.x. It seems that it's better behavior than it was before, so perhaps documentation will be sufficient. 2b> If "no", then we're free to put this in both trunk and 5x (5.5 if one comes out) if we think the behavior is better with this patch. Still add a note to CHANGES.txt, along with the issue... I'm tending at this point to put it in the 5x code line and make a point of this change in CHANGES.txt as I think it's a change that'll be seen as positive. I'm not wedded to the idea though. was (Author: erickerickson): [~gpendleb] OK, we're using the same patch which was what I wanted to be sure of. Sorry for the mis-attribution. Regardless of what we call it, my questions are: 1> Are we going to include this change in a point release, i.e. 5.4.x? I have no intention of doing so, that ship has already sailed. 2> Does this change behavior that we've supported in the past? 2a> If "yes", then we need to concern ourselves with back-compat and it's arguable whether this should be put in 5.x. It seems that it's better behavior than it was before, so perhaps documentation will be sufficient. 2b> If "no", then we're free to put this in both trunk and 5x (5.5 if one comes out) if we think the behavior is better with this patch. Still add a note to CHANGES.txt, along with the issue... I'm tending at this point to put it in the 5x code line and make a point of this change in CHANGES.txt as I think it's a change that'll be seen as positive. I'm not wedded to the idea though. > MM ignored in edismax queries with operators > > > Key: SOLR-2649 > URL: https://issues.apache.org/jira/browse/SOLR-2649 > Project: Solr > Issue Type: Improvement > Components: query parsers >Reporter: Magnus Bergmark >Assignee: Erick Erickson > Fix For: 4.9, Trunk > > Attachments: SOLR-2649-with-Qop.patch, SOLR-2649-with-Qop.patch, > SOLR-2649.diff, SOLR-2649.patch > > > Hypothetical scenario: > 1. User searches for "stocks oil gold" with MM set to "50%" > 2. User adds "-stockings" to the query: "stocks oil gold -stockings" > 3. User gets no hits since MM was ignored and all terms where AND-ed > together > The behavior seems to be intentional, although the reason why is never > explained: > // For correct lucene queries, turn off mm processing if there > // were explicit operators (except for AND). > boolean doMinMatched = (numOR + numNOT + numPluses + numMinuses) == 0; > (lines 232-234 taken from > tags/lucene_solr_3_3/solr/src/java/org/apache/solr/search/ExtendedDismaxQParserPlugin.java) > This makes edismax unsuitable as an replacement to dismax; mm is one of the > primary features of dismax. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-2649) MM ignored in edismax queries with operators
[ https://issues.apache.org/jira/browse/SOLR-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038738#comment-15038738 ] Mike edited comment on SOLR-2649 at 12/3/15 10:33 PM: -- I know this is a spammy comment, but I can't help but say how excited I am this is finally getting fixed. It's been a thorn in my side for YEARS. Thanks to everybody that's pushing this forward today. Y'all are fantastic. was (Author: mlissner): I know this is a spamming comment, but I can't help but say how excited I am this is finally getting fixed. It's been a thorn in my side for YEARS. Thanks to everybody that's pushing this forward today. Y'all are fantastic. > MM ignored in edismax queries with operators > > > Key: SOLR-2649 > URL: https://issues.apache.org/jira/browse/SOLR-2649 > Project: Solr > Issue Type: Improvement > Components: query parsers >Reporter: Magnus Bergmark >Assignee: Erick Erickson > Fix For: 4.9, Trunk > > Attachments: SOLR-2649-with-Qop.patch, SOLR-2649-with-Qop.patch, > SOLR-2649.diff, SOLR-2649.patch > > > Hypothetical scenario: > 1. User searches for "stocks oil gold" with MM set to "50%" > 2. User adds "-stockings" to the query: "stocks oil gold -stockings" > 3. User gets no hits since MM was ignored and all terms where AND-ed > together > The behavior seems to be intentional, although the reason why is never > explained: > // For correct lucene queries, turn off mm processing if there > // were explicit operators (except for AND). > boolean doMinMatched = (numOR + numNOT + numPluses + numMinuses) == 0; > (lines 232-234 taken from > tags/lucene_solr_3_3/solr/src/java/org/apache/solr/search/ExtendedDismaxQParserPlugin.java) > This makes edismax unsuitable as an replacement to dismax; mm is one of the > primary features of dismax. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-2649) MM ignored in edismax queries with operators
[ https://issues.apache.org/jira/browse/SOLR-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038626#comment-15038626 ] Greg Pendlebury edited comment on SOLR-2649 at 12/3/15 9:31 PM: I tried Jan's patch, and (whilst it is technically correct) it did not improve the usefulness of edismax without also addressing how q.op is handled. We continued to see absurd search results that failed UAT. The combined patch with both has been on our prod servers since May 2014 without any problems, but I have not heard any feedback from others that might have tried it. The corpus is nearly 200 million fulltext newspaper articles: http://trove.nla.gov.au/newspaper/result?q= was (Author: gpendleb): I tried Jan's patch, and (whilst it is technically correct) it did not improve the usefulness of edismax without also addressing how q.op is handled. We continued to see absurd search results that failed UAT. The combined patch with both has been on our prod servers since May 2014 without any problems, but I have not heard any feedback from others that might have tried it. The corpus is nearly 200 million fulltext newspapers: http://trove.nla.gov.au/newspaper/result?q= > MM ignored in edismax queries with operators > > > Key: SOLR-2649 > URL: https://issues.apache.org/jira/browse/SOLR-2649 > Project: Solr > Issue Type: Improvement > Components: query parsers >Reporter: Magnus Bergmark >Assignee: Erick Erickson > Fix For: 4.9, Trunk > > Attachments: SOLR-2649-with-Qop.patch, SOLR-2649-with-Qop.patch, > SOLR-2649.diff, SOLR-2649.patch > > > Hypothetical scenario: > 1. User searches for "stocks oil gold" with MM set to "50%" > 2. User adds "-stockings" to the query: "stocks oil gold -stockings" > 3. User gets no hits since MM was ignored and all terms where AND-ed > together > The behavior seems to be intentional, although the reason why is never > explained: > // For correct lucene queries, turn off mm processing if there > // were explicit operators (except for AND). > boolean doMinMatched = (numOR + numNOT + numPluses + numMinuses) == 0; > (lines 232-234 taken from > tags/lucene_solr_3_3/solr/src/java/org/apache/solr/search/ExtendedDismaxQParserPlugin.java) > This makes edismax unsuitable as an replacement to dismax; mm is one of the > primary features of dismax. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-2649) MM ignored in edismax queries with operators
[
https://issues.apache.org/jira/browse/SOLR-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055567#comment-14055567
]
Shawn Heisey edited comment on SOLR-2649 at 7/8/14 9:31 PM:
We got bitten by this bug today. Consider this filter query, constructed from
multiple entry boxes on an advanced search form:
{code}
{!edismax}publish:(0) AND caption:(john lennon)
{code}
The mm value is 100%, but we get results as if mm were 0.
What we're going to do for a solution is create multiple fq parameters instead
of combining with AND.
was (Author: elyograg):
We got bitten by this bug today. Consider this filter query, constructed from
multiple entry boxes on an advanced search form:
{!edismax}publish:(0) AND caption:(john lennon)
The mm value is 100%, but we get results as if mm were 0.
What we're going to do for a solution is create multiple fq parameters instead
of combining with AND.
> MM ignored in edismax queries with operators
>
>
> Key: SOLR-2649
> URL: https://issues.apache.org/jira/browse/SOLR-2649
> Project: Solr
> Issue Type: Bug
> Components: query parsers
>Reporter: Magnus Bergmark
>Priority: Minor
> Fix For: 4.9, 5.0
>
> Attachments: SOLR-2649-with-Qop.patch, SOLR-2649.diff, SOLR-2649.patch
>
>
> Hypothetical scenario:
> 1. User searches for "stocks oil gold" with MM set to "50%"
> 2. User adds "-stockings" to the query: "stocks oil gold -stockings"
> 3. User gets no hits since MM was ignored and all terms where AND-ed
> together
> The behavior seems to be intentional, although the reason why is never
> explained:
> // For correct lucene queries, turn off mm processing if there
> // were explicit operators (except for AND).
> boolean doMinMatched = (numOR + numNOT + numPluses + numMinuses) == 0;
> (lines 232-234 taken from
> tags/lucene_solr_3_3/solr/src/java/org/apache/solr/search/ExtendedDismaxQParserPlugin.java)
> This makes edismax unsuitable as an replacement to dismax; mm is one of the
> primary features of dismax.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-2649) MM ignored in edismax queries with operators
[ https://issues.apache.org/jira/browse/SOLR-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986408#comment-13986408 ] Greg Pendlebury edited comment on SOLR-2649 at 5/1/14 6:54 AM: --- I applied this patch to 4.7.2 Yesterday and tried it out on our dev servers. At first I thought it was pretty bad and failed completely... but then I had a good think and re-read everything on this ticket and this[1] article and realised my understanding of the problem was flawed. Using just this patch in isolation it converted all of the OR operators to AND operators with mm=100%. Very confusing behaviour for our business area, but I realise now that it is correct. Perhaps the confusion stems from the way the q.op and mm parameters interact. If the behaviour was to instead separate them more clearly then we could change the config entirely. At the moment our mm is 100% because we effectively want q.op=AND, but if q.op was instead applied 1) always, 2) first and 3) independently from mm (ie. insert AND wherever an operator is missing) we could set mm=1 and achieve what we want by respecting the OR parameters provided by the user. I've added this on top of the patch already here and deployed again to our dev servers using 'q.op=AND & mm=1' and now everything appears to function as it should. I'll upload the patch in a minute, and it includes several unit tests with different mm and q.op values. From my perspective I think the two parameters are interacting appropriately, but perhaps someone with more convoluted mm settings could give it a try? The change is simply in the constructor of the ExtendedSolrQueryParser class where it was hardcoded to force the default operator to OR (presumably so that mm would take care of things) I've made it look at the parameter provided with the query (copied the code from the Simple QParser and adjusted to fit). The unit test from the first patch that was marked TODO I have tweaked slightly. I think not finding a result in that case is entirely appropriate if the user can now tweak q.op. Opinions may vary of course. [1] http://searchhub.org/2011/12/28/why-not-and-or-and-not/ was (Author: gpendleb): I applied this patch to 4.7.2 Yesterday and tried it out on or dev servers. At first I thought it was pretty bad and failed completely... but then I had a good think and re-read everything on this ticket and this[1] article and realised my understanding of the problem was flawed. Using just this patch in isolation it converted all of the OR operators to AND operators with mm=100%. Very confusing behaviour for our business area, but I realise now that it is correct. Perhaps the confusion stems from the way the q.op and mm parameters interact. If the behaviour was to instead separate them more clearly then we could change the config entirely. At the moment our mm is 100% because we effectively want q.op=AND, but if q.op was instead applied 1) always, 2) first and 3) independently from mm (ie. insert AND wherever an operator is missing) we could set mm=1 and achieve what we want by respecting the OR parameters provided by the user. I've added this on top of the patch already here and deployed again to our dev servers using 'q.op=AND & mm=1' and now everything appears to function as it should. I'll upload the patch in a minute, and it includes several unit tests with different mm and q.op values. From my perspective I think the two parameters are interacting appropriately, but perhaps someone with more convoluted mm settings could give it a try? The change is simply in the constructor of the ExtendedSolrQueryParser class where it was hardcoded to force the default operator to OR (presumably so that mm would take care of things) I've made it look at the parameter provided with the query (copied the code from the Simple QParser and adjusted to fit). The unit test from the first patch that was marked TODO I have tweaked slightly. I think not finding a result in that case is entirely appropriate if the user can now tweak q.op. Opinions may vary of course. [1] http://searchhub.org/2011/12/28/why-not-and-or-and-not/ > MM ignored in edismax queries with operators > > > Key: SOLR-2649 > URL: https://issues.apache.org/jira/browse/SOLR-2649 > Project: Solr > Issue Type: Bug > Components: query parsers >Reporter: Magnus Bergmark >Priority: Minor > Fix For: 4.9, 5.0 > > Attachments: SOLR-2649-with-Qop.patch, SOLR-2649.diff, SOLR-2649.patch > > > Hypothetical scenario: > 1. User searches for "stocks oil gold" with MM set to "50%" > 2. User adds "-stockings" to the query: "stocks oil gold -stockings" > 3. User gets no hits since MM was ignored and all terms where AND-ed > together > The beh
[jira] [Comment Edited] (SOLR-2649) MM ignored in edismax queries with operators
[
https://issues.apache.org/jira/browse/SOLR-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13822482#comment-13822482
]
Anca Kopetz edited comment on SOLR-2649 at 11/21/13 4:10 PM:
-
We need to apply Min should match for edismax query strings with operators
(AND,OR) and mm parameter.
For example, when the below query was parsed, the mm was not applied
{code}
&q=(((termA AND termB) OR specialTerm) (termC AND termD)
(termE))&mm=2&defType=edismax&qf=title
{code}
Therefore we developed our custom query parser.
The code is below, maybe it is useful for somebody who has the same
requirements.
{code:title=CustomExtendedDismaxQParser.java}
public class CustomExtendedDismaxQParser extends ExtendedDismaxQParser {
public CustomExtendedDismaxQParser(String qstr, SolrParams localParams,
SolrParams params, SolrQueryRequest req) {
super(qstr, localParams, params, req);
}
@Override
protected Query parseOriginalQuery(ExtendedSolrQueryParser up, String
mainUserQuery, List clauses, ExtendedDismaxConfiguration config) {
Query query = super.parseOriginalQuery(up, mainUserQuery, clauses,
config);
String mmValue = this.params.get(DisMaxParams.MM);
if(!Strings.isNullOrEmpty(mmValue)) {
if (query instanceof BooleanQuery) {
SolrPluginUtils.setMinShouldMatch((BooleanQuery)query, mmValue);
}
}
return query;
}
}
{code}
{code:title=solrconfig.xml}
{code}
Then we set defType=customEdismax in the query parameters.
With these configuration, mm is applied on top-level clauses. In our example,
there are 3 top-level SHOULD clauses :
{code} ((termA AND termB) OR specialTerm), (termC AND termD), (termE)
{code}
And the parsed query is :
{code}
+((
((+(title:termA) +(title:termB)) (title:specialTerm))
(+(title:termC) +(title:termD))
(title:termE)
)~2)
{code}
was (Author: agh):
We need to apply Min should match for edismax query strings with operators
(AND,OR) and mm parameter.
Therefore we developed our custom query parser.
The code is below, maybe it is useful for somebody who has the same
requirements.
{code:title=CustomExtendedDismaxQParser.java}
public class CustomExtendedDismaxQParser extends ExtendedDismaxQParser {
public CustomExtendedDismaxQParser(String qstr, SolrParams localParams,
SolrParams params, SolrQueryRequest req) {
super(qstr, localParams, params, req);
}
@Override
protected Query parseOriginalQuery(ExtendedSolrQueryParser up, String
mainUserQuery, List clauses, ExtendedDismaxConfiguration config) {
Query query = super.parseOriginalQuery(up, mainUserQuery, clauses,
config);
String mmValue = this.params.get(DisMaxParams.MM);
if(!Strings.isNullOrEmpty(mmValue)) {
if (query instanceof BooleanQuery) {
SolrPluginUtils.setMinShouldMatch((BooleanQuery)query, mmValue);
}
}
return query;
}
}
{code}
{code:title=solrconfig.xml}
{code}
> MM ignored in edismax queries with operators
>
>
> Key: SOLR-2649
> URL: https://issues.apache.org/jira/browse/SOLR-2649
> Project: Solr
> Issue Type: Bug
> Components: query parsers
>Reporter: Magnus Bergmark
>Priority: Minor
> Fix For: 4.6
>
>
> Hypothetical scenario:
> 1. User searches for "stocks oil gold" with MM set to "50%"
> 2. User adds "-stockings" to the query: "stocks oil gold -stockings"
> 3. User gets no hits since MM was ignored and all terms where AND-ed
> together
> The behavior seems to be intentional, although the reason why is never
> explained:
> // For correct lucene queries, turn off mm processing if there
> // were explicit operators (except for AND).
> boolean doMinMatched = (numOR + numNOT + numPluses + numMinuses) == 0;
> (lines 232-234 taken from
> tags/lucene_solr_3_3/solr/src/java/org/apache/solr/search/ExtendedDismaxQParserPlugin.java)
> This makes edismax unsuitable as an replacement to dismax; mm is one of the
> primary features of dismax.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-2649) MM ignored in edismax queries with operators
[ https://issues.apache.org/jira/browse/SOLR-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662455#comment-13662455 ] Naomi Dushay edited comment on SOLR-2649 at 5/20/13 11:41 PM: -- Our dismax mm setting is 6<-1 6<90%. I would like our mm to be honored for the top-level SHOULD clauses. Oh please, oh please? EDISMAX q=customer driven academic library: +(((custom)~0.01 (driven)~0.01 (academ)~0.01 (librari)~0.01)~4) 4 hits customer NOT driven academic library: +((custom)~0.01 -(driven)~0.01 (academ)~0.01 (librari)~0.01) 984300 hits <= INSANE customer -driven academic library: +((custom)~0.01 -(driven)~0.01 (academ)~0.01 (librari)~0.01) 984300 hits <= INSANE customer OR academic OR library NOT driven: +((custom)~0.01 (academ)~0.01 (librari)~0.01 -(driven)~0.01) 984300 hits customer academic library: +(((custom)~0.01 (academ)~0.01 (librari)~0.01)~3) 100 hits DISMAX (plausible results!): customer driven academic library: +(((custom)~0.01 (driven)~0.01 (academ)~0.01 (librari)~0.01)~4) () 4 hits customer NOT driven academic library: +(((custom)~0.01 -(driven)~0.01 (academ)~0.01 (librari)~0.01)~3) () 96 hits customer -driven academic library: +(((custom)~0.01 -(driven)~0.01 (academ)~0.01 (librari)~0.01)~3) () 96 hits customer academic library: +(((custom)~0.01 (academ)~0.01 (librari)~0.01)~3)() 100 hits was (Author: ndushay): Our dismax mm setting is 6<-1 6<90%. I would like our mm to be honored for the top-level SHOULD clauses. Oh please, oh please? EDISMAX q=customer driven academic library: +(((custom)~0.01 (driven)~0.01 (academ)~0.01 (librari)~0.01)~4) 4 hits customer NOT driven academic library: +((custom)~0.01 -(driven)~0.01 (academ)~0.01 (librari)~0.01) 984300 hits <= INSANE customer -driven academic library: +((custom)~0.01 -(driven)~0.01 (academ)~0.01 (librari)~0.01) 984300 hits <= INSANE customer OR academic OR library NOT driven: +((custom)~0.01 (academ)~0.01 (librari)~0.01 -(driven)~0.01) 984300 hits customer academic library: +(((custom)~0.01 (academ)~0.01 (librari)~0.01)~3) 100 hits DISMAX (plausible results!): customer driven academic library: +(((custom)~0.01 (driven)~0.01 (academ)~0.01 (librari)~0.01)~4) ()4 hits customer NOT driven academic library: +(((custom)~0.01 -(driven)~0.01 (academ)~0.01 (librari)~0.01)~3) () 96 hits customer -driven academic library: +(((custom)~0.01 -(driven)~0.01 (academ)~0.01 (librari)~0.01)~3) () 96 hits customer academic library: +(((custom)~0.01 (academ)~0.01 (librari)~0.01)~3)() 100 hits > MM ignored in edismax queries with operators > > > Key: SOLR-2649 > URL: https://issues.apache.org/jira/browse/SOLR-2649 > Project: Solr > Issue Type: Bug > Components: query parsers >Reporter: Magnus Bergmark >Priority: Minor > Fix For: 4.4 > > > Hypothetical scenario: > 1. User searches for "stocks oil gold" with MM set to "50%" > 2. User adds "-stockings" to the query: "stocks oil gold -stockings" > 3. User gets no hits since MM was ignored and all terms where AND-ed > together > The behavior seems to be intentional, although the reason why is never > explained: > // For correct lucene queries, turn off mm processing if there > // were explicit operators (except for AND). > boolean doMinMatched = (numOR + numNOT + numPluses + numMinuses) == 0; > (lines 232-234 taken from > tags/lucene_solr_3_3/solr/src/java/org/apache/solr/search/ExtendedDismaxQParserPlugin.java) > This makes edismax unsuitable as an replacement to dismax; mm is one of the > primary features of dismax. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
