Re: Query generation is different for search terms with and without "-"
Ages ago at Netflix, I fixed this with a few hundred synonyms. If you are working with a fixed vocabulary (movie titles, product names), that can work just fine. babysitter, baby-sitter, baby sitter fullmetal, full-metal, full metal manhunter, man-hunter, man hunter spiderman, spider-man, spider man wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Nov 25, 2020, at 9:26 AM, Erick Erickson wrote: > > Parameters, no. You could use a PatternReplaceCharFilterFactory. NOTE: > > *FilterFactory are _not_ what you want in this case, they are applied to > individual tokens after parsing > > *CharFiterFactory are invoked on the entire input to the field, although I > can’t say for certain that even that’s early enough. > > There are two other options to consider: > StatelessScriptUpdateProcessor > FieldMutatingUpdateProcessor > > Stateless... is probably easiest… > > Best, > ERick > >> On Nov 24, 2020, at 1:44 PM, Samuel Gutierrez >> wrote: >> >> Are there any good workarounds/parameters we can use to fix this so it >> doesn't have to be solved client side? >> >> On Tue, Nov 24, 2020 at 7:50 AM matthew sporleder >> wrote: >> >>> Is the normal/standard solution here to regex remove the '-'s and >>> combine them into a single token? >>> >>> On Tue, Nov 24, 2020 at 8:00 AM Erick Erickson >>> wrote: This is a common point of confusion. There are two phases for creating a >>> query, query _parsing_ first, then the analysis chain for the parsed result. So what e-dismax sees in the two cases is: Name_enUS:“high tech” -> two tokens, since there are two of them pf2 >>> comes into play. Name_enUS:“high-tech” -> there’s only one token so pf2 doesn’t apply, >>> splitting it on the hyphen comes later. It’s especially confusing since the field analysis then breaks up >>> “high-tech” into two tokens that look the same as “high tech” in the debug response, just without the >>> phrase query. Name_enUS:high Name_enUS:tech Best, Erick > On Nov 23, 2020, at 8:32 PM, Samuel Gutierrez < >>> samuel.gutier...@iherb.com.INVALID> wrote: > > I am troubleshooting an issue with ranking for search terms that >>> contain a > "-" vs the same query that does not contain the dash e.g. "high-tech" >>> vs > "high tech". The field that I am querying is using the standard >>> tokenizer, > so I would expect that the underlying lucene query should be the same >>> for > both versions of the query, however when printing the debug, it appears > they are generated differently. I know "-" must be escaped as it has > special meaning in lucene, however escaping does not fix the problem. >>> It > appears that with the "-" present, the pf2 edismax parameter is not > respected and omitted from the final query. We use sow=false as we have > multiterm synonyms and need to ensure they are included in the final >>> lucene > query. My expectation is that the final underlying lucene query should >>> be > based on the output of the field analyzer, however after briefly >>> looking > at the code for ExtendedDismaxQParser, it appears that there is some >>> string > processing happening outside of the analysis step which causes the > unexpected lucene query. > > > Solr Debug for "high tech": > > parsedquery: "+(DisjunctionMaxQuery((Name_enUS:high)~0.4) > DisjunctionMaxQuery((Name_enUS:tech)~0.4))~2 > DisjunctionMaxQuery((Name_enUS:"high tech"~5)~0.4) > DisjunctionMaxQuery((Name_enUS:"high tech"~4)~0.4)", > parsedquery_toString: "+(((Name_enUS:high)~0.4 > (Name_enUS:tech)~0.4)~2) (Name_enUS:"high tech"~5)~0.4 > (Name_enUS:"high tech"~4)~0.4", > > > Solr Debug for "high-tech" > > parsedquery: "+DisjunctionMaxQueryName_enUS:high > Name_enUS:tech)~2))~0.4) DisjunctionMaxQuery((Name_enUS:"high > tech"~5)~0.4)", > parsedquery_toString: "+(((Name_enUS:high Name_enUS:tech)~2))~0.4 > (Name_enUS:"high tech"~5)~0.4" > > SolrConfig: > > > >true >true >json >375% >Name_enUS >Name_enUS >5 >Name_enUS >4 >3 >0.4 >explicit >100 >false > > >edismax > > > > Schema: > > >> positionIncrementGap="100"> > > > > > > > > > > Using Solr 8.6.3 > >>> >> >> -- >> *The information contained in this message is the sole and exclusive >> property of ***iHerb Inc.*** and may be privileged and confidential. It may >> not be disseminated or distributed to persons or entities other than the >> ones intended without the written authority of ***iHerb Inc.** *If you have >> received this e-mail in error or are
Re: Query generation is different for search terms with and without "-"
Parameters, no. You could use a PatternReplaceCharFilterFactory. NOTE: *FilterFactory are _not_ what you want in this case, they are applied to individual tokens after parsing *CharFiterFactory are invoked on the entire input to the field, although I can’t say for certain that even that’s early enough. There are two other options to consider: StatelessScriptUpdateProcessor FieldMutatingUpdateProcessor Stateless... is probably easiest… Best, ERick > On Nov 24, 2020, at 1:44 PM, Samuel Gutierrez > wrote: > > Are there any good workarounds/parameters we can use to fix this so it > doesn't have to be solved client side? > > On Tue, Nov 24, 2020 at 7:50 AM matthew sporleder > wrote: > >> Is the normal/standard solution here to regex remove the '-'s and >> combine them into a single token? >> >> On Tue, Nov 24, 2020 at 8:00 AM Erick Erickson >> wrote: >>> >>> This is a common point of confusion. There are two phases for creating a >> query, >>> query _parsing_ first, then the analysis chain for the parsed result. >>> >>> So what e-dismax sees in the two cases is: >>> >>> Name_enUS:“high tech” -> two tokens, since there are two of them pf2 >> comes into play. >>> >>> Name_enUS:“high-tech” -> there’s only one token so pf2 doesn’t apply, >> splitting it on the hyphen comes later. >>> >>> It’s especially confusing since the field analysis then breaks up >> “high-tech” into two tokens that >>> look the same as “high tech” in the debug response, just without the >> phrase query. >>> >>> Name_enUS:high >>> Name_enUS:tech >>> >>> Best, >>> Erick >>> On Nov 23, 2020, at 8:32 PM, Samuel Gutierrez < >> samuel.gutier...@iherb.com.INVALID> wrote: I am troubleshooting an issue with ranking for search terms that >> contain a "-" vs the same query that does not contain the dash e.g. "high-tech" >> vs "high tech". The field that I am querying is using the standard >> tokenizer, so I would expect that the underlying lucene query should be the same >> for both versions of the query, however when printing the debug, it appears they are generated differently. I know "-" must be escaped as it has special meaning in lucene, however escaping does not fix the problem. >> It appears that with the "-" present, the pf2 edismax parameter is not respected and omitted from the final query. We use sow=false as we have multiterm synonyms and need to ensure they are included in the final >> lucene query. My expectation is that the final underlying lucene query should >> be based on the output of the field analyzer, however after briefly >> looking at the code for ExtendedDismaxQParser, it appears that there is some >> string processing happening outside of the analysis step which causes the unexpected lucene query. Solr Debug for "high tech": parsedquery: "+(DisjunctionMaxQuery((Name_enUS:high)~0.4) DisjunctionMaxQuery((Name_enUS:tech)~0.4))~2 DisjunctionMaxQuery((Name_enUS:"high tech"~5)~0.4) DisjunctionMaxQuery((Name_enUS:"high tech"~4)~0.4)", parsedquery_toString: "+(((Name_enUS:high)~0.4 (Name_enUS:tech)~0.4)~2) (Name_enUS:"high tech"~5)~0.4 (Name_enUS:"high tech"~4)~0.4", Solr Debug for "high-tech" parsedquery: "+DisjunctionMaxQueryName_enUS:high Name_enUS:tech)~2))~0.4) DisjunctionMaxQuery((Name_enUS:"high tech"~5)~0.4)", parsedquery_toString: "+(((Name_enUS:high Name_enUS:tech)~2))~0.4 (Name_enUS:"high tech"~5)~0.4" SolrConfig: true true json 375% Name_enUS Name_enUS 5 Name_enUS 4 3 0.4 explicit 100 false edismax Schema: > positionIncrementGap="100"> Using Solr 8.6.3 >> > > -- > *The information contained in this message is the sole and exclusive > property of ***iHerb Inc.*** and may be privileged and confidential. It may > not be disseminated or distributed to persons or entities other than the > ones intended without the written authority of ***iHerb Inc.** *If you have > received this e-mail in error or are not the intended recipient, you may > not use, copy, disseminate or distribute it. Do not open any attachments. > Please delete it immediately from your system and notify the sender > promptly by e-mail that you have done so.*
Re: Query generation is different for search terms with and without "-"
Are there any good workarounds/parameters we can use to fix this so it doesn't have to be solved client side? On Tue, Nov 24, 2020 at 7:50 AM matthew sporleder wrote: > Is the normal/standard solution here to regex remove the '-'s and > combine them into a single token? > > On Tue, Nov 24, 2020 at 8:00 AM Erick Erickson > wrote: > > > > This is a common point of confusion. There are two phases for creating a > query, > > query _parsing_ first, then the analysis chain for the parsed result. > > > > So what e-dismax sees in the two cases is: > > > > Name_enUS:“high tech” -> two tokens, since there are two of them pf2 > comes into play. > > > > Name_enUS:“high-tech” -> there’s only one token so pf2 doesn’t apply, > splitting it on the hyphen comes later. > > > > It’s especially confusing since the field analysis then breaks up > “high-tech” into two tokens that > > look the same as “high tech” in the debug response, just without the > phrase query. > > > > Name_enUS:high > > Name_enUS:tech > > > > Best, > > Erick > > > > > On Nov 23, 2020, at 8:32 PM, Samuel Gutierrez < > samuel.gutier...@iherb.com.INVALID> wrote: > > > > > > I am troubleshooting an issue with ranking for search terms that > contain a > > > "-" vs the same query that does not contain the dash e.g. "high-tech" > vs > > > "high tech". The field that I am querying is using the standard > tokenizer, > > > so I would expect that the underlying lucene query should be the same > for > > > both versions of the query, however when printing the debug, it appears > > > they are generated differently. I know "-" must be escaped as it has > > > special meaning in lucene, however escaping does not fix the problem. > It > > > appears that with the "-" present, the pf2 edismax parameter is not > > > respected and omitted from the final query. We use sow=false as we have > > > multiterm synonyms and need to ensure they are included in the final > lucene > > > query. My expectation is that the final underlying lucene query should > be > > > based on the output of the field analyzer, however after briefly > looking > > > at the code for ExtendedDismaxQParser, it appears that there is some > string > > > processing happening outside of the analysis step which causes the > > > unexpected lucene query. > > > > > > > > > Solr Debug for "high tech": > > > > > > parsedquery: "+(DisjunctionMaxQuery((Name_enUS:high)~0.4) > > > DisjunctionMaxQuery((Name_enUS:tech)~0.4))~2 > > > DisjunctionMaxQuery((Name_enUS:"high tech"~5)~0.4) > > > DisjunctionMaxQuery((Name_enUS:"high tech"~4)~0.4)", > > > parsedquery_toString: "+(((Name_enUS:high)~0.4 > > > (Name_enUS:tech)~0.4)~2) (Name_enUS:"high tech"~5)~0.4 > > > (Name_enUS:"high tech"~4)~0.4", > > > > > > > > > Solr Debug for "high-tech" > > > > > > parsedquery: "+DisjunctionMaxQueryName_enUS:high > > > Name_enUS:tech)~2))~0.4) DisjunctionMaxQuery((Name_enUS:"high > > > tech"~5)~0.4)", > > > parsedquery_toString: "+(((Name_enUS:high Name_enUS:tech)~2))~0.4 > > > (Name_enUS:"high tech"~5)~0.4" > > > > > > SolrConfig: > > > > > > > > > > > > true > > > true > > > json > > > 375% > > > Name_enUS > > > Name_enUS > > > 5 > > > Name_enUS > > > 4 > > > 3 > > > 0.4 > > > explicit > > > 100 > > > false > > > > > > > > > edismax > > > > > > > > > > > > Schema: > > > > > > positionIncrementGap="100"> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Using Solr 8.6.3 > > > > -- *The information contained in this message is the sole and exclusive property of ***iHerb Inc.*** and may be privileged and confidential. It may not be disseminated or distributed to persons or entities other than the ones intended without the written authority of ***iHerb Inc.** *If you have received this e-mail in error or are not the intended recipient, you may not use, copy, disseminate or distribute it. Do not open any attachments. Please delete it immediately from your system and notify the sender promptly by e-mail that you have done so.*
Re: Query generation is different for search terms with and without "-"
Is the normal/standard solution here to regex remove the '-'s and combine them into a single token? On Tue, Nov 24, 2020 at 8:00 AM Erick Erickson wrote: > > This is a common point of confusion. There are two phases for creating a > query, > query _parsing_ first, then the analysis chain for the parsed result. > > So what e-dismax sees in the two cases is: > > Name_enUS:“high tech” -> two tokens, since there are two of them pf2 comes > into play. > > Name_enUS:“high-tech” -> there’s only one token so pf2 doesn’t apply, > splitting it on the hyphen comes later. > > It’s especially confusing since the field analysis then breaks up “high-tech” > into two tokens that > look the same as “high tech” in the debug response, just without the phrase > query. > > Name_enUS:high > Name_enUS:tech > > Best, > Erick > > > On Nov 23, 2020, at 8:32 PM, Samuel Gutierrez > > wrote: > > > > I am troubleshooting an issue with ranking for search terms that contain a > > "-" vs the same query that does not contain the dash e.g. "high-tech" vs > > "high tech". The field that I am querying is using the standard tokenizer, > > so I would expect that the underlying lucene query should be the same for > > both versions of the query, however when printing the debug, it appears > > they are generated differently. I know "-" must be escaped as it has > > special meaning in lucene, however escaping does not fix the problem. It > > appears that with the "-" present, the pf2 edismax parameter is not > > respected and omitted from the final query. We use sow=false as we have > > multiterm synonyms and need to ensure they are included in the final lucene > > query. My expectation is that the final underlying lucene query should be > > based on the output of the field analyzer, however after briefly looking > > at the code for ExtendedDismaxQParser, it appears that there is some string > > processing happening outside of the analysis step which causes the > > unexpected lucene query. > > > > > > Solr Debug for "high tech": > > > > parsedquery: "+(DisjunctionMaxQuery((Name_enUS:high)~0.4) > > DisjunctionMaxQuery((Name_enUS:tech)~0.4))~2 > > DisjunctionMaxQuery((Name_enUS:"high tech"~5)~0.4) > > DisjunctionMaxQuery((Name_enUS:"high tech"~4)~0.4)", > > parsedquery_toString: "+(((Name_enUS:high)~0.4 > > (Name_enUS:tech)~0.4)~2) (Name_enUS:"high tech"~5)~0.4 > > (Name_enUS:"high tech"~4)~0.4", > > > > > > Solr Debug for "high-tech" > > > > parsedquery: "+DisjunctionMaxQueryName_enUS:high > > Name_enUS:tech)~2))~0.4) DisjunctionMaxQuery((Name_enUS:"high > > tech"~5)~0.4)", > > parsedquery_toString: "+(((Name_enUS:high Name_enUS:tech)~2))~0.4 > > (Name_enUS:"high tech"~5)~0.4" > > > > SolrConfig: > > > > > > > > true > > true > > json > > 375% > > Name_enUS > > Name_enUS > > 5 > > Name_enUS > > 4 > > 3 > > 0.4 > > explicit > > 100 > > false > > > > > > edismax > > > > > > > > Schema: > > > > > positionIncrementGap="100"> > > > > > > > > > > > > > > > > > > > > Using Solr 8.6.3 > >
Re: Query generation is different for search terms with and without "-"
This is a common point of confusion. There are two phases for creating a query, query _parsing_ first, then the analysis chain for the parsed result. So what e-dismax sees in the two cases is: Name_enUS:“high tech” -> two tokens, since there are two of them pf2 comes into play. Name_enUS:“high-tech” -> there’s only one token so pf2 doesn’t apply, splitting it on the hyphen comes later. It’s especially confusing since the field analysis then breaks up “high-tech” into two tokens that look the same as “high tech” in the debug response, just without the phrase query. Name_enUS:high Name_enUS:tech Best, Erick > On Nov 23, 2020, at 8:32 PM, Samuel Gutierrez > wrote: > > I am troubleshooting an issue with ranking for search terms that contain a > "-" vs the same query that does not contain the dash e.g. "high-tech" vs > "high tech". The field that I am querying is using the standard tokenizer, > so I would expect that the underlying lucene query should be the same for > both versions of the query, however when printing the debug, it appears > they are generated differently. I know "-" must be escaped as it has > special meaning in lucene, however escaping does not fix the problem. It > appears that with the "-" present, the pf2 edismax parameter is not > respected and omitted from the final query. We use sow=false as we have > multiterm synonyms and need to ensure they are included in the final lucene > query. My expectation is that the final underlying lucene query should be > based on the output of the field analyzer, however after briefly looking > at the code for ExtendedDismaxQParser, it appears that there is some string > processing happening outside of the analysis step which causes the > unexpected lucene query. > > > Solr Debug for "high tech": > > parsedquery: "+(DisjunctionMaxQuery((Name_enUS:high)~0.4) > DisjunctionMaxQuery((Name_enUS:tech)~0.4))~2 > DisjunctionMaxQuery((Name_enUS:"high tech"~5)~0.4) > DisjunctionMaxQuery((Name_enUS:"high tech"~4)~0.4)", > parsedquery_toString: "+(((Name_enUS:high)~0.4 > (Name_enUS:tech)~0.4)~2) (Name_enUS:"high tech"~5)~0.4 > (Name_enUS:"high tech"~4)~0.4", > > > Solr Debug for "high-tech" > > parsedquery: "+DisjunctionMaxQueryName_enUS:high > Name_enUS:tech)~2))~0.4) DisjunctionMaxQuery((Name_enUS:"high > tech"~5)~0.4)", > parsedquery_toString: "+(((Name_enUS:high Name_enUS:tech)~2))~0.4 > (Name_enUS:"high tech"~5)~0.4" > > SolrConfig: > > > > true > true > json > 375% > Name_enUS > Name_enUS > 5 > Name_enUS > 4 > 3 > 0.4 > explicit > 100 > false > > > edismax > > > > Schema: > > > > > > > > > > > > Using Solr 8.6.3 > > -- > *The information contained in this message is the sole and exclusive > property of ***iHerb Inc.*** and may be privileged and confidential. It may > not be disseminated or distributed to persons or entities other than the > ones intended without the written authority of ***iHerb Inc.** *If you have > received this e-mail in error or are not the intended recipient, you may > not use, copy, disseminate or distribute it. Do not open any attachments. > Please delete it immediately from your system and notify the sender > promptly by e-mail that you have done so.*
Query generation is different for search terms with and without "-"
I am troubleshooting an issue with ranking for search terms that contain a "-" vs the same query that does not contain the dash e.g. "high-tech" vs "high tech". The field that I am querying is using the standard tokenizer, so I would expect that the underlying lucene query should be the same for both versions of the query, however when printing the debug, it appears they are generated differently. I know "-" must be escaped as it has special meaning in lucene, however escaping does not fix the problem. It appears that with the "-" present, the pf2 edismax parameter is not respected and omitted from the final query. We use sow=false as we have multiterm synonyms and need to ensure they are included in the final lucene query. My expectation is that the final underlying lucene query should be based on the output of the field analyzer, however after briefly looking at the code for ExtendedDismaxQParser, it appears that there is some string processing happening outside of the analysis step which causes the unexpected lucene query. Solr Debug for "high tech": parsedquery: "+(DisjunctionMaxQuery((Name_enUS:high)~0.4) DisjunctionMaxQuery((Name_enUS:tech)~0.4))~2 DisjunctionMaxQuery((Name_enUS:"high tech"~5)~0.4) DisjunctionMaxQuery((Name_enUS:"high tech"~4)~0.4)", parsedquery_toString: "+(((Name_enUS:high)~0.4 (Name_enUS:tech)~0.4)~2) (Name_enUS:"high tech"~5)~0.4 (Name_enUS:"high tech"~4)~0.4", Solr Debug for "high-tech" parsedquery: "+DisjunctionMaxQueryName_enUS:high Name_enUS:tech)~2))~0.4) DisjunctionMaxQuery((Name_enUS:"high tech"~5)~0.4)", parsedquery_toString: "+(((Name_enUS:high Name_enUS:tech)~2))~0.4 (Name_enUS:"high tech"~5)~0.4" SolrConfig: true true json 375% Name_enUS Name_enUS 5 Name_enUS 4 3 0.4 explicit 100 false edismax Schema: Using Solr 8.6.3 -- *The information contained in this message is the sole and exclusive property of ***iHerb Inc.*** and may be privileged and confidential. It may not be disseminated or distributed to persons or entities other than the ones intended without the written authority of ***iHerb Inc.** *If you have received this e-mail in error or are not the intended recipient, you may not use, copy, disseminate or distribute it. Do not open any attachments. Please delete it immediately from your system and notify the sender promptly by e-mail that you have done so.*
