I use 3 word shingles with stopwords for my MLT ML trainer that worked pretty well for such a solution, but for a full index the size became prohibitive
On Fri, Nov 8, 2019 at 12:13 PM Walter Underwood <wun...@wunderwood.org> wrote: > If we had IDF for phrases, they would be super effective. The 2X weight is > a hack that mostly works. > > Infoseek had phrase IDF and it was a killer algorithm for relevance. > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > > On Nov 8, 2019, at 11:08 AM, David Hastings < > hastings.recurs...@gmail.com> wrote: > > > > the pf and qf fields are REALLY nice for this > > > > On Fri, Nov 8, 2019 at 12:02 PM Walter Underwood <wun...@wunderwood.org> > > wrote: > > > >> I always enable phrase searching in edismax for exactly this reason. > >> > >> Something like: > >> > >> <str name="qf”>title^8 keywords^4 text</str> > >> <str name="pf”>title^16 keywords^8 text^2</str> > >> > >> To deal with concepts in queries, a classifier and/or named entity > >> extractor can be helpful. If you have a list of concepts (“controlled > >> vocabulary”) that includes “Lamin A”, and that shows up in a query, that > >> term can be queried against the field matching that vocabulary. > >> > >> This is how LinkedIn separates people, companies, and places, for > example. > >> > >> wunder > >> Walter Underwood > >> wun...@wunderwood.org > >> http://observer.wunderwood.org/ (my blog) > >> > >>> On Nov 8, 2019, at 10:48 AM, Erick Erickson <erickerick...@gmail.com> > >> wrote: > >>> > >>> Look at the “mm” parameter, try setting it to 100%. Although that’t not > >> entirely likely to do what you want either since virtually every doc > will > >> have “a” in it. But at least you’d get docs that have both terms. > >>> > >>> you may also be able to search for things like “Lamin A” _only as a > >> phrase_ and have some luck. But this is a gnarly problem in general. > Some > >> people have been able to substitute synonyms and/or shingles to make > this > >> work at the expense of a larger index. > >>> > >>> This is a generic problem with context. “Lamin A” is really a > “concept”, > >> not just two words that happen to be near each other. Searching as a > phrase > >> is an OOB-but-naive way to try to make it more likely that the ranked > >> results refer to the _concept_ of “Lamin A”. The assumption here is “if > >> these two words appear next to each other, they’re more likely to be > what I > >> want”. I say “naive” because “Lamins: A new approach to...” would > _also_ be > >> found for a naive phrase search. (I have no idea whether such a title > makes > >> sense or not, but you figured that out already)... > >>> > >>> To do this well you’d have to dive in to NLP/Machine learning. > >>> > >>> I truly wish we could have the DWIM search algorithm (Do What I Mean)…. > >>> > >>>> On Nov 8, 2019, at 11:29 AM, Guilherme Viteri <gvit...@ebi.ac.uk> > >> wrote: > >>>> > >>>> HI Walter and Paras > >>>> > >>>> I indexed it removing all the references to StopWordFilter and I went > >> from 121 results to near 20K as the search term q="Lymphoid and a > >> non-Lymphoid cell" is matching entities such as "IFT A" or "Lamin A". > So I > >> don't think removing it completely is the way to go from the scenario we > >> have, but I appreciate the suggestion… > >>>> > >>>> Yes the response is using fl=* > >>>> I am trying some combinations at the moment, but yet no success. > >>>> > >>>> defType=edismax > >>>> q.alt=Lymphoid and a non-Lymphoid cell > >>>> Number of results=1599 > >>>> Quite a considerable increase, even though reasonable meaningful > >> results. > >>>> > >>>> I am sorry but I didn't understand what do you want me to do exactly > >> with the lst (??) and qf and bf. > >>>> > >>>> Thanks everyone with their inputs > >>>> > >>>> > >>>>> On 8 Nov 2019, at 06:45, Paras Lehana <paras.leh...@indiamart.com> > >> wrote: > >>>>> > >>>>> Hi Guilherme > >>>>> > >>>>> By accident, I ended up querying the using the default handler > >> (/select) and it worked. > >>>>> > >>>>> You've just found the culprit. Thanks for giving the material I > >> requested. Your analysis chain is working as expected. I don't see any > >> issue in either StopWordFilter or your boosts. I also use a boost of 50 > >> when boosting contextual suggestions (boosting "gold iphone" on a page > of > >> iphone) but I take Walter's suggestion and would try to optimize my > >> weights. I agree that this 50 thing was not researched much about by us > as > >> well (we never faced performance or relevance issues). > >>>>> > >>>>> See the major difference in both the handlers - edismax. I'm pretty > >> sure that your problem lies in the parsing of queries (you can confirm > that > >> from parsedquery key in debug of both JSON responses). I hope you have > >> provided the response with fl=*. Replace q with q.alt in your /search > >> handler query and I think you should start getting responses. That's > >> because q.alt uses standard parser. If you want to keep using edisMax, I > >> suggest you to test the responses removing some combination of lst (qf, > bf) > >> and find what's restricting the documents to come up. I'm out of office > >> today - would have certainly tried analyzing the field values of the > >> document in /select request and compare it with qf/bq in solrconfig.xml > >> /search. Do this for me and you'd certainly find something. > >>>>> > >>>>> On Thu, 7 Nov 2019 at 21:00, Walter Underwood <wun...@wunderwood.org > >> <mailto:wun...@wunderwood.org>> wrote: > >>>>> I normally use a weight of 8 for the most important field, like > title. > >> Other fields might get a 4 or 2. > >>>>> > >>>>> I add a “pf” field with the weights doubled, so that phrase matches > >> have a higher weight. > >>>>> > >>>>> The weight of 8 comes from experience at Infoseek and Inktomi, two > >> early web search engines. With different relevance algorithms and > totally > >> different evaluation and tuning systems, they settled on weights of 8 > and > >> 7.5 for HTML titles. With the the two radically different system getting > >> the same number, I decided that was a property of the documents, not of > the > >> search engines. > >>>>> > >>>>> wunder > >>>>> Walter Underwood > >>>>> wun...@wunderwood.org <mailto:wun...@wunderwood.org> > >>>>> http://observer.wunderwood.org/ <http://observer.wunderwood.org/> > >> (my blog) > >>>>> > >>>>>> On Nov 7, 2019, at 9:03 AM, Guilherme Viteri <gvit...@ebi.ac.uk > >> <mailto:gvit...@ebi.ac.uk>> wrote: > >>>>>> > >>>>>> Hi Wunder, > >>>>>> > >>>>>> My indexer takes quite a few hours to be executed I am shortening it > >> to run faster, but I also need to make sure it gives what we are > expecting. > >> This implementation's been there for >4y, and massively used. > >>>>>> > >>>>>>> In your edismax handlers, weights of 20, 50, and 100 are extremely > >> high. I don’t think I’ve ever used a weight higher than 16 in a dozen > years > >> of configuring Solr. > >>>>>> I've inherited that implementation and I am really keen to adequate > >> it, what would you recommend ? > >>>>>> > >>>>>> Cheers > >>>>>> Guilherme > >>>>>> > >>>>>>> On 7 Nov 2019, at 14:43, Walter Underwood <wun...@wunderwood.org > >> <mailto:wun...@wunderwood.org>> wrote: > >>>>>>> > >>>>>>> Thanks for posting the files. Looking at schema.xml, I see that you > >> still are using StopFilterFactory. The first advice we gave you was to > >> remove that. > >>>>>>> > >>>>>>> Remove StopFilterFactory everywhere and reindex. > >>>>>>> > >>>>>>> You will continue to have problems matching stopwords until you do > >> that. > >>>>>>> > >>>>>>> In your edismax handlers, weights of 20, 50, and 100 are extremely > >> high. I don’t think I’ve ever used a weight higher than 16 in a dozen > years > >> of configuring Solr. > >>>>>>> > >>>>>>> wunder > >>>>>>> Walter Underwood > >>>>>>> wun...@wunderwood.org <mailto:wun...@wunderwood.org> > >>>>>>> http://observer.wunderwood.org/ <http://observer.wunderwood.org/> > >> (my blog) > >>>>>>> > >>>>>>>> On Nov 7, 2019, at 6:56 AM, Guilherme Viteri <gvit...@ebi.ac.uk > >> <mailto:gvit...@ebi.ac.uk>> wrote: > >>>>>>>> > >>>>>>>> Hi Paras, everyone > >>>>>>>> > >>>>>>>> Thank you again for your inputs and suggestions. I sorry to hear > >> you had trouble with the attachments I will host it somewhere and share > the > >> links. > >>>>>>>> I don't tweak my index, I get the data from the graph database, > >> create a document as they are and save to solr. > >>>>>>>> > >>>>>>>> So, I am sending the new analysis screen querying the way you > >> suggested. Also the results with params and solr query url. > >>>>>>>> > >>>>>>>> During the process of querying what you asked I found something > >> really weird (at least for me). By accident, I ended up querying the > using > >> the default handler (/select) and it worked. Then If I use the one I > must > >> use, then sadly doesn't work. I am posting both results and I will also > >> post the handlers as well. > >>>>>>>> > >>>>>>>> Here is the link with all the files mentioned before > >>>>>>>> > >> > https://www.dropbox.com/sh/fymfm1q94zum1lx/AADwU1c9EUf2A4d7FtzSKR54a?dl=0< > >> > https://www.dropbox.com/sh/fymfm1q94zum1lx/AADwU1c9EUf2A4d7FtzSKR54a?dl=0> > >> < > https://www.dropbox.com/sh/fymfm1q94zum1lx/AADwU1c9EUf2A4d7FtzSKR54a?dl=0 > >> < > https://www.dropbox.com/sh/fymfm1q94zum1lx/AADwU1c9EUf2A4d7FtzSKR54a?dl=0 > >>>> > >>>>>>>> If the link doesn't work www dot dropbox dot com slash sh slash > >> fymfm1q94zum1lx/AADwU1c9EUf2A4d7FtzSKR54a ? dl equals 0 > >>>>>>>> > >>>>>>>> Thanks > >>>>>>>> > >>>>>>>>> On 7 Nov 2019, at 05:23, Paras Lehana < > paras.leh...@indiamart.com > >> <mailto:paras.leh...@indiamart.com>> wrote: > >>>>>>>>> > >>>>>>>>> Hi Guilherme. > >>>>>>>>> > >>>>>>>>> I am sending they analysis result and the json result as > requested. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Thanks for the effort. Luckily, I can see your attachments (low > >> quality > >>>>>>>>> though). > >>>>>>>>> > >>>>>>>>> From the analysis screen, the analysis is working as expected. > One > >> of the > >>>>>>>>> reasons for query="lymphoid and *a* non-lymphoid cell" not > matching > >>>>>>>>> document containing "Lymphoid and a non-Lymphoid cell" I can > >> initially > >>>>>>>>> think of is: the stopword "a" is probably present in > post-analysis > >> either > >>>>>>>>> of query or index. Did you tweak your index time analysis after > >> indexing? > >>>>>>>>> > >>>>>>>>> Do two things: > >>>>>>>>> > >>>>>>>>> 1. Post the analysis screen for and index=*"Immunoregulatory > >>>>>>>>> interactions between a Lymphoid and a non-Lymphoid cell"* and > >>>>>>>>> "query=*"lymphoid > >>>>>>>>> and a non-lymphoid cell"*. Try hosting the image and providing > the > >> link > >>>>>>>>> here. > >>>>>>>>> 2. Give the same JSON output as you have sent but this time with > >>>>>>>>> *"echoParams=all"*. Also, post the exact Solr query url. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Wed, 6 Nov 2019 at 21:07, Erick Erickson < > >> erickerick...@gmail.com <mailto:erickerick...@gmail.com>> wrote: > >>>>>>>>> > >>>>>>>>>> I don’t see the attachments, maybe I deleted old e-mails or some > >> such. The > >>>>>>>>>> Apache server is fairly aggressive about stripping attachments > >> though, so > >>>>>>>>>> it’s also possible they didn’t make it through. > >>>>>>>>>> > >>>>>>>>>>> On Nov 6, 2019, at 9:28 AM, Guilherme Viteri < > gvit...@ebi.ac.uk > >> <mailto:gvit...@ebi.ac.uk>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Thanks Erick. > >>>>>>>>>>> > >>>>>>>>>>>> First, your index and analysis chains are considerably > >> different, this > >>>>>>>>>> can easily be a source of problems. In particular, using two > >> different > >>>>>>>>>> tokenizers is a huge red flag. I _strongly_ recommend against > >> this unless > >>>>>>>>>> you’re totally sure you understand the consequences. > >> Additionally, your use > >>>>>>>>>> of the length filter is suspicious, especially since your > problem > >> statement > >>>>>>>>>> is about the addition of a single letter term and the min length > >> allowed on > >>>>>>>>>> that filter is 2. That said, it’s reasonable to suppose that the > >> ’a’ is > >>>>>>>>>> filtered out in both cases, but maybe you’ve found something odd > >> about the > >>>>>>>>>> interactions. > >>>>>>>>>>> I will investigate the min length and post the results later. > >>>>>>>>>>> > >>>>>>>>>>>> Second, I have no idea what this will do. Are the equal signs > >> typos? > >>>>>>>>>> Used by custom code? > >>>>>>>>>>> This the url in my application, not solr params. That's the > >> query string. > >>>>>>>>>>> > >>>>>>>>>>>> What does “species=“ do? That’s not Solr syntax, so it’s > likely > >> that > >>>>>>>>>> all the params with an equal-sign are totally ignored unless > it’s > >> just a > >>>>>>>>>> typo. > >>>>>>>>>>> This is part of the application. Species will be used later on > >> in solr > >>>>>>>>>> to filter out the result. That's not solr. That my app params. > >>>>>>>>>>> > >>>>>>>>>>>> Third, the easiest way to see what’s happening under the > covers > >> is to > >>>>>>>>>> add “&debug=true” to the query and look at the parsed query. > >> Ignore all the > >>>>>>>>>> relevance calculations for the nonce, or specify “&debug=query” > >> to skip > >>>>>>>>>> that part. > >>>>>>>>>>> The two json files i've sent, they are debugQuery=on and the > >> explain tag > >>>>>>>>>> is present. > >>>>>>>>>>> I will try the searching the way you mentioned. > >>>>>>>>>>> > >>>>>>>>>>> Thank for your inputs > >>>>>>>>>>> > >>>>>>>>>>> Guilherme > >>>>>>>>>>> > >>>>>>>>>>>> On 6 Nov 2019, at 14:14, Erick Erickson < > >> erickerick...@gmail.com <mailto:erickerick...@gmail.com>> > >>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> Fwd to another server > >>>>>>>>>>>> > >>>>>>>>>>>> First, your index and analysis chains are considerably > >> different, this > >>>>>>>>>> can easily be a source of problems. In particular, using two > >> different > >>>>>>>>>> tokenizers is a huge red flag. I _strongly_ recommend against > >> this unless > >>>>>>>>>> you’re totally sure you understand the consequences. > >> Additionally, your use > >>>>>>>>>> of the length filter is suspicious, especially since your > problem > >> statement > >>>>>>>>>> is about the addition of a single letter term and the min length > >> allowed on > >>>>>>>>>> that filter is 2. That said, it’s reasonable to suppose that the > >> ’a’ is > >>>>>>>>>> filtered out in both cases, but maybe you’ve found something odd > >> about the > >>>>>>>>>> interactions. > >>>>>>>>>>>> > >>>>>>>>>>>> Second, I have no idea what this will do. Are the equal signs > >> typos? > >>>>>>>>>> Used by custom code? > >>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>> > >> > https://dev.reactome.org/content/query?q=lymphoid+and+a+non-lymphoid+cell&species=Homo+sapiens&species=Entries+without+species&cluster=true > >> < > >> > https://dev.reactome.org/content/query?q=lymphoid+and+a+non-lymphoid+cell&species=Homo+sapiens&species=Entries+without+species&cluster=true > >>> > >>>>>>>>>>>> > >>>>>>>>>>>> What does “species=“ do? That’s not Solr syntax, so it’s > likely > >> that > >>>>>>>>>> all the params with an equal-sign are totally ignored unless > it’s > >> just a > >>>>>>>>>> typo. > >>>>>>>>>>>> > >>>>>>>>>>>> Third, the easiest way to see what’s happening under the > covers > >> is to > >>>>>>>>>> add “&debug=true” to the query and look at the parsed query. > >> Ignore all the > >>>>>>>>>> relevance calculations for the nonce, or specify “&debug=query” > >> to skip > >>>>>>>>>> that part. > >>>>>>>>>>>> > >>>>>>>>>>>> 90% + of the time, the question “why didn’t this query do > what I > >>>>>>>>>> expect” is answered by looking at the “&debug=query” output and > >> the > >>>>>>>>>> analysis page in the admin UI. NOTE: for the analysis page be > >> sure to look > >>>>>>>>>> at _both_ the query and index output. Also, and very important > >> about the > >>>>>>>>>> analysis page (and this is confusing) is that this _assumes_ > that > >> what you > >>>>>>>>>> put in the text boxes have made it through the query parser > >> intact and is > >>>>>>>>>> analyzed by the field selected. Consider the search > >> "q=field:word1 word2". > >>>>>>>>>> Now you type “word1 word2” into the analysis text box and it > >> looks like > >>>>>>>>>> what you expect. That’s misleading because the query is _parsed_ > >> as > >>>>>>>>>> "field:word1 default_search_field:word2”. This is where > >> “&debug=query” > >>>>>>>>>> helps. > >>>>>>>>>>>> > >>>>>>>>>>>> Best, > >>>>>>>>>>>> Erick > >>>>>>>>>>>> > >>>>>>>>>>>>> On Nov 6, 2019, at 2:36 AM, Paras Lehana < > >> paras.leh...@indiamart.com <mailto:paras.leh...@indiamart.com>> > >>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hi Walter, > >>>>>>>>>>>>> > >>>>>>>>>>>>> The solr.StopFilter removes all tokens that are stopwords. > >> Those words > >>>>>>>>>> will > >>>>>>>>>>>>>> not be in the index, so they can never match a query. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> I think the OP's concern is different results when adding a > >> stopword. I > >>>>>>>>>>>>> think he's using the filter factory correctly - the query > chain > >>>>>>>>>> includes > >>>>>>>>>>>>> the filter as well so it should remove "a" while querying. > >>>>>>>>>>>>> > >>>>>>>>>>>>> *@Guilherme*, please post results for both the query, the > >> document in > >>>>>>>>>>>>> result you are concerned about and post full result of > >> analysis screen > >>>>>>>>>> (for > >>>>>>>>>>>>> both query and index). > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Tue, 5 Nov 2019 at 21:38, Walter Underwood < > >> wun...@wunderwood.org <mailto:wun...@wunderwood.org>> > >>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> No. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The solr.StopFilter removes all tokens that are stopwords. > >> Those words > >>>>>>>>>>>>>> will not be in the index, so they can never match a query. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 1. Remove the lines with solr.StopFilter from every analysis > >> chain in > >>>>>>>>>>>>>> schema.xml. > >>>>>>>>>>>>>> 2. Reload the collection, restart Solr, or whatever to read > >> the new > >>>>>>>>>> config. > >>>>>>>>>>>>>> 3. Reindex all of the documents. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> When indexed with the new analysis chain, the stopwords will > >> not be > >>>>>>>>>>>>>> removed and they will be searchable. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> wunder > >>>>>>>>>>>>>> Walter Underwood > >>>>>>>>>>>>>> wun...@wunderwood.org <mailto:wun...@wunderwood.org> > >>>>>>>>>>>>>> http://observer.wunderwood.org/ < > >> http://observer.wunderwood.org/> (my blog) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Nov 5, 2019, at 8:56 AM, Guilherme Viteri < > >> gvit...@ebi.ac.uk <mailto:gvit...@ebi.ac.uk>> > >>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Ok. I am kind a lost now. > >>>>>>>>>>>>>>> If I open up the console > analysis and perform it, that's > >> the final > >>>>>>>>>>>>>> result. > >>>>>>>>>>>>>>> <Screenshot 2019-11-05 at 14.54.16.png> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Your suggestion is: get rid of the <filter stopword.txt> in > >> the > >>>>>>>>>>>>>> schema.xml and during index phase replaceAll("in > >> stopwords.txt"," ") > >>>>>>>>>> then > >>>>>>>>>>>>>> add to solr. Is that correct ? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks David > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 5 Nov 2019, at 14:48, David Hastings < > >>>>>>>>>> hastings.recurs...@gmail.com <mailto: > hastings.recurs...@gmail.com > >>> > >>>>>>>>>>>>>> <mailto:hastings.recurs...@gmail.com <mailto: > >> hastings.recurs...@gmail.com>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Fwd to another server > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> no, > >>>>>>>>>>>>>>>> <filter class="solr.StopFilterFactory" > >> ignoreCase="true" > >>>>>>>>>>>>>>>> words="stopwords.txt"/> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> is still using stopwords and should be removed, in my > >> opinion of > >>>>>>>>>> course, > >>>>>>>>>>>>>>>> based on your use case may be different, but i generally > >> axe any > >>>>>>>>>>>>>> reference > >>>>>>>>>>>>>>>> to them at all > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Tue, Nov 5, 2019 at 9:47 AM Guilherme Viteri < > >> gvit...@ebi.ac.uk <mailto:gvit...@ebi.ac.uk> > >>>>>>>>>>>>>> <mailto:gvit...@ebi.ac.uk <mailto:gvit...@ebi.ac.uk>>> > wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Thanks. > >>>>>>>>>>>>>>>>> Haven't I done this here ? > >>>>>>>>>>>>>>>>> <fieldType name="text_field" class="solr.TextField" > >>>>>>>>>>>>>>>>> positionIncrementGap="100" omitNorms="false" > > >>>>>>>>>>>>>>>>> <analyzer type="index"> > >>>>>>>>>>>>>>>>> <tokenizer class="solr.StandardTokenizerFactory"/> > >>>>>>>>>>>>>>>>> <filter class="solr.ClassicFilterFactory"/> > >>>>>>>>>>>>>>>>> <filter class="solr.LengthFilterFactory" min="2" > >>>>>>>>>>>>>> max="20"/> > >>>>>>>>>>>>>>>>> <filter class="solr.LowerCaseFilterFactory"/> > >>>>>>>>>>>>>>>>> <filter class="solr.StopFilterFactory" > >> ignoreCase="true" > >>>>>>>>>>>>>>>>> words="stopwords.txt"/> > >>>>>>>>>>>>>>>>> </analyzer> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 5 Nov 2019, at 14:15, David Hastings < > >>>>>>>>>> hastings.recurs...@gmail.com <mailto: > hastings.recurs...@gmail.com > >>> > >>>>>>>>>>>>>> <mailto:hastings.recurs...@gmail.com <mailto: > >> hastings.recurs...@gmail.com>>> > >>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Fwd to another server > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> The first thing you should do is remove any reference to > >> stop > >>>>>>>>>> words > >>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>> never use them, then re-index your data and try it > again. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On Tue, Nov 5, 2019 at 9:14 AM Guilherme Viteri < > >>>>>>>>>> gvit...@ebi.ac.uk <mailto:gvit...@ebi.ac.uk> > >>>>>>>>>>>>>> <mailto:gvit...@ebi.ac.uk <mailto:gvit...@ebi.ac.uk>>> > >>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I am performing a search to match a name (text_field), > >> however > >>>>>>>>>> this > >>>>>>>>>>>>>> term > >>>>>>>>>>>>>>>>>>> contains 'and' and 'a' and it doesn't return any > >> records. If i > >>>>>>>>>> remove > >>>>>>>>>>>>>>>>> 'a' > >>>>>>>>>>>>>>>>>>> then it works. > >>>>>>>>>>>>>>>>>>> e.g > >>>>>>>>>>>>>>>>>>> Search Term: lymphoid and a non-lymphoid cell > >>>>>>>>>>>>>>>>>>> doesn't work: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>> > >> > https://dev.reactome.org/content/query?q=lymphoid+and+a+non-lymphoid+cell&species=Homo+sapiens&species=Entries+without+species&cluster=true > >> < > >> > https://dev.reactome.org/content/query?q=lymphoid+and+a+non-lymphoid+cell&species=Homo+sapiens&species=Entries+without+species&cluster=true > >>> > >>>>>>>>>>>>>> < > >>>>>>>>>>>>>> > >>>>>>>>>> > >> > https://dev.reactome.org/content/query?q=lymphoid+and+a+non-lymphoid+cell&species=Homo+sapiens&species=Entries+without+species&cluster=true > >> < > >> > https://dev.reactome.org/content/query?q=lymphoid+and+a+non-lymphoid+cell&species=Homo+sapiens&species=Entries+without+species&cluster=true > >>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>> > >> > https://dev.reactome.org/content/query?q=lymphoid+and+a+non-lymphoid+cell&species=Homo+sapiens&species=Entries+without+species&cluster=true > >> < > >> > https://dev.reactome.org/content/query?q=lymphoid+and+a+non-lymphoid+cell&species=Homo+sapiens&species=Entries+without+species&cluster=true > >>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Search term: lymphoid and non-lymphoid cell > >>>>>>>>>>>>>>>>>>> works: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>> > >> > https://dev.reactome.org/content/query?q=lymphoid+and+non-lymphoid+cell&species=Homo+sapiens&species=Entries+without+species&cluster=true > >> < > >> > https://dev.reactome.org/content/query?q=lymphoid+and+non-lymphoid+cell&species=Homo+sapiens&species=Entries+without+species&cluster=true > >>> > >>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>> > >> > https://dev.reactome.org/content/query?q=lymphoid+and+non-lymphoid+cell&species=Homo+sapiens&species=Entries+without+species&cluster=true > >> < > >> > https://dev.reactome.org/content/query?q=lymphoid+and+non-lymphoid+cell&species=Homo+sapiens&species=Entries+without+species&cluster=true > >>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> interested in the first result > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> schema.xml > >>>>>>>>>>>>>>>>>>> <field name="name" > >> type="text_field" > >>>>>>>>>>>>>>>>>>> indexed="true" stored="true" omitNorms="false" > >>>>>>>>>> required="true" > >>>>>>>>>>>>>>>>>>> multiValued="false"/> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> <analyzer type="query"> > >>>>>>>>>>>>>>>>>>> <tokenizer class="solr.PatternTokenizerFactory" > >>>>>>>>>>>>>>>>>>> pattern="[^a-zA-Z0-9/._:]"/> > >>>>>>>>>>>>>>>>>>> <filter class="solr.PatternReplaceFilterFactory" > >>>>>>>>>>>>>>>>>>> pattern="^[/._:]+" replacement=""/> > >>>>>>>>>>>>>>>>>>> <filter class="solr.PatternReplaceFilterFactory" > >>>>>>>>>>>>>>>>>>> pattern="[/._:]+$" replacement=""/> > >>>>>>>>>>>>>>>>>>> <filter class="solr.PatternReplaceFilterFactory" > >>>>>>>>>>>>>>>>>>> pattern="[_]" replacement=" "/> > >>>>>>>>>>>>>>>>>>> <filter class="solr.LengthFilterFactory" min="2" > >>>>>>>>>>>>>>>>> max="20"/> > >>>>>>>>>>>>>>>>>>> <filter class="solr.LowerCaseFilterFactory"/> > >>>>>>>>>>>>>>>>>>> <filter class="solr.StopFilterFactory" > >>>>>>>>>>>>>> ignoreCase="true" > >>>>>>>>>>>>>>>>>>> words="stopwords.txt"/> > >>>>>>>>>>>>>>>>>>> </analyzer> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> <fieldType name="text_field" class="solr.TextField" > >>>>>>>>>>>>>>>>>>> positionIncrementGap="100" omitNorms="false" > > >>>>>>>>>>>>>>>>>>> <analyzer type="index"> > >>>>>>>>>>>>>>>>>>> <tokenizer class="solr.StandardTokenizerFactory"/> > >>>>>>>>>>>>>>>>>>> <filter class="solr.ClassicFilterFactory"/> > >>>>>>>>>>>>>>>>>>> <filter class="solr.LengthFilterFactory" min="2" > >>>>>>>>>>>>>>>>> max="20"/> > >>>>>>>>>>>>>>>>>>> <filter class="solr.LowerCaseFilterFactory"/> > >>>>>>>>>>>>>>>>>>> <filter class="solr.StopFilterFactory" > >>>>>>>>>>>>>> ignoreCase="true" > >>>>>>>>>>>>>>>>>>> words="stopwords.txt"/> > >>>>>>>>>>>>>>>>>>> </analyzer> > >>>>>>>>>>>>>>>>>>> <analyzer type="query"> > >>>>>>>>>>>>>>>>>>> <tokenizer class="solr.PatternTokenizerFactory" > >>>>>>>>>>>>>>>>>>> pattern="[^a-zA-Z0-9/._:]"/> > >>>>>>>>>>>>>>>>>>> <filter class="solr.PatternReplaceFilterFactory" > >>>>>>>>>>>>>>>>>>> pattern="^[/._:]+" replacement=""/> > >>>>>>>>>>>>>>>>>>> <filter class="solr.PatternReplaceFilterFactory" > >>>>>>>>>>>>>>>>>>> pattern="[/._:]+$" replacement=""/> > >>>>>>>>>>>>>>>>>>> <filter class="solr.PatternReplaceFilterFactory" > >>>>>>>>>>>>>>>>>>> pattern="[_]" replacement=" "/> > >>>>>>>>>>>>>>>>>>> <filter class="solr.LengthFilterFactory" min="2" > >>>>>>>>>>>>>>>>> max="20"/> > >>>>>>>>>>>>>>>>>>> <filter class="solr.LowerCaseFilterFactory"/> > >>>>>>>>>>>>>>>>>>> <filter class="solr.StopFilterFactory" > >>>>>>>>>>>>>> ignoreCase="true" > >>>>>>>>>>>>>>>>>>> words="stopwords.txt"/> > >>>>>>>>>>>>>>>>>>> </analyzer> > >>>>>>>>>>>>>>>>>>> </fieldType> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> stopwords.txt > >>>>>>>>>>>>>>>>>>> #Standard english stop words taken from Lucene's > >> StopAnalyzer > >>>>>>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>> b > >>>>>>>>>>>>>>>>>>> c > >>>>>>>>>>>>>>>>>>> .... > >>>>>>>>>>>>>>>>>>> an > >>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>> are > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Running SolR 6.6.2. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Is there anything I could do to prevent this ? > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>>>>>> Guilherme > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> Regards, > >>>>>>>>>>>>> > >>>>>>>>>>>>> *Paras Lehana* [65871] > >>>>>>>>>>>>> Development Engineer, Auto-Suggest, > >>>>>>>>>>>>> IndiaMART Intermesh Ltd. > >>>>>>>>>>>>> > >>>>>>>>>>>>> 8th Floor, Tower A, Advant-Navis Business Park, Sector 142, > >>>>>>>>>>>>> Noida, UP, IN - 201303 > >>>>>>>>>>>>> > >>>>>>>>>>>>> Mob.: +91-9560911996 > >>>>>>>>>>>>> Work: 01203916600 | Extn: *8173* > >>>>>>>>>>>>> > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> IMPORTANT: > >>>>>>>>>>>>> NEVER share your IndiaMART OTP/ Password with anyone. > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> -- > >>>>>>>>> Regards, > >>>>>>>>> > >>>>>>>>> *Paras Lehana* [65871] > >>>>>>>>> Development Engineer, Auto-Suggest, > >>>>>>>>> IndiaMART Intermesh Ltd. > >>>>>>>>> > >>>>>>>>> 8th Floor, Tower A, Advant-Navis Business Park, Sector 142, > >>>>>>>>> Noida, UP, IN - 201303 > >>>>>>>>> > >>>>>>>>> Mob.: +91-9560911996 > >>>>>>>>> Work: 01203916600 | Extn: *8173* > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> IMPORTANT: > >>>>>>>>> NEVER share your IndiaMART OTP/ Password with anyone. > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> -- > >>>>> Regards, > >>>>> > >>>>> Paras Lehana [65871] > >>>>> Development Engineer, Auto-Suggest, > >>>>> IndiaMART Intermesh Ltd. > >>>>> > >>>>> 8th Floor, Tower A, Advant-Navis Business Park, Sector 142, > >>>>> Noida, UP, IN - 201303 > >>>>> > >>>>> Mob.: +91-9560911996 <tel:+91-9560911996> > >>>>> Work: 01203916600 | Extn: 8173 > >>>>> > >>>>> IMPORTANT: > >>>>> NEVER share your IndiaMART OTP/ Password with anyone. > >>> > >> > >> > >
