You could try experimenting with CommonGramsFilterFactory and CommonGramsQueryFilter (slightly different). There is actually a lot of cool analyzers bundled with Solr. You can find full list on my site at: http://www.solr-start.com/info/analyzers
Regards, Alex. Personal: http://www.outerthoughts.com/ and @arafalov Solr resources: http://www.solr-start.com/ and @solrstart Solr popularizers community: https://www.linkedin.com/groups?gid=6713853 On Tue, Jul 15, 2014 at 8:42 AM, Teague James <teag...@insystechinc.com> wrote: > Alex, > > Thanks! Great suggestion. I figured out that it was the > EdgeNGramFilterFactory. Taking that out of the mix did it. > > -Teague > > -----Original Message----- > From: Alexandre Rafalovitch [mailto:arafa...@gmail.com] > Sent: Monday, July 14, 2014 9:14 PM > To: solr-user > Subject: Re: Of, To, and Other Small Words > > Have you tried the Admin UI's Analyze screen. Because it will show you what > happens to the text as it progresses through the tokenizers and filters. No > need to reindex. > > Regards, > Alex. > Personal: http://www.outerthoughts.com/ and @arafalov Solr resources: > http://www.solr-start.com/ and @solrstart Solr popularizers community: > https://www.linkedin.com/groups?gid=6713853 > > > On Tue, Jul 15, 2014 at 8:10 AM, Teague James <teag...@insystechinc.com> > wrote: >> Hi Anshum, >> >> Thanks for replying and suggesting this, but the field type I am using (a >> modified text_general) in my schema has the file set to 'stopwords.txt'. >> >> <fieldType name="text_general" class="solr.TextField" >> positionIncrementGap="100"> >> <analyzer type="index"> >> <tokenizer class="solr.StandardTokenizerFactory"/> >> <filter class="solr.StopFilterFactory" >> ignoreCase="true" words="stopwords.txt" /> >> <!-- in this example, we will only use synonyms at >> query time >> <filter class="solr.SynonymFilterFactory" >> synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>--> >> <filter class="solr.LowerCaseFilterFactory"/> >> <!-- CHANGE: The NGramFilterFactory was added to >> provide partial word search. This can be changed to >> EdgeNGramFilterFactory side="front" to only match >> front sided partial searches if matching any >> part of a word is undesireable.--> >> <filter class="solr.NGramFilterFactory" >> minGramSize="3" maxGramSize="10" /> >> <!-- CHANGE: The PorterStemFilterFactory was added >> to allow matches for 'cat' and 'cats' by searching for 'cat' --> >> <filter class="solr.PorterStemFilterFactory"/> >> </analyzer> >> <analyzer type="query"> >> <tokenizer class="solr.StandardTokenizerFactory"/> >> <filter class="solr.StopFilterFactory" >> ignoreCase="true" words="stopwords.txt" /> >> <filter class="solr.SynonymFilterFactory" >> synonyms="synonyms.txt" ignoreCase="true" expand="true"/> >> <filter class="solr.LowerCaseFilterFactory"/> >> <!-- CHANGE: The PorterStemFilterFactory was added >> to allow matches for 'cat' and 'cats' by searching for 'cat' --> >> <filter class="solr.PorterStemFilterFactory"/> >> </analyzer> >> </fieldType> >> >> Just to be double sure I cleared the list in stopwords_en.txt, restarted >> Solr, re-indexed, and searched with still zero results. Any other >> suggestions on where I might be able to control this behavior? >> >> -Teague >> >> >> -----Original Message----- >> From: Anshum Gupta [mailto:ans...@anshumgupta.net] >> Sent: Monday, July 14, 2014 4:04 PM >> To: solr-user@lucene.apache.org >> Subject: Re: Of, To, and Other Small Words >> >> Hi Teague, >> >> The StopFilterFactory (which I think you're using) by default uses >> lang/stopwords_en.txt (which wouldn't be empty if you check). >> What you're looking at is the stopword.txt. You could either empty that file >> out or change the field type for your field. >> >> >> On Mon, Jul 14, 2014 at 12:53 PM, Teague James <teag...@insystechinc.com> >> wrote: >>> Hello all, >>> >>> I am working with Solr 4.9.0 and am searching for phrases that >>> contain words like "of" or "to" that Solr seems to be ignoring at index >>> time. >>> Here's what I tried: >>> >>> curl http://localhost/solr/update?commit=true -H "Content-Type: text/xml" >>> --data-binary '<add><doc><field name="id">100</field><field >>> name="content">blah blah blah knowledge of science blah blah >>> blah</field></doc></add>' >>> >>> Then, using a broswer: >>> >>> http://localhost/solr/collection1/select?q="knowledge+of+science"&fq= >>> i >>> d:100 >>> >>> I get zero hits. Search for "knowledge" or "science" and I'll get hits. >>> "knowledge of" or "of science" and I get zero hits. I don't want to >>> use proximity if I can avoid it, as this may introduce too many >>> undesireable results. Stopwords.txt is blank, yet clearly Solr is ignoring >>> "of" and "to" >>> and possibly more words that I have not discovered through testing >>> yet. Is there some other configuration file that contains these small >>> words? Is there any way to force Solr to pay attention to them and >>> not drop them from the phrase? Any advice is appreciated! Thanks! >>> >>> -Teague >>> >>> >> >> >> >> -- >> >> Anshum Gupta >> http://www.anshumgupta.net >> >
