The spellchecker handler doesn't seem to work with multi-word query. For example, when I tried to spellcheck "Java developar", it returns nothing while if I tried "developar", spellchecker correctly returns "developer". I followed the setup on the wiki.
Regards, Cuong Hoang On 7/10/07, Charles Hornberger <[EMAIL PROTECTED]> wrote: > > For what it's worth, I recently did a quick implementation of the > spellchecker feature, and I simply created another field in my schema > (Iike 'spell' in Tristan's example below). After feeding content into > my search index, I used the spell field into add one single-field > document for every distinct word in my document collection (I'm > assuming the content folks have run spell-checkers :-)). E.g.: > > <doc><field name="spell">aardvark</field></doc> > <doc><field name="spell">abacus</field></doc> > <doc><field name="spell">abbot</field></doc> > <doc><field name="spell">acacia</field></doc> > etc. > > I also added some extra documents for proper names that appear in my > documents. For instance, there are a couple fields that have > comma-separated list of names, so I for each of those -- in addition > to documents for "john", "doe", and "jane", which were generated by > the naive word-splitting done in the first pass -- I added documents > like so: > > <doc><field name="spell">john doe</field></doc> > <doc><field name="spell">jane doe</field></doc> > etc. > > You could do the same for other searchable multi-word tokens in your > input -- song/album/book/movie titles, publisher names, geographic > names (cities, neighborhoods, etc.), product names, and so on. > > -Charlie > > On 7/9/07, Tristan Vittorio <[EMAIL PROTECTED]> wrote: > > I think there is some confusion regarding how the spell checker actually > > uses the termSourceField. It is suggested that you use a simple field > type > > such a "string", however since this field type does not tokenize or > split > > words, it is only useful in situations where the whole field is > considered a > > dictionary "word": > > > > <add> > > <doc> > > <field name="title">Accountant</field> > > < > http://localhost:8984/solr/select/?q=Accountent&qt=spellchecker&cmd=rebuildand > ><field > > name="title">Auditor</field> > > <field name="title">Solicitor</field> > > </doc > > </add> > > > > The follow example case will not work with spell checker since the whole > > field is considered a single word or string: > > > > <add> > > <doc> > > <field name="title">Accountant reveals that Accounting is boring</field> > > </doc > > </add> > > > > I might suggest that you create an additional field in your schema that > > takes advantage of the StandardTokenizer and StandardFilter which > doesn't > > perform a great deal of processing on the field yet should provide > decent > > results when used with the spell checker: > > > > <fieldType name="spell" class="solr.TextField" > positionIncrementGap="100"> > > <analyzer type="index"> > > <tokenizer class="solr.StandardTokenizerFactory"/> > > <filter class="solr.StopFilterFactory" ignoreCase="true" words=" > > stopwords.txt"/> > > <filter class="solr.StandardFilterFactory"/> > > <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> > > </analyzer> > > <analyzer type="query"> > > <tokenizer class="solr.StandardTokenizerFactory"/> > > <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" > > ignoreCase="true" expand="true"/> > > <filter class="solr.StopFilterFactory" ignoreCase="true" words=" > > stopwords.txt"/> > > <filter class="solr.StandardFilterFactory"/> > > <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> > > </analyzer> > > </fieldType> > > > > If you want this field to be automatically populated with the contents > of > > the title field when a document is added to the index, simply use a > > copyField: > > > > <copyField source="title" dest="spell"/> > > > > Hope this helps, let me know if this is still not clear, I probably will > add > > it to the wiki page soon. > > > > cheers, > > Tristan > > > > > > > > On 7/9/07, climbingrose <[EMAIL PROTECTED]> wrote: > > > > > > Thanks for the quick reply. However, I'm still not able to setup > > > spellchecker. Solr does create spell directory under data but doesn't > seem > > > to build the spellchecker index. Here are snippets of my schema.xml: > > > > > > <field name="title" type="string" indexed="true" stored="true"/> > > > > > > <requestHandler name="spellchecker" class=" > solr.SpellCheckerRequestHandler > > > " > > > startup="lazy"> > > > <!-- default values for query parameters --> > > > <lst name="defaults"> > > > <int name="suggestionCount">1</int> > > > <float name="accuracy">0.5</float> > > > </lst> > > > > > > <!-- Main init params for handler --> > > > > > > <!-- The directory where your SpellChecker Index should live. > --> > > > <!-- May be absolute, or relative to the Solr "dataDir" > directory. > > > --> > > > <!-- If this option is not specified, a RAM directory will be > used > > > --> > > > <str name="spellcheckerIndexDir">spell</str> > > > > > > <!-- the field in your schema that you want to be able to build > --> > > > <!-- your spell index on. This should be a field that uses a very > --> > > > <!-- simple FieldType without a lot of Analysis (ie: string) --> > > > <str name="termSourceField">title</str> > > > > > > </requestHandler> > > > > > > I tried this url: > > > > > > > http://localhost:8984/solr/select/?q=Accountent&qt=spellchecker&cmd=rebuildand > > > receive this: > > > > > > <response> > > > <lst name="responseHeader"> > > > <int name="status">0</int> > > > <int name="QTime">2</int> > > > </lst> > > > <str name="cmdExecuted">rebuild</str> > > > <arr name="suggestions"/> > > > </response> > > > > > > > > > On 7/9/07, Tristan Vittorio <[EMAIL PROTECTED]> wrote: > > > > > > > > The spellchecker should be available in 1.2 release, your query is > > > > incorrect, try the following: > > > > > > > > > > > > > > > > http://localhost:8984/solr/select/?q=java&qt=spellchecker&termSourceField=title_text&cmd=rebuild > > > > > > > > the 'q' parameter must only contain the word being checked; you must > > > > specify > > > > the field separately. You can set "termSourceField" in your > > > > solrconfig.xmlfile so you do not need to explicitly set it each time > > > > you want to run a > > > > spell check query. Also make sure your field isn't heavily processed > ( > > > i.e. > > > > with porter stemmer analyzers) otherwise the suggestions will look a > bit > > > > weird / mangled. Take a look at the wiki page for more info: > > > > > > > > http://wiki.apache.org/solr/SpellCheckerRequestHandler > > > > > > > > cheers, > > > > Tristan > > > > > > > > > > > > > > > > On 7/9/07, climbingrose <[EMAIL PROTECTED]> wrote: > > > > > > > > > > Hi Tristan, > > > > > > > > > > Is this spellchecker available in 1.2 release or I have to build > the > > > > > trunk. > > > > > I tried your instructions but Solr returns nothing: > > > > > > > > > > > > > > > > > > > > > > > http://localhost:8984/solr/select/?q=title_text:java&qt=spellchecker&cmd=rebuild > > > > > > > > > > Result: > > > > > > > > > > <response> > > > > > <lst name="responseHeader"> > > > > > <int name="status">0</int> > > > > > <int name="QTime">3</int> > > > > > </lst> > > > > > <str name="cmdExecuted">rebuild</str> > > > > > <arr name="suggestions"/> > > > > > </response> > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > On 7/8/07, Tristan Vittorio <[EMAIL PROTECTED]> wrote: > > > > > > > > > > > > Hi Otis, > > > > > > > > > > > > I have written a draft wiki entry for the spell checker: > > > > > > http://wiki.apache.org/solr/SpellCheckerRequestHandler > > > > > > > > > > > > I've learned that my initial observation about the suggestion > > > ordering > > > > > was > > > > > > incorrect, it does in fact order the results by popularity (or > term > > > > > > frequency) of the word in the termSourceField, the problem I > > > > experienced > > > > > > was > > > > > > caused by setting termSourceField to a field of type "text", > which > > > > > heavily > > > > > > stemmed and analyzed the words. I found that using the > > > > > StandardTokenizer > > > > > > and StandardFilter and removing the PorterStemmer and > > > LowerCaseFilter > > > > > from > > > > > > the field schema really improved the spell checker performance. > > > > > > > > > > > > I haven't included this info on the wiki page yet, I'll try to > > > update > > > > it > > > > > > soon when I have a bit more time. > > > > > > > > > > > > cheers, > > > > > > Tristan > > > > > > > > > > > > > > > > > > > > > > > > On 7/8/07, Otis Gospodnetic <[EMAIL PROTECTED]> wrote: > > > > > > > > > > > > > > Tristan - good summary - want to copy that to the Solr Wiki? > > > > > > > > > > > > > > Thanks, > > > > > > > Otis > > > > > > > > > > > > > > . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . > > > > > > > Simpy -- http://www.simpy.com/ - Tag - Search - Share > > > > > > > > > > > > > > ----- Original Message ---- > > > > > > > From: Tristan Vittorio <[EMAIL PROTECTED]> > > > > > > > To: solr-user@lucene.apache.org > > > > > > > Sent: Saturday, July 7, 2007 1:51:15 AM > > > > > > > Subject: Re: Spell Check Handler > > > > > > > > > > > > > > I couldn't find any documention on the spell check handler > either > > > > but > > > > > > > found > > > > > > > enough information from the solrconfig.xml file, simply search > for > > > > > > > "SpellCheckerRequestHandler" (online version here): > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://svn.apache.org/repos/asf/lucene/solr/trunk/example/solr/conf/solrconfig.xml > > > > > > > > > > > > > > You can view the original development discussion from JIRA > (not > > > sure > > > > > how > > > > > > > helpful that will be for you though): > > > > > > > https://issues.apache.org/jira/browse/SOLR-81 > > > > > > > > > > > > > > In a nutshell, the configuration parameters available are:: > > > > > > > > > > > > > > suggestionCount: determines how many spelling suggestions are > > > > > returned. > > > > > > > accuracy: a float value between 1.0 and 0.0 on how close the > > > > suggested > > > > > > > words > > > > > > > should match the original word being checked. > > > > > > > spellcheckerIndexDir and termSourceField: check > solrconfig.xmlfor > > > > a > > > > > > full > > > > > > > explanation. > > > > > > > > > > > > > > In order to use the spell checking hander for the first time, > you > > > > need > > > > > > to > > > > > > > explicitly build the spelling index with a sample query > something > > > > like > > > > > > > this: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://localhost:8080/solr/select/?q=macrosoft&qt=spellchecker&cmd=rebuild > > > > > > > < > http://localhost:8080/solr/select/?q=macrosoft&qt=spellchecker> > > > > > > > Depending on how large you main index is, this rebuild > operation > > > > could > > > > > > > take > > > > > > > a while. Subsequent queries can omit '&cmd=rebuild' and will > > > return > > > > > > > results > > > > > > > much faster: > > > > > > > > > > > > > > http://localhost:8080/solr/select/?q=macrosoft&qt=spellchecker > > > > > > > < > http://localhost:8080/solr/select/?q=macrosoft&qt=spellchecker> > > > > > > > The order of the suggestions returned seems to be based on the > > > > > accuracy > > > > > > > figure (i.e. how close it matches the original word). it would > be > > > > > great > > > > > > to > > > > > > > be able to sort these suggested results based on term > frequency / > > > > > > document > > > > > > > frequency of the suggested word in the main index, since the > most > > > > > > accurate > > > > > > > suggestion may not always be the most relevant. > > > > > > > > > > > > > > As far as I can tell there is currently no way of doing this > using > > > > the > > > > > > > spellchecker handler alone (you could always run seperate > standard > > > > > > queries > > > > > > > on each word suggestion and order by numDocs, but that would > be > > > very > > > > > > > inefficient), has anybody else tried to achieve this? > > > > > > > > > > > > > > cheers, > > > > > > > Tristan > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 7/7/07, Andrew Nagy <[EMAIL PROTECTED] > wrote: > > > > > > > > > > > > > > > > Hello, is there any documentation on how to use the new > spell > > > > check > > > > > > > > module? > > > > > > > > > > > > > > > > Thanks > > > > > > > > Andrew > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Regards, > > > > > > > > > > Cuong Hoang > > > > > > > > > > > > > > > > > > > > > -- > > > Regards, > > > > > > Cuong Hoang > > > > > > -- Regards, Cuong Hoang
