For what it's worth, I recently did a quick implementation of the spellchecker feature, and I simply created another field in my schema (Iike 'spell' in Tristan's example below). After feeding content into my search index, I used the spell field into add one single-field document for every distinct word in my document collection (I'm assuming the content folks have run spell-checkers :-)). E.g.:
<doc><field name="spell">aardvark</field></doc> <doc><field name="spell">abacus</field></doc> <doc><field name="spell">abbot</field></doc> <doc><field name="spell">acacia</field></doc> etc. I also added some extra documents for proper names that appear in my documents. For instance, there are a couple fields that have comma-separated list of names, so I for each of those -- in addition to documents for "john", "doe", and "jane", which were generated by the naive word-splitting done in the first pass -- I added documents like so: <doc><field name="spell">john doe</field></doc> <doc><field name="spell">jane doe</field></doc> etc. You could do the same for other searchable multi-word tokens in your input -- song/album/book/movie titles, publisher names, geographic names (cities, neighborhoods, etc.), product names, and so on. -Charlie On 7/9/07, Tristan Vittorio <[EMAIL PROTECTED]> wrote:
I think there is some confusion regarding how the spell checker actually uses the termSourceField. It is suggested that you use a simple field type such a "string", however since this field type does not tokenize or split words, it is only useful in situations where the whole field is considered a dictionary "word": <add> <doc> <field name="title">Accountant</field> <http://localhost:8984/solr/select/?q=Accountent&qt=spellchecker&cmd=rebuildand><field name="title">Auditor</field> <field name="title">Solicitor</field> </doc </add> The follow example case will not work with spell checker since the whole field is considered a single word or string: <add> <doc> <field name="title">Accountant reveals that Accounting is boring</field> </doc </add> I might suggest that you create an additional field in your schema that takes advantage of the StandardTokenizer and StandardFilter which doesn't perform a great deal of processing on the field yet should provide decent results when used with the spell checker: <fieldType name="spell" class="solr.TextField" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words=" stopwords.txt"/> <filter class="solr.StandardFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words=" stopwords.txt"/> <filter class="solr.StandardFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType> If you want this field to be automatically populated with the contents of the title field when a document is added to the index, simply use a copyField: <copyField source="title" dest="spell"/> Hope this helps, let me know if this is still not clear, I probably will add it to the wiki page soon. cheers, Tristan On 7/9/07, climbingrose <[EMAIL PROTECTED]> wrote: > > Thanks for the quick reply. However, I'm still not able to setup > spellchecker. Solr does create spell directory under data but doesn't seem > to build the spellchecker index. Here are snippets of my schema.xml: > > <field name="title" type="string" indexed="true" stored="true"/> > > <requestHandler name="spellchecker" class="solr.SpellCheckerRequestHandler > " > startup="lazy"> > <!-- default values for query parameters --> > <lst name="defaults"> > <int name="suggestionCount">1</int> > <float name="accuracy">0.5</float> > </lst> > > <!-- Main init params for handler --> > > <!-- The directory where your SpellChecker Index should live. --> > <!-- May be absolute, or relative to the Solr "dataDir" directory. > --> > <!-- If this option is not specified, a RAM directory will be used > --> > <str name="spellcheckerIndexDir">spell</str> > > <!-- the field in your schema that you want to be able to build --> > <!-- your spell index on. This should be a field that uses a very --> > <!-- simple FieldType without a lot of Analysis (ie: string) --> > <str name="termSourceField">title</str> > > </requestHandler> > > I tried this url: > > http://localhost:8984/solr/select/?q=Accountent&qt=spellchecker&cmd=rebuildand > receive this: > > <response> > <lst name="responseHeader"> > <int name="status">0</int> > <int name="QTime">2</int> > </lst> > <str name="cmdExecuted">rebuild</str> > <arr name="suggestions"/> > </response> > > > On 7/9/07, Tristan Vittorio <[EMAIL PROTECTED]> wrote: > > > > The spellchecker should be available in 1.2 release, your query is > > incorrect, try the following: > > > > > > > http://localhost:8984/solr/select/?q=java&qt=spellchecker&termSourceField=title_text&cmd=rebuild > > > > the 'q' parameter must only contain the word being checked; you must > > specify > > the field separately. You can set "termSourceField" in your > > solrconfig.xmlfile so you do not need to explicitly set it each time > > you want to run a > > spell check query. Also make sure your field isn't heavily processed ( > i.e. > > with porter stemmer analyzers) otherwise the suggestions will look a bit > > weird / mangled. Take a look at the wiki page for more info: > > > > http://wiki.apache.org/solr/SpellCheckerRequestHandler > > > > cheers, > > Tristan > > > > > > > > On 7/9/07, climbingrose <[EMAIL PROTECTED]> wrote: > > > > > > Hi Tristan, > > > > > > Is this spellchecker available in 1.2 release or I have to build the > > > trunk. > > > I tried your instructions but Solr returns nothing: > > > > > > > > > > > > http://localhost:8984/solr/select/?q=title_text:java&qt=spellchecker&cmd=rebuild > > > > > > Result: > > > > > > <response> > > > <lst name="responseHeader"> > > > <int name="status">0</int> > > > <int name="QTime">3</int> > > > </lst> > > > <str name="cmdExecuted">rebuild</str> > > > <arr name="suggestions"/> > > > </response> > > > > > > Thanks. > > > > > > > > > On 7/8/07, Tristan Vittorio <[EMAIL PROTECTED]> wrote: > > > > > > > > Hi Otis, > > > > > > > > I have written a draft wiki entry for the spell checker: > > > > http://wiki.apache.org/solr/SpellCheckerRequestHandler > > > > > > > > I've learned that my initial observation about the suggestion > ordering > > > was > > > > incorrect, it does in fact order the results by popularity (or term > > > > frequency) of the word in the termSourceField, the problem I > > experienced > > > > was > > > > caused by setting termSourceField to a field of type "text", which > > > heavily > > > > stemmed and analyzed the words. I found that using the > > > StandardTokenizer > > > > and StandardFilter and removing the PorterStemmer and > LowerCaseFilter > > > from > > > > the field schema really improved the spell checker performance. > > > > > > > > I haven't included this info on the wiki page yet, I'll try to > update > > it > > > > soon when I have a bit more time. > > > > > > > > cheers, > > > > Tristan > > > > > > > > > > > > > > > > On 7/8/07, Otis Gospodnetic <[EMAIL PROTECTED]> wrote: > > > > > > > > > > Tristan - good summary - want to copy that to the Solr Wiki? > > > > > > > > > > Thanks, > > > > > Otis > > > > > > > > > > . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . > > > > > Simpy -- http://www.simpy.com/ - Tag - Search - Share > > > > > > > > > > ----- Original Message ---- > > > > > From: Tristan Vittorio <[EMAIL PROTECTED]> > > > > > To: solr-user@lucene.apache.org > > > > > Sent: Saturday, July 7, 2007 1:51:15 AM > > > > > Subject: Re: Spell Check Handler > > > > > > > > > > I couldn't find any documention on the spell check handler either > > but > > > > > found > > > > > enough information from the solrconfig.xml file, simply search for > > > > > "SpellCheckerRequestHandler" (online version here): > > > > > > > > > > > > > > > > > > > > http://svn.apache.org/repos/asf/lucene/solr/trunk/example/solr/conf/solrconfig.xml > > > > > > > > > > You can view the original development discussion from JIRA (not > sure > > > how > > > > > helpful that will be for you though): > > > > > https://issues.apache.org/jira/browse/SOLR-81 > > > > > > > > > > In a nutshell, the configuration parameters available are:: > > > > > > > > > > suggestionCount: determines how many spelling suggestions are > > > returned. > > > > > accuracy: a float value between 1.0 and 0.0 on how close the > > suggested > > > > > words > > > > > should match the original word being checked. > > > > > spellcheckerIndexDir and termSourceField: check solrconfig.xmlfor > > a > > > > full > > > > > explanation. > > > > > > > > > > In order to use the spell checking hander for the first time, you > > need > > > > to > > > > > explicitly build the spelling index with a sample query something > > like > > > > > this: > > > > > > > > > > > > > > > > > > > > http://localhost:8080/solr/select/?q=macrosoft&qt=spellchecker&cmd=rebuild > > > > > <http://localhost:8080/solr/select/?q=macrosoft&qt=spellchecker> > > > > > Depending on how large you main index is, this rebuild operation > > could > > > > > take > > > > > a while. Subsequent queries can omit '&cmd=rebuild' and will > return > > > > > results > > > > > much faster: > > > > > > > > > > http://localhost:8080/solr/select/?q=macrosoft&qt=spellchecker > > > > > <http://localhost:8080/solr/select/?q=macrosoft&qt=spellchecker> > > > > > The order of the suggestions returned seems to be based on the > > > accuracy > > > > > figure (i.e. how close it matches the original word). it would be > > > great > > > > to > > > > > be able to sort these suggested results based on term frequency / > > > > document > > > > > frequency of the suggested word in the main index, since the most > > > > accurate > > > > > suggestion may not always be the most relevant. > > > > > > > > > > As far as I can tell there is currently no way of doing this using > > the > > > > > spellchecker handler alone (you could always run seperate standard > > > > queries > > > > > on each word suggestion and order by numDocs, but that would be > very > > > > > inefficient), has anybody else tried to achieve this? > > > > > > > > > > cheers, > > > > > Tristan > > > > > > > > > > > > > > > > > > > > On 7/7/07, Andrew Nagy <[EMAIL PROTECTED] > wrote: > > > > > > > > > > > > Hello, is there any documentation on how to use the new spell > > check > > > > > > module? > > > > > > > > > > > > Thanks > > > > > > Andrew > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Regards, > > > > > > Cuong Hoang > > > > > > > > > -- > Regards, > > Cuong Hoang >
