Query regarding spelling suggestions
Hi, I have defined a Search Component in solrconfig.xml containing two entries for spellchecker. One is the indexed field - "title" . Second is a text file - "mywords.txt". When I enable spellchecking and query for some word, I expect that the spelling suggestions will come from both the dictionaries. But my observation is that the suggestions come from the indexed field only if some exist otherwise I do not get any suggestions. If I explicitly add spellcheck.dictionary=file to get suggestion from text file then I get suggestions from the file but in that case I do not get suggestions from the indexed field. How can I get spelling suggestions from both dictionaries. Thanks, Mukta
Setting Request Handler
Hi,
I have a request handler in my solrconfig.xml : /spellCheckCompRH
It utilizes the search component "spellcheck".
When I specify following query in browser, I get correct spelling
suggestions from the file dictionary.
http://localhost:8080/solr/spellCheckCompRH/?q=SolrDocs&spellcheck.q=rel
evancy&spellcheck=true&fl=title,score&spellcheck.dictionary=file
Now I write a java program to achieve the same result:
Code snippet
.
.
server = new CommonsHttpSolrServer("http://localhost:8080/solr";);
.
.
SolrQuery query = new SolrQuery();
query.setQuery("solr" );
query.setFields("*,score");
query.set("qt", spellCheckCompRH);
query.set("spellcheck", "true");
query.set(SpellingParams.SPELLCHECK_DICT, "file");
query.set(SpellingParams.SPELLCHECK_Q , "solt");
.
.
QueryResponse rsp = server.query( query );
SolrDocumentList docs = rsp.getResults();
SpellCheckResponse srsp = rsp.getSpellCheckResponse();
I get documents for my query but I do not get any spelling suggestions.
I think that the request handler is not getting set for the query
correctly.
Can someone please help.
Best Regards,
Mukta
RE: Setting Request Handler
Hi Grant,
Thanks for the help.
So now I can have multiple components, configured as "last-components"
of standard request handler.
Best Regards,
Mukta
-Original Message-
From: Grant Ingersoll [mailto:[EMAIL PROTECTED]
Sent: Wednesday, December 10, 2008 9:25 PM
To: solr-user@lucene.apache.org
Subject: Re: Setting Request Handler
Inline below...
Also, though, you should note that the /spellCheckCompRH that is
packaged with the example is not necessarily the best way to actually
use the SpellCheckComponent. It is intended to be used as a
component in whatever your MAIN Request Handler is, it merely shows the
how of hooking it in.
On Dec 10, 2008, at 7:51 AM, Deshpande, Mukta wrote:
> Hi,
>
> I have a request handler in my solrconfig.xml : /spellCheckCompRH It
> utilizes the search component "spellcheck".
>
> When I specify following query in browser, I get correct spelling
> suggestions from the file dictionary.
>
> http://localhost:8080/solr/spellCheckCompRH/?q=SolrDocs&spellcheck.q=r
> el evancy&spellcheck=true&fl=title,score&spellcheck.dictionary=file
>
> Now I write a java program to achieve the same result:
>
> Code snippet
>
> .
> .
>
> server = new CommonsHttpSolrServer("http://localhost:8080/solr";);
> .
> .
> SolrQuery query = new SolrQuery();
> query.setQuery("solr" );
> query.setFields("*,score");
> query.set("qt", spellCheckCompRH);
Is spellCheckCompRH a variable? Does it equal "/spellCheckCompRH"?
>
> query.set("spellcheck", "true");
> query.set(SpellingParams.SPELLCHECK_DICT, "file");
> query.set(SpellingParams.SPELLCHECK_Q , "solt"); .
> .
> QueryResponse rsp = server.query( query ); SolrDocumentList docs =
> rsp.getResults(); SpellCheckResponse srsp =
> rsp.getSpellCheckResponse();
>
> I get documents for my query but I do not get any spelling
> suggestions.
> I think that the request handler is not getting set for the query
> correctly.
>
> Can someone please help.
>
> Best Regards,
> Mukta
--
Grant Ingersoll
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
Multiple language support
Hi All, I have a multiple language supporting schema in which there is a separate field for every language. I have a field "product_name" to store product name and its description that can be in any user preferred language. This can be stored in fields product_name_EN if user prefers English language, product_name_SCH if user prefers Simplified Chinese language. The WhitespaceTokenizerFactory and filter EnglishPorterFilterFactory are applied on product_name_EN. The CJKAnalyzer and CJKTokenizer are applied on product_name_SCH. e.g. Value can be : ElectrolyticCapacitor - 被对立的电容器以价值220µF Now my problem is: Which field do I store the above value? product_name_EN OR product_name_SCH OR should it be something else? How do I find out which analyzers should get applied for this field. Did any one face a similar situation before. Please help ASAP. Thanks, ~Mukta
Query regarding Spelling Suggestions
Hi, I am using Wordnet dictionary for spelling suggestions. The dictionary is converted to Solr index with only one field "word" and stored in location /data/syn_index, using syns2Index.java program available at http://www.tropo.com/techno/java/lucene/wordnet.html I have added the "word" field in my "schema.xml" as My application data indexes are in /data I am trying to use solr.IndexBasedSpellChecker to get spelling suggestions. My spell check component is configured as: textSpell dict solr.IndexBasedSpellChecker word UTF-8 ./syn_index I have added this component to my standard request handler as: explicit spellcheck With the above configuration, I do not get any spelling suggestions. Can somebody help ASAP. Thanks, ~Mukta
RE: Query regarding Spelling Suggestions
Yes. I send the build command as:
http://localhost:8080/solr/select/?q=documnet&spellcheck=true&spellcheck
.build=true&spellcheck.count=2&spellcheck.q=parfect&spellcheck.dictionar
y=dict
The Tomcat log shows:
Jan 9, 2009 9:55:19 AM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select/
params={spellcheck=true&q=documnet&spellcheck.q=parfect&spellcheck.dicti
onary=dict&spellcheck.count=2&spellcheck.build=true} hits=0 status=0
QTime=141
Even after sending the build command I do not get any suggestions.
Can you please check.
Thanks,
~Mukta
-Original Message-
From: Grant Ingersoll [mailto:gsing...@apache.org]
Sent: Thursday, January 08, 2009 7:42 PM
To: solr-user@lucene.apache.org
Subject: Re: Query regarding Spelling Suggestions
Did you send in the build command? See
http://wiki.apache.org/solr/SpellCheckComponent
On Jan 8, 2009, at 5:14 AM, Deshpande, Mukta wrote:
> Hi,
>
> I am using Wordnet dictionary for spelling suggestions.
>
> The dictionary is converted to Solr index with only one field "word"
> and stored in location /data/syn_index, using
> syns2Index.java program available at
> http://www.tropo.com/techno/java/lucene/wordnet.html
>
> I have added the "word" field in my "schema.xml" as type="textSpell" indexed="true" stored="true"/>
>
> My application data indexes are in /data
>
> I am trying to use solr.IndexBasedSpellChecker to get spelling
> suggestions.
>
> My spell check component is configured as:
>
>
> textSpell
>
> dict
> solr.IndexBasedSpellChecker
> word
> UTF-8
> ./syn_index
>
>
>
> I have added this component to my standard request handler as:
>
> default="true">
>
> explicit
>
>
> spellcheck
>
>
>
> With the above configuration, I do not get any spelling suggestions.
> Can
> somebody help ASAP.
>
> Thanks,
> ~Mukta
--
Grant Ingersoll
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
RE: Query regarding Spelling Suggestions
I am using the example schema that comes with the Solr installation downloaded
from http://www.mirrorgeek.com/apache.org/lucene/solr/.
I have added the "word" field with "textSpell" fieldtype in the schema.xml
file, as specified in the below mail.
My spelling index exist under /data/
If I open my index in Luke I can see the entries against "word" field.
Thanks,
~Mukta
From: Grant Ingersoll [mailto:gsing...@apache.org]
Sent: Fri 1/9/2009 8:29 AM
To: solr-user@lucene.apache.org
Subject: Re: Query regarding Spelling Suggestions
Can you put the full log (as short as possibly demonstrates the
problem) somewhere where I can take a look? Likewise, can you share
your schema?
Also, does the spelling index exist under /data/index? If
you open it w/ Luke, does it have entries?
Thanks,
Grant
On Jan 8, 2009, at 11:30 PM, Deshpande, Mukta wrote:
>
> Yes. I send the build command as:
> http://localhost:8080/solr/select/?q=documnet&spellcheck=true&spellcheck
> .build
> =true&spellcheck.count=2&spellcheck.q=parfect&spellcheck.dictionar
> y=dict
>
> The Tomcat log shows:
> Jan 9, 2009 9:55:19 AM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/select/
> params
> ={spellcheck=true&q=documnet&spellcheck.q=parfect&spellcheck.dicti
> onary=dict&spellcheck.count=2&spellcheck.build=true} hits=0 status=0
> QTime=141
>
> Even after sending the build command I do not get any suggestions.
> Can you please check.
>
> Thanks,
> ~Mukta
>
> -Original Message-
> From: Grant Ingersoll [mailto:gsing...@apache.org]
> Sent: Thursday, January 08, 2009 7:42 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Query regarding Spelling Suggestions
>
> Did you send in the build command? See
> http://wiki.apache.org/solr/SpellCheckComponent
>
> On Jan 8, 2009, at 5:14 AM, Deshpande, Mukta wrote:
>
>> Hi,
>>
>> I am using Wordnet dictionary for spelling suggestions.
>>
>> The dictionary is converted to Solr index with only one field "word"
>> and stored in location /data/syn_index, using
>> syns2Index.java program available at
>> http://www.tropo.com/techno/java/lucene/wordnet.html
>>
>> I have added the "word" field in my "schema.xml" as > name="word"
>
>> type="textSpell" indexed="true" stored="true"/>
>>
>> My application data indexes are in /data
>>
>> I am trying to use solr.IndexBasedSpellChecker to get spelling
>> suggestions.
>>
>> My spell check component is configured as:
>>
>>
>> textSpell
>>
>> dict
>> solr.IndexBasedSpellChecker
>> word
>> UTF-8
>> ./syn_index
>>
>>
>>
>> I have added this component to my standard request handler as:
>>
>> > default="true">
>>
>>explicit
>>
>>
>>spellcheck
>>
>>
>>
>> With the above configuration, I do not get any spelling suggestions.
>> Can
>> somebody help ASAP.
>>
>> Thanks,
>> ~Mukta
>
> --
> Grant Ingersoll
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
>
>
>
>
>
>
>
--
Grant Ingersoll
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
RE: Query regarding Spelling Suggestions
Hi,
Could you please send me the needful entries in log4j.properties to
enable logging, explicitly for SpellCheckComponent.
My current log4j.properties looks like:
log4j.rootLogger=INFO,console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p
%c{2}: %m%n
log4j.logger.org.apache.solr=DEBUG
With these settings I can only see the INFO level logs.
I tried to change the log level for SpellCheckComponent to "FINE" using
the admin logging page http://localhost:8080/solr/admin/logging but did
not see any difference in logging.
Thanks,
~Mukta
-Original Message-
From: Grant Ingersoll [mailto:gsing...@apache.org]
Sent: Monday, January 12, 2009 3:22 AM
To: solr-user@lucene.apache.org
Subject: Re: Query regarding Spelling Suggestions
Can you send the full log?
On Jan 11, 2009, at 1:51 PM, Deshpande, Mukta wrote:
> I am using the example schema that comes with the Solr installation
> downloaded from http://www.mirrorgeek.com/apache.org/lucene/solr/.
> I have added the "word" field with "textSpell" fieldtype in the
> schema.xml file, as specified in the below mail.
>
> My spelling index exist under /data/ If I open my index in
> Luke I can see the entries against "word"
> field.
>
> Thanks,
> ~Mukta
>
>
>
>
> From: Grant Ingersoll [mailto:gsing...@apache.org]
> Sent: Fri 1/9/2009 8:29 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Query regarding Spelling Suggestions
>
>
>
> Can you put the full log (as short as possibly demonstrates the
> problem) somewhere where I can take a look? Likewise, can you share
> your schema?
>
> Also, does the spelling index exist under /data/index? If
> you open it w/ Luke, does it have entries?
>
> Thanks,
> Grant
>
> On Jan 8, 2009, at 11:30 PM, Deshpande, Mukta wrote:
>
>>
>> Yes. I send the build command as:
>> http://localhost:8080/solr/select/?q=documnet&spellcheck=true&spellch
>> eck
>> .build
>> =true&spellcheck.count=2&spellcheck.q=parfect&spellcheck.dictionar
>> y=dict
>>
>> The Tomcat log shows:
>> Jan 9, 2009 9:55:19 AM org.apache.solr.core.SolrCore execute
>> INFO: [] webapp=/solr path=/select/
>> params
>> ={spellcheck=true&q=documnet&spellcheck.q=parfect&spellcheck.dicti
>> onary=dict&spellcheck.count=2&spellcheck.build=true} hits=0 status=0
>> QTime=141
>>
>> Even after sending the build command I do not get any suggestions.
>> Can you please check.
>>
>> Thanks,
>> ~Mukta
>>
>> -Original Message-
>> From: Grant Ingersoll [mailto:gsing...@apache.org]
>> Sent: Thursday, January 08, 2009 7:42 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Query regarding Spelling Suggestions
>>
>> Did you send in the build command? See
>> http://wiki.apache.org/solr/SpellCheckComponent
>>
>> On Jan 8, 2009, at 5:14 AM, Deshpande, Mukta wrote:
>>
>>> Hi,
>>>
>>> I am using Wordnet dictionary for spelling suggestions.
>>>
>>> The dictionary is converted to Solr index with only one field
>>> "word"
>>> and stored in location /data/syn_index, using
>>> syns2Index.java program available at
>>> http://www.tropo.com/techno/java/lucene/wordnet.html
>>>
>>> I have added the "word" field in my "schema.xml" as >> name="word"
>>
>>> type="textSpell" indexed="true" stored="true"/>
>>>
>>> My application data indexes are in /data
>>>
>>> I am trying to use solr.IndexBasedSpellChecker to get spelling
>>> suggestions.
>>>
>>> My spell check component is configured as:
>>>
>>>
>>> textSpell
>>>
>>>dict
>>>solr.IndexBasedSpellChecker
>>>word
>>>UTF-8
>>>./syn_index
>>>
>>>
>>>
>>> I have added this component to my standard request handler as:
>>>
>>> >> default="true">
>>>
>>> explicit
>>>
>>>
>>> spellcheck
>>>
>>>
>>>
>>> With the above configuration, I do not get any spelling suggestions.
>>> Can
>>> somebody help ASAP.
>>>
>>> Thanks,
>>> ~Mukta
>>
>>
RE: Query regarding Spelling Suggestions
Hi Grant,
My spellcheck is now working fine with the following configuration:
word
solr.IndexBasedSpellChecker
word
UTF-8
d:\solr-tomcat\solr\data\syn_index
./spellcheckerFile1
Earlier I configured the lucene-index (dictionary) "syn_index" to the
spellcheckIndexDir as interpreted from the wiki page.
Then I was looking into the file IndexBasedSpellChecker.java and found
the usage of "sourceLocation".
When I configured my lucene-index (dictionary) "syn_index" as
"sourceLocation" the IndexBasedSpellChecker worked.
I have following question / observation : (just to ensure that my
configurations are correct)
The lucene-index (dictionary) "syn_index" is already an index so do we
have to specify the spellcheckIndexDir again?
(If I do not give the spellcheckIndexDir I do not get any
suggestions.)
When I give the build command the spellcheckIndexDir gets populated
reading the "syn_index". Can we avoid this duplication?
If the "sourceLocation" is mandatory when using a third party index for
spelling suggestions, may I update the Solr WIKI to include this
important information.
Thanks & Best Regards,
~Mukta
-Original Message-
From: Grant Ingersoll [mailto:gsing...@apache.org]
Sent: Monday, January 12, 2009 10:15 PM
To: solr-user@lucene.apache.org
Subject: Re: Query regarding Spelling Suggestions
Solr 1.3 doesn't use Log4J, it uses Java Utility Logging (JUL). I
believe the info level in the logs is sufficient. Let's start by
posting what you have?
Also, are you able to get the sample spellchecking to work?
On Jan 12, 2009, at 2:16 AM, Deshpande, Mukta wrote:
> Hi,
>
> Could you please send me the needful entries in log4j.properties to
> enable logging, explicitly for SpellCheckComponent.
>
> My current log4j.properties looks like:
>
> log4j.rootLogger=INFO,console
> log4j.appender.console=org.apache.log4j.ConsoleAppender
> log4j.appender.console.target=System.err
> log4j.appender.console.layout=org.apache.log4j.PatternLayout
> log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd
> HH:mm:ss} %p
> %c{2}: %m%n
> log4j.logger.org.apache.solr=DEBUG
>
> With these settings I can only see the INFO level logs.
>
> I tried to change the log level for SpellCheckComponent to "FINE"
> using
> the admin logging page http://localhost:8080/solr/admin/logging but
> did not see any difference in logging.
>
> Thanks,
> ~Mukta
>
> -Original Message-
> From: Grant Ingersoll [mailto:gsing...@apache.org]
> Sent: Monday, January 12, 2009 3:22 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Query regarding Spelling Suggestions
>
> Can you send the full log?
>
> On Jan 11, 2009, at 1:51 PM, Deshpande, Mukta wrote:
>
>> I am using the example schema that comes with the Solr installation
>> downloaded from http://www.mirrorgeek.com/apache.org/lucene/solr/.
>> I have added the "word" field with "textSpell" fieldtype in the
>> schema.xml file, as specified in the below mail.
>>
>> My spelling index exist under /data/ If I open my index in
>> Luke I can see the entries against "word"
>> field.
>>
>> Thanks,
>> ~Mukta
>>
>>
>>
>>
>> From: Grant Ingersoll [mailto:gsing...@apache.org]
>> Sent: Fri 1/9/2009 8:29 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Query regarding Spelling Suggestions
>>
>>
>>
>> Can you put the full log (as short as possibly demonstrates the
>> problem) somewhere where I can take a look? Likewise, can you share
>> your schema?
>>
>> Also, does the spelling index exist under /data/index? If
>> you open it w/ Luke, does it have entries?
>>
>> Thanks,
>> Grant
>>
>> On Jan 8, 2009, at 11:30 PM, Deshpande, Mukta wrote:
>>
>>>
>>> Yes. I send the build command as:
>>> http://localhost:8080/solr/select/?
>>> q=documnet&spellcheck=true&spellch
>>> eck
>>> .build
>>> =true&spellcheck.count=2&spellcheck.q=parfect&spellcheck.dictionar
>>> y=dict
>>>
>>> The Tomcat log shows:
>>> Jan 9, 2009 9:55:19 AM org.apache.solr.core.SolrCore execute
>>> INFO: [] webapp=/solr path=/select/
>>> params
>>> ={spellcheck=true&q=documnet&spellcheck.q=parfect&spellcheck.dicti
>>> onary=dict&spellcheck.count=2&spellcheck.build=true} hits=0 status=0
>>> QTime=141
>>>
>>> Even after sending the build command I do not get any suggestions.
>>> Can you please check.
>
RE: Query regarding Spelling Suggestions
Thanks all for the help and information. Best Regards, ~Mukta -Original Message- From: Shalin Shekhar Mangar [mailto:shalinman...@gmail.com] Sent: Tuesday, January 13, 2009 6:50 PM To: solr-user@lucene.apache.org Subject: Re: Query regarding Spelling Suggestions On Tue, Jan 13, 2009 at 5:16 PM, Deshpande, Mukta wrote: > I have following question / observation : (just to ensure that my > configurations are correct) > > The lucene-index (dictionary) "syn_index" is already an index so do we > have to specify the spellcheckIndexDir again? >(If I do not give the spellcheckIndexDir I do not get any > suggestions.) The "syn_index" here is your Lucene index from which you want to use as the source for words. Spell checker processes each token to create n-grams which are then stored into a lucene index at the "spellCheckIndexDir" or in memory. This is why you need to specify both sourceLocation and spellcheckIndexDir. If you do not give spellCheckIndexDir, spell checker will create a Lucene index in-memory, so it should still work. Are you sure you gave a build command before issuing the query? > When I give the build command the spellcheckIndexDir gets populated > reading the "syn_index". Can we avoid this duplication? Spell checker needs a Lucene index to work. It creates a new one and adds tokens after some processing to this index. There is no way to avoid creation of another index at present. However, it should be possible to modify it to store it's fields inside an existing Lucene index (maybe even Solr's own index). Contributions are always welcome :) > If the "sourceLocation" is mandatory when using a third party index > for spelling suggestions, may I update the Solr WIKI to include this > important information. Sure, please go ahead. Thanks! -- Regards, Shalin Shekhar Mangar.
