Hello,
Further investigation shows the following pattern, for both DirectIndex and
wordbreak spellchekers.
Assume that in all cases there are spellchecker results when distrib=false
In distributed mode (distrib=true)
case when matches=0
1. group=true, no spellcheck results
2. group=false , there are spellcheck results
case when matches>0
1. group=true, there are spellcheck results
2. group =false, there are spellcheck results
Do these constitute a failing test case?
Thanks.
Alex.
-----Original Message-----
From: alxsss <alx...@aim.com>
To: solr-user <solr-user@lucene.apache.org>
Sent: Thu, Mar 21, 2013 6:50 pm
Subject: Re: strange behaviour of wordbreak spellchecker in solr cloud
Hello,
I am debugging the SpellCheckComponent#finishStage.
>From the responses I see that not only wordbreak, but also directSpellchecker
does not return some results in distributed mode.
The request handler I was using had
<str name="group">true</str>
So, I desided to turn of grouping and I see spellcheck results in distributed
mode.
curl
'server1:8983/solr/test/testhandler?q=paulusoles&indent=true&rows=10&shards.qt=testhandler'
has no spellchek results
but
curl
'server1:8983/solr/test/testhandler?q=paulusoles&indent=true&rows=10&shards.qt=testhandler
&group=false'
returns results.
So, the conclusion is that grouping causes the distributed spellcheker to fail.
Could please you point me to the class that may be responsible to this issue?
Thanks.
Alex.
-----Original Message-----
From: Dyer, James <james.d...@ingramcontent.com>
To: solr-user <solr-user@lucene.apache.org>
Sent: Thu, Mar 21, 2013 11:23 am
Subject: RE: strange behaviour of wordbreak spellchecker in solr cloud
The shard responses get combined in SpellCheckComponent#finishStage . I highly
recommend you file a JIRA bug report for this at
https://issues.apache.org/jira/browse/SOLR
. If you write a failing unit test, it would make it much more likely that
others would help you with a fix. Of course, if you solve the issue entirely,
a
patch would be much appreciated.
James Dyer
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: alx...@aim.com [mailto:alx...@aim.com]
Sent: Thursday, March 21, 2013 12:45 PM
To: solr-user@lucene.apache.org
Subject: Re: strange behaviour of wordbreak spellchecker in solr cloud
Hello,
We need this feature be fixed ASAP. So, please let me know which class is
responsible for combining spellcheck results from all shards. I will try to
debug the code.
Thanks in advance.
Alex.
-----Original Message-----
From: alxsss <alx...@aim.com>
To: solr-user <solr-user@lucene.apache.org>
Sent: Tue, Mar 19, 2013 11:34 am
Subject: Re: strange behaviour of wordbreak spellchecker in solr cloud
-- distributed environment. But to nail it down, we probably need to see both
-- the applicable <requestHandler />
Not sure what this is?
I have
<searchComponent name="spellcheck" class="solr.SpellCheckComponent">
<str name="queryAnalyzerFieldType">spell</str>
<!-- Multiple "Spell Checkers" can be declared and used by this
component
-->
<!-- a spellchecker built from a field of the main index -->
<lst name="spellchecker">
<str name="name">direct</str>
<str name="field">spell</str>
<str name="classname">solr.DirectSolrSpellChecker</str>
<!-- the spellcheck distance measure used, the default is the internal
levenshtein -->
<str name="distanceMeasure">internal</str>
<!-- minimum accuracy needed to be considered a valid spellcheck
suggestion -->
<float name="accuracy">0.5</float>
<!-- the maximum #edits we consider when enumerating terms: can be 1 or 2
-->
<int name="maxEdits">2</int>
<!-- the minimum shared prefix when enumerating terms -->
<int name="minPrefix">1</int>
<!-- maximum number of inspections per result. -->
<int name="maxInspections">5</int>
<!-- minimum length of a query term to be considered for correction -->
<int name="minQueryLength">4</int>
<!-- maximum threshold of documents a query term can appear to be
considered for correction -->
<float name="maxQueryFrequency">0.01</float>
<!-- uncomment this to require suggestions to occur in 1% of the documents
<float name="thresholdTokenFrequency">.01</float>
-->
</lst>
<!-- a spellchecker that can break or combine words. See "/spell" handler
below for usage -->
<lst name="spellchecker">
<str name="name">wordbreak</str>
<str name="classname">solr.WordBreakSolrSpellChecker</str>
<str name="field">spell</str>
<str name="combineWords">true</str>
<str name="breakWords">true</str>
<int name="maxChanges">10</int>
</lst>
<!-- a spellchecker that uses a different distance measure -->
<!--
<lst name="spellchecker">
<str name="name">jarowinkler</str>
<str name="field">spell</str>
<str name="classname">solr.DirectSolrSpellChecker</str>
<str name="distanceMeasure">
org.apache.lucene.search.spell.JaroWinklerDistance
</str>
</lst>
-->
<!-- a spellchecker that use an alternate comparator
comparatorClass be one of:
1. score (default)
2. freq (Frequency first, then score)
3. A fully qualified class name
-->
<!--
<lst name="spellchecker">
<str name="name">freq</str>
<str name="field">lowerfilt</str>
<str name="classname">solr.DirectSolrSpellChecker</str>
<str name="comparatorClass">freq</str>
-->
<!-- A spellchecker that reads the list of words from a file -->
<!--
<lst name="spellchecker">
<str name="classname">solr.FileBasedSpellChecker</str>
<str name="name">file</str>
<str name="sourceLocation">spellings.txt</str>
<str name="characterEncoding">UTF-8</str>
<str name="spellcheckIndexDir">spellcheckerFile</str>
</lst>
-->
</searchComponent>
spell filed in our schema is called spell and its type also is called spell.
Here are requests
curl
'server1:8983/solr/test/testhandler?q=paulusoles&indent=true&rows=10&shards.qt=testhandler&distrib=false'
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">32</int>
<lst name="params">
<str name="indent">true</str>
<str name="shards.qt">testhandler</str>
<str name="q">paulusoles</str>
<str name="distrib">false</str>
<str name="rows">10</str>
</lst>
</lst>
<lst name="grouped">
<lst name="site">
<int name="matches">0</int>
<int name="ngroups">0</int>
<arr name="groups"/>
</lst>
</lst>
<lst name="highlighting"/>
<lst name="spellcheck">
<lst name="suggestions"/>
</lst>
</response>
curl
'server2:8983/solr/test/testhandler?q=paulusoles&indent=true&rows=10&shards.qt=testhandler&distrib=false'
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">26</int>
<lst name="params">
<str name="indent">true</str>
<str name="shards.qt">testhandler</str>
<str name="q">paulusoles</str>
<str name="distrib">false</str>
<str name="rows">10</str>
</lst>
</lst>
<lst name="grouped">
<lst name="site">
<int name="matches">0</int>
<int name="ngroups">0</int>
<arr name="groups"/>
</lst>
</lst>
<lst name="highlighting"/>
<lst name="spellcheck">
<lst name="suggestions">
<lst name="paulusoles">
<int name="numFound">1</int>
<int name="startOffset">0</int>
<int name="endOffset">11</int>
<arr name="suggestion">
<str>paul u soles</str>
</arr>
</lst>
<str name="collation">(paul u soles)</str>
</lst>
</lst>
</response>
No distrib param
curl
'server1:8983/solr/test/testhandler?q=paulusoles&indent=true&rows=10&shards.qt=testhandler'
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">24</int>
<lst name="params">
<str name="indent">true</str>
<str name="shards.qt">testhandler</str>
<str name="q">paulusoles</str>
<str name="distrib">false</str>
<str name="rows">10</str>
</lst>
</lst>
<lst name="grouped">
<lst name="site">
<int name="matches">0</int>
<int name="ngroups">0</int>
<arr name="groups"/>
</lst>
</lst>
<lst name="highlighting"/>
<lst name="spellcheck">
<lst name="suggestions"/>
</lst>
</response>
curl
'server2:8983/solr/test/testhandler?q=paulusoles&indent=true&rows=10&shards.qt=testhandler'
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">24</int>
<lst name="params">
<str name="indent">true</str>
<str name="shards.qt">testhandler</str>
<str name="q">paulusoles</str>
<str name="rows">10</str>
</lst>
</lst>
<lst name="grouped">
<lst name="site">
<int name="matches">0</int>
<int name="ngroups">0</int>
<arr name="groups"/>
</lst>
</lst>
<lst name="highlighting"/>
<lst name="spellcheck">
<lst name="suggestions"/>
</lst>
</response>
Thanks.
Alex.
---Original Message-----
From: Dyer, James <james.d...@ingramcontent.com>
To: solr-user <solr-user@lucene.apache.org>
Sent: Tue, Mar 19, 2013 11:10 am
Subject: RE: strange behaviour of wordbreak spellchecker in solr cloud
You may likely be hitting on a bug with WordBreakSolrSpellChecker in a
distributed environment. But to nail it down, we probably need to see both the
applicable <requestHandler /> section of your config and also this section:
<searchComponent name="spellcheck" class="solr.SpellCheckComponent" />. Also
need an example of a query that succeeds non-distributed (with the exact query
url and output you get) vs the same query url and output in the distributed
scenario. Then, without access to your actual index, it might be possible to
come up with a failing unit test. With a failing unit test in hand, we have a
good shot at getting a fix.
James Dyer
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: alx...@aim.com [mailto:alx...@aim.com]
Sent: Tuesday, March 19, 2013 12:39 PM
To: solr-user@lucene.apache.org
Subject: Re: strange behaviour of wordbreak spellchecker in solr cloud
Hello,
I was testing my custom testhandler. Direct spellchecker also was not working in
cloud. After I added
<arr name="last-components">
<str>spellcheck</str>
</arr>
to /select requestHandler it worked but the wordbreak spellchecker. I have added
shards.qt=testhanlder to curl request but it did not solve the issue.
Thanks.
Alex.
-----Original Message-----
From: Dyer, James <james.d...@ingramcontent.com>
To: solr-user <solr-user@lucene.apache.org>
Sent: Tue, Mar 19, 2013 10:30 am
Subject: RE: strange behaviour of wordbreak spellchecker in solr cloud
Mark,
I wasn't sure if Alex is actually testing /select, or if the problem is just
coming up in /testhandler. Just wanted to verify that before we get into bug
reports.
DistributedSpellCheckComponentTest does have 1 little Word Break test scenario
in it, so we know WordBreakSolrSpellChecker at least works some of the time in a
Distributed environment :) . Ideally, we should probably use a random test for
stuff like this as adding a bunch of test scenarios would make this
already-slower-than-molasses test even slower. On the other hand, we want to
test as many possibilities as we can. Based on DSCCT and it being so
superficial, I really can't vouch too much for my spell check enhancements
working as well with shards as they do with a single index.
James Dyer
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: Mark Miller [mailto:markrmil...@gmail.com]
Sent: Tuesday, March 19, 2013 11:49 AM
To: solr-user@lucene.apache.org
Subject: Re: strange behaviour of wordbreak spellchecker in solr cloud
My first thought too, but then I saw that he had the spell component in both his
custom testhander and the /select handler, so I'd expect that to work as well.
- Mark
On Mar 19, 2013, at 12:18 PM, "Dyer, James" <james.d...@ingramcontent.com>
wrote:
> Can you try including in your request the "shards.qt" parameter? In your
case, I think you should set it to "testhandler". See
http://wiki.apache.org/solr/SpellCheckComponent?highlight=%28shards\.qt%29#Distributed_Search_Support
for a brief discussion.
>
> James Dyer
> Ingram Content Group
> (615) 213-4311
>
>
> -----Original Message-----
> From: alx...@aim.com [mailto:alx...@aim.com]
> Sent: Monday, March 18, 2013 4:07 PM
> To: solr-user@lucene.apache.org
> Subject: strange behaviour of wordbreak spellchecker in solr cloud
>
> Hello,
>
> I try to use wordbreak spellchecker in solr-4.2 with cloud feature. We have
two server with one shard in each of them.
>
> curl 'server1:8983/solr/test/testhandler?q=paulusoles&indent=true&rows=10'
> curl 'server2:8983/solr/test/testhandler?q=paulusoles&indent=true&rows=10'
>
> does not return any results in spellchecker. However, if I specify
distrib=false only one of these has spellchecker results.
>
> curl
> 'server1:8983/solr/test/testhandler?q=paulusoles&indent=true&rows=10&distrib=false'
>
> no spellcheler results
>
> curl
> 'server2:8983/solr/test/testhandler?q=paulusoles&indent=true&rows=10&distrib=false'
> returns spellcheker results.
>
>
> My testhandler and select handlers are as follows
>
>
> <requestHandler name="/testhandler" class="solr.SearchHandler" >
> <lst name="defaults">
> <str name="defType">edismax</str>
> <str name="echoParams">explicit</str>
> <float name="tie">0.01</float>
> <str name="qf">host^30 content^0.5 title^1.2 </str>
> <str name="pf">site^25 content^10 title^22</str>
> <str name="fl">url,id,title</str>
> <!-- <str name="mm">2<-1 5<-3 6<90%</str> -->
> <str name="mm">3<-1 5<-3 6<90%</str>
> <int name="ps">1</int>
>
> <str name="hl">true</str>
> <str name="hl.fl">content</str>
> <str name="f.content.hl.fragmenter">regex</str>
> <str name="hl.fragsize">165</str>
> <str name="hl.fragmentsBuilder">default</str>
>
>
> <str name="spellcheck.dictionary">direct</str>
> <str name="spellcheck.dictionary">wordbreak</str>
> <str name="spellcheck">on</str>
> <str name="spellcheck.collate">true</str>
> <str name="spellcheck.onlyMorePopular">false</str>
> <str name="spellcheck.count">2</str>
>
> </lst>
>
> <arr name="last-components">
> <str>spellcheck</str>
> </arr>
>
> </requestHandler>
>
>
> <requestHandler name="/select" class="solr.SearchHandler">
> <!-- default values for query parameters can be specified, these
> will be overridden by parameters in the request
> -->
> <lst name="defaults">
> <str name="echoParams">explicit</str>
> <int name="rows">10</int>
> <!-- <str name="df">text</str> -->
> </lst>
> <!-- In addition to defaults, "appends" params can be specified
> to identify values which should be appended to the list of
> multi-val params from the query (or the existing "defaults").
> -->
> <!-- In this example, the param "fq=instock:true" would be appended to
> any query time fq params the user may specify, as a mechanism for
> partitioning the index, independent of any user selected filtering
> that may also be desired (perhaps as a result of faceted searching).
>
> NOTE: there is *absolutely* nothing a client can do to prevent these
> "appends" values from being used, so don't use this mechanism
> unless you are sure you always want it.
> -->
> <!--
> <lst name="appends">
> <str name="fq">inStock:true</str>
> </lst>
> -->
> <!-- "invariants" are a way of letting the Solr maintainer lock down
> the options available to Solr clients. Any params values
> specified here are used regardless of what values may be specified
> in either the query, the "defaults", or the "appends" params.
>
> In this example, the facet.field and facet.query params would
> be fixed, limiting the facets clients can use. Faceting is
> not turned on by default - but if the client does specify
> facet=true in the request, these are the only facets they
> will be able to see counts for; regardless of what other
> facet.field or facet.query params they may specify.
>
> NOTE: there is *absolutely* nothing a client can do to prevent these
> "invariants" values from being used, so don't use this mechanism
> unless you are sure you always want it.
> -->
> <!--
> <lst name="invariants">
> <str name="facet.field">cat</str>
> <str name="facet.field">manu_exact</str>
> <str name="facet.query">price:[* TO 500]</str>
> <str name="facet.query">price:[500 TO *]</str>
> </lst>
> -->
> <!-- If the default list of SearchComponents is not desired, that
> list can either be overridden completely, or components can be
> prepended or appended to the default list. (see below)
> -->
> <!--
> <arr name="components">
> <str>nameOfCustomComponent1</str>
> <str>nameOfCustomComponent2</str>
> </arr>
> -->
> <arr name="last-components">
> <str>spellcheck</str>
> </arr>
> </requestHandler>
>
>
>
> is this a bug or something else has to be done?
>
>
> Thanks.
> Alex.
>
