Re: Understanding query behaviour in LBHttpSolrServer

[search engn dev]

Thanks Shawn,

I am also not sure about query flow , 

>From my understanding solr and solrj works as below, 
1. LBHttpSolrServer keeps pinging above list of servers and maintains list
of live servers. 
2. Every time query arives it picks one server from the list (round-robin
fashion) 
3. Sends query to selected server server. 
4. When query arives at solr node it internally distributes query to
remaining shards , collects,merges,ranks results and sends response back to
the user. 

are these steps correct ?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Understanding-query-behaviour-in-LBHttpSolrServer-tp4147835p4147846.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Understanding query behaviour in LBHttpSolrServer

[Shawn Heisey]

On 7/17/2014 11:15 PM, search engn dev wrote:
> I just want to understand query flow and how load balancing works in case of
> LBHttpSolrServer. We have setup SolrCloud with one collection, and that
> collection has 4 shards and each shard has two nodes i.e one master and one
> replica.

If you're running SolrCloud and building client applications with SolrJ,
just use CloudSolrServer.  You just pass it the same zkHost value that
you give to SolrCloud itself, listing all your zookeeper servers.  The
CloudSolrServer object is a zookeeper client, so my understanding is
that it will dynamically adjust to the current clusterstate -- if
servers go down, get added, or get removed, the client will know as soon
as SolrCloud itself does, without restarting the application or building
a new client object.

CloudSolrServer will automatically load balance requests across the
nodes that comprise the collection that is being queried.  Newer
versions of the client will also route updates directly to the leader of
the correct shard, which reduces load on the servers and speeds up indexing.

Internally, CloudSolrServer uses an instance of LBHttpSolrServer, but
the list of URLs is dynamically managed, your program doesn't need to
worry about it.

http://lucene.apache.org/solr/4_9_0/solr-solrj/org/apache/solr/client/solrj/impl/CloudSolrServer.html

Side questions/comments for experienced committers, I can file some
issues and work on these:

The javadoc for the first CloudSolrServer constructor just mentions
HOST:PORT for the format of zkHost.  Should that be expanded so that
it's apparent that if multiple ZK servers are present, they all need to
be listed?  All of the constructor javadocs could do with a little more
substance.

I think that "throws MalformedURLException" needs to be removed from the
second CloudSolrServer constructor.  When I tried that, eclipse didn't
show any errors.

Although it's not complex code, the various CloudSolrServer constructors
are very similar, the actual work should probably be done by one
constructor that is called by all the others.

Thanks,
Shawn

SolrCloud performance issues regarding hardware configuration

[search engn dev]

I need help regarding hardware configuration of my SolrCloud setup. most of
the queries to our SolrCloud are boolean queries involving 6-7 fields and
spatial and facet queries. currently for facet queries solr throws OOM
errors.

below are my solrcloud stats,
solr version 4.6.1
1 collection : 4 shards : each shard has one master and one replica
total documents : 700 million
total index size is 150 GB, i.e 37.5GB at each shard
total fields in index are 30,  25 fields are stored and indexed, remaining
are only indexed.
hardware of each solr node is quad-core cpu and 16 gb ram
JVM parameters : -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Xms8192M
-Xmx8192M

Now what we planned is instead of 4 shards we will create 8 shards of 32GB
ram and quad-core cpu, then each shard will have ~19GB of index. 

What will be correct approach here, creating 4 shards with 64 gb ram each or
creating 8 shards with 32 gb ram each?




--
View this message in context: 
http://lucene.472066.n3.nabble.com/SolrCloud-performance-issues-regarding-hardware-configuration-tp4147843.html
Sent from the Solr - User mailing list archive at Nabble.com.

Query join on multiple fields

[Haiying Wang]

Can Solr query join on multiple fields? Or, anyway to achieve equivalent 
function with current join syntax?

From the Solr Wiki and examples found on the web, the syntax only supports join 
on one field from a doc to another field of other docs, like below:


{!join from=customer_name to=name}state:TX

We have two type of docs and want logically join on more than one field, and we 
want to retrieve one type of doc with filters on the other type of docs; the 
equivalent SQL query will be like

select * from order where exists (select 1 from customer where customer.name = 
order.customer_name and customer.address=order.ship_from_address and 
customer.state='TX')

Can the Solr query achieve this type of join? Or, any technical reason not 
support join on multiple fields?


Thanks,

Haiying

Re: Memory leak for debugQuery?

[Umesh Prasad]

Histogram by itself isn't sufficient to root cause the JVM heap issue.
We have found JVM heap memory  issues multiple times in our system and each
time it was due to a different reasons.  I would recommend taking  heap
dumps at regular interval (using jmap/visual vm) and analyze those heap
dumps. That will give a definite answer to memory issues.

 I have regularly analyzed heap dump of size > 32 GB with eclipse memory
analyzer. The linux version comes with a command line script
ParseHeapDump.sh inside mat directory.

# Usage: ParseHeapDump.sh  [report]*
#
# The leak report has the id org.eclipse.mat.api:suspects
# The top component report has the id org.eclipse.mat.api:top_components
Increase the memory by setting Xmx and Xms param in MemoryAnalyzer.ini (in
same directory).

The leak suspect report is quite good. For checking detailed allocation
pattern etc , you can copy the index files generated from parsing and open
it in GUI.




On 17 July 2014 05:36, Tomás Fernández Löbbe  wrote:

> Also, is this trunk? Solr 4.x? Single shard, right?
>
>
> On Wed, Jul 16, 2014 at 2:24 PM, Erik Hatcher 
> wrote:
>
> > Tom -
> >
> > You could maybe isolate it a little further by seeing using the “debug"
> > parameter with values of timing|query|results
> >
> > Erik
> >
> > On May 15, 2014, at 5:50 PM, Tom Burton-West  wrote:
> >
> > > Hello all,
> > >
> > > I'm trying to get relevance scoring information for each of 1,000 docs
> > returned for each of 250 queries.If I run the query (appended below)
> > without debugQuery=on, I have no problem with getting all the results
> with
> > under 4GB of memory use.  If I add the parameter &debugQuery=on, memory
> use
> > goes up continuously and after about 20 queries (with 1,000 results
> each),
> > memory use reaches about 29.1 GB and the garbage collector gives up:
> > >
> > > " org.apache.solr.common.SolrException;
> null:java.lang.RuntimeException:
> > java.lang.OutOfMemoryError: GC overhead limit exceeded"
> > >
> > > I've attached a jmap -histo, exgerpt below.
> > >
> > > Is this a known issue with debugQuery?
> > >
> > > Tom
> > > 
> > > query:
> > >
> > >
> >
> q=Abraham+Lincoln&fl=id,score&indent=on&wt=json&start=0&rows=1000&version=2.2&debugQuery=on
> > >
> > > without debugQuery=on:
> > >
> > >
> >
> q=Abraham+Lincoln&fl=id,score&indent=on&wt=json&start=0&rows=1000&version=2.2
> > >
> > > num   #instances#bytes  Class description
> > >
> >
> --
> > > 1:  585,559 10,292,067,456  byte[]
> > > 2:  743,639 18,874,349,592  char[]
> > > 3:  53,821  91,936,328  long[]
> > > 4:  70,430  69,234,400  int[]
> > > 5:  51,348  27,111,744
> >  org.apache.lucene.util.fst.FST$Arc[]
> > > 6:  286,357 20,617,704
> >  org.apache.lucene.util.fst.FST$Arc
> > > 7:  715,364 17,168,736  java.lang.String
> > > 8:  79,561  12,547,792  * ConstMethodKlass
> > > 9:  18,909  11,404,696  short[]
> > > 10: 345,854 11,067,328  java.util.HashMap$Entry
> > > 11: 8,823   10,351,024  * ConstantPoolKlass
> > > 12: 79,561  10,193,328  * MethodKlass
> > > 13: 228,587 9,143,480
> > org.apache.lucene.document.FieldType
> > > 14: 228,584 9,143,360
> org.apache.lucene.document.Field
> > > 15: 368,423 8,842,152   org.apache.lucene.util.BytesRef
> > > 16: 210,342 8,413,680   java.util.TreeMap$Entry
> > > 17: 81,576  8,204,648   java.util.HashMap$Entry[]
> > > 18: 107,921 7,770,312
> > org.apache.lucene.util.fst.FST$Arc
> > > 19: 13,020  6,874,560
> > org.apache.lucene.util.fst.FST$Arc[]
> > >
> > > 
> >
> >
>



-- 
---
Thanks & Regards
Umesh Prasad

Understanding query behaviour in LBHttpSolrServer

[search engn dev]

I just want to understand query flow and how load balancing works in case of
LBHttpSolrServer. We have setup SolrCloud with one collection, and that
collection has 4 shards and each shard has two nodes i.e one master and one
replica. 

I have configured LBHttpSolrServer as below.
SolrServer lbHttpSolrServer = new
LBHttpSolrServer("http://shard1_master:8080/solr/","http://shard2_master:8080/solr/","http://shard3_master:8080/solr/","http://shard4_master:8080/solr/","http://shard1_replica:8080/solr/","http://shard2_replica:8080/solr/","http://shard3_replica:8080/solr/","http://shard4_replica:8080/solr/";,);

>From my understanding solr and solrj works as below,
1. LBHttpSolrServer keeps pinging above list of servers and maintains list
of live servers.
2. Every time query arives it picks one server from the list (round-robin
fashion)
3. Sends query to selected server server.
4. When query arives at solr node it internally distributes query to
remaining shards , collects,merges,ranks results and sends response back to
the user.

Here my confusion is at point number 4, is my understanding correct? if not
please correct. And do i need to pass all 8 nodes to LBHttpSolrServer or
just 4 will be sufficient . 
 



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Understanding-query-behaviour-in-LBHttpSolrServer-tp4147835.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How do I get faceting to work with Solr JOINs

[Umesh Prasad]

Hi Vinay,

You can customize the FacetsComponent. Basically FacetComponent uses
SimpleFacets to compute the facet count. It passes matched docset present
in responsebuilder to  SimpleFacets's constructor.

1.  Build a mapping between parent space and auxiliary document space in
(say an int array) and cache it in your own custom cache in
SolrIndexSearcher.  You will need to rebuild this mapping on every commit
have to define a CacheRegenerator for that.

2.  You can map the matched docset (which is in parent space) to auxiliary
document space.
 The catch is that facets from non matching auxililary docs also would
be counted.

3. You can then pass on this mapped auxiliary document to SimpleFacets for
faceting.

I have doing something similar for our needs .. Basically, we have a parent
document with text attributes and changes very less. And we have child
documents with inventory attributes which changes extremely fast. The
search results requires child documents but faceting has to be done on text
attributes which belong to parents. So we do this mapping by customizing
the FacetComponent.






On 18 July 2014 04:11, Vinay B,  wrote:

> Some Background info :
> In our application, we have a requirement to update large number of records
> often.  I investigated solr child documents but it requires updating both
> the child and the parent document . Therefore, I'm investigating adding
> frequently updated information in an "auxillary document" with a custom
> defined "parent-id" field that can be used to join with the static "parent
> document". - basically rolling my own child document functionality.
>
> This approach has satisfied all my requirements, except one. How can I
> facet upon a field present in the auxillary document?
>
> First, here's a gist dump of my test core index (4 docs + 4 aux docs)
> https://gist.github.com/anonymous/2774b54e667778c71492
>
> Next, here's a simple facet query only on the aux . While this works, it
> only returns auxillary documents
> https://gist.github.com/anonymous/a58b87576b895e467c68
>
> Finally, I tweak the query using a SOLR join (
> https://wiki.apache.org/solr/Join ) to return the main documents (which it
> does), but the faceting returns no results. This is what I'm hoping someone
> on this list can answer .
> Here is the gist of that query
> https://gist.github.com/anonymous/f3a287ab726f35b142cf
>
> Any answers, suggestions ?
>
> Thanks
>



-- 
---
Thanks & Regards
Umesh Prasad

Re: Inconsistant result's on solr cloud 4.8

[Joel Bernstein]

Also, it would be a good idea to verify if the numDocs are in sync between
leaders and replicas. You can view this from the admin screens.

Joel Bernstein
Search Engineer at Heliosearch


On Thu, Jul 17, 2014 at 9:53 PM, Erick Erickson 
wrote:

> Is there any chance that there's a commit happening in between these calls
> causing the number of files to change? Is there any indexing going on at
> all?
>
> Best,
> Erick
>
>
> On Wed, Jul 16, 2014 at 7:26 PM, Cool Techi 
> wrote:
>
> > Hi,
> > We are using solr cloud with solr version 4.8, we have 2 shard/2 replica
> > servers in Solr Cloud. During two consecutive request to the solr cloud,
> > the "total results" number varies.
> > 1) As per my understanding this can happen when the leader and the
> replica
> > have inconsistant number of results.
> > 2) This inconsistant number of docs between leader and replica can happen
> > only when replica is recovering. Should a request be sent to a node which
> > is recovering.
> > Since this is happening on our live setup, we tend to question how much
> > can we rely on solr. What could be causing this and what's the fix.
> > Regards
>

Re: problem with replication/solrcloud - getting 'missing required field' during update intermittently (SOLR-6251)

[nneul]

There are several required fields - timestamp is one of them. All the required 
fields are present in the document, 
including a single element in the 'channel' array.

SOLR-6255 created. I'll try to reproduce with a minimal schema asap.

-- Nathan

On 07/17/2014 09:07 PM, Erick Erickson [via Lucene] wrote:
> Hmmm, is "channel" a multiValued field or not? I'm guessing that "id" is the
> only required field, and if so the error message is misleading at best.
>
> So yeah, it's probably a reasonable thing to open a JIRA. Please include the
> schema file (as small as you can make it) and the JSON as well. Not
> promising
> that the behavior will change, but at least we'll have a record of the
> behavior and
> any subsequent discussion.
>
> Erick
>
>
> On Thu, Jul 17, 2014 at 3:27 PM, nneul <[hidden email] 
> > wrote:
>
>  > Should I go ahead and submit this as a distinct bug issue w/ the query
>  > parsing/json now that it's more clear what the problem is?
>  >
>  >
>  > Jack Krupansky-2 wrote
>  > > At least parts of Solr use semi-custom JSON parsing that allows repeating
>  > > a
>  > > map key, so either this particular feature didn't use that parsing
>  > > technique, or didn't have the logic to kick out the problem, or didn't
>  > > process it properly. So, I think this is SOME kind of issue on the Solr
>  > > side, if only better error reporting at a minimum.
>  >
>  >
>  >
>  >
>  >
>  > --
>  > View this message in context:
>  >
> http://lucene.472066.n3.nabble.com/problem-with-replication-solrcloud-getting-missing-required-field-during-update-intermittently-SOLR--tp4147395p4147781.html
>  > Sent from the Solr - User mailing list archive at Nabble.com.
>  >
>
>
> 
> If you reply to this email, your message will be added to the discussion 
> below:
> http://lucene.472066.n3.nabble.com/problem-with-replication-solrcloud-getting-missing-required-field-during-update-intermittently-SOLR--tp4147395p4147812.html
>
> To unsubscribe from problem with replication/solrcloud - getting 'missing 
> required field' during update intermittently
> (SOLR-6251), click here
> .
> NAML
> 
>

-- 

Nathan Neulinger   nn...@neulinger.org
Neulinger Consulting   (573) 612-1412




--
View this message in context: 
http://lucene.472066.n3.nabble.com/problem-with-replication-solrcloud-getting-missing-required-field-during-update-intermittently-SOLR--tp4147395p4147815.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: problem with replication/solrcloud - getting 'missing required field' during update intermittently (SOLR-6251)

[Erick Erickson]

Hmmm, is "channel" a multiValued field or not? I'm guessing that "id" is the
only required field, and if so the error message is misleading at best.

So yeah, it's probably a reasonable thing to open a JIRA. Please include the
schema file (as small as you can make it) and the JSON as well. Not
promising
that the behavior will change, but at least we'll have a record of the
behavior and
any subsequent discussion.

Erick


On Thu, Jul 17, 2014 at 3:27 PM, nneul  wrote:

> Should I go ahead and submit this as a distinct bug issue w/ the query
> parsing/json now that it's more clear what the problem is?
>
>
> Jack Krupansky-2 wrote
> > At least parts of Solr use semi-custom JSON parsing that allows repeating
> > a
> > map key, so either this particular feature didn't use that parsing
> > technique, or didn't have the logic to kick out the problem, or didn't
> > process it properly. So, I think this is SOME kind of issue on the Solr
> > side, if only better error reporting at a minimum.
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/problem-with-replication-solrcloud-getting-missing-required-field-during-update-intermittently-SOLR--tp4147395p4147781.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Shard Replicas not getting replicated data from leader

[Erick Erickson]

Right, so did that fix your problem?


On Thu, Jul 17, 2014 at 7:10 AM, Marc Campeau  wrote:

> Turns out updateLog had been disabled in solrconfig.xml
>
> Marc
>
>
> 2014-07-16 16:44 GMT-04:00 Marc Campeau :
>
> > Hi,
> >
> > I have setup 4 Solr (4.9.0) Nodes into a single shard for a given
> > collection, meaning I should have 4 replicated nodes. I have 3 Zookeepers
> > in ensemble managing the configs for this collection. I have a load
> > balancer in front of the 4 nodes to split traffic between them.
> >
> > I start this collection with an empty data/index directory.
> >
> > When I send /update requests to the load balancers I see these going to
> > all 4 nodes. Also, I can see that all FOLLOWERs distribute the requests
> > they receive to the LEADER as is expected. But for some reason the
> > FOLLOWERS are not getting /replication requests from the LEADER.  So the
> > collection for the leader contains many thousand of documents and is on
> the
> > 8th generation. I see that it's replicable in the admin interface, yet
> all
> > FOLLOWER nodes have an empty index.
> >
> > Hence, I need your insights please.
> >
> > Thanks,
> >
> > Marc
> >
> > To Note:
> >
> > When I startup my nodes I see the following error in solr.log:
> > 1) When Zookeeper does a clusterstate update, all nodes have their starte
> > "DOWN", why? This I means that in the Solr Admin interface they show up
> has
> > down. This never updates to active.
> >
> > 2) I have a warning :  org.apache.solr.rest.ManagedResource; No
> registered
> > observers for /rest/managed, which I need to update solrconfig.xml to fix
> >
> > 3) I have the following error:
> > ERROR - 2014-07-16 19:49:25.336; org.apache.solr.cloud.SyncStrategy; No
> > UpdateLog found - cannot sync
> >
> > SOLR.LOG
> > -
> > []
> > INFO  - 2014-07-16 19:47:30.870;
> > org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state
> > numShards=null message={
> >   "operation":"state",
> >   "state":"down",
> >   "base_url":"http://192.168.150.90:8983/solr";,
> >   "core":"collection_name",
> >   "roles":null,
> >   "node_name":"192.168.150.90:8983_solr",
> >   "shard":null,
> >   "collection":"collection_name",
> >   "numShards":null,
> >   "core_node_name":null}
> > INFO  - 2014-07-16 19:47:30.871;
> > org.apache.solr.cloud.Overseer$ClusterStateUpdater; node=core_node1 is
> > already registered
> > []
> > WARN  - 2014-07-16 19:47:34.535; org.apache.solr.rest.ManagedResource; No
> > registered observers for /rest/managed
> > []
> > INFO  - 2014-07-16 19:48:25.135;
> > org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (2)
> > INFO  - 2014-07-16 19:48:25.287;
> > org.apache.solr.cloud.DistributedQueue$LatchChildWatcher;
> LatchChildWatcher
> > fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> > INFO  - 2014-07-16 19:48:25.291;
> > org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from
> > ZooKeeper...
> > INFO  - 2014-07-16 19:48:25.293;
> > org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state
> > numShards=null message={
> >   "operation":"state",
> >   "state":"down",
> >   "base_url":"http://192.168.200.90:8983/solr";,
> >   "core":"collection_name",
> >   "roles":null,
> >   "node_name":"192.168.200.90:8983_solr",
> >   "shard":null,
> >   "collection":"collection_name",
> >   "numShards":null,
> >   "core_node_name":null}
> > INFO  - 2014-07-16 19:48:25.293;
> > org.apache.solr.cloud.Overseer$ClusterStateUpdater; node=core_node2 is
> > already registered
> > INFO  - 2014-07-16 19:48:25.293;
> > org.apache.solr.cloud.Overseer$ClusterStateUpdater; shard=shard1 is
> already
> > registered
> > []
> > INFO  - 2014-07-16 19:49:00.188;
> > org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (3)
> > INFO  - 2014-07-16 19:49:00.322;
> > org.apache.solr.cloud.DistributedQueue$LatchChildWatcher;
> LatchChildWatcher
> > fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> > INFO  - 2014-07-16 19:49:00.335;
> > org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from
> > ZooKeeper...
> > INFO  - 2014-07-16 19:49:00.337;
> > org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state
> > numShards=null message={
> >   "operation":"state",
> >   "state":"down",
> >   "base_url":"http://192.168.200.91:8983/solr";,
> >   "core":"collection_name",
> >   "roles":null,
> >   "node_name":"192.168.200.91:8983_solr",
> >   "shard":null,
> >   "collection":"collection_name",
> >   "numShards":null,
> >   "core_node_name":null}
> > INFO  - 2014-07-16 19:49:00.337;
> > org.apache.solr.cloud.Overseer$ClusterStateUpdater; node=core_node3 is
> > already registered
> > INFO  - 2014-07-16 19:49:00.337;
> > org.apache.solr.cloud.Overseer$ClusterStateUpdater; shard=shard1 is
> already
> > registered
> > []
> > INFO  - 2014-07-16 19:49:21.220;
> > org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (4)
> > 

Re: questions on Solr WordBreakSolrSpellChecker and WordDelimiterFilterFactory

[Erick Erickson]

Zeroth, take a look at the admin/analysis page with that input and see if
your
field in analyzing x-box and xbox like you expect.

First, try adding &debug=all to the URL, that'll show you exactly what the
parsed query
was. It may surprise you.

Second, examine what's actually _in_ the index with the
admin/schema-browser or
TermsComponent or Luke to see if _that's_ what you expect.

My bet is it'll be pretty obvious in one of those three steps... but I've
lost bets before.


On Thu, Jul 17, 2014 at 5:42 AM,  wrote:

> Hi Ahmet,
>
> using  or  didn't
> make any difference. Still running into the same issues aforementioned :(
>
> Thanks,
> Jia
>
> On 7/16/2014, "Ahmet Arslan"  wrote:
>
> >Hi Jia,
> >
> >What happens when you use
> >
> > 
> >
> >instead of
> >
> > 
> >
> >Ahmet
> >
> >
> >On Wednesday, July 16, 2014 3:07 AM, "j...@ece.ubc.ca" 
> wrote:
> >
> >
> >
> >Hello everyone :)
> >
> >I have a product called "xbox" indexed, and when the user search for
> >either "x-box" or "x box" i want the "xbox" product to be
> >returned.  I'm new to Solr, and from reading online, I thought I need
> >to use WordDelimiterFilterFactory for "x-box" case, and
> >WordBreakSolrSpellChecker for "x box" case. Is this correct?
> >
> >(1) In my schema file, this is what I changed:
> > >generateNumberParts="1" catenateWords="1" catenateNumbers="1"
> >catenateAll="1" splitOnCaseChange="0" preserveOriginal="1"/>
> >
> >But I don't see the xbox product returned when the search term is
> >"x-box", so I must have missed something
> >
> >(2) I tried to use  WordBreakSolrSpellChecker together with
> >DirectSolrSpellChecker as shown below, but the WordBreakSolrSpellChecker
> >never got used:
> >
> > >class="solr.SpellCheckComponent">
> >wc_textSpell
> >
> >
> >  default
> >  spellCheck
> >  solr.DirectSolrSpellChecker
> >  internal
> >  0.3
> >2
> >1
> >5
> >3
> >0.01
> >0.004
> >
> >
> >wordbreak
> >solr.WordBreakSolrSpellChecker
> >spellCheck
> >true
> >true
> >10
> >  
> >  
> >
> >   >class="org.apache.solr.handler.component.SearchHandler">
> >
> >SpellCheck
> >true
> >   default
> >wordbreak
> > true
> >   false
> >   10
> >   true
> >   false
> >
> >
> >  wc_spellcheck
> >
> >  
> >
> >I tried to build the dictionary this way:
> >
> http://localhost/solr/coreName/select?spellcheck=true&spellcheck.build=true
> ,
> >but the response returned is this:
> >
> >
> >0
> >0
> >
> >true
> >true
> >
> >
> >build
> >
> >
> >
> >What's the correct way to build the dictionary?
> >Even though my requestHandler's name="/spellcheck", i wasn't able to
> >use
> >
> http://localhost/solr/coreName/spellcheck?spellcheck=true&spellcheck.build=true
> >.. is there something wrong with my definition above?
> >
> >(3) I also tried to use WordBreakSolrSpellChecker without the
> >DirectSolrSpellChecker as shown below:
> > >class="solr.SpellCheckComponent">
> >
> >  wc_textSpell
> >
> >default
> >solr.WordBreakSolrSpellChecker
> >spellCheck
> >true
> >true
> >10
> >  
> >   
> >
> >>class="org.apache.solr.handler.component.SearchHandler">
> >
> >SpellCheck
> >true
> >   default
> >
> > true
> >   false
> >   10
> >   true
> >   false
> >
> >
> >  wc_spellcheck
> >
> >  
> >
> >And still unable to see WordBreakSolrSpellChecker being called anywhere.
> >
> >Would someone kindly help me?
> >
> >Many thanks,
> >Jia
> >
>

Re: Inconsistant result's on solr cloud 4.8

[Erick Erickson]

Is there any chance that there's a commit happening in between these calls
causing the number of files to change? Is there any indexing going on at
all?

Best,
Erick


On Wed, Jul 16, 2014 at 7:26 PM, Cool Techi  wrote:

> Hi,
> We are using solr cloud with solr version 4.8, we have 2 shard/2 replica
> servers in Solr Cloud. During two consecutive request to the solr cloud,
> the "total results" number varies.
> 1) As per my understanding this can happen when the leader and the replica
> have inconsistant number of results.
> 2) This inconsistant number of docs between leader and replica can happen
> only when replica is recovering. Should a request be sent to a node which
> is recovering.
> Since this is happening on our live setup, we tend to question how much
> can we rely on solr. What could be causing this and what's the fix.
> Regards

Re: weird drastic query latency during performance testing and DIH import delay after performance testing

[Erick Erickson]

This is very strange. I have no idea why DIH is
taking so long. What happens if you execute the DIH
query in some SQL front end? It's possible that DIH
is just taking forever to execute the SQL query because
of how it's written.

I'm having trouble following the query results, but again
this is super-slow. How many docs do you have in
your index? How much memory have you allocated
for your JVM? Your query is actually pretty simple, so I have
no clue whatsoever why your response times are
so slow. But this is such bad performance that something
major is wrong.

I'd approach it one problem at a time. Understand what's
happening with your queries, _then_ tackle DIH (or the other
way around)...

Best,
Erick


On Wed, Jul 16, 2014 at 2:03 AM, YouPeng Yang 
wrote:

> Hi
>   I build my SolrCloud using Solr 4.6.0 (java version:1.7.0_45). In my
> cloud,I have a collection with 30 shard,and each shard has one replica.
> each core of the shard contains nearly  50 million docs  that is 15GB in
> size,so does the replica.
>   Before applying my cloud in the real world,I do a performance test with
> JMeter 2.11.
>   The scenario of the my test is simple:100 threads sending requests for 20
> seconds ,and these requests are only sent to  a specific core of a specific
> shard.the request is similar to the following :
>  http://IP:port/solr/tv_201407/select?q=*:*&fq=BEGINTIME:["2014-06-01
> 00:00:00"+TO+*]+AND+(CONTACT:${user})+AND (TV_STATE:"00")&shards=tv_201407
> &rows=2000&sort=BEGINTIME+desc.
>
>   I encountered the drastic  query latency during performance testing and
> DIH import delay after performance testing.Please help me. I have tested
>  several times and get the same problem and can not handle it by myself.Any
> suggestion will be apprecaited.
>
>  The following steps describes what I have done .
>
> Step 1: Before the test,the DIH import job is very fast.As the statistics
> [1], the DIH importing takes only 1s for 10 docs.
> [1]---
> Indexing completed. Added/Updated: 10 documents. Deleted 0 documents.
> (Duration: 01s)
> Requests: 1 (1/s), Fetched: 10 (10/s), Skipped: 0, Processed: 10 (10/s)
> Started: less than a minute ago
> ---
>
> Step 2:  Then ,Doing the test under the caches are cleaned. The summery
> statistics data is as [2]. Although I have clean the caches,I never think
> the query latency becomes so drastic that it cannot be acceptable in my
> real application.
>   The red font describes the latency of the query performance test on the
> core tv_201407 of the shard tv_201407 .
>
>   So would you experts can give some hints about the drastic  query latency
> ?
>
> [2]---
> [solr@solr2 test]$ ../bin/jmeter.sh  -n -t solrCoudKala20140401.jmx  -l
> logfile_solrCloud_20.jtl
> Creating summariser 
> Created the tree successfully using solrCoudKala20140401.jmx
> Starting the test @ Wed Jul 16 15:59:28 CST 2014 (1405497568104)
> Waiting for possible shutdown message on port 4445
> aggregate +  1 in   8.1s =0.1/s Avg:  8070 Min:  8070 Max:  8070
> Err:
> 0 (0.00%) Active: 100 Started: 100 Finished: 0
> aggregate +103 in  13.4s =7.7/s Avg:  8027 Min:  4191 Max:  8434
> Err:
> 0 (0.00%) Active: 97 Started: 100 Finished: 3
> aggregate =104 in  13.4s =7.7/s Avg:  8027 Min:  4191 Max:  8434
> Err:
> 0 (0.00%)
> aggregate + 96 in 7s =   14.5/s Avg:  6160 Min:  5295 Max:  6625
> Err:
> 0 (0.00%) Active: 0 Started: 100 Finished: 100
> aggregate =200 in15s =   13.6/s Avg:  7131 Min:  4191 Max:  8434
> Err:
> 0 (0.00%)
> Tidying up ...@ Wed Jul 16 15:59:43 CST 2014 (1405497583461)
> ... end of run
> [solr@solr2 test]$
> ---
> Step 3:To be continued,after the test,I do the DIH importing job again
> using  the same import expresion.However the performance of the DIH becomes
> so unacceptable.
> to import  the 10 docs takes 2 m 15 s [3]!
>   Having noticing that ,solr can fetched the 10 docs fast,the processing is
> slow.
>
> [3]---
> *Indexing completed. Added/Updated: 10 documents. Deleted 0 documents.
> (Duration: 2m 15s)*
> Requests: 1 (0/s), Fetched: 10 (0/s), Skipped: 0, Processed: 10 (0/s)
> Started: about an hour ago
> ---
>
>  By the way. jvm gc goes normal,and there is no long full gc during the
> test. the load of my system(rhel 6.5) are also normal.
>
> Regards
>

Re: Contiguous Phrase Highlighting Example

[Koji Sekiguchi]

Hi Teague,

If you want phrase-unit tagging for highlighter, you need to use
FastVectorHighlighter instead of the ordinary Highlighter.

To turn on FVH, set hl.useFastVectorHighlighter=on when querying.
In addition, when indexing, you need to set termVectors=on, termPositions=on
and termOffsets=on on content field in your schema.xml.

http://wiki.apache.org/solr/HighlightingParameters#hl.useFastVectorHighlighter

Koji
--
http://soleami.com/blog/comparing-document-classification-functions-of-lucene-and-mahout.html

(2014/07/18 3:19), Teague James wrote:

Hi everyone!

Does anyone have any good examples of generating a contiguous highlight for
a phrase? Here's what I have done:

curl http://localhost/solr/collection1/update?commit=true -H "Content-Type:
text/xml" --data-binary '100blah blah blah knowledge of science blah blah
blah'

Then, using a browser:

http://localhost/solr/collection1/select?q="knowledge+of+science"&fq=id:100

What I get back in highlighting is:
blah blah blah knowledge of science blah blah
blah

What I want to get back is:
blah blah blah knowledge of science blah blah blah

I have the following highlighting configurations in my requestHandler in
addition to hl, hl.fl, etc.:
false
true
true
None of the last two seemed to have any impact on the output. I've tried
every permutation of those three, but the output is the same. Any
suggestions or examples of getting highlights to come back this way? I'd
appreciate any advice on this! Thanks!

-Teague

Re: Solr 4.7.2 auto suggestion

[Alexandre Rafalovitch]

On Thu, Jul 17, 2014 at 11:01 PM, benjelloun  wrote:
> i'm trying to do auto suggestion but thats not working.

Not working specifically how? What do you put in, what did you get
back, what did you expect?

Regards,
   Alex.

Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853

Re: Boost using date and a field value

[Hakim Benoudjit]

The real issue is that I'm having issues to balance my two boosted fields,
although I've added a coefficient to them.
What I've done is that I have a boost query with the two fields:
field1:value1 AND date:[ TO ], to have all today's docs satisfying the
condition field1 = value1 first of the list.
But, it seems to me that (except those boosted documents) all other scores
have similar scores. Is there a way to have all other documents have
different scores? (maybe I should use boost fields for this)?

My other question, is in case that I want to penalize docs having a certain
field value in favor of other docs having this value. how should I do to
achieve this? Do I need to add a field affected at index-time? or a dynamic
field pre-computed depending on this field value?


2014-07-17 10:23 GMT+01:00 Alexandre Rafalovitch :

> Well, to start, it would help to know what "can't combine the  boosts"
> mean in your case. Are you saying you can only apply one at a time?
> Are you saying that you see the effect of both but having troubles
> balancing them? Something else?
>
> The more complex use cases become, the more precise information people
> need to be able to help you. And Relevancy is a complex case even in
> the basic scenarios.
>
> Regards,
>Alex.
> Personal: http://www.outerthoughts.com/ and @arafalov
> Solr resources: http://www.solr-start.com/ and @solrstart
> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
>
>
> On Thu, Jul 17, 2014 at 3:57 PM, Hakim Benoudjit 
> wrote:
> > Any idea please?
> >
> >
> > 2014-07-15 15:00 GMT+01:00 Hakim Benoudjit :
> >
> >> Hi,
> >> I want to boost recent (*today's*) documents having a certain *field
> >> value*. The two fields to be bosted are respectively: '*date*' &
> '*site*'.
> >> But I dont want to penalize *recent *documents not satisfying the field
> >> value ('*site*'), in favor of *older* documents satisfying this field
> >> value ('*site*').
> >>
> >> - I've boosted documents having this field value ('*site*'), using
> *dismax
> >> boost query*.
> >> - And I've found on solr doc how to boost *recent *docs:
> >>
> https://wiki.apache.org/solr/SolrRelevancyFAQ#How_can_I_boost_the_score_of_newer_documents
> >>
> >> - But I cant combine these two boosts.
> >>
> >>
> >> --
> >> Hakim Benoudjit.
> >>
> >
> >
> >
> > --
> > Hakim Benoudjit.
>



-- 
Hakim Benoudjit.

How do I get faceting to work with Solr JOINs

[Vinay B,]

Some Background info :
In our application, we have a requirement to update large number of records
often.  I investigated solr child documents but it requires updating both
the child and the parent document . Therefore, I'm investigating adding
frequently updated information in an "auxillary document" with a custom
defined "parent-id" field that can be used to join with the static "parent
document". - basically rolling my own child document functionality.

This approach has satisfied all my requirements, except one. How can I
facet upon a field present in the auxillary document?

First, here's a gist dump of my test core index (4 docs + 4 aux docs)
https://gist.github.com/anonymous/2774b54e667778c71492

Next, here's a simple facet query only on the aux . While this works, it
only returns auxillary documents
https://gist.github.com/anonymous/a58b87576b895e467c68

Finally, I tweak the query using a SOLR join (
https://wiki.apache.org/solr/Join ) to return the main documents (which it
does), but the faceting returns no results. This is what I'm hoping someone
on this list can answer .
Here is the gist of that query
https://gist.github.com/anonymous/f3a287ab726f35b142cf

Any answers, suggestions ?

Thanks

Re: problem with replication/solrcloud - getting 'missing required field' during update intermittently (SOLR-6251)

[nneul]

Should I go ahead and submit this as a distinct bug issue w/ the query
parsing/json now that it's more clear what the problem is?


Jack Krupansky-2 wrote
> At least parts of Solr use semi-custom JSON parsing that allows repeating
> a 
> map key, so either this particular feature didn't use that parsing 
> technique, or didn't have the logic to kick out the problem, or didn't 
> process it properly. So, I think this is SOME kind of issue on the Solr 
> side, if only better error reporting at a minimum.





--
View this message in context: 
http://lucene.472066.n3.nabble.com/problem-with-replication-solrcloud-getting-missing-required-field-during-update-intermittently-SOLR--tp4147395p4147781.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr/Tomcat (embedded) Replication

[Shawn Heisey]

On 7/17/2014 12:56 PM, SWP wrote:
> Trying to get replication working with embedded Solr (4.4.0) running in a
> Tomcat6 container with the below master & slave configuration ->

If you are using EmbeddedSolrServer, regardless of whether that code is
running in Tomcat or not, there is no HTTP access to the Solr server.

You must have Tomcat extract and run the solr war for normal access via
HTTP.  Or even better, use the Jetty that's included in the example
instead of Tomcat.

Thanks,
Shawn

Solr/Tomcat (embedded) Replication

[SWP]

Hello,

Trying to get replication working with embedded Solr (4.4.0) running in a
Tomcat6 container with the below master & slave configuration ->

Master (solrconfig.xml):


   
true
startup
commit

 

Slave (solrconfig.xml):


  
${enable.slave:true}
http://$search_master:8181/$core_name
00:00:60
  


We've tried various solr auto-commit & soft auto-commit configuration
combinations on the master in addition to application layer commits to force
replication but none of our changes have been successful.  Current
autoCommit config (solrconfig.xml):


  ${solr.autoCommit.maxDocs:1}
  ${solr.autoCommit.maxTime:1000}
  true


We've read that replication with embedded Solr is not possible without HTTP
but by using Tomcat as a Solr container, we're assuming this shouldn't be a
problem.  Is this assumption accurate?  Is there a problem with our
configuration?

Thanks.

Re: problem with replication/solrcloud - getting 'missing required field' during update intermittently (SOLR-6251)

[nneul]

Actually, turns out the 'intermittent' was our misinterpretation - we never
noticed that we were only occcasionally adding to two channels... Once I
figured that out, symptom is 100% repeatable and consistent.


Yonik Seeley-5 wrote
> On Wed, Jul 16, 2014 at 10:20 PM, Nathan Neulinger <

> nneul@

> > wrote:
>> [{"id":"4b2c4d09-31e2-4fe2-b767-3868efbdcda1","channel": {"add":
>> "preet"},"channel": {"add": "adam"}}]
>>
>> Look at the JSON... It's trying to add two "channel" array elements...
>> Should have been:
> [...]
>> From what I'm reading on JSON - this isn't valid syntax at all.
> 
> It is valid... repeated keys are actually allowed by the JSON spec.
> How we want to handle this particular situation at the Solr level is
> another question.  It's also not clear how this causes intermittent
> failures.
> 
> 
> -Yonik
> http://heliosearch.org - native code faceting, facet functions,
> sub-facets, off-heap data





--
View this message in context: 
http://lucene.472066.n3.nabble.com/problem-with-replication-solrcloud-getting-missing-required-field-during-update-intermittently-SOLR--tp4147395p4147724.html
Sent from the Solr - User mailing list archive at Nabble.com.

Contiguous Phrase Highlighting Example

[Teague James]

Hi everyone!

Does anyone have any good examples of generating a contiguous highlight for
a phrase? Here's what I have done:

curl http://localhost/solr/collection1/update?commit=true -H "Content-Type:
text/xml" --data-binary '100blah blah blah knowledge of science blah blah
blah'

Then, using a browser:

http://localhost/solr/collection1/select?q="knowledge+of+science"&fq=id:100

What I get back in highlighting is:
blah blah blah knowledge of science blah blah
blah

What I want to get back is:
blah blah blah knowledge of science blah blah blah

I have the following highlighting configurations in my requestHandler in
addition to hl, hl.fl, etc.:
false
true
true
None of the last two seemed to have any impact on the output. I've tried
every permutation of those three, but the output is the same. Any
suggestions or examples of getting highlights to come back this way? I'd
appreciate any advice on this! Thanks!

-Teague

Fwd: Script Transformer Help

[pavan patharde]

Hi Alex,

I tried putting the transformer on the inner entity. Now i can not access
the parent entity fields.

Can you tell me how to read the parent fields ?

Regards,
Pavan .P.Patharde



On Thu, Jul 17, 2014 at 10:08 AM, Alexandre Rafalovitch 
wrote:

> Have you tried putting the transformer on the inner entity definition?
> It's like a nested loop and you just put it in the outer loop.
>
> Regards,
>Alex.
> Personal: http://www.outerthoughts.com/ and @arafalov
> Solr resources: http://www.solr-start.com/ and @solrstart
> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
>
>
> On Thu, Jul 17, 2014 at 11:29 AM, pavan patharde
>  wrote:
> > Hi All,
> >
> > I have data-config.xml as below:Script Transformer is omitted.
> > 
> >  > url="jdbc:hsqldb:/temp/example/ex"­­ user="sa" />
> >
> >  >row.put('message', 'Hello World!');return
> > row;}]]>
> >
> >
> > 
> >  > transformer="script:f1">
> > 
> > 
> >
> > 
> > 
> > 
> > 
> >
> > 
> > 
> > 
> >
> > I am able to access NAME and BSIN in the function f1. I am not able to
> > access the brand and cname. Is there any way i can access brand and cname
> > from child entity in script transformer ?
> > Thanks in advance.
> >
> > Regards,
> > Pavan .P.Patharde
>

Re: solr wiki: 'Support for Solr' page edit policy

[jmlucjav]

appreciated Stefan. Done updating.


On Thu, Jul 17, 2014 at 5:36 PM, Stefan Matheis 
wrote:

> Xavi
>
> It’s the former :) I’ve adding you to the contributors group
>
> -Stefan
>
>
> On Thursday, July 17, 2014 at 5:19 PM, jmlucjav wrote:
>
> > Hi guys,
> >
> > I don't remember anymore what is the policy to have someone added to this
> > page:
> >
> > - ask for edit rights and add your own line where needed
> > - send someone your line and they'll add it for you.
> >
> > If the former, could I get edit permissions for the wiki? My login is
> > jmlucjav. If the later, who could I send it to?
> >
> > thanks!
> > xavi
> >
> >
>
>
>

Re: Script Transformer Help

[pavan patharde]

Hi Alex,

I tried putting the transformer on the inner entity. Now i can not access
the parent entity fields.

Can you tell me how to read the parent fields ?

Regards,
Pavan .P.Patharde



On Thu, Jul 17, 2014 at 10:08 AM, Alexandre Rafalovitch 
wrote:

> Have you tried putting the transformer on the inner entity definition?
> It's like a nested loop and you just put it in the outer loop.
>
> Regards,
>Alex.
> Personal: http://www.outerthoughts.com/ and @arafalov
> Solr resources: http://www.solr-start.com/ and @solrstart
> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
>
>
> On Thu, Jul 17, 2014 at 11:29 AM, pavan patharde
>  wrote:
> > Hi All,
> >
> > I have data-config.xml as below:Script Transformer is omitted.
> > 
> >  > url="jdbc:hsqldb:/temp/example/ex"­­ user="sa" />
> >
> >  >row.put('message', 'Hello World!');return
> > row;}]]>
> >
> >
> > 
> >  > transformer="script:f1">
> > 
> > 
> >
> > 
> > 
> > 
> > 
> >
> > 
> > 
> > 
> >
> > I am able to access NAME and BSIN in the function f1. I am not able to
> > access the brand and cname. Is there any way i can access brand and cname
> > from child entity in script transformer ?
> > Thanks in advance.
> >
> > Regards,
> > Pavan .P.Patharde
>

Re: problem with replication/solrcloud - getting 'missing required field' during update intermittently (SOLR-6251)

[Yonik Seeley]

On Wed, Jul 16, 2014 at 10:20 PM, Nathan Neulinger  wrote:
> [{"id":"4b2c4d09-31e2-4fe2-b767-3868efbdcda1","channel": {"add":
> "preet"},"channel": {"add": "adam"}}]
>
> Look at the JSON... It's trying to add two "channel" array elements...
> Should have been:
[...]
> From what I'm reading on JSON - this isn't valid syntax at all.

It is valid... repeated keys are actually allowed by the JSON spec.
How we want to handle this particular situation at the Solr level is
another question.  It's also not clear how this causes intermittent
failures.


-Yonik
http://heliosearch.org - native code faceting, facet functions,
sub-facets, off-heap data

Solr 4.7.2 auto suggestion

[benjelloun]

Hello,

i'm trying to do auto suggestion but thats not working.
here is my configuration:

schema:



 
 











   


















SolrConfig:



   explicit
   10
   
   edismax
   
   AllChamp^2.0 AllChamp_ar^2.0 AllChamp_en^2.0 AllChamp_fr^5.0
   
   true
   AND
mySuggester
on
mySuggester
DocumentDictionaryFactory
FuzzyLookupFactory
spell
suggestion
textSpell

  
  true
  true
  6


  suggest  
  





  mySuggester
  FuzzyLookupFactory 
  DocumentDictionaryFactory 
  spell
  IdDocument
  suggestion
  text_general

  

Thanks best regards :)






--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-4-7-2-auto-suggestion-tp4147677.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Indexing database in Solr using Data Import Handler

[Sam Barber]

Hi,



You have the wrong varname in your sub query.



select favouritedby from filefav where id=
'${filemetadata.id}'



should be

select favouritedby from filefav where id=
'${restaurant.id}'

Re: solr boosting any perticular URL

[Michael Della Bitta]

Rahul,

Check out the relevancy FAQ. You probably want to boost that field value at
index time, or use the query elevation component.

http://wiki.apache.org/solr/SolrRelevancyFAQ

Michael Della Bitta

Applications Developer

o: +1 646 532 3062

appinions inc.

“The Science of Influence Marketing”

18 East 41st Street

New York, NY 10017

t: @appinions  | g+:
plus.google.com/appinions

w: appinions.com 


On Thu, Jul 17, 2014 at 10:28 AM, rahulmodi  wrote:

> Hi There,
>
> I am new to Solr. My client is asking me to boost a particular URL so that
> it should appear on the top of the results.
> I have already searched on various websites but i did not found boosting
> for
> particular URL.
>
> Please tell me whether this feature is available or not, if available then
> how to achieve it.
>
> Thanks
> Rahul Modi
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/solr-boosting-any-perticular-URL-tp4147657.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Strange Scoring Results

[Michael Carlson]

Okay, not sure if the following will help with troubleshooting, but here are a 
couple of links that show visual representations of how the scores for these 
results are calculated. For these queries I got rid of my boosts to make the 
results easier to read.

Here are the top 10 scoring results:
http://explain.solr.pl/explains/6mfdyixa

Here is the page which I think should score higher than all of them:
http://explain.solr.pl/explains/6xo77shs

Why does the second link show 0 for all of the matches despite the existence of 
matches?



On Jul 16, 2014, at 11:36 AM, Michael Carlson  
wrote:

> Hey All - 
> 
> I’m a Solr newbie in need of some help.
> 
> I’m using Apache Nutch to crawl a site and populate a Solr core, which we 
> then use to query search results. I’ve got it all up and running, but the 
> Solr scoring results I get don’t seem to make any sense. Let’s take the 
> following query as an example:
> 
> content:devlearn 2014 registration information
> 
> I have a page with a title of "DevLearn 2014 Conference & Expo - Registration 
> Information” and a url of 
> "www.mydomain.com/DevLearn/content/3426/devlearn-2014-conference--expo--registration-information/“
>  which has multiple instances of all terms in the content field. I would 
> expect this document to be returned at the top of the list, since in addition 
> to being in the content field, all terms are in both the title and the url, 
> which I’m boosting for. Instead, it returns as number 3320 in the results 
> with a score of 0. Meanwhile, 3319 other pages return with higher scores, and 
> all of these have fewer instances of the terms in the content field, and one 
> or fewer of the terms in the title or url.
> 
> Below is the select requestHandler section from my solrconfig.xml which shows 
> the query select defaults. Let me know if I should include more of this file 
> or any other information:
> 
> 
> 
> 
>   explicit
>   10
>   text
>  
>   on
>   content
>   html
>   
>   
>   1
>   200
>   content
>   750
> 
>   edismax
>   
>  content^0.5 url^10.0 title^10.0
>   
>   content
>   100%
>   *:*
>   10
>   *,score
>   
>   content^0.5 url^10.0 title^10.0
>   
>   100
> 
> 
> 
> 
> 
> 
> 
> 

Re: solr wiki: 'Support for Solr' page edit policy

[Stefan Matheis]

Xavi  

It’s the former :) I’ve adding you to the contributors group

-Stefan  


On Thursday, July 17, 2014 at 5:19 PM, jmlucjav wrote:

> Hi guys,
>  
> I don't remember anymore what is the policy to have someone added to this
> page:
>  
> - ask for edit rights and add your own line where needed
> - send someone your line and they'll add it for you.
>  
> If the former, could I get edit permissions for the wiki? My login is
> jmlucjav. If the later, who could I send it to?
>  
> thanks!
> xavi
>  
>  

Re: problem with replication/solrcloud - getting 'missing required field' during update intermittently (SOLR-6251)

[Jack Krupansky]

At least parts of Solr use semi-custom JSON parsing that allows repeating a 
map key, so either this particular feature didn't use that parsing 
technique, or didn't have the logic to kick out the problem, or didn't 
process it properly. So, I think this is SOME kind of issue on the Solr 
side, if only better error reporting at a minimum.


-- Jack Krupansky

-Original Message- 
From: Shalin Shekhar Mangar

Sent: Thursday, July 17, 2014 12:40 AM
To: solr-user@lucene.apache.org
Subject: Re: problem with replication/solrcloud - getting 'missing required 
field' during update intermittently (SOLR-6251)


Phew, thanks for tracking it down.


On Thu, Jul 17, 2014 at 7:50 AM, Nathan Neulinger 
wrote:


FYI. We finally tracked down the problem at least 99.9% sure at this
point, and it was staring me in the face the whole time - just never
noticed:

[{"id":"4b2c4d09-31e2-4fe2-b767-3868efbdcda1","channel": {"add":
"preet"},"channel": {"add": "adam"}}]

Look at the JSON... It's trying to add two "channel" array elements...
Should have been:

[{"id":"4b2c4d09-31e2-4fe2-b767-3868efbdcda1","channel": {"add":
"preet"}},
 {"id":"4b2c4d09-31e2-4fe2-b767-3868efbdcda1","channel": {"add": "adam"}}]

I half wonder how it chose to interpret that particular chunk of json, but
either way, I think the origin of our issue is resolved.


From what I'm reading on JSON - this isn't valid syntax at all. I'm
guessing that SOLR doesn't actually validate the JSON, and it's parser is
just creating something weird in that situation like a new request for a
whole new document.

-- Nathan



On 07/15/2014 07:19 PM, Nathan Neulinger wrote:


Issue was closed in Jira requesting it be discussed here first. Looking
for any diagnostic assistance on this issue with
4.8.0 since it is intermittent and occurs without warning.

Setup is two nodes, with external zk ensemble. Nodes are accessed
round-robin on EC2 behind an ELB.

Schema has:


...

...


Most of the updates are working without issue, but randomly we'll get the
above failure, even though searches before and
after the update clearly indicate that the document had the timestamp
field in it. The error occurs when the second node
does it's distrib operation against the first node.

Diagnostic details are all in the jira issue. Can provide more as needed,
but would appreciate any suggestions on what
to try or to help diagnose this other than just trying to throw thousands
of requests at it in round-robin between the
two instances to see if it's possible to reproduce the issue.

-- Nathan


Nathan Neulinger   nn...@neulinger.org
Neulinger Consulting   (573) 612-1412



--

Nathan Neulinger   nn...@neulinger.org
Neulinger Consulting   (573) 612-1412





--
Regards,
Shalin Shekhar Mangar. 

solr wiki: 'Support for Solr' page edit policy

[jmlucjav]

Hi guys,

I don't remember anymore what is the policy to have someone added to this
page:

- ask for edit rights and add your own line where needed
- send someone your line and they'll add it for you.

If the former, could I get edit permissions for the wiki? My login is
jmlucjav. If the later, who could I send it to?

thanks!
xavi

solr boosting any perticular URL

[rahulmodi]

Hi There,

I am new to Solr. My client is asking me to boost a particular URL so that
it should appear on the top of the results.
I have already searched on various websites but i did not found boosting for
particular URL.

Please tell me whether this feature is available or not, if available then
how to achieve it.

Thanks
Rahul Modi



--
View this message in context: 
http://lucene.472066.n3.nabble.com/solr-boosting-any-perticular-URL-tp4147657.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Shard Replicas not getting replicated data from leader

[Marc Campeau]

Turns out updateLog had been disabled in solrconfig.xml

Marc


2014-07-16 16:44 GMT-04:00 Marc Campeau :

> Hi,
>
> I have setup 4 Solr (4.9.0) Nodes into a single shard for a given
> collection, meaning I should have 4 replicated nodes. I have 3 Zookeepers
> in ensemble managing the configs for this collection. I have a load
> balancer in front of the 4 nodes to split traffic between them.
>
> I start this collection with an empty data/index directory.
>
> When I send /update requests to the load balancers I see these going to
> all 4 nodes. Also, I can see that all FOLLOWERs distribute the requests
> they receive to the LEADER as is expected. But for some reason the
> FOLLOWERS are not getting /replication requests from the LEADER.  So the
> collection for the leader contains many thousand of documents and is on the
> 8th generation. I see that it's replicable in the admin interface, yet all
> FOLLOWER nodes have an empty index.
>
> Hence, I need your insights please.
>
> Thanks,
>
> Marc
>
> To Note:
>
> When I startup my nodes I see the following error in solr.log:
> 1) When Zookeeper does a clusterstate update, all nodes have their starte
> "DOWN", why? This I means that in the Solr Admin interface they show up has
> down. This never updates to active.
>
> 2) I have a warning :  org.apache.solr.rest.ManagedResource; No registered
> observers for /rest/managed, which I need to update solrconfig.xml to fix
>
> 3) I have the following error:
> ERROR - 2014-07-16 19:49:25.336; org.apache.solr.cloud.SyncStrategy; No
> UpdateLog found - cannot sync
>
> SOLR.LOG
> -
> []
> INFO  - 2014-07-16 19:47:30.870;
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state
> numShards=null message={
>   "operation":"state",
>   "state":"down",
>   "base_url":"http://192.168.150.90:8983/solr";,
>   "core":"collection_name",
>   "roles":null,
>   "node_name":"192.168.150.90:8983_solr",
>   "shard":null,
>   "collection":"collection_name",
>   "numShards":null,
>   "core_node_name":null}
> INFO  - 2014-07-16 19:47:30.871;
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; node=core_node1 is
> already registered
> []
> WARN  - 2014-07-16 19:47:34.535; org.apache.solr.rest.ManagedResource; No
> registered observers for /rest/managed
> []
> INFO  - 2014-07-16 19:48:25.135;
> org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (2)
> INFO  - 2014-07-16 19:48:25.287;
> org.apache.solr.cloud.DistributedQueue$LatchChildWatcher; LatchChildWatcher
> fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> INFO  - 2014-07-16 19:48:25.291;
> org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from
> ZooKeeper...
> INFO  - 2014-07-16 19:48:25.293;
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state
> numShards=null message={
>   "operation":"state",
>   "state":"down",
>   "base_url":"http://192.168.200.90:8983/solr";,
>   "core":"collection_name",
>   "roles":null,
>   "node_name":"192.168.200.90:8983_solr",
>   "shard":null,
>   "collection":"collection_name",
>   "numShards":null,
>   "core_node_name":null}
> INFO  - 2014-07-16 19:48:25.293;
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; node=core_node2 is
> already registered
> INFO  - 2014-07-16 19:48:25.293;
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; shard=shard1 is already
> registered
> []
> INFO  - 2014-07-16 19:49:00.188;
> org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (3)
> INFO  - 2014-07-16 19:49:00.322;
> org.apache.solr.cloud.DistributedQueue$LatchChildWatcher; LatchChildWatcher
> fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> INFO  - 2014-07-16 19:49:00.335;
> org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from
> ZooKeeper...
> INFO  - 2014-07-16 19:49:00.337;
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state
> numShards=null message={
>   "operation":"state",
>   "state":"down",
>   "base_url":"http://192.168.200.91:8983/solr";,
>   "core":"collection_name",
>   "roles":null,
>   "node_name":"192.168.200.91:8983_solr",
>   "shard":null,
>   "collection":"collection_name",
>   "numShards":null,
>   "core_node_name":null}
> INFO  - 2014-07-16 19:49:00.337;
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; node=core_node3 is
> already registered
> INFO  - 2014-07-16 19:49:00.337;
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; shard=shard1 is already
> registered
> []
> INFO  - 2014-07-16 19:49:21.220;
> org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (4)
> INFO  - 2014-07-16 19:49:21.350;
> org.apache.solr.cloud.DistributedQueue$LatchChildWatcher; LatchChildWatcher
> fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> INFO  - 2014-07-16 19:49:21.357;
> org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from
> ZooKeeper...
> INFO  - 2014-07-16 19:49:21.359;
> org.apache.so

RE: SolrCloud Issues

[Cool Techi]

1) The Zookeepers are on the same node as Solr, should we move them out? What 
would be the basic config's of machine just running Zookeepers?
2) The servers are pretty big, 
 2 X Quad Core processors 64 bit 96Gb of RAM 500GB SSD 
drive on which solr resides 

The index side between the two shards is 250GB,  how can we detect performance 
problems if any. Also, what keep the node in recovering mode for most of the 
time.  We index about 2K documents/ minute.
What would be the ideal configuration for such load and increasing index size.
Regards,Ayush



> From: dan.mur...@buy4now.com
> To: solr-user@lucene.apache.org
> Subject: RE: SolrCloud Issues
> Date: Thu, 17 Jul 2014 13:27:17 +
> 
> Have you deployed ZooKeeper on servers other than the Solr nodes?
> If you have them on the Solr nodes, then you may be getting elections when 
> under pressure.
> 
> -Original Message-
> From: Shawn Heisey [mailto:s...@elyograg.org] 
> Sent: 17 July 2014 14:25
> To: solr-user@lucene.apache.org
> Subject: Re: SolrCloud Issues
> 
> On 7/17/2014 6:57 AM, Cool Techi wrote:
> > We have moved to SolrCloud (version 4.8) and are facing several issues 
> > in this setup compared to the master/slave setup we have had for a 
> > while now,
> > 1) We have a 2 shard set-up with one replica each, we notice that most of 
> > the time the replica's are in recovering status. What could be causing this?
> > 2) In-consistant results count, especially when one of the nodes is 
> > recovering, I have asked another question earlier on this. To our 
> > understanding the recovering node, doesn't returns any results, what else 
> > can cause this?
> > 3) Solr node goes down very frequently, there is no OOM or otherwise in the 
> > logs, but a node keeps going down. Also, at times we have noticed that 
> > tomcat stops responding. Since there are so many parts to solr cloud, 
> > checking what's causing the issue is difficult in a quick time, so if 
> > anyone else has faced this, it would be very helpful.
> 
> I don't have anything specific for you, but if you are having any kind of 
> performance issues at all, it can lead to bizarre SolrCloud behavior.
> 
> The basic zookeeper client timeout defaults to 15 seconds.  This is a very 
> long timeout, but if anything is happening that makes any part of SolrCloud 
> wait longer than 15 seconds, SolrCloud will think there's a problem that 
> needs recovery.
> 
> Here's a summary of common performance problems and some possible solutions:
> 
> http://wiki.apache.org/solr/SolrPerformanceProblems
> 
> Thanks,
> Shawn
> 
  

RE: SolrCloud Issues

[Dan Murphy]

Have you deployed ZooKeeper on servers other than the Solr nodes?
If you have them on the Solr nodes, then you may be getting elections when 
under pressure.

-Original Message-
From: Shawn Heisey [mailto:s...@elyograg.org] 
Sent: 17 July 2014 14:25
To: solr-user@lucene.apache.org
Subject: Re: SolrCloud Issues

On 7/17/2014 6:57 AM, Cool Techi wrote:
> We have moved to SolrCloud (version 4.8) and are facing several issues 
> in this setup compared to the master/slave setup we have had for a 
> while now,
> 1) We have a 2 shard set-up with one replica each, we notice that most of the 
> time the replica's are in recovering status. What could be causing this?
> 2) In-consistant results count, especially when one of the nodes is 
> recovering, I have asked another question earlier on this. To our 
> understanding the recovering node, doesn't returns any results, what else can 
> cause this?
> 3) Solr node goes down very frequently, there is no OOM or otherwise in the 
> logs, but a node keeps going down. Also, at times we have noticed that tomcat 
> stops responding. Since there are so many parts to solr cloud, checking 
> what's causing the issue is difficult in a quick time, so if anyone else has 
> faced this, it would be very helpful.

I don't have anything specific for you, but if you are having any kind of 
performance issues at all, it can lead to bizarre SolrCloud behavior.

The basic zookeeper client timeout defaults to 15 seconds.  This is a very long 
timeout, but if anything is happening that makes any part of SolrCloud wait 
longer than 15 seconds, SolrCloud will think there's a problem that needs 
recovery.

Here's a summary of common performance problems and some possible solutions:

http://wiki.apache.org/solr/SolrPerformanceProblems

Thanks,
Shawn

Re: SolrCloud Issues

[Shawn Heisey]

On 7/17/2014 6:57 AM, Cool Techi wrote:
> We have moved to SolrCloud (version 4.8) and are facing several issues in 
> this setup compared to the master/slave setup we have had for a while now,
> 1) We have a 2 shard set-up with one replica each, we notice that most of the 
> time the replica's are in recovering status. What could be causing this?
> 2) In-consistant results count, especially when one of the nodes is 
> recovering, I have asked another question earlier on this. To our 
> understanding the recovering node, doesn't returns any results, what else can 
> cause this?
> 3) Solr node goes down very frequently, there is no OOM or otherwise in the 
> logs, but a node keeps going down. Also, at times we have noticed that tomcat 
> stops responding. Since there are so many parts to solr cloud, checking 
> what's causing the issue is difficult in a quick time, so if anyone else has 
> faced this, it would be very helpful.

I don't have anything specific for you, but if you are having any kind
of performance issues at all, it can lead to bizarre SolrCloud behavior.

The basic zookeeper client timeout defaults to 15 seconds.  This is a
very long timeout, but if anything is happening that makes any part of
SolrCloud wait longer than 15 seconds, SolrCloud will think there's a
problem that needs recovery.

Here's a summary of common performance problems and some possible solutions:

http://wiki.apache.org/solr/SolrPerformanceProblems

Thanks,
Shawn

Re: Slow inserts when using Solr Cloud

[ian]

Hi Tim

Thanks for the info about the bug.  I've just looked at the CPU usage for
the leader using JConsole, while my bulk load process was running, inserting
documents into my Solr cloud.  Is that what you meant by profiling and
looking for hotspots?   I find the CPU usage goes up quite a lot when the
replica is enabled, compared to when it is disabled:  

 

In the above chart, the dip in CPU usage in the middle was while the replica
(which lives on a different VM) was disabled.

Thanks
Ian


Timothy Potter wrote
> Hi Ian,
> 
> What's the CPU doing on the leader? Have you tried attaching a
> profiler to the leader while running and then seeing if there are any
> hotspots showing. Not sure if this is related but we recently fixed an
> issue in the area of leader forwarding to replica that used too many
> CPU cycles inefficiently - see SOLR-6136.
> 
> Tim





--
View this message in context: 
http://lucene.472066.n3.nabble.com/Slow-inserts-when-using-Solr-Cloud-tp4146087p4147645.html
Sent from the Solr - User mailing list archive at Nabble.com.

SolrCloud Issues

[Cool Techi]

Hi,
We have moved to SolrCloud (version 4.8) and are facing several issues in this 
setup compared to the master/slave setup we have had for a while now,
1) We have a 2 shard set-up with one replica each, we notice that most of the 
time the replica's are in recovering status. What could be causing this?
2) In-consistant results count, especially when one of the nodes is recovering, 
I have asked another question earlier on this. To our understanding the 
recovering node, doesn't returns any results, what else can cause this?
3) Solr node goes down very frequently, there is no OOM or otherwise in the 
logs, but a node keeps going down. Also, at times we have noticed that tomcat 
stops responding. Since there are so many parts to solr cloud, checking what's 
causing the issue is difficult in a quick time, so if anyone else has faced 
this, it would be very helpful.

Since this is happening on our UAT environment, we need to get a fix soon. 
Regards,Ayush 

Re: questions on Solr WordBreakSolrSpellChecker and WordDelimiterFilterFactory

[jiag]

Hi Ahmet,

using  or  didn't
make any difference. Still running into the same issues aforementioned :(

Thanks,
Jia

On 7/16/2014, "Ahmet Arslan"  wrote:

>Hi Jia,
>
>What happens when you use 
>
> 
>
>instead of 
>
> 
>
>Ahmet
>
>
>On Wednesday, July 16, 2014 3:07 AM, "j...@ece.ubc.ca"  wrote:
>
>
>
>Hello everyone :)
>
>I have a product called "xbox" indexed, and when the user search for
>either "x-box" or "x box" i want the "xbox" product to be
>returned.  I'm new to Solr, and from reading online, I thought I need
>to use WordDelimiterFilterFactory for "x-box" case, and
>WordBreakSolrSpellChecker for "x box" case. Is this correct?
>
>(1) In my schema file, this is what I changed:
>generateNumberParts="1" catenateWords="1" catenateNumbers="1"
>catenateAll="1" splitOnCaseChange="0" preserveOriginal="1"/>
>
>But I don't see the xbox product returned when the search term is
>"x-box", so I must have missed something
>
>(2) I tried to use  WordBreakSolrSpellChecker together with
>DirectSolrSpellChecker as shown below, but the WordBreakSolrSpellChecker
>never got used:
>
>class="solr.SpellCheckComponent">
>    wc_textSpell
>
>    
>      default
>      spellCheck
>      solr.DirectSolrSpellChecker
>      internal
>          0.3
>            2
>            1
>            5
>            3
>            0.01
>            0.004
>    
>
>    wordbreak
>    solr.WordBreakSolrSpellChecker
>    spellCheck
>    true
>    true
>    10
>  
>  
>
>  class="org.apache.solr.handler.component.SearchHandler">
>    
>        SpellCheck
>        true
>       default
>        wordbreak
>         true
>       false
>       10
>       true
>       false
>    
>    
>      wc_spellcheck
>    
>  
>
>I tried to build the dictionary this way:
>http://localhost/solr/coreName/select?spellcheck=true&spellcheck.build=true,
>but the response returned is this:
>
>
>0
>0
>
>true
>true
>
>
>build
>
>
>
>What's the correct way to build the dictionary?
>Even though my requestHandler's name="/spellcheck", i wasn't able to
>use
>http://localhost/solr/coreName/spellcheck?spellcheck=true&spellcheck.build=true
>.. is there something wrong with my definition above?
>
>(3) I also tried to use WordBreakSolrSpellChecker without the
>DirectSolrSpellChecker as shown below:
>class="solr.SpellCheckComponent">
>
>  wc_textSpell
>    
>    default
>    solr.WordBreakSolrSpellChecker
>    spellCheck
>    true
>    true
>    10
>  
>   
>
>   class="org.apache.solr.handler.component.SearchHandler">
>    
>        SpellCheck
>        true
>       default
>        
>         true
>       false
>       10
>       true
>       false
>    
>    
>      wc_spellcheck
>    
>  
>
>And still unable to see WordBreakSolrSpellChecker being called anywhere.
>
>Would someone kindly help me?
>
>Many thanks,
>Jia
>

Re: Request to be added to the ContrubitorsGroup

[Stefan Matheis]

Hey Vitaly  

i’ve added you to the contributors group

-Stefan  


On Thursday, July 17, 2014 at 12:58 PM, p...@satisware.com wrote:

> Hi there,
> Could you please add me as a contributor?
> My username is VitaliyVerbenko
> Thanks in advance!
> -
> Vitaliy Verbenko
> Marketing Guy
> Helprace by Satisware (http://satisware.com/help-desk-software)

Request to be added to the ContrubitorsGroup

[pr]

Hi there,
Could you please add me as a contributor?
My username is VitaliyVerbenko
Thanks in advance!
-
Vitaliy Verbenko
Marketing Guy
Helprace_by_Satisware

Re: Boost using date and a field value

[Alexandre Rafalovitch]

Well, to start, it would help to know what "can't combine the  boosts"
mean in your case. Are you saying you can only apply one at a time?
Are you saying that you see the effect of both but having troubles
balancing them? Something else?

The more complex use cases become, the more precise information people
need to be able to help you. And Relevancy is a complex case even in
the basic scenarios.

Regards,
   Alex.
Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853


On Thu, Jul 17, 2014 at 3:57 PM, Hakim Benoudjit  wrote:
> Any idea please?
>
>
> 2014-07-15 15:00 GMT+01:00 Hakim Benoudjit :
>
>> Hi,
>> I want to boost recent (*today's*) documents having a certain *field
>> value*. The two fields to be bosted are respectively: '*date*' & '*site*'.
>> But I dont want to penalize *recent *documents not satisfying the field
>> value ('*site*'), in favor of *older* documents satisfying this field
>> value ('*site*').
>>
>> - I've boosted documents having this field value ('*site*'), using *dismax
>> boost query*.
>> - And I've found on solr doc how to boost *recent *docs:
>> https://wiki.apache.org/solr/SolrRelevancyFAQ#How_can_I_boost_the_score_of_newer_documents
>>
>> - But I cant combine these two boosts.
>>
>>
>> --
>> Hakim Benoudjit.
>>
>
>
>
> --
> Hakim Benoudjit.

Re: Boost using date and a field value

[Hakim Benoudjit]

Any idea please?


2014-07-15 15:00 GMT+01:00 Hakim Benoudjit :

> Hi,
> I want to boost recent (*today's*) documents having a certain *field
> value*. The two fields to be bosted are respectively: '*date*' & '*site*'.
> But I dont want to penalize *recent *documents not satisfying the field
> value ('*site*'), in favor of *older* documents satisfying this field
> value ('*site*').
>
> - I've boosted documents having this field value ('*site*'), using *dismax
> boost query*.
> - And I've found on solr doc how to boost *recent *docs:
> https://wiki.apache.org/solr/SolrRelevancyFAQ#How_can_I_boost_the_score_of_newer_documents
>
> - But I cant combine these two boosts.
>
>
> --
> Hakim Benoudjit.
>



-- 
Hakim Benoudjit.