AW: Hard commits, soft commits and transaction logs

Thanks Erick. I guess I'll go the 3>-way, i.e. optimize the index "whenever appropriate". Could I alternatively ("whenever appropriate") issue a '/suggest?spellcheck.build=true'-request? > bq: Suggestions are re-built on commit Agree. That was for unitTesting purposes only. In production we have

Solr 5: not loading shards from symlinked directories

I've tried to upgrade from Solr 4.10.3 to 5.4.1. Solr shards are placed on different disks and symlinks (ln -s) are created to SOLR_HOME (SOLR_HOME itself is set as an absolute path and works fine). When Solr starts, it loads only shards placed in home directory, but not symlinked ones. If I copy s

Re: Multi-level nested documents query

Hi Mikhail, Thank you for the link. I will check that blog post. On Friday 05 February 2016 01:42 AM, Mikhail Khludnev wrote: Hello, I'm not sure that it's achievable overall, but at least you need to use different parent fields/terms/filters across levels like in http://blog.

Re: ​Securing fields and documents with Shield | Elastic

Thanks for replying, Alex. At the moment, my requirement is to show public/published documents as well as unpublished documents based on the user issuing the query. Or just the user's documents, with no public documents. I've implemented this with a JOIN and my last post on this is here: http://luc

Re: Solr segment merging in different replica

Hi Shawn, Thanks for your reply. Yes, we were planning for such instance where the replica went down during indexing, and when it re-started, it will start to copy the index over to the main node. Regards, Edwin On 5 February 2016 at 03:35, Shawn Heisey wrote: > On 2/4/2016 9:27 AM, Zheng L

hitratio vs cumulative_hitratio

Solr caching : What does it mean have lookup=0, hits=0 and hitratio=0 but cumulative_hitratio=0.75 and cumulative_lookups >100,000 with cumulative_inserts >20k and cumulative_evictions =0, maxSize of cache objects is 512

Re: ​Securing fields and documents with Shield | Elastic

I have not used Shield yet, so this is based just on the document you sent. I would use different Request Handler endpoints for different users and put the restrictions there, in the invariants section. For field restrictions, I would use 'uf' parameter. As for example here (from my old book): ht

​Securing fields and documents with Shield | Elastic

Does Solr have anything like this? https://www.elastic.co/blog/securing-fields-and-documents-with-shield Or is it on the roadmap?

Re: Change in EXPLAIN info since Solr 5

On 2/4/2016 2:54 PM, Burgmans, Tom wrote: > While exploring Solr 5.4.0, I noticed a subtle difference in the EXPLAIN > debug information, compared to the version we currently use (4.10.1). > The difference is the removal of (MATCH) in some of the EXPLAIN lines. That > is causing issues for us si

Re: How to assign a hash range via zookeeper?

On 2/4/2016 2:12 PM, Aki Balogh wrote: > I found the state.json file and it indeed shows that the range for shard1 > is null. > > In order to fix, do I need to upload a corrected state.json file with > corrected hash ranges? How can I do that? (zkcli.sh?) The easiest way to figure out the correc

Re: How to assign a hash range via zookeeper?

Hash ranges should have been assigned automatically when you created the collection unless you created the collection with the implicit router. What was the command you used to create the collection? Best, Erick On Thu, Feb 4, 2016 at 1:21 PM, Aki Balogh wrote: > I'm not sure how these hash rang

Change in EXPLAIN info since Solr 5

Hi group, While exploring Solr 5.4.0, I noticed a subtle difference in the EXPLAIN debug information, compared to the version we currently use (4.10.1). Solr 4.10.1: 2.0739748 = (MATCH) max plus 1.0 times others of: 2.0739748 = (MATCH) weight(text:test in 30) [DefaultSimilarity], result of:

Re: How to assign a hash range via zookeeper?

I'm not sure how these hash ranges were determined, so I'm not sure if I should be manually setting them or somehow allowing solr to pick them for this shard. Thanks, Aki On Thu, Feb 4, 2016 at 4:12 PM, Aki Balogh wrote: > Shawn, > > Thanks - this is very helpful. > > I found the state.json f

Re: Errors During Load Test

Tiwari, Shailendra wrote: > We are on Solr 4.10.3. Got 2 load balanced RedHat with 16 GB > memory on each. Memory assigned to JVM 4 GB, 2 Shards, > total docs 60 K, and 4 replicas. As you are chasing throughput, you should aim to lower the overall resources needed for a single request, potentia

Re: How to assign a hash range via zookeeper?

Shawn, Thanks - this is very helpful. I found the state.json file and it indeed shows that the range for shard1 is null. In order to fix, do I need to upload a corrected state.json file with corrected hash ranges? How can I do that? (zkcli.sh?) Thanks, Aki On Thu, Feb 4, 2016 at 4:10 PM, Sh

Re: How to assign a hash range via zookeeper?

On 2/4/2016 1:37 PM, Aki Balogh wrote: > Specifically, they suggest getting clusterstate.json. But I've tried that > and when I get that file, I only get an empty file {} > > Is there another way to ask Zookeeper to cover the missing hash range? Solr 5.x changed how the clusterstate is managed.

Re: How to assign a hash range via zookeeper?

PS - confirmed: in the GUI, I go to Admin->Cloud->Tree, click on clusterstate.json and it's empty {} On Thu, Feb 4, 2016 at 3:37 PM, Aki Balogh wrote: > One of our shards went down. We brought it back up but it doesn't have a > hash range: > > > > active > > > > marketmuse_shard1_repli

Re: Use SqlEntityProcessor in cached mode to repeat a query for a nested child element

you're right, that was a mistake in my code - I did actually using cacheKey but that didn't work so I was looking at the Java class for DIHCacheSupport to see if there were any other settings I could use https://lucene.apache.org/solr/5_4_0/solr-dataimporthandler/index.html?org/apache/solr/handl

How to assign a hash range via zookeeper?

One of our shards went down. We brought it back up but it doesn't have a hash range: active marketmuse_shard1_replica1 http://172.30.0.254:8080/solr 172.30.0.254:8080_solr active active marketmuse_shard1_replica2 172.30.0.89:8080_solr http://172.30.0.89:8080/solr true This results in

Re: Use SqlEntityProcessor in cached mode to repeat a query for a nested child element

Where did cachePrimaryKey comes from? The documentation has cacheKey : https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler Regards, Alex. Newsletter and resources for Solr beginners and intermediates: http://www.solr-start.com

Re: Multi-level nested documents query

Hello, I'm not sure that it's achievable overall, but at least you need to use different parent fields/terms/filters across levels like in http://blog.griddynamics.com/2013/12/grandchildren-and-siblings-with-block.html On Thu, Feb 4, 2016 at 8:39 PM, Pranaya Behera wrote: > Hi, > I have

RE: Errors During Load Test

We are on Solr 4.10.3. Got 2 load balanced RedHat with 16 GB memory on each. Memory assigned to JVM 4 GB, 2 Shards, total docs 60 K, and 4 replicas. Thanks Shail -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Thursday, February 04, 2016 1:27 PM To: solr

Re: Solr segment merging in different replica

On 2/4/2016 9:27 AM, Zheng Lin Edwin Yeo wrote: > Yes, I'm already on SolrCloud, so I'll probably stick to that. > > Regarding the network, I am just afraid that when the replica code copies > the index over from the main node, it will use up all the available > bandwidth, and causes the search que

Re: Hard commits, soft commits and transaction logs

bq: and suggestions of deleted docs are... OK, this is something different than I read the first time. I'm assuming that when you mention suggestions, you're using one of the suggesters that works off the indexed terms, which will include data from deleted docs. There's really not a good mechanism

Re: Errors During Load Test

The short form is "add more replicas", assuming you're using SolrCloud. If older-style master/slave, then "add more slaves". Solr request processing scales pretty linearly with the number of replicas (or slaves). Note that this is _not_ adding shards (assuming SolrCloud). You usually add shards w

Multi-level nested documents query

Hi, I have documents that are indexed are like this: product -isParent:true - child1 -isParent:true - child1_1 - child1_2 - child1_3 - child2 -isParent:true - child2_1 - child2_2 - child2_3 I have used fl=*,[child parentFilter=isParent:t

Re: Solr 4.10 with Jetty 8.1.10 & Tomcat 7

On 2/4/2016 9:48 AM, Shahzad Masud wrote: > Thank you Shawn for your response. I have been using manual shards (old > mechanism) i.e. seperate context for each shard and each shard pointing to > seperate data and indexing folder. > > Shard 1 = localhost:8983/solr_2014 > Shard 2 = localhost:8983/so

Re: "I was asked to wait on state recovering for shard.... but I still do not see the request state"

Only INFO level, so I suspect not bad... If that Overseer closed, another node should have picked up where it left off. See that in another log? Generally an Overseer close means a node or cluster restart. This can cause a lot of DOWN state publishing. If it's a cluster restart, a lot of those D

Re: Solr 4.10 with Jetty 8.1.10 & Tomcat 7

Thank you Shawn for your response. I have been using manual shards (old mechanism) i.e. seperate context for each shard and each shard pointing to seperate data and indexing folder. Shard 1 = localhost:8983/solr_2014 Shard 2 = localhost:8983/solr_2015 Shard 3 = localhost:8983/solr_2016 Do you thi

Re: Solr segment merging in different replica

Thanks Emir and Benedetti. Yes, I'm already on SolrCloud, so I'll probably stick to that. Regarding the network, I am just afraid that when the replica code copies the index over from the main node, it will use up all the available bandwidth, and causes the search query to have little bandwidth l

Use SqlEntityProcessor in cached mode to repeat a query for a nested child element

Hi everyone, Is it possible to use SqlEntityProcessor in cached mode to repeat a query for a nested child element? I'd like to use the entity query once to consolidate information from the children to the parent, then another to actually index the entities as children. Here's an example of wh

Re: Solr 4.10 with Jetty 8.1.10 & Tomcat 7

On 2/4/2016 7:29 AM, Shahzad Masud wrote: > Q: Is this a normal that one node support one shard in Jetty? > Q: Can anyone point to appropriate guideline; if jetty is better than > tomcat? > Q: Have anyone else experienced similar migration, and concluded that > tomcat is better. Solr 5.x is more d

commitReserveDuration vs maxWriteMBPerSec

Hi, I would like to find out, what is the difference between commitReserveDuration and maxWriteMBPerSec under the /replication requestHandler? Will there be any impact if we set a long commitReserveDuration but a high maxWriteMBPerSec? I am using Solr 5.4.0 Regards, Edwin

Re: Errors During Load Test

What is your solr setup -- nodes/shards/specs? 7221 requests/min is a lot so it's likely that your solr setup simply isn't able to support this kind of load which results in the requests timing out which is why you keep seeing the timeout and connect exceptions. On Thu, 4 Feb 2016, 20:30 Tiwari, S

Errors During Load Test

Hi All, We did our first load test on Search (Solr) API, and started to see some errors after 2000 Users. Errors used to go away after 30 seconds, but keep happening frequently. Errors were "java.net.SocketTimeoutException" and "org.apache.http.conn.HttpHostConnectException". We were using JMet

Solr 4.10 with Jetty 8.1.10 & Tomcat 7

I have been running Solr 4.10 with Tomcat 7 with manual shard scheme (i.e. 4 Tomcats with 16 shards - Each tomcat having 4 contexts / instances in it to represent shards). It was working fairly good for last 4 years, but with few OOM (Out of memory) on random servers. This situation get back normal

Re: Solr for real time analytics system

Hi Rohit, Please take a loot at Streaming expressions & Parallel SQL Interface. That should meet many of your analytics requirement (aggregation queries like sum/average/groupby etc). https://cwiki.apache.org/confluence/display/solr/Streaming+Expressions https://cwiki.apache.org/confluence/displa

Re: Tutorial or Code Samples to explain how to Write Solr Plugins

I used those links to learn to write my first plugin as well. I might have that code still lying around somewhere. Let me take a look and get back. On Thu, 4 Feb 2016, 19:32 Gian Maria Ricci - aka Alkampfer < alkamp...@nablasoft.com> wrote: > I've already found these two presentation, sadly enoug

RE: Tutorial or Code Samples to explain how to Write Solr Plugins

I've already found these two presentation, sadly enough link for source code is broken, it seems that the domain www.searchbox.com is completely down :| -- Gian Maria Ricci Cell: +39 320 0136949 -Original Message- From: Binoy Dalal [mailto:binoydala...@gmail.com] Sent: mercoledì 3

RE: Tutorial or Code Samples to explain how to Write Solr Plugins

Thanks to everyone for the really useful links. The problem is that, googling around does not produces really good results. In the past when I wrote my first plugin it was a real pain :). Thanks. -- Gian Maria Ricci Cell: +39 320 0136949 -Original Message- From: Alexandre Rafalovi

Re: Out of memory error during full import

On 2/4/2016 12:18 AM, Srinivas Kashyap wrote: > I have implemented 'SortedMapBackedCache' in my SqlEntityProcessor for the > child entities in data-config.xml. When i try to do full import, i'm getting > OutOfMemory error(Java Heap Space). I increased the HEAP allocation to the > maximum extent

Re: implement exact match for one of the search fields only?

The desired architecture is that you use a middle app layer that clients send queries to and that middle app layer then constructs the formal query and sends it on to Solr proper. This architecture also enables breaking a user query into multiple Solr queries and then aggregating the results. Besid

Re: filters to work with dates

Hi Markus At first, I thought keep the original field and create a new field using function "Copying Fields ". For this reason, I thought it was better choice to use a filter function in destiny field. However I am going to stud

Out of memory error during full import

Hello, I have implemented 'SortedMapBackedCache' in my SqlEntityProcessor for the child entities in data-config.xml. When i try to do full import, i'm getting OutOfMemory error(Java Heap Space). I increased the HEAP allocation to the maximum extent possible. Is there a workaround to do initial

Re: Solr for real time analytics system

Thanks Bhimavarapu for the information. We are creating our own dashboard, so probably wont need kibana/banana. I was more curious about Solr support for fast aggregation query over very large data set. As suggested, I guess elasticsearch has this capability. Is there any published metrics or dat

Loading Solr Analyzer from RuntimeLib Blob

Hi, Did you find a solution to your problem? We are also having similar problem. We tried with ‘sharedLib’ attribute but to no avail so far. -Ravi

Re: Solr for real time analytics system

A few people did a real time analytics system with solr and talked about it at conferences. Maybe you'll find their presentations useful: https://www.youtube.com/results?search_query=solr%20real%20time%20analytics&oq=&gs_l= (esp. the first one: https://www.youtube.com/watch?v=PkoyCxBXAiA ) On Thu,

Re: implement exact match for one of the search fields only?

Hi Erick << The manual way of doing this would be to construct an elaborate query, like q=spp_keyword_exact:"dvd bracket" OR P_ShortDescription:(dvd bracket) OR NOTE: the parens are necessary or the last part of the above would be parsed as P_ShortDescription:dvd default_searchfield:bracke