Re: SolrCloud scaling/optimization for high request rate

2019-05-16 Thread Sofiya Strochyk
14:19 +0200, Sofiya Strochyk wrote: I'll check if the filter queries or the main query tokenizers/filters might have anything to do with this, but I'm afraid query optimization can only get us so far. Why do you think that? As you tried eliminating sorting and retrieval previously, the

Re: **SPAM** Re: SolrCloud scaling/optimization for high request rate

2018-11-12 Thread Sofiya Strochyk
find the relevant references right now, but just something worth checking out. Regards, Ere Sofiya Strochyk kirjoitti 6.11.2018 klo 16.38: Hi Toke, sorry for the late reply. The query i wrote here is edited to hide production details, but I can post additional info if this helps. I have t

Re: SolrCloud scaling/optimization for high request rate

2018-11-12 Thread Sofiya Strochyk
s to be a way to scale this setup and deal with both higher load and more complex queries. On 08.11.18 10:53, Toke Eskildsen wrote: On Tue, 2018-11-06 at 16:38 +0200, Sofiya Strochyk wrote: I have tested all of the suggested changes none of these seem to make a noticeable difference (usuall

Re: SolrCloud scaling/optimization for high request rate

2018-11-06 Thread Sofiya Strochyk
Hi Toke, sorry for the late reply. The query i wrote here is edited to hide production details, but I can post additional info if this helps. I have tested all of the suggested changes none of these seem to make a noticeable difference (usually response time and other metrics fluctuate over

Re: Re: SolrCloud scaling/optimization for high request rate

2018-10-31 Thread Sofiya Strochyk
The logfiles on your servers should be verbose enough to indicate what machines are handling which parts of the request. Yes, generally i see the following entries in logs: 1. df=_text_&distrib=false&fl=_id&fl=score&shards.purpose=4&start=0&fsv=true&sort=fq=&shard.url=&rows=24&version=2&q=&NO

Re: SolrCloud scaling/optimization for high request rate

2018-10-30 Thread Sofiya Strochyk
book.com/deicool LinkedIn: www.linkedin.com/in/deicool <http://www.linkedin.com/in/deicool> "Plant a Tree, Go Green" Make In India : http://www.makeinindia.com/home On Tue, Oct 30, 2018 at 5:21 PM Sofiya Strochyk <mailto:s...@interlogic.com.ua>> wrote: My swappin

Re: SolrCloud scaling/optimization for high request rate

2018-10-30 Thread Sofiya Strochyk
500 12833 deic...@gmail.com <mailto:deic...@gmail.com> Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool <http://www.linkedin.com/in/deicool> "Plant a Tree, Go Green" Make In India : http://www.makeinindia.com/home On Mon, Oct 29, 2018 a

Re: SolrCloud scaling/optimization for high request rate

2018-10-29 Thread Sofiya Strochyk
, 29 Oct 2018, 20:30 Sofiya Strochyk, <mailto:s...@interlogic.com.ua>> wrote: Hi Deepak and thanks for your reply, On 27.10.18 10:35, Deepak Goel wrote: Last, what is the nature of your request. Are the queries the same? Or they are very random? Random queries would

Re: SolrCloud scaling/optimization for high request rate

2018-10-29 Thread Sofiya Strochyk
at speculation is borne out, you have something that'll help. Or you have another blind alley ;) Best Erick On Mon, Oct 29, 2018 at 8:00 AM Sofiya Strochyk wrote: Hi Deepak and thanks for your reply, On 27.10.18 10:35, Deepak Goel wrote: Last, what is the nature of your request. Are the que

Re: Re: SolrCloud scaling/optimization for high request rate

2018-10-29 Thread Sofiya Strochyk
ve to be fetched from all shards. This consumes a relatively high amount of memory, and even if the servers are able to handle a certain number of these queries simultaneously, you'd run into garbage collection trouble with more queries being served. So just one more thing to be aware of!

Re: SolrCloud scaling/optimization for high request rate

2018-10-29 Thread Sofiya Strochyk
Hi Deepak and thanks for your reply, On 27.10.18 10:35, Deepak Goel wrote: Last, what is the nature of your request. Are the queries the same? Or they are very random? Random queries would need more tuning than if the queries the same. The search term (q) is different for each query, and fil

Re: SolrCloud scaling/optimization for high request rate

2018-10-29 Thread Sofiya Strochyk
Hi Shalin, these are stats for caches used: *documentCache* class:org.apache.solr.search.LRUCache description:LRU Cache(maxSize=128, initialSize=128) stats: CACHE.searcher.documentCache.cumulative_evictions:234923643 CACHE.searcher.documentCache.cumulative_hitratio:0 CACHE.searcher.documentCache

Re: SolrCloud scaling/optimization for high request rate

2018-10-29 Thread Sofiya Strochyk
node has 12 cores that would mean it can process 12 concurrent searches? And since every request is sent to all shards to check if there are results, does this also mean the whole cluster can handle 12 concurrent requests on average? On 27.10.18 09:00, Toke Eskildsen wrote: Sofiya Strochyk

Re: SolrCloud scaling/optimization for high request rate

2018-10-29 Thread Sofiya Strochyk
Erick, thanks, i've been pulling my hair out over this for a long time and gathered a lot of information :) Doesn't there exist a setting for maxIndexingThreads in solrconfig with default value of about 8? It's not clear if my updates are being executed in parallel or not but i would expect

Re: **SPAM** Re: SolrCloud scaling/optimization for high request rate

2018-10-29 Thread Sofiya Strochyk
get very, very slow. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) On Oct 26, 2018, at 10:21 AM, Sofiya Strochyk wrote: Thanks Erick, 1. We already use Solr 7.5, upgraded some of our nodes only recently to see if this eliminates the difference in

Re: SolrCloud scaling/optimization for high request rate

2018-10-26 Thread Sofiya Strochyk
onfiguration would be helpful. There's been talk of separate thread pools for indexing and querying to give queries a better shot at the CPU, but that's not in place yet. 7> G1GC may also help rather than CMS, but as you're well aware GC tuning "is more art than science" ;)

SolrCloud scaling/optimization for high request rate

2018-10-26 Thread Sofiya Strochyk
Hi everyone, We have a SolrCloud setup with the following configuration: * 4 nodes (3x128GB RAM Intel Xeon E5-1650v2, 1x64GB RAM Intel Xeon E5-1650v2, 12 cores, with SSDs) * One collection, 4 shards, each has only a single replica (so 4 replicas in total), using compositeId router * Tot