Re: Solr performance issue

On 2/15/2018 2:00 AM, Srinivas Kashyap wrote: > I have implemented 'SortedMapBackedCache' in my SqlEntityProcessor for the > child entities in data-config.xml. And i'm using the same for full-import > only. And in the beginning of my implementation, i had written delta-import > query to index

Re: Solr performance issue

Srinivas: Not an answer to your question, but when DIH starts getting this complicated, I start to seriously think about SolrJ, see: https://lucidworks.com/2012/02/14/indexing-with-solrj/ IN particular, it moves the heavy lifting of acquiring the data from a Solr node (which I'm assuming also

Solr performance issue

Hi, I have implemented 'SortedMapBackedCache' in my SqlEntityProcessor for the child entities in data-config.xml. And i'm using the same for full-import only. And in the beginning of my implementation, i had written delta-import query to index the modified changes. But my requirement grew and

Re: Solr performance issue on querying --> Solr 6.5.1

Hi Erick, As suggested, I did try nonHDFS solr cloud instance and it response looks to be really better. From the configuration side to, I am mostly using default configurations and with block.cache.direct.memory.allocation as false. On analysis of hdfs cache, evictions seems to be on higher

Re: Solr performance issue on querying --> Solr 6.5.1

Hi Arun, It is hard to measure something without affecting it, but we could use debug results and combine with QTime without debug: If we ignore merging results, it seems that majority of time is spent for retrieving docs (~500ms). You should consider reducing number of rows if you want better

Re: Solr performance issue on querying --> Solr 6.5.1

Hi Emir, Please find the response without bq parameter and debugQuery set to true. Also it was noted that Qtime comes down drastically without the debug parameter to about 700-800. true 0 3446 ("hybrid electric powerplant" "hybrid electric powerplants" "Electric" "Electrical" "Electricity"

Re: Solr performance issue on querying --> Solr 6.5.1

Hi Erick, Qtime comes down with rows set as 1. Also it was noted that qtime comes down when debug parameter is not added with the query. It comes to about 900. Thanks, Arun -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr performance issue on querying --> Solr 6.5.1

On Tue, 2017-09-26 at 07:43 -0700, sasarun wrote: > Allocated heap size for young generation is about 8 gb and old  > generation is about 24 gb. And gc analysis showed peak > size utlisation is really low compared to these values. That does not come as a surprise. Your collections would normally

Re: Solr performance issue on querying --> Solr 6.5.1

Hi Arun, This is not the most simple query either - a dozen of phrase queries on several fields + the same query as bq. Can you provide debugQuery info. I did not look much into debug times and what includes what, but one thing that is strange to me is that QTime is 4s while query in debug is

Re: Solr performance issue on querying --> Solr 6.5.1

Well, 15 second responses are not what I'd expect either. But two things (just looked again) 1> note that the time to assemble the debug information is a large majority of your total time (14 of 15.3 seconds). 2> you're specifying 600 rows which is quite a lot as each one requires that a 16K

Re: Solr performance issue on querying --> Solr 6.5.1

Hi Erick, Thank you for the quick response. Query time was relatively faster once it is read from memory. But personally I always felt response time could be far better. As suggested, We will try and set up in a non HDFS environment and update on the results. Thanks, Arun -- Sent from:

Re: Solr performance issue on querying --> Solr 6.5.1

Does the query time _stay_ low? Once the data is read from HDFS it should pretty much stay in memory. So my question is whether, once Solr warms up you see this kind of query response time. Have you tried this on a non HDFS system? That would be useful to help figure out where to look. And given

Solr performance issue on querying --> Solr 6.5.1

Hi All, I have been using Solr for some time now but mostly in standalone mode. Now my current project is using Solr 6.5.1 hosted on hadoop. My solrconfig.xml has the following configuration. In the prod environment the performance on querying seems to really slow. Can anyone help me with few

RE: Solr performance issue on indexing

> Also we will try to decouple tika to solr. +1 -Original Message- From: tstusr [mailto:ulfrhe...@gmail.com] Sent: Friday, March 31, 2017 4:31 PM To: solr-user@lucene.apache.org Subject: Re: Solr performance issue on indexing Hi, thanks for the feedback. Yes, it is about OOM, ind

Re: Solr performance issue on indexing

r. > > By the way, make it available with solr cloud will improve performance? Or > there will be no perceptible improvement? > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Solr-performance-issue-on-indexing-tp4327886p4327914.html > Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr performance issue on indexing

? -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-performance-issue-on-indexing-tp4327886p4327914.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr performance issue on indexing

> -- > View this message in context: > http://lucene.472066.n3.nabble.com/Solr-performance-issue-on-indexing-tp4327886.html > Sent from the Solr - User mailing list archive at Nabble.com.

Solr performance issue on indexing

shes. So, what are some general tips about improving performance for this scenario? -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-performance-issue-on-indexing-tp4327886.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr performance issue

1 million document isn't considered big for Solr. How much RAM does your machine have? Regards, Edwin On 8 February 2016 at 23:45, Susheel Kumar wrote: > 1 million document shouldn't have any issues at all. Something else is > wrong with your hw/system configuration. >

Re: solr performance issue

1 million document shouldn't have any issues at all. Something else is wrong with your hw/system configuration. Thanks, Susheel On Mon, Feb 8, 2016 at 6:45 AM, sara hajili wrote: > On Mon, Feb 8, 2016 at 3:04 AM, sara hajili wrote: > > > sorry i

solr performance issue

hi all. i have a problem with my solr performance and usage hardware like a ram,cup... i have a lot of document and so indexed file about 1000 doc in solr that every doc has about 8 field in average. and each field has about 60 char. i set my field as a storedfield = "false" except of 1 field. //

Re: solr performance issue

Hi Sara, It is still considered to be small index. Can you give us bit details about your setup? Thanks, Emir On 08.02.2016 12:04, sara hajili wrote: sorry i made a mistake i have a bout 1000 K doc. i mean about 100 doc. On Mon, Feb 8, 2016 at 1:35 AM, Emir Arnautovic <

Re: solr performance issue

Hi Sara, Not sure if I am reading this right, but I read it as you have 1000 doc index and issues? Can you tell us bit more about your setup: number of servers, hw, index size, number of shards, queries that you run, do you index at the same time... It seems to me that you are running Solr

Re: solr performance issue

sorry i made a mistake i have a bout 1000 K doc. i mean about 100 doc. On Mon, Feb 8, 2016 at 1:35 AM, Emir Arnautovic < emir.arnauto...@sematext.com> wrote: > Hi Sara, > Not sure if I am reading this right, but I read it as you have 1000 doc > index and issues? Can you tell us bit more

Re: solr performance issue

On Mon, Feb 8, 2016 at 3:04 AM, sara hajili wrote: > sorry i made a mistake i have a bout 1000 K doc. > i mean about 100 doc. > > On Mon, Feb 8, 2016 at 1:35 AM, Emir Arnautovic < > emir.arnauto...@sematext.com> wrote: > >> Hi Sara, >> Not sure if I am reading this

Re: Solr Performance Issue

Hi; Erick and Shawn have explained that we need more information about your infrastructure. I should add that: I had test data at my SolrCloud nearly as much as yours and I did not have any problems except for when indexing at a huge index rate and it can be solved with turning. You should

Re: Solr Performance Issue

Hi Furkan, Just curious what was the index rate that you were able to achieve?   Regards, Hien On Thursday, December 5, 2013 3:06 PM, Furkan KAMACI furkankam...@gmail.com wrote: Hi; Erick and Shawn have explained that we need more information about your infrastructure. I should add that:

Re: Solr Performance Issue

On 12/5/2013 4:08 PM, Hien Luu wrote: Just curious what was the index rate that you were able to achieve? What I've usually seen based on my experience and what people have said here and on IRC is that the data source is usually the bottleneck - Solr typically indexes VERY fast, as long as

Re: Solr Performance Issue

Hi Hien; Actually high index rate is a relative concept. I could index such kind of data within a few hours. I aim to index much much more data within same time soon. I can share my test results when I do. Thanks; Furkan KAMACI 6 Aralık 2013 Cuma tarihinde Hien Luu h...@yahoo.com adlı kullanıcı

Re: Solr Performance Issue

Thanks Furkan. Looking forward to seeing your test results. Sent from Yahoo Mail on Android

Solr Performance Issue

[java.lang.IllegalStateException: Cannot call sendError() after the response has been committed] with root cause Can anybody help me how can i solve this problem. Kumar. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Performance-Issue-tp4104907.html Sent from the Solr - User mailing list

Re: Solr Performance Issue

. Kumar. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Performance-Issue-tp4104907.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Performance Issue

On 12/4/2013 6:31 AM, kumar wrote: I am having almost 5 to 6 crores of indexed documents in solr. And when i am going to change anything in the configuration file solr server is going down. If you mean crore and not core, then you are talking about 50 to 60 million documents. That's a lot.

Re: Solr performance issue

Hello, The problem turned out to be some sort of sharding/searching weirdness. We modified some code in sharding but I don't think it is related. In any case, we just added a new server that just shards (but doesn't do any searching / doesn't contain any index) and performance is very very good.

Re: Solr performance issue

Btw, I am monitoring output via jconsole with 8gb of ram and it still goes to 8gb every 20 seconds or so, gc runs, falls down to 1gb. Hmm, jvm is eating 8Gb for 20 seconds - sounds a lot. Do you return all results (ids) for your queries? Any tricky faceting/sorting/function queries?

Re: Solr performance issue

My solr+jetty+java6 install seems to work well with these GC options. It's a dual processor environment: -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode I've never had a real problem with memory, so I've not done any kind of auditing. I probably should, but time is a limited resource.

Re: Solr performance issue

CMS is very good for multicore CPU's. Use incremental mode only when you have a single CPU with only one or two cores. On Tuesday 15 March 2011 16:03:38 Shawn Heisey wrote: My solr+jetty+java6 install seems to work well with these GC options. It's a dual processor environment:

Re: Solr performance issue

The host is dual quad-core, each Xen VM has been given two CPUs. Not counting dom0, two of the hosts have 10/8 CPUs allocated, two of them have 8/8. The dom0 VM is also allocated two CPUs. I'm not really sure how that works out when it comes to Java running on the VM, but if at all

Solr performance issue

Hello everyone, First of all here is our Solr setup: - Solr nightly build 986158 - Running solr inside the default jetty comes with solr build - 1 write only Master , 4 read only Slaves (quad core 5640 with 24gb of RAM) - Index replicated (on optimize) to slaves via Solr Replication - Size of

Re: Solr performance issue

Hi Doğacan, Are you, at some point, running out of heap space? In my experience, that's the common cause of increased load and excessivly high response times (or time outs). Cheers, Hello everyone, First of all here is our Solr setup: - Solr nightly build 986158 - Running solr inside

Re: Solr performance issue

Hello, 2011/3/14 Markus Jelsma markus.jel...@openindex.io Hi Doğacan, Are you, at some point, running out of heap space? In my experience, that's the common cause of increased load and excessivly high response times (or time outs). How much of a heap size would be enough? Our index size

Re: Solr performance issue

Hello, 2011/3/14 Markus Jelsma markus.jel...@openindex.io Hi Doğacan, Are you, at some point, running out of heap space? In my experience, that's the common cause of increased load and excessivly high response times (or time outs). How much of a heap size would be enough? Our

Re: Solr performance issue

I've definitely had cases in 1.4.1 where even though I didn't have an OOM error, Solr was being weirdly slow, and increasing the JVM heap size fixed it. I can't explain why it happened, or exactly how you'd know this was going on, I didn't see anything odd in the logs to indicate, I just

Re: Solr performance issue

Hello again, 2011/3/14 Markus Jelsma markus.jel...@openindex.io Hello, 2011/3/14 Markus Jelsma markus.jel...@openindex.io Hi Doğacan, Are you, at some point, running out of heap space? In my experience, that's the common cause of increased load and excessivly high response

Re: Solr performance issue

Nope, no OOM errors. That's a good start! Insanity count is 0 and fieldCAche has 12 entries. We do use some boosting functions. Btw, I am monitoring output via jconsole with 8gb of ram and it still goes to 8gb every 20 seconds or so, gc runs, falls down to 1gb. Hmm, maybe the garbage

Re: Solr performance issue

It's actually, as I understand it, expected JVM behavior to see the heap rise to close to it's limit before it gets GC'd, that's how Java GC works. Whether that should happen every 20 seconds or what, I don't nkow. Another option is setting better JVM garbage collection arguments, so GC

Re: Solr performance issue

You might also want to add the following switches for your GC log. JAVA_OPTS=$JAVA_OPTS -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails - Xloggc:/var/log/tomcat6/gc.log -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime Also, what JVM version are you using and

Re: Solr performance issue

That depends on your GC settings and generation sizes. And, instead of UseParallelGC you'd better use UseParNewGC in combination with CMS. See 22: http://java.sun.com/docs/hotspot/gc1.4.2/faq.html It's actually, as I understand it, expected JVM behavior to see the heap rise to close to it's

Re: Solr performance issue

Hello, 2011/3/14 Markus Jelsma markus.jel...@openindex.io That depends on your GC settings and generation sizes. And, instead of UseParallelGC you'd better use UseParNewGC in combination with CMS. JConsole now shows a different profile output but load is still high and performance is still

Re: Solr performance issue

Mmm. SearchHander.handleRequestBody takes care of sharding. Could your system suffer from http://wiki.apache.org/solr/DistributedSearch#Distributed_Deadlock ? I'm not sure, i haven't seen a similar issue in a sharded environment, probably because it was a controlled environment. Hello,

Re: Solr performance issue

2011/3/14 Markus Jelsma markus.jel...@openindex.io Mmm. SearchHander.handleRequestBody takes care of sharding. Could your system suffer from http://wiki.apache.org/solr/DistributedSearch#Distributed_Deadlock ? We increased thread limit (which was 1 before) but it did not help. Anyway,