Re: solr performance with >1 NUMAs

2020-10-22 Thread matthew sporleder
Great updates. Thanks for keeping us all in the loop! On Thu, Oct 22, 2020 at 7:43 PM Wei wrote: > > Hi Shawn, > > I.m circling back with some new findings with our 2 NUMA issue. After a > few iterations, we do see improvement with the useNUMA flag and other JVM > setting changes. Here are the

Re: solr performance with >1 NUMAs

2020-10-22 Thread Wei
Hi Shawn, I.m circling back with some new findings with our 2 NUMA issue. After a few iterations, we do see improvement with the useNUMA flag and other JVM setting changes. Here are the current settings, with Java 11: -XX:+UseNUMA -XX:+UseG1GC -XX:+AlwaysPreTouch -XX:+UseTLAB -XX:G1MaxNewSiz

Re: solr performance with >1 NUMAs

2020-09-28 Thread Shawn Heisey
On 9/28/2020 12:17 PM, Wei wrote: Thanks Shawn. Looks like Java 11 is the way to go with -XX:+UseNUMA. Do you see any backward compatibility issue for Solr 8 with Java 11? Can we run Solr 8 built with JDK 8 in Java 11 JRE, or need to rebuild solr with Java 11 JDK? I do not know of any problems

Re: solr performance with >1 NUMAs

2020-09-28 Thread Wei
Thanks Shawn. Looks like Java 11 is the way to go with -XX:+UseNUMA. Do you see any backward compatibility issue for Solr 8 with Java 11? Can we run Solr 8 built with JDK 8 in Java 11 JRE, or need to rebuild solr with Java 11 JDK? Best, Wei On Sat, Sep 26, 2020 at 6:44 PM Shawn Heisey wrote: >

Re: solr performance with >1 NUMAs

2020-09-26 Thread Shawn Heisey
On 9/26/2020 1:39 PM, Wei wrote: Thanks Shawn! Currently we are still using the CMS collector for solr with Java 8. When last evaluated with Solr 7, CMS performs better than G1 for our case. When using G1, is it better to upgrade from Java 8 to Java 11? From https://lucene.apache.org/solr/guide/

Re: solr performance with >1 NUMAs

2020-09-26 Thread Wei
Thanks Shawn! Currently we are still using the CMS collector for solr with Java 8. When last evaluated with Solr 7, CMS performs better than G1 for our case. When using G1, is it better to upgrade from Java 8 to Java 11? >From https://lucene.apache.org/solr/guide/8_4/solr-system-requirements.html,

Re: solr performance with >1 NUMAs

2020-09-25 Thread Shawn Heisey
On 9/23/2020 7:42 PM, Wei wrote: Recently we deployed solr 8.4.1 on a batch of new servers with 2 NUMAs. I noticed that query latency almost doubled compared to deployment on single NUMA machines. Not sure what's causing the huge difference. Is there any tuning to boost the performance on multipl

Re: solr performance with >1 NUMAs

2020-09-25 Thread Wei
Thanks Dominique. I'll start with the -XX:+UseNUMA option. Best, Wei On Fri, Sep 25, 2020 at 7:04 AM Dominique Bejean wrote: > Hi, > > This would be a Java VM option, not something Solr itself can know about. > Take a look at this article in comments. May be it will help. > > https://blog.theta

Re: solr performance with >1 NUMAs

2020-09-25 Thread Dominique Bejean
Hi, This would be a Java VM option, not something Solr itself can know about. Take a look at this article in comments. May be it will help. https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html?showComment=1347033706559#c229885263664926125 Regards Dominique Le jeu. 24 sept.

solr performance with >1 NUMAs

2020-09-23 Thread Wei
Hi, Recently we deployed solr 8.4.1 on a batch of new servers with 2 NUMAs. I noticed that query latency almost doubled compared to deployment on single NUMA machines. Not sure what's causing the huge difference. Is there any tuning to boost the performance on multiple NUMA machines? Any pointer i

Re: question about setup for maximizing solr performance

2020-06-01 Thread Shawn Heisey
On 6/1/2020 9:29 AM, Odysci wrote: Hi, I'm looking for some advice on improving performance of our solr setup. Does anyone have any insights on what would be better for maximizing throughput on multiple searches being done at the same time? thanks! In almost all cases, adding memory will pr

question about setup for maximizing solr performance

2020-06-01 Thread Odysci
Hi, I'm looking for some advice on improving performance of our solr setup. In particular, about the trade-offs between applying larger machines, vs more smaller machines. Our full index has just over 100 million docs, and we do almost all searches using fq's (with q=*:*) and facets. We are using s

Re: Solr performance using fq with multiple values

2020-04-18 Thread Shawn Heisey
On 4/18/2020 12:20 PM, Odysci wrote: We don't used this field for general queries (q:*), only for fq and faceting. Do you think making it indexed="true" would make a difference in fq performance? fq means "filter query". It's still a query. So yes, the field should be indexed. The query you

Re: Solr performance using fq with multiple values

2020-04-18 Thread Odysci
We don't used this field for general queries (q:*), only for fq and faceting. Do you think making it indexed="true" would make a difference in fq performance? Thanks Reinaldo On Sat, Apr 18, 2020 at 3:06 PM Sylvain James wrote: > Hi Reinaldo, > > Involved fields should be indexed for better per

Re: Solr performance using fq with multiple values

2020-04-18 Thread Sylvain James
Hi Reinaldo, Involved fields should be indexed for better performance ? Sylvain Le sam. 18 avr. 2020 à 18:46, Odysci a écrit : > Hi, > > We are seeing significant performance degradation on single queries that > use fq with multiple values as in: > > fq=field1_name:(V1 V2 V3 ...) > > If we u

Solr performance using fq with multiple values

2020-04-18 Thread Odysci
Hi, We are seeing significant performance degradation on single queries that use fq with multiple values as in: fq=field1_name:(V1 V2 V3 ...) If we use only one value in the fq (say only V1) we get Qtime = T ms As we increase the number of values, say to 5 values, Qtime more than triples, even i

Re: SOLR PERFORMANCE Warning

2020-02-20 Thread Emir Arnautović
Hi, It means that you are either committing too frequently or your warming up takes too long. If you are committing on every bulk, stop doing that and use autocommit. Regards, Emir -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - ht

SOLR PERFORMANCE Warning

2020-02-20 Thread Akreeti Agarwal
Hi All, I am using SOLR 7.5 version with master slave architecture. I am getting : "PERFORMANCE WARNING: Overlapping onDeckSearchers=2" continuously on my master logs for all cores. Please help me to resolve this. Thanks & Regards, Akreeti Agarwal ::DISCLAIMER:: ___

Re: solr SSL encryption degardes solr performance

2019-02-06 Thread Zheng Lin Edwin Yeo
Hi, Which Solr version are you using? Also, how many collections do you have, and how many records have you indexed in those collections? Regards, Edwin On Mon, 4 Feb 2019 at 23:33, Anchal Sharma2 wrote: > > > Hi All, > > We had recently enabled SSL on solr. But afterwards ,our application >

solr SSL encryption degardes solr performance

2019-02-04 Thread Anchal Sharma2
Hi All, We had recently enabled SSL on solr. But afterwards ,our application performance has degraded significantly i.e the time for the source application to fetch a record from solr has increased from approx 4 ms to 200 ms(this is for a single record) .This amounts to a lot of time ,when mu

Re: SOLR Performance Statistics

2018-11-21 Thread Shawn Heisey
On 11/21/2018 8:59 AM, Marc Schöchlin wrote: Is it possible to modify the log4j appender to also log other query attributes like response/request size in bytes and number of resulted documents? Changing the log4j config might not do anything useful at all.  In order for such a change to be us

SOLR Performance Statistics

2018-11-21 Thread Marc Schöchlin
Hello list, i am using the pretty old solr 4.7 *sigh* release and i am currently in investigation of performance problems. The solr instance runs currently very expensive queries with huge results and i want to find the most promising queries for optimization. I am currently using the solr logf

Re: Live publishing and solr performance optimization

2018-11-20 Thread Zheng Lin Edwin Yeo
Sharding can be one of the option. But what is the size of your documents? And which Solr version are you using? Regards, Edwin On Tue, 20 Nov 2018 at 01:40, Balanathagiri Ayyasamypalanivel < bala.cit...@gmail.com> wrote: > Hi, > We are in the process for live Publishing document in solr and th

Live publishing and solr performance optimization

2018-11-19 Thread Balanathagiri Ayyasamypalanivel
Hi, We are in the process for live Publishing document in solr and the same time we have to maintain the search performance. Total existing docs : 120 million Expected data for live publishing : 1 million For every 1 hour, we will get 1m docs to publish in live to the hot solr collection, can you

Re: Solr performance issue

2018-02-15 Thread Shawn Heisey
On 2/15/2018 2:00 AM, Srinivas Kashyap wrote: > I have implemented 'SortedMapBackedCache' in my SqlEntityProcessor for the > child entities in data-config.xml. And i'm using the same for full-import > only. And in the beginning of my implementation, i had written delta-import > query to index th

Re: Solr performance issue

2018-02-15 Thread Erick Erickson
Srinivas: Not an answer to your question, but when DIH starts getting this complicated, I start to seriously think about SolrJ, see: https://lucidworks.com/2012/02/14/indexing-with-solrj/ IN particular, it moves the heavy lifting of acquiring the data from a Solr node (which I'm assuming also has

Solr performance issue

2018-02-15 Thread Srinivas Kashyap
Hi, I have implemented 'SortedMapBackedCache' in my SqlEntityProcessor for the child entities in data-config.xml. And i'm using the same for full-import only. And in the beginning of my implementation, i had written delta-import query to index the modified changes. But my requirement grew and i

Re: Solr performance issue on querying --> Solr 6.5.1

2017-09-30 Thread sasarun
Hi Erick, As suggested, I did try nonHDFS solr cloud instance and it response looks to be really better. From the configuration side to, I am mostly using default configurations and with block.cache.direct.memory.allocation as false. On analysis of hdfs cache, evictions seems to be on higher sid

Re: Solr performance issue on querying --> Solr 6.5.1

2017-09-27 Thread Emir Arnautović
Hi Arun, It is hard to measure something without affecting it, but we could use debug results and combine with QTime without debug: If we ignore merging results, it seems that majority of time is spent for retrieving docs (~500ms). You should consider reducing number of rows if you want better r

Re: Solr performance issue on querying --> Solr 6.5.1

2017-09-27 Thread sasarun
Hi Emir, Please find the response without bq parameter and debugQuery set to true. Also it was noted that Qtime comes down drastically without the debug parameter to about 700-800. true 0 3446 ("hybrid electric powerplant" "hybrid electric powerplants" "Electric" "Electrical" "Electricity"

Re: Solr performance issue on querying --> Solr 6.5.1

2017-09-27 Thread sasarun
Hi Erick, Qtime comes down with rows set as 1. Also it was noted that qtime comes down when debug parameter is not added with the query. It comes to about 900. Thanks, Arun -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr performance issue on querying --> Solr 6.5.1

2017-09-27 Thread Toke Eskildsen
On Tue, 2017-09-26 at 07:43 -0700, sasarun wrote: > Allocated heap size for young generation is about 8 gb and old  > generation is about 24 gb. And gc analysis showed peak > size utlisation is really low compared to these values. That does not come as a surprise. Your collections would normally b

Re: Solr performance issue on querying --> Solr 6.5.1

2017-09-27 Thread Emir Arnautović
Hi Arun, This is not the most simple query either - a dozen of phrase queries on several fields + the same query as bq. Can you provide debugQuery info. I did not look much into debug times and what includes what, but one thing that is strange to me is that QTime is 4s while query in debug is 1.3

Re: Solr performance issue on querying --> Solr 6.5.1

2017-09-26 Thread Erick Erickson
Well, 15 second responses are not what I'd expect either. But two things (just looked again) 1> note that the time to assemble the debug information is a large majority of your total time (14 of 15.3 seconds). 2> you're specifying 600 rows which is quite a lot as each one requires that a 16K block

Re: Solr performance issue on querying --> Solr 6.5.1

2017-09-26 Thread sasarun
Hi Erick, Thank you for the quick response. Query time was relatively faster once it is read from memory. But personally I always felt response time could be far better. As suggested, We will try and set up in a non HDFS environment and update on the results. Thanks, Arun -- Sent from: http

Re: Solr performance issue on querying --> Solr 6.5.1

2017-09-26 Thread Erick Erickson
Does the query time _stay_ low? Once the data is read from HDFS it should pretty much stay in memory. So my question is whether, once Solr warms up you see this kind of query response time. Have you tried this on a non HDFS system? That would be useful to help figure out where to look. And given

Solr performance issue on querying --> Solr 6.5.1

2017-09-26 Thread sasarun
Hi All, I have been using Solr for some time now but mostly in standalone mode. Now my current project is using Solr 6.5.1 hosted on hadoop. My solrconfig.xml has the following configuration. In the prod environment the performance on querying seems to really slow. Can anyone help me with few poin

Re: Suggestions on scaling up Solr performance.

2017-05-11 Thread Erick Erickson
Impossible to answer as Shawn says. Or even recommend. For instance, you say "but once we launch our application all across the world it may give performance issues." You haven't defined at all what changes when you "launch our application all across the world". Increasing your query traffic 10 fo

Re: Suggestions on scaling up Solr performance.

2017-05-11 Thread Venkateswarlu Bommineni
Thanks, Shawn. As of now, we don't have any performance issues, We are just working for the future purpose. So I was looking for any general architecture which is agreed by many of Solr experts. Thanks, Venkat. On Thu, May 11, 2017 at 8:19 PM, Shawn Heisey wrote: > On 5/11/2017 7:39 AM, Venka

Re: Suggestions on scaling up Solr performance.

2017-05-11 Thread Shawn Heisey
On 5/11/2017 7:39 AM, Venkateswarlu Bommineni wrote: > In current design we have below configuration: *One collection with > one shard with 4 replication factor with 4 nodes.* as of now, it is > working fine.but once we launch our application all across the world > it may give performance issues. T

Suggestions on scaling up Solr performance.

2017-05-11 Thread Venkateswarlu Bommineni
Hello Guys, In current design we have below configuration: *One collection with one shard with 4 replication factor with 4 nodes.* as of now, it is working fine.but once we launch our application all across the world it may give performance issues. To improve the performance below is our thoug

Re: Solr performance on EC2 linux

2017-05-03 Thread Walter Underwood
Already have a Jira issue for next week. I have a script to run prod logs against a cluster. I’ll be testing a four shard by two replica cluster with 17 million docs and very long queries. We are working on getting the 95th percentile under one second, so we should exercise the timeAllowed featu

Re: Solr performance on EC2 linux

2017-05-03 Thread Rick Leir
+Walter test it Jeff, How much CPU does the EC2 hypervisor use? I have heard 5% but that is for a normal workload, and is mostly consumed during system calls or context changes. So it is quite understandable that frequent time calls would take a bigger bite in the AWS cloud compared to bare met

Re: Solr performance on EC2 linux

2017-05-03 Thread Jeff Wartes
It’s presumably not a small degradation - this guy very recently suggested it’s 77% slower: https://blog.packagecloud.io/eng/2017/03/08/system-calls-are-much-slower-on-ec2/ The other reason that blog post is interesting to me is that his benchmark utility showed the work of entering the kernel

Re: Solr performance on EC2 linux

2017-05-02 Thread Tomás Fernández Löbbe
I remember seeing some performance impact (even when not using it) and it was attributed to the calls to System.nanoTime. See SOLR-7875 and SOLR-7876 (fixed for 5.3 and 5.4). Those two Jiras fix the impact when timeAllowed is not used, but I don't know if there were more changes to improve the perf

Re: Solr performance on EC2 linux

2017-05-02 Thread Walter Underwood
Hmm, has anyone measured the overhead of timeAllowed? We use it all the time. If nobody has, I’ll run a benchmark with and without it. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On May 2, 2017, at 9:52 AM, Chris Hostetter wrote: > > > : I speci

Re: Solr performance on EC2 linux

2017-05-02 Thread Chris Hostetter
: I specify a timeout on all queries, Ah -- ok, yeah -- you mean using "timeAllowed" correct? If the root issue you were seeing is in fact clocksource related, then using timeAllowed would probably be a significant compounding factor there since it would involve a lot of time checks in a s

Re: Solr performance on EC2 linux

2017-05-01 Thread Jeff Wartes
Yes, that’s the Xenial I tried. Ubuntu 16.04.2 LTS. On 5/1/17, 7:22 PM, "Will Martin" wrote: Ubuntu 16.04 LTS - Xenial (HVM) Is this your Xenial version? On 5/1/2017 6:37 PM, Jeff Wartes wrote: > I tried a few variations of various things before we found

Re: Solr performance on EC2 linux

2017-05-01 Thread Jeff Wartes
I started with the same three-node 15-shard configuration I’d been used to, in an RF1 cluster. (the index is almost 700G so this takes three r4.8xlarge’s if I want to be entirely memory-resident) I eventually dropped down to a 1/3rd size index on a single node (so 5 shards, 100M docs each) so I

Re: Solr performance on EC2 linux

2017-05-01 Thread Will Martin
Ubuntu 16.04 LTS - Xenial (HVM) Is this your Xenial version? On 5/1/2017 6:37 PM, Jeff Wartes wrote: > I tried a few variations of various things before we found and tried that > linux/EC2 tuning page, including: >- EC2 instance type: r4, c4, and i3 >- Ubuntu version: Xenial and Trust

Re: Solr performance on EC2 linux

2017-05-01 Thread Walter Underwood
Might want to measure the single CPU performance of your EC2 instance. The last time I checked, my MacBook was twice as fast as the EC2 instance I was using. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On May 1, 2017, at 6:24 PM, Chris Hostetter w

Re: Solr performance on EC2 linux

2017-05-01 Thread Chris Hostetter
: tldr: Recently, I tried moving an existing solrcloud configuration from : a local datacenter to EC2. Performance was roughly 1/10th what I’d : expected, until I applied a bunch of linux tweaks. How many total nodes in your cluster? How many of them running ZooKeeper? Did you observe the hea

Re: Solr performance on EC2 linux

2017-05-01 Thread Jeff Wartes
I tried a few variations of various things before we found and tried that linux/EC2 tuning page, including: - EC2 instance type: r4, c4, and i3 - Ubuntu version: Xenial and Trusty - EBS vs local storage - Stock openjdk vs Zulu openjdk (Recent java8 in both cases - I’m aware of the issues

Re: Solr performance on EC2 linux

2017-05-01 Thread John Bickerstaff
It's also very important to consider the type of EC2 instance you are using... We settled on the R4.2XL... The R series is labeled "High-Memory" Which instance type did you end up using? On Mon, May 1, 2017 at 8:22 AM, Shawn Heisey wrote: > On 4/28/2017 10:09 AM, Jeff Wartes wrote: > > tldr:

Re: Solr performance on EC2 linux

2017-05-01 Thread Shawn Heisey
On 4/28/2017 10:09 AM, Jeff Wartes wrote: > tldr: Recently, I tried moving an existing solrcloud configuration from a > local datacenter to EC2. Performance was roughly 1/10th what I’d expected, > until I applied a bunch of linux tweaks. How very strange. I knew virtualization would have overhe

Re: Solr performance on EC2 linux

2017-04-30 Thread Jeff Wartes
I’d like to think I helped a little with the metrics upgrade that got released in 6.4, so I was already watching that and I’m aware of the resulting performance issue. This was 5.4 though, patched with https://github.com/whitepages/SOLR-4449 - an index we’ve been running for some time now. Mgan

Re: Solr performance on EC2 linux

2017-04-29 Thread mganeshs
We use Solr 6.2 in EC2 instance with Cent OS 6.2 and we don't see any difference in performance between EC2 and in local environment. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-performance-on-EC2-linux-tp4332467p4332553.html Sent from the Solr - User mailing

Re: Solr performance on EC2 linux

2017-04-28 Thread Erick Erickson
Well, 6.4.0 had a pretty severe performance issue so if you were using that release you might see this, 6.4.2 is the most recent 6.4 release. But I have no clue how changing linux settings would alter that and I sure can't square that issue with you having such different performance between local a

Solr performance on EC2 linux

2017-04-28 Thread Jeff Wartes
tldr: Recently, I tried moving an existing solrcloud configuration from a local datacenter to EC2. Performance was roughly 1/10th what I’d expected, until I applied a bunch of linux tweaks. This should’ve been a straight port: one datacenter server -> one EC2 node. Solr 5.4, Solrcloud, Ubuntu

RE: Solr performance issue on indexing

2017-04-04 Thread Allison, Timothy B.
> Also we will try to decouple tika to solr. +1 -Original Message- From: tstusr [mailto:ulfrhe...@gmail.com] Sent: Friday, March 31, 2017 4:31 PM To: solr-user@lucene.apache.org Subject: Re: Solr performance issue on indexing Hi, thanks for the feedback. Yes, it is about OOM, ind

Re: Solr performance issue on indexing

2017-03-31 Thread Erick Erickson
decouple tika to > solr. > > By the way, make it available with solr cloud will improve performance? Or > there will be no perceptible improvement? > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Solr-performance-issue-on-indexing-tp4327886p4327914.html > Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr performance issue on indexing

2017-03-31 Thread tstusr
ble improvement? -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-performance-issue-on-indexing-tp4327886p4327914.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr performance issue on indexing

2017-03-31 Thread Erick Erickson
with 4gb of JVM > Memory and ~50gb of physical memory (reported through dashboard) we are > using a single instance. > > I don't think is a normal behaviour that handler crashes. So, what are some > general tips about improving performance for this scenario? > > > > --

Solr performance issue on indexing

2017-03-31 Thread tstusr
andler crashes. So, what are some general tips about improving performance for this scenario? -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-performance-issue-on-indexing-tp4327886.html Sent from the Solr - User mailing list archive at Nabble.com.

RE: solr | performance warning

2016-11-21 Thread Prateek Jain J
Thanks EricK Regards, Prateek Jain -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: 21 November 2016 04:32 PM To: solr-user Subject: Re: solr | performance warning _when_ are you seeing this? I see this on startup upon occasion, and I _think_ there

Re: solr | performance warning

2016-11-21 Thread Erick Erickson
_when_ are you seeing this? I see this on startup upon occasion, and I _think_ there's a JIRA about startup opening more than one searcher on startup. If it _is_ on startup, you can simply ignore it. If it's after the system is up and running, then you're probably committing too frequently. "Too f

solr | performance warning

2016-11-21 Thread Prateek Jain J
Hi All, I am observing following error in logs, any clues about this: 2016-11-06T23:15:53.066069+00:00@solr@@ org.apache.solr.core.SolrCore:1650 - [my_custom_core] PERFORMANCE WARNING: Overlapping onDeckSearchers=2 Slight web search suggests that it could be a case of too-frequent commits. I

Re: Disable hyper-threading for better Solr performance?

2016-03-09 Thread Ilan Schwarts
> > Sent: Wednesday, March 09, 2016 9:09 AM > > To: solr-user@lucene.apache.org > > Subject: Re: Disable hyper-threading for better Solr performance? > > > > What is the solr version and shard config? Standalone? Multiple cores? > > Spread over RAID ? > &g

Re: Disable hyper-threading for better Solr performance?

2016-03-09 Thread Vincenzo D'Amore
iginal Message- > From: Ilan Schwarts [mailto:ila...@gmail.com] > Sent: Wednesday, March 09, 2016 9:09 AM > To: solr-user@lucene.apache.org > Subject: Re: Disable hyper-threading for better Solr performance? > > What is the solr version and shard config? Standalone? Multiple co

RE: Disable hyper-threading for better Solr performance?

2016-03-09 Thread Avner Levy
reads will perform better. Thanks, Avner -Original Message- From: Ilan Schwarts [mailto:ila...@gmail.com] Sent: Wednesday, March 09, 2016 9:09 AM To: solr-user@lucene.apache.org Subject: Re: Disable hyper-threading for better Solr performance? What is the solr version and shard c

RE: Disable hyper-threading for better Solr performance?

2016-03-09 Thread Markus Jelsma
ssage- > From:Avner Levy > Sent: Wednesday 9th March 2016 8:00 > To: solr-user@lucene.apache.org > Subject: Disable hyper-threading for better Solr performance? > > I have a machine with 16 real cores (32 with HT enabled). > I'm running on it a Solr server and trying to reach maxi

Re: Disable hyper-threading for better Solr performance?

2016-03-08 Thread Ilan Schwarts
What is the solr version and shard config? Standalone? Multiple cores? Spread over RAID ? On Mar 9, 2016 9:00 AM, "Avner Levy" wrote: > I have a machine with 16 real cores (32 with HT enabled). > I'm running on it a Solr server and trying to reach maximum performance > for indexing and queries (i

Disable hyper-threading for better Solr performance?

2016-03-08 Thread Avner Levy
I have a machine with 16 real cores (32 with HT enabled). I'm running on it a Solr server and trying to reach maximum performance for indexing and queries (indexing 20k documents/sec by a number of threads). I've read on multiple places that in some scenarios / products disabling the hyper-thread

Re: solr performance issue

2016-02-09 Thread Zheng Lin Edwin Yeo
where the issue is. > > >> > > >> Thanks, > > >> Emir > > >> > > >> -- > > >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management > > >> Solr & Elasticsearch Support * http://sematext.com/ > > &g

Re: solr performance issue

2016-02-08 Thread Susheel Kumar
>> > >> Thanks, > >> Emir > >> > >> -- > >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management > >> Solr & Elasticsearch Support * http://sematext.com/ > >> > >> > >> > >> On 08.02.2016 10:27,

Re: solr performance issue

2016-02-08 Thread sara hajili
; Thanks, >> Emir >> >> -- >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management >> Solr & Elasticsearch Support * http://sematext.com/ >> >> >> >> On 08.02.2016 10:27, sara hajili wrote: >> >>> hi all.

Re: solr performance issue

2016-02-08 Thread Emir Arnautovic
nomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On 08.02.2016 10:27, sara hajili wrote: hi all. i have a problem with my solr performance and usage hardware like a ram,cup... i have a lot of document and so indexed file about 1000 doc in solr

Re: solr performance issue

2016-02-08 Thread sara hajili
ch Support * http://sematext.com/ > > > > On 08.02.2016 10:27, sara hajili wrote: > >> hi all. >> i have a problem with my solr performance and usage hardware like a >> ram,cup... >> i have a lot of document and so indexed file about 1000 doc in solr t

Re: solr performance issue

2016-02-08 Thread Emir Arnautovic
* Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On 08.02.2016 10:27, sara hajili wrote: hi all. i have a problem with my solr performance and usage hardware like a ram,cup... i have a lot of document and so indexed file about 1000 doc in solr that every doc has a

solr performance issue

2016-02-08 Thread sara hajili
hi all. i have a problem with my solr performance and usage hardware like a ram,cup... i have a lot of document and so indexed file about 1000 doc in solr that every doc has about 8 field in average. and each field has about 60 char. i set my field as a storedfield = "false" except o

Re: Solr performance is slow with just 1GB of data indexed

2015-08-26 Thread Zheng Lin Edwin Yeo
Thanks for your recommendation Toke. Will try to ask in the carrot forum. Regards, Edwin On 26 August 2015 at 18:45, Toke Eskildsen wrote: > On Wed, 2015-08-26 at 15:47 +0800, Zheng Lin Edwin Yeo wrote: > > > Now I've tried to increase the carrot.fragSize to 75 and > > carrot.summarySnippets t

Re: Solr performance is slow with just 1GB of data indexed

2015-08-26 Thread Toke Eskildsen
On Wed, 2015-08-26 at 15:47 +0800, Zheng Lin Edwin Yeo wrote: > Now I've tried to increase the carrot.fragSize to 75 and > carrot.summarySnippets to 2, and set the carrot.produceSummary to > true. With this setting, I'm mostly able to get the cluster results > back within 2 to 3 seconds when I set

Re: Solr performance is slow with just 1GB of data indexed

2015-08-26 Thread Zheng Lin Edwin Yeo
Hi Toke, Thank you for the link. I'm using Solr 5.2.1 but I think the carrot2 bundled will be slightly older version, as I'm using the latest carrot2-workbench-3.10.3, which is only released recently. I've changed all the settings like fragSize and desiredCluserCountBase to be the same on both si

Re: Solr performance is slow with just 1GB of data indexed

2015-08-25 Thread Toke Eskildsen
On Wed, 2015-08-26 at 10:10 +0800, Zheng Lin Edwin Yeo wrote: > I'm currently trying out on the Carrot2 Workbench and get it to call Solr > to see how they did the clustering. Although it still takes some time to do > the clustering, but the results of the cluster is much better than mine. I > thin

Re: Solr performance is slow with just 1GB of data indexed

2015-08-25 Thread Zheng Lin Edwin Yeo
Hi Toke, Thank you for your reply. I'm currently trying out on the Carrot2 Workbench and get it to call Solr to see how they did the clustering. Although it still takes some time to do the clustering, but the results of the cluster is much better than mine. I think its probably due to the differe

Re: Solr performance is slow with just 1GB of data indexed

2015-08-25 Thread Toke Eskildsen
On Tue, 2015-08-25 at 10:40 +0800, Zheng Lin Edwin Yeo wrote: > Would like to confirm, when I set rows=100, does it mean that it only build > the cluster based on the first 100 records that are returned by the search, > and if I have 1000 records that matches the search, all the remaining 900 > rec

Re: Solr performance is slow with just 1GB of data indexed

2015-08-24 Thread Zheng Lin Edwin Yeo
Thank you Upayavira for your reply. Would like to confirm, when I set rows=100, does it mean that it only build the cluster based on the first 100 records that are returned by the search, and if I have 1000 records that matches the search, all the remaining 900 records will not be considered for c

Re: Solr performance is slow with just 1GB of data indexed

2015-08-24 Thread Upayavira
I honestly suspect your performance issue is down to the number of terms you are passing into the clustering algorithm, not to memory usage as such. If you have *huge* documents and cluster across them, performance will be slower, by definition. Clustering is usually done offline, for example on a

Re: Solr performance is slow with just 1GB of data indexed

2015-08-23 Thread Zheng Lin Edwin Yeo
Hi Alexandre, I've tried to use just index=true, and the speed is still the same and not any faster. If I set to store=false, there's no results that came back with the clustering. Is this due to the index are not stored, and the clustering requires indexed that are stored? I've also increase my

Re: Solr performance is slow with just 1GB of data indexed

2015-08-23 Thread Zheng Lin Edwin Yeo
Yes, I'm using store=true. However, this field needs to be stored as my program requires this field to be returned during normal searching. I tried the lazyLoading=true, but it's not working. Will you do a copy field for the content, and not to set stored="true" for that field. So that field wil

Re: Solr performance is slow with just 1GB of data indexed

2015-08-23 Thread Bill Bell
We use 8gb to 10gb for those size indexes all the time. Bill Bell Sent from mobile > On Aug 23, 2015, at 8:52 AM, Shawn Heisey wrote: > >> On 8/22/2015 10:28 PM, Zheng Lin Edwin Yeo wrote: >> Hi Shawn, >> >> Yes, I've increased the heap size to 4GB already, and I'm using a machine >> with 32

Re: Solr performance is slow with just 1GB of data indexed

2015-08-23 Thread Jimmy Lin
unsubscribe On Sat, Aug 22, 2015 at 9:31 PM, Zheng Lin Edwin Yeo wrote: > Hi, > > I'm using Solr 5.2.1, and I've indexed about 1GB of data into Solr. > > However, I find that clustering is exceeding slow after I index this 1GB of > data. It took almost 30 seconds to return the cluster results wh

Re: Solr performance is slow with just 1GB of data indexed

2015-08-23 Thread Upayavira
And be aware that I'm sure the more terms in your documents, the slower clustering will be. So it isn't just the number of docs, the size of them counts in this instance. A simple test would be to build an index with just the first 1000 terms of your clustering fields, and see if that makes a diff

Re: Solr performance is slow with just 1GB of data indexed

2015-08-23 Thread Erick Erickson
You're confusing clustering with searching. Sure, Solr can index and lots of data, but clustering is essentially finding ad-hoc similarities between arbitrary documents. It must take each of the documents in the result size you specify from your result set and try to find commonalities. For perf i

Re: Solr performance is slow with just 1GB of data indexed

2015-08-23 Thread Alexandre Rafalovitch
Are you by any chance doing store=true on the fields you want to search? If so, you may want to switch to just index=true. Of course, they will then not come back in the results, but do you really want to sling huge content fields around. The other option is to do lazyLoading=true and not request

Re: Solr performance is slow with just 1GB of data indexed

2015-08-23 Thread Zheng Lin Edwin Yeo
Hi Shawn and Toke, I only have 520 docs in my data, but each of the documents is quite big in size, In the Solr, it is using 221MB. So when i set to read from the top 1000 rows, it should just be reading all the 520 docs that are indexed? Regards, Edwin On 23 August 2015 at 22:52, Shawn Heisey

Re: Solr performance is slow with just 1GB of data indexed

2015-08-23 Thread Shawn Heisey
On 8/22/2015 10:28 PM, Zheng Lin Edwin Yeo wrote: > Hi Shawn, > > Yes, I've increased the heap size to 4GB already, and I'm using a machine > with 32GB RAM. > > Is it recommended to further increase the heap size to like 8GB or 16GB? Probably not, but I know nothing about your data. How many So

Re: Solr performance is slow with just 1GB of data indexed

2015-08-23 Thread Toke Eskildsen
Zheng Lin Edwin Yeo wrote: > However, I find that clustering is exceeding slow after I index this 1GB of > data. It took almost 30 seconds to return the cluster results when I set it > to cluster the top 1000 records, and still take more than 3 seconds when I > set it to cluster the top 100 record

Re: Solr performance is slow with just 1GB of data indexed

2015-08-22 Thread Zheng Lin Edwin Yeo
Hi Shawn, Yes, I've increased the heap size to 4GB already, and I'm using a machine with 32GB RAM. Is it recommended to further increase the heap size to like 8GB or 16GB? Regards, Edwin On 23 Aug 2015 10:23, "Shawn Heisey" wrote: > On 8/22/2015 7:31 PM, Zheng Lin Edwin Yeo wrote: > > I'm usin

Re: Solr performance is slow with just 1GB of data indexed

2015-08-22 Thread Shawn Heisey
On 8/22/2015 7:31 PM, Zheng Lin Edwin Yeo wrote: > I'm using Solr 5.2.1, and I've indexed about 1GB of data into Solr. > > However, I find that clustering is exceeding slow after I index this 1GB of > data. It took almost 30 seconds to return the cluster results when I set it > to cluster the top

  1   2   3   4   5   >