Re: SolrCloud 6.6.2 suddenly crash due to slow queries and Log4j issue

Shawn, According to the log4j description ( https://bz.apache.org/bugzilla/show_bug.cgi?id=57714), the issue is related to lock during appenders collection process. In addition to CONSOLE and file appenders in the default log4j.properties, my customer added 2 extra FileAppender dedicated to all

Re: SolrCloud 6.6.2 suddenly crash due to slow queries and Log4j issue

Hi Shawn, Thank you for your response. You are confirming my diagnosis. This is in fact a 8 nodes cluster with one single collection with 4 shards and 1 replica (8 cores). 4 Gb heap and 90 Gb Ram When no issue occurs nearly 50% of the heap is used. Num Docs in collection : 10.000.000 Num

Re: SolrCloud 6.6.2 suddenly crash due to slow queries and Log4j issue

On 10/18/2020 3:22 AM, Dominique Bejean wrote: A few months ago, I reported an issue with Solr nodes crashing due to the old generation heap growing suddenly and generating OOM. This problem occurred again this week. I have threads dumps for each minute during the 3 minutes the problem occured.

SolrCloud 6.6.2 suddenly crash due to slow queries and Log4j issue

cute(AbstractHttpClient.java:816) - waiting to lock <0x00070087b910> (a org.apache.solr.util.stats.InstrumentedHttpClient) === 15h58 -> Young Gen heap full -> OOM The global scenario is (6 solr nodes) 1/ The problem starts with very slow queries (30 seconds to 2 minutes) on all

Re: Slow queries until core is reindexed

I'm pretty sure we found the problem. It's related to memory. Sometimes Windows seems to unmap index files from memory, because other processes need it. To force Windows to map index files again, we need to rebuild the index. We can clearly see this behaviour with tools like RAMMap. With servers

RE: Log slow queries to SQL Database using Log4j2 (JDBC)

Message- From: Walter Underwood Sent: Dienstag, 26. Mai 2020 02:06 To: solr-user@lucene.apache.org Subject: Re: Log slow queries to SQL Database using Log4j2 (JDBC) I would back up and do this a different way, with off-the-shelf parts. Send the logs to syslog or your favorite log aggregator. From

Re: Log slow queries to SQL Database using Log4j2 (JDBC)

I would back up and do this a different way, with off-the-shelf parts. Send the logs to syslog or your favorite log aggregator. From there, configure something that puts them into an ELK stack (Elasticsearch, Logstash, Kibana). A commercial version of this is logz.io .

Log slow queries to SQL Database using Log4j2 (JDBC)

Hi everyone, For our Solr instance I have the requirement that all queries should be logged, so that we can later on analyze, which search texts were queried most often. Were using solr 8.3.1 using the official docker image, hosted on Azure. My approach for implementing this, was now to

Slow queries until core is reindexed

Hi all, We have a solr 8.4.1 server running on Windows. (Very simple setup.) 16GB RAM / JVM-Mem set to 4GB Solr host 4 cores. (2 GB + 1GB + 75MB + 75MB) Full data import every night. No delta import. This server is used for tests by 2 people. (very low request rate) We have an issue we don't

Re: CommonTerms & slow queries

> > All great advice thanks Michael, have an excellent weekend! Testing the > common grams > -Craig

Re: CommonTerms & slow queries

fast <1 sec however we have noticed queries > > > containing "common" words are quite slow sometimes 10+sec , currently > > using > > > edismax with 2 text_general fields,. qf, and pf, qs=0,ps=0 > > > > > > I came across these which describe the

Re: CommonTerms & slow queries

" words are quite slow sometimes 10+sec , currently > using > > edismax with 2 text_general fields,. qf, and pf, qs=0,ps=0 > > > > I came across these which describe the issue. > > > > > https://www.hathitrust.org/blogs/large-scale-search/slow-queries-and-common-words-pa

Re: CommonTerms & slow queries

re quite slow sometimes 10+sec , currently using > edismax with 2 text_general fields,. qf, and pf, qs=0,ps=0 > > I came across these which describe the issue. > > https://www.hathitrust.org/blogs/large-scale-search/slow-queries-and-common-words-part-2 > > > https://lu

CommonTerms & slow queries

l fields,. qf, and pf, qs=0,ps=0 I came across these which describe the issue. https://www.hathitrust.org/blogs/large-scale-search/slow-queries-and-common-words-part-2 https://lucene.apache.org/core/5_5_3/queries/org/apache/lucene/queries/CommonTermsQuery.html Test queries with issues : 1.

Re: slow queries

duct of:\n 1.0 > = > > queryNorm\n" }, "QParser": "ExtendedDismaxQParser", "altquerystring": > null, > > "boost_queries": null, "parsed_boost_queries": [], "boostfuncs": null, " > > filter_queries": [ "{!co

Re: slow queries

csQuery, product of:\n >> 1.0 >> >> = >> >> > queryNorm\n", "Product:3375982": "\n1.0 = (MATCH) MatchAllDocsQuery, >> >> > product of:\n 1.0 = queryNorm\n", "Product:38276777": "\n1.0 = (MATCH) >> >> > Matc

Re: slow queries

1.0 = queryNorm\n", "Product:13702130": >> > "\n1.0 >> > = (MATCH) MatchAllDocsQuery, product of:\n 1.0 = queryNorm\n", " >> > Product:25679953": "\n1.0 = (MATCH) MatchAllDocsQuery, product of:\n 1.0 >> = >> > queryNor

Re: slow queries

>> > "\n1.0 > >> > = (MATCH) MatchAllDocsQuery, product of:\n 1.0 = queryNorm\n", " > >> > Product:50827742": "\n1.0 = (MATCH) MatchAllDocsQuery, product of:\n > 1.0 > >> = > >> > queryNorm\n", "Product

Re: slow queries

. But are these slow queries constant or intermittent? In other words, are all queries of this general form slow or just the first few? In particular is the first query that mentions sorting on this field slow but subsequent ones faster? In that case consider adding a query to the newSearcher event

Re: slow queries

utoCommit.maxTime:15000} >false > > > > >${solr.autoSoftCommit.maxTime:60} > > > and the following stats for filters: > > lookups = 3602 > hits = 3148 > hit ratio = 0.87 > inserts = 455 > evictions = 400 > size = 63 > warmupTime = 770 > > *Problem: *a lot of

Re: slow queries

lr.autoCommit.maxTime:15000} >false > > > > >${solr.autoSoftCommit.maxTime:60} > > > and the following stats for filters: > > lookups = 3602 > hits = 3148 > hit ratio = 0.87 > inserts = 455 > evictions = 400

Re: slow queries

{!cost=1 cache=true}type_s:Product AND > is_valid_b:true", > "{!cost=50 cache=true}in_languages_t:de", "{!cost=99 > cache=false}(shipping_country_codes_mt: (DE OR EURO OR EUR OR ALL)) AND > (cents_ri: [* TO 3000])" ], "parsed_filter_queries": [ &qu

Re: slow queries

archer$2@169ee0fd) > > > > > > > >${solr.autoCommit.maxTime:15000} > >false > > > > > > > > > >${solr.autoSoftCommit.maxTime:60} > > > > > > and the following stats for f

Re: slow queries

ND is_valid_b:true", "{!cost=50 cache=true}in_languages_t:de", "{!cost=99 cache=false}(shipping_country_codes_mt: (DE OR EURO OR EUR OR ALL)) AND (cents_ri: [* TO 3000])" ], "parsed_filter_queries": [ "+type_s:Product +is_valid_b:true", "in_langua

Re: slow queries

;${solr.autoCommit.maxTime:15000} >> >false >> > >> > >> > >> > >> >${solr.autoSoftCommit.maxTime:60} >> > >> > >> > and the following stats for filters: >> > &g

slow queries

) ${solr.autoCommit.maxTime:15000} false ${solr.autoSoftCommit.maxTime:60} and the following stats for filters: lookups = 3602 hits = 3148 hit ratio = 0.87 inserts = 455 evictions = 400 size = 63 warmupTime = 770 *Problem: *a lot of slow queries, for example: {q=*:*=1.0

Re: Slow queries

, -- View this message in context: http://lucene.472066.n3.nabble.com/Slow-queries-tp4172032p4173092.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Slow queries

for the performance, SHould I use an independant tomcat engine? rgds, -- View this message in context: http://lucene.472066.n3.nabble.com/Slow-queries-tp4172032p4173092.html Sent from the Solr - User mailing list archive at Nabble.com.

Slow queries

to optimize the collection and speed up querying it is it normal with this volume of data? is sharding is a good solution? regards, -- View this message in context: http://lucene.472066.n3.nabble.com/Slow-queries-tp4172032.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Slow queries

solution? regards, -- View this message in context: http://lucene.472066.n3.nabble.com/Slow-queries-tp4172032.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Slow queries

this message in context: http://lucene.472066.n3.nabble.com/Slow-queries-tp4172032p4172039.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Slow queries

? rgds, -- View this message in context: http://lucene.472066.n3.nabble.com/Slow-queries-tp4172032p4172039.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Slow queries

? Is there any thing else that can be done to the solr instance for example deviding the collection? rgds, -- View this message in context: http://lucene.472066.n3.nabble.com/Slow-queries-tp4172032p4172039.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Slow queries for common terms

-West [mailto:tburt...@umich.edu] Sent: Saturday, March 23, 2013 1:38 AM To: solr-user@lucene.apache.org Subject: Re: Slow queries for common terms Hi David and Jan, I wrote the blog post, and David, you are right, the problem we had was with phrase queries because our positions lists are so

RE: Slow queries for common terms

] Sent: Monday, March 25, 2013 10:04 PM To: solr-user@lucene.apache.org Subject: Re: Slow queries for common terms take a look here: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html looking at memory consumption can be a bit tricky to interpret with MMapDirectory. But you say I

RE: Slow queries for common terms

-Original Message- From: Tom Burton-West [mailto:tburt...@umich.edu] Sent: Saturday, March 23, 2013 1:38 AM To: solr-user@lucene.apache.org Subject: Re: Slow queries for common terms Hi David and Jan, I wrote the blog post, and David, you are right, the problem we had was with phrase queries

Re: Slow queries for common terms

users. If you could elaborate a bit on your thinking I'd be quite grateful. David -Original Message- From: Jan Høydahl [mailto:jan@cominvent.com] Sent: Thursday, March 21, 2013 8:01 PM To: solr-user@lucene.apache.org Subject: Re: Slow queries for common terms Hi

Re: Slow queries for common terms

Hi David and Jan, I wrote the blog post, and David, you are right, the problem we had was with phrase queries because our positions lists are so huge. Boolean queries don't need to read the positions lists. I think you need to determine whether you are CPU bound or I/O bound.It is possible

Slow queries for common terms

I've got a query that takes 15 seconds to return whenever I have the term book in a query that isn't cached. That's a pretty common term in our search index. We're indexing about 120 GB of text data. We only store terms and IDs, no document data, and the disk is virtually unused, it's all CPU

Re: Slow queries for common terms

Hi, I think you can start by reading this blog http://www.hathitrust.org/blogs/large-scale-search/slow-queries-and-common-words-part-2 and try out the approach using a dictionary of the most common words in your index. You don't say how many documents, avg. doc size, the IDF value of book

RE: Slow queries for common terms

. -Original Message- From: Jan Høydahl [mailto:jan@cominvent.com] Sent: Thursday, March 21, 2013 5:43 PM To: solr-user@lucene.apache.org Subject: Re: Slow queries for common terms Hi, I think you can start by reading this blog http://www.hathitrust.org/blogs/large-scale-search/slow

Re: Slow queries for common terms

] Sent: Thursday, March 21, 2013 5:43 PM To: solr-user@lucene.apache.org Subject: Re: Slow queries for common terms Hi, I think you can start by reading this blog http://www.hathitrust.org/blogs/large-scale-search/slow-queries-and-common-w ords-part-2 and try out the approach using

Slow queries for common terms

I've got a query that takes 15 seconds to return whenever I have the term book in a query that isn't cached. That's a pretty common term in our search index. We're indexing about 120 GB of text data. We only store terms and IDs, no document data, and the disk is virtually unused, it's all CPU

Re: Slow queries for common terms

: Slow queries for common terms To: solr-user@lucene.apache.org solr-user@lucene.apache.org Date: Thursday, March 21, 2013, 11:40 AM I've got a query that takes 15 seconds to return whenever I have the term book in a query that isn't cached. That's a pretty common term in our search index

RE: Slow queries for common terms

for a few users. If you could elaborate a bit on your thinking I'd be quite grateful. David -Original Message- From: Jan Høydahl [mailto:jan@cominvent.com] Sent: Thursday, March 21, 2013 8:01 PM To: solr-user@lucene.apache.org Subject: Re: Slow queries for common terms Hi, If you say

Re: Dropping slow queries

in the shard, network issues between the shards, etc. thanks -- View this message in context: http://lucene.472066.n3.nabble.com/Dropping-slow-queries-tp4043074p4043592.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Dropping slow queries

On Wed, Feb 27, 2013 at 9:29 AM, adm1n evgeni.evg...@gmail.com wrote: Anybody? -- View this message in context: http://lucene.472066.n3.nabble.com/Dropping-slow-queries-tp4043074p4043387.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Dropping slow queries

Am I missing something? which answer do you mean? And actually, just to clarify, the question was how can I force the partial results and drop the results from the slow shard? -- View this message in context: http://lucene.472066.n3.nabble.com/Dropping-slow-queries-tp4043074p4043549.html Sent

Dropping slow queries

this message in context: http://lucene.472066.n3.nabble.com/Dropping-slow-queries-tp4043074.html Sent from the Solr - User mailing list archive at Nabble.com.

Very slow queries

Hello everyone, All of a sudden, I am experiencing some very slow queries with solr. I have 13GB of indexed documents, each averaging 50-100kb. They have an id key, so I expect to be getting results really fast if I execute id:7cd6cb99fd239c1d743a51bb85a48f790f4a6d3c as the query with no other

Re: Very slow queries

...@simpleweb.co.uk wrote: Hello everyone, All of a sudden, I am experiencing some very slow queries with solr. I have 13GB of indexed documents, each averaging 50-100kb. They have an id key, so I expect to be getting results really fast if I execute id:7cd6cb99fd239c1d743a51bb85a48f790f4a6d3c