solr crashes

Hello all, We have a configuration with a single node with 30gb of RAM. We use it to index ~40MLN of documents. We perform queries with edismax parser that contain often edismax parser subqueries with the syntax '_query_:{!edismax mm=X v=$subqueryN}' Often X == 1. This solves the "too many

SOlr crashes

It crashes giving this output. =java.net.SocketException: Too many open files at java.net.ServerSocket.createImpl(ServerSocket.java:255) at java.net.ServerSocket.getImpl(ServerSocket.java:205) at java.net.ServerSocket.bind(ServerSocket.java:319) at java.net.ServerSocke

Re: solr crashes

Hi Danilo, you have to give more infos about your system and the config. - 30gb RAM (physical RAM?) how much heap do you have for JAVA? - how large (in GByte) are your 40 million raw data being indexed? - how large is your index (in GByte) with 40 million docs indexed? - which version of Solr an

Re: solr crashes

Hello Bernd, Here I list the extra info you requested: - actually the virtual machine has 22GB of RAM and 16GB of heap - my 40 million raw data takes about 1364GB on filesystem (in xml format) - my index optimized (1 segment, 0 deleted docs) takes about 555GB - solr 7.3, openjdk 1.8.0_181 -

Re: solr crashes

Hi Danilo, Full GC points out that you need more heap which also implies that you need more RAM. Raise your heap to 24GB and your physical RAM to about 75GB or better 96GB. RAM should be about 3 to 4 times heap size. Regards, Bernd Am 04.12.18 um 13:37 schrieb Danilo Tomasoni: Hello Bernd,

Re: solr crashes

Hello Bernd, Thanks for the suggestion, the problem is that we don't have 75 GB of RAM. Are you aware of any way to reduce solr memory usage? Thanks Danilo On 04/12/18 15:06, Bernd Fehling wrote: Hi Danilo, Full GC points out that you need more heap which also implies that you need more R

Re: solr crashes

you can set the -Xms value lower on startup but your still going to run into this issue. Really you just need to go buy more ram, hardware is cheap so you may as well max out the number of sockets for memory and get a couple TB sized SSD's. On Tue, Dec 4, 2018 at 10:47 AM Danilo Tomasoni wrote:

Re: solr crashes

Am 04.12.18 um 16:47 schrieb Danilo Tomasoni: Hello Bernd, Thanks for the suggestion, the problem is that we don't have 75 GB of RAM. Are you aware of any way to reduce solr memory usage? Yes, remove all Faceting, especially those for Fields with high cardinality. Don't use huge Synonym

Re: solr crashes

I’ve never heard a recommendation to have three times as much RAM as the heap. That doesn’t make sense to me. You might need 3X as much disk space as the index size. For RAM, it is best to have the sum of: * JVM heap * A couple of gigabytes for OS and demons * RAM for other processes needed on

Re: solr crashes

3x heap is larger than usual, but significant RAM beyond heap is a good idea if you can't fit the whole index in 31 GB of memory, since the OS will cache files in ram. Note also the use of 32 GB through about 45 GB heap settings gives you LESS heap than 31 GB due to an increase in pointer sizes nee

Re: solr crashes

But it is silly to base non-heap RAM on the size of the heap. Get the RAM needed for the non-heap usage. That has nothing to do with the size of the Java heap. Non-heap RAM is mostly used for two things: other programs and file buffers for the Solr indexes. Base the RAM needs on those. wunder Wal

Re: solr crashes

Am 05.12.18 um 17:11 schrieb Walter Underwood: I’ve never heard a recommendation to have three times as much RAM as the heap. That doesn’t make sense to me. https://wiki.apache.org/solr/SolrPerformanceProblems#RAM You might need 3X as much disk space as the index size. For RAM, it is be

Re: solr crashes

> On Dec 6, 2018, at 12:59 AM, Bernd Fehling > wrote: > > Am 05.12.18 um 17:11 schrieb Walter Underwood: >> I’ve never heard a recommendation to have three times as much RAM as the >> heap. That doesn’t make sense to me. > > https://wiki.apache.org/solr/SolrPerformanceProblems#RAM >

Re: solr crashes

On 12/3/2018 2:09 AM, Danilo Tomasoni wrote: Unfortunately in this scenario solr often crashes while performing a query, even with a single query and no other source of system load. What do you mean by "crashes"?  Because what I think of as a crash is exceedingly rare with Java programs.  I wo

RE: SOlr crashes

m: farhan diwan [mailto:[EMAIL PROTECTED] Sent: 14 August 2006 07:29 To: solr-user@lucene.apache.org Subject: SOlr crashes It crashes giving this output. =java.net.SocketException: Too many open files at java.net.ServerSocket.createImpl(ServerSocket.java:255) at java.net.Serve

RE: SOlr crashes

example for "useCompoundFile" : Date: Mon, 14 Aug 2006 07:36:00 +0100 : From: David Wiblin <[EMAIL PROTECTED]> : Reply-To: solr-user@lucene.apache.org : To: solr-user@lucene.apache.org : Subject: RE: SOlr crashes : : : A common problem with any server running lucene becasue it needs

Re: SOlr crashes

On 8/14/06, Chris Hostetter <[EMAIL PROTECTED]> wrote: Something else to consider is using the compound file format to reduce the number of files for your index. this is mentioned in the Lucen FAQ... Yeah, although unless you have a *lot* of fields with norms, I'd sooner reduce the mergeFacto