Well.. it is strange that when I use the default GC I don't get any errors. If I'm so close to run out of memory I should see those OOM exceptions as well with the standard GC.BTW I'm faceting on around 13 fields and my total number of unique values is around 30000. One of the fields with the biggest amount of unique values has almost 16000 unique values.
On Sun, Sep 27, 2009 at 4:32 PM, Fuad Efendi <f...@efendi.ca> wrote: > Mark, > > > Nothing against orange-hat :) > > Nothing against GC tuning; but if SOLR needs application-specific settings > it should be well-documented. > > GC-tuning: for instance, we need it for 'realtime' Online Trading > applications. However, even Online Banking doesn't need; primary reason - > GC > must happen 'outside of current transaction', GC 'must be predictable', and > (for instance) Oracle/BEA JRockit has specific 'realtime' version for > that... Does SOLR need that? > > > Having load-stress simulator (multithreaded!!!) will definitely help to > predict any possible bottleneck... it's even better to write it from > scratch > (depends on schema!), by sending random requests to SOLR in-parallel... > instead of waiting when FieldCache tries to add new FieldImpl to cache > (unpredictable!) > > > Tomcat is multithreaded; what if end-users need to load 1000s large > documents (in parallel! 1000s concurrent users), can you predict memory > requirements and GC options without application-specific knowledge? What > about new SOLR-Caches warming up? > > > -Fuad > > > > -----Original Message----- > > From: Mark Miller [mailto:markrmil...@gmail.com] > > Sent: September-27-09 2:46 PM > > To: solr-user@lucene.apache.org > > Subject: Re: Solr and Garbage Collection > > > > If he needed double the RAM, he'd likely know by now :) The JVM likes to > > throw OOM exceptions when you need more RAM. Until it does - thats an > > odd path to focus on. There has been no indication he has ever seen an > > OOM with his over 10 GB heap. It sounds like he has run Solr in his > > environment for quite a long time - after running for that long, until > > he gets an OOM, its about as good as chasing ghost to worry about it. > > > > I like to think of GC tuning as orange-hat. Mostly because I like the > > color orange. > > > > Fuad Efendi wrote: > > >>> Ok. After the server ran for more than 12 hours, the time spent on GC > > >>> decreased from 11% to 3,4%, but 5 hours later it crashed. > > >>> > > > > > > All this 'black-hat' GC tuning and 'fast' object moving (especially > objects > > > accessing by some thread during GC-defragmentation) > > > > > > - try to use multithreaded load-stress tools (at least 100 requests > > > in-parallel) and see that you need at least double memory if 12Gb is > > > threshold for your FieldCache (largest objects) > > > > > > > > > Also, don't trust this counters: > > > > > >> So I logged the Garbage Collection activity to check if it's because > of > > >> that. It seems like 11% of the time the application runs, it is > stopped > > >> because of GC. > > >> > > > > > > > > > Stopped? Of course, locking/unlocking in order to move objects > currently > > > accessesd in multiuser-multithreaded Tomcat... you can easily create > crash > > > scenario proving that latest-greatest JVMs are buggy too. > > > > > > > > > > > > Don't forget: Tomcat is multithreaded, and if 'core' needs 10Gb in > order > to > > > avoid OOM, you need to double it (in order to warm new cash instances > on > > > index replica / update). > > > > > > > > > http://www.linkedin.com/in/liferay > > > > > > > > > > > > > > > -- > > - Mark > > > > http://www.lucidimagination.com > > > > > > > >
