Maybe completely wrong here, but I would be ready to bet that you see high thread churn
We were bitten by it many times at other places. We have some memory heavy analyzers and once you experiance high thread-churn gc() cannot keep up anymore. This was the reason to move to tomcat from jetty, because we know how to configure ThreadPools there to alleviate the problem (not perfect, just slightly better!)…. What Adrian said somehow points at similar pattern "reuses the buffer used for decompression across calls in the same thread" solr should really do something OOTB to limit the number of used threads and reuse them, but this is container "turf" and increases dependancy … On May 9, 2013, at 12:45 PM, Adrien Grand <[email protected]> wrote: > Hi Erick, > > The stored fields reader reuses the buffer used for decompression across > calls in the same thread, so I'm thinking that this kind of behaviour could > happen if some documents are very large. Is it the case? > > Adrien > > Le 8 mai 2013 17:22, "Erick Erickson" <[email protected]> a écrit : > I'm seeing a case where (reported to me, not on my personal machine) > where Solr's heap is being exhausted apparently by compressed field > reader. Here's a sample line from a summary of a heap dump: > > CompressingStoredFieldReader 735 instances > > where the CompressingStoredFieldReader instances are taking up 7.5G of > heap space (out of 8 allocated). The GC cycles are a pattern I've seen > before > gc for a long time, recover a few meg > run for a _very_ short time > gc again, recover a few meg > > This app happens to be doing a lot of atomic updates, running with CMS > collector, java 1.7. There is very little querying going on, and it is > happening on several different machines. Solr 4.2. > > Now, there is a bit of custom update code involved, we're testing whether > 1> if not compressing the stored fields changes things > 2> if disabling the custom code changes things > 3> if Solr 4.3 changes things. > 4> whether memory usage grows over time linearly or spikes (pretty > sure the former). > > This is an index-heavy/query-light application, and we're hosing a > _bunch_ of documents at the indexing process. But still, assuming that > the memory reporting above is accurate, each > CompressingStoredFieldReader is apparently taking up 10M each and > there are over 700 of them. The documents are being batched (don't > quite know how many per batch, will find out). > > Mainly, I'm asking if this rings any bells. I don't see anything on a > quick look at the JIRAs. > > Thanks, > Erick > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] >
