That makes sense about all bets being off. I wanted to make sure that people whose systems are behaving sensibly weren't going to have problems.
I think I need to tame the base amount of memory the field cache takes. We currently do boosting on several fields during most queries. We boost by at least 64 long fields (only one or two per query, but that many fields in total) so with a maxDocs of around 10m on each shard we're talking about nearly 5GB of heap just for the field cache. Then I still want to have a filter cache which protects me from chewing too much CPU during requests. I've seen a lot of concurrent mode failures flying by while staring at the GC log so I think I need to get GC to kick in a bit sooner and try to limit those (is it possible to eliminate them completely?). Performance is definitely suffering while that's happening. I had a go with G1 and no special options and things slowed down a bit. I'm beginning to understand what to look for in the CMS GC logs so I'll stick with that and see where I get to. Thanks for the link. I'll give it a proper read when I get into work tomorrow. I have a feeling that five shards might not be the sweet spot for the spec of the machines I'm running on. Our goal was to replace five 96GB physicals with 48GB heaps doing master/slave replication for an index which is at least 120GB in size. At the moment we're using ten VMs with 24GB of RAM, 8GB heaps and around 10GB of index. These machines are managing to get the whole index into the Linux OS cache. Hopefully the 5GB minimum for field cache and 8GB heap is what's causing this trouble right now. On 24 July 2013 19:06, Shawn Heisey <s...@elyograg.org> wrote: > On 7/24/2013 10:33 AM, Neil Prosser wrote: > > The log for server09 starts with it throwing OutOfMemoryErrors. At this > > point I externally have it listed as recovering. Unfortunately I haven't > > got the GC logs for either box in that time period. > > There's a lot of messages in this thread, so I apologize if this has > been dealt with already by previous email messages. > > All bets are off when you start throwing OOM errors. The state of any > java program pretty much becomes completely undefined when you run out > of heap memory. > > It just so happens that I just finished updating a wiki page about > reducing heap requirements for another message on the list. GC pause > problems have already been mentioned, so increasing the heap may not be > a real option here. Take a look at the following for ways to reduce > your heap requirements: > > > https://wiki.apache.org/solr/SolrPerformanceProblems#Reducing_heap_requirements > > Thanks, > Shawn > >
