About caches. The queryResultCache is only useful when you expect there to be a number of _identical_ queries. Think of this cache as a map where the key is the query and the value is just a list of N document IDs (internal) where N is your window size. Paging is often the place where this is used. Take a look at your admin page for this cache, you can see the hit rates. But, the take-away is that this is a very small cache memory-wise, varying it is probably not a great predictor of memory usage.
The filterCache is more intense memory wise, it's another map where the key is the fq clause and the value is bounded by maxDoc/8. Take a close look at this in the admin screen and see what the hit ratio is. It may be that you can make it much smaller and still get a lot of benefit. _Especially_ considering it could occupy about 44G of memory. (43,000,000 / 8) * 8192........ And the autowarm count is excessive in most cases from what I've seen. Cutting the autowarm down to, say, 16 may not make a noticeable difference in your response time. And if you're using NOW in your fq clauses, it's almost totally useless, see: http://searchhub.org/2012/02/23/date-math-now-and-filter-queries/ Also, read Uwe's excellent blog about MMapDirectory here: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html for some problems with over-allocating memory to the JVM. Of course if you're hitting OOMs, well..... bq: order them by one of their fields. This is one place I'd look first. How many unique values are in each field that you sort on? This is one of the major memory consumers. You can get a sense of this by looking at admin/schema-browser and selecting the fields you sort on. There's a text box with the number of terms returned, then a / ### where ### is the total count of unique terms in the field. NOTE: in 4.4 this will be -1 for multiValued fields, but you shouldn't be sorting on those anyway. How many fields are you sorting on anyway, and of what types? For your SolrCloud experiments, what are your soft and hard commit intervals? Because something is really screwy here. Your sharding moving the number of docs down this low per shard should be fast. Back to the point above, the only good explanation I can come up with from this remove is that the fields you sort on have a LOT of unique values. It's possible that the total number of unique values isn't scaling with sharding. That is, each shard may have, say, 90% of all unique terms (number from thin air). Worth checking anyway, but a stretch. This is definitely unusual... Best, Erick On Thu, Sep 19, 2013 at 8:20 AM, Neil Prosser <neil.pros...@gmail.com> wrote: > Apologies for the giant email. Hopefully it makes sense. > > We've been trying out SolrCloud to solve some scalability issues with our > current setup and have run into problems. I'd like to describe our current > setup, our queries and the sort of load we see and am hoping someone might > be able to spot the massive flaw in the way I've been trying to set things > up. > > We currently run Solr 4.0.0 in the old style Master/Slave replication. We > have five slaves, each running Centos with 96GB of RAM, 24 cores and with > 48GB assigned to the JVM heap. Disks aren't crazy fast (i.e. not SSDs) but > aren't slow either. Our GC parameters aren't particularly exciting, just > -XX:+UseConcMarkSweepGC. Java version is 1.7.0_11. > > Our index size ranges between 144GB and 200GB (when we optimise it back > down, since we've had bad experiences with large cores). We've got just > over 37M documents some are smallish but most range between 1000-6000 > bytes. We regularly update documents so large portions of the index will be > touched leading to a maxDocs value of around 43M. > > Query load ranges between 400req/s to 800req/s across the five slaves > throughout the day, increasing and decreasing gradually over a period of > hours, rather than bursting. > > Most of our documents have upwards of twenty fields. We use different > fields to store territory variant (we have around 30 territories) values > and also boost based on the values in some of these fields (integer ones). > > So an average query can do a range filter by two of the territory variant > fields, filter by a non-territory variant field. Facet by a field or two > (may be territory variant). Bring back the values of 60 fields. Boost query > on field values of a non-territory variant field. Boost by values of two > territory-variant fields. Dismax query on up to 20 fields (with boosts) and > phrase boost on those fields too. They're pretty big queries. We don't do > any index-time boosting. We try to keep things dynamic so we can alter our > boosts on-the-fly. > > Another common query is to list documents with a given set of IDs and > select documents with a common reference and order them by one of their > fields. > > Auto-commit every 30 minutes. Replication polls every 30 minutes. > > Document cache: > * initialSize - 32768 > * size - 32768 > > Filter cache: > * autowarmCount - 128 > * initialSize - 8192 > * size - 8192 > > Query result cache: > * autowarmCount - 128 > * initialSize - 8192 > * size - 8192 > > After a replicated core has finished downloading (probably while it's > warming) we see requests which usually take around 100ms taking over 5s. GC > logs show concurrent mode failure. > > I was wondering whether anyone can help with sizing the boxes required to > split this index down into shards for use with SolrCloud and roughly how > much memory we should be assigning to the JVM. Everything I've read > suggests that running with a 48GB heap is way too high but every attempt > I've made to reduce the cache sizes seems to wind up causing out-of-memory > problems. Even dropping all cache sizes by 50% and reducing the heap by 50% > caused problems. > > I've already tried using SolrCloud 10 shards (around 3.7M documents per > shard, each with one replica) and kept the cache sizes low: > > Document cache: > * initialSize - 1024 > * size - 1024 > > Filter cache: > * autowarmCount - 128 > * initialSize - 512 > * size - 512 > > Query result cache: > * autowarmCount - 32 > * initialSize - 128 > * size - 128 > > Even when running on six machines in AWS with SSDs, 24GB heap (out of 60GB > memory) and four shards on two boxes and three on the rest I still see > concurrent mode failure. This looks like it's causing ZooKeeper to mark the > node as down and things begin to struggle. > > Is concurrent mode failure just something that will inevitably happen or is > it avoidable by dropping the CMSInitiatingOccupancyFraction? > > If anyone has anything that might shove me in the right direction I'd be > very grateful. I'm wondering whether our set-up will just never work and > maybe we're expecting too much. > > Many thanks, > > Neil
