Hmmmm, I confess I haven't had a chance to play with this yet, but have you considered docValues for some of your fields? See: http://wiki.apache.org/solr/DocValues
And just to tantalize you: > Since Solr4.2 to build a forward index for a field, for purposes of sorting, > faceting, grouping, function queries, etc. > You can specify a different docValuesFormat on the fieldType > (docValuesFormat="Disk") to only load minimal data on the heap, keeping other > data structures on disk. Do note, though: > Not a huge improvement for a static index this latter isn't a problem though since you don't have a static index.... Erick On Tue, Sep 24, 2013 at 4:13 AM, Neil Prosser <neil.pros...@gmail.com> wrote: > Shawn: unfortunately the current problems are with facet.method=enum! > > Erick: We already round our date queries so they're the same for at least > an hour so thankfully our fq entries will be reusable. However, I'll take a > look at reducing the cache and autowarming counts and see what the effect > on hit ratios and performance are. > > For SolrCloud our soft commit (openSearcher=false) interval is 15 seconds > and our hard commit is 15 minutes. > > You're right about those sorted fields having a lot of unique values. They > can be any number between 0 and 10,000,000 (it's sparsely populated across > the documents) and could appear in several variants across multiple > documents. This is probably a good area for seeing what we can bend with > regard to our requirements for sorting/boosting. I've just looked at two > shards and they've each got upwards of 1000 terms showing in the schema > browser for one (potentially out of 60) fields. > > > > On 21 September 2013 20:07, Erick Erickson <erickerick...@gmail.com> wrote: > >> About caches. The queryResultCache is only useful when you expect there >> to be a number of _identical_ queries. Think of this cache as a map where >> the key is the query and the value is just a list of N document IDs >> (internal) >> where N is your window size. Paging is often the place where this is used. >> Take a look at your admin page for this cache, you can see the hit rates. >> But, the take-away is that this is a very small cache memory-wise, varying >> it is probably not a great predictor of memory usage. >> >> The filterCache is more intense memory wise, it's another map where the >> key is the fq clause and the value is bounded by maxDoc/8. Take a >> close look at this in the admin screen and see what the hit ratio is. It >> may >> be that you can make it much smaller and still get a lot of benefit. >> _Especially_ considering it could occupy about 44G of memory. >> (43,000,000 / 8) * 8192........ And the autowarm count is excessive in >> most cases from what I've seen. Cutting the autowarm down to, say, 16 >> may not make a noticeable difference in your response time. And if >> you're using NOW in your fq clauses, it's almost totally useless, see: >> http://searchhub.org/2012/02/23/date-math-now-and-filter-queries/ >> >> Also, read Uwe's excellent blog about MMapDirectory here: >> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html >> for some problems with over-allocating memory to the JVM. Of course >> if you're hitting OOMs, well..... >> >> bq: order them by one of their fields. >> This is one place I'd look first. How many unique values are in each field >> that you sort on? This is one of the major memory consumers. You can >> get a sense of this by looking at admin/schema-browser and selecting >> the fields you sort on. There's a text box with the number of terms >> returned, >> then a / ### where ### is the total count of unique terms in the field. >> NOTE: >> in 4.4 this will be -1 for multiValued fields, but you shouldn't be >> sorting on >> those anyway. How many fields are you sorting on anyway, and of what types? >> >> For your SolrCloud experiments, what are your soft and hard commit >> intervals? >> Because something is really screwy here. Your sharding moving the >> number of docs down this low per shard should be fast. Back to the point >> above, the only good explanation I can come up with from this remove is >> that the fields you sort on have a LOT of unique values. It's possible that >> the total number of unique values isn't scaling with sharding. That is, >> each >> shard may have, say, 90% of all unique terms (number from thin air). Worth >> checking anyway, but a stretch. >> >> This is definitely unusual... >> >> Best, >> Erick >> >> >> On Thu, Sep 19, 2013 at 8:20 AM, Neil Prosser <neil.pros...@gmail.com> >> wrote: >> > Apologies for the giant email. Hopefully it makes sense. >> > >> > We've been trying out SolrCloud to solve some scalability issues with our >> > current setup and have run into problems. I'd like to describe our >> current >> > setup, our queries and the sort of load we see and am hoping someone >> might >> > be able to spot the massive flaw in the way I've been trying to set >> things >> > up. >> > >> > We currently run Solr 4.0.0 in the old style Master/Slave replication. We >> > have five slaves, each running Centos with 96GB of RAM, 24 cores and with >> > 48GB assigned to the JVM heap. Disks aren't crazy fast (i.e. not SSDs) >> but >> > aren't slow either. Our GC parameters aren't particularly exciting, just >> > -XX:+UseConcMarkSweepGC. Java version is 1.7.0_11. >> > >> > Our index size ranges between 144GB and 200GB (when we optimise it back >> > down, since we've had bad experiences with large cores). We've got just >> > over 37M documents some are smallish but most range between 1000-6000 >> > bytes. We regularly update documents so large portions of the index will >> be >> > touched leading to a maxDocs value of around 43M. >> > >> > Query load ranges between 400req/s to 800req/s across the five slaves >> > throughout the day, increasing and decreasing gradually over a period of >> > hours, rather than bursting. >> > >> > Most of our documents have upwards of twenty fields. We use different >> > fields to store territory variant (we have around 30 territories) values >> > and also boost based on the values in some of these fields (integer >> ones). >> > >> > So an average query can do a range filter by two of the territory variant >> > fields, filter by a non-territory variant field. Facet by a field or two >> > (may be territory variant). Bring back the values of 60 fields. Boost >> query >> > on field values of a non-territory variant field. Boost by values of two >> > territory-variant fields. Dismax query on up to 20 fields (with boosts) >> and >> > phrase boost on those fields too. They're pretty big queries. We don't do >> > any index-time boosting. We try to keep things dynamic so we can alter >> our >> > boosts on-the-fly. >> > >> > Another common query is to list documents with a given set of IDs and >> > select documents with a common reference and order them by one of their >> > fields. >> > >> > Auto-commit every 30 minutes. Replication polls every 30 minutes. >> > >> > Document cache: >> > * initialSize - 32768 >> > * size - 32768 >> > >> > Filter cache: >> > * autowarmCount - 128 >> > * initialSize - 8192 >> > * size - 8192 >> > >> > Query result cache: >> > * autowarmCount - 128 >> > * initialSize - 8192 >> > * size - 8192 >> > >> > After a replicated core has finished downloading (probably while it's >> > warming) we see requests which usually take around 100ms taking over 5s. >> GC >> > logs show concurrent mode failure. >> > >> > I was wondering whether anyone can help with sizing the boxes required to >> > split this index down into shards for use with SolrCloud and roughly how >> > much memory we should be assigning to the JVM. Everything I've read >> > suggests that running with a 48GB heap is way too high but every attempt >> > I've made to reduce the cache sizes seems to wind up causing >> out-of-memory >> > problems. Even dropping all cache sizes by 50% and reducing the heap by >> 50% >> > caused problems. >> > >> > I've already tried using SolrCloud 10 shards (around 3.7M documents per >> > shard, each with one replica) and kept the cache sizes low: >> > >> > Document cache: >> > * initialSize - 1024 >> > * size - 1024 >> > >> > Filter cache: >> > * autowarmCount - 128 >> > * initialSize - 512 >> > * size - 512 >> > >> > Query result cache: >> > * autowarmCount - 32 >> > * initialSize - 128 >> > * size - 128 >> > >> > Even when running on six machines in AWS with SSDs, 24GB heap (out of >> 60GB >> > memory) and four shards on two boxes and three on the rest I still see >> > concurrent mode failure. This looks like it's causing ZooKeeper to mark >> the >> > node as down and things begin to struggle. >> > >> > Is concurrent mode failure just something that will inevitably happen or >> is >> > it avoidable by dropping the CMSInitiatingOccupancyFraction? >> > >> > If anyone has anything that might shove me in the right direction I'd be >> > very grateful. I'm wondering whether our set-up will just never work and >> > maybe we're expecting too much. >> > >> > Many thanks, >> > >> > Neil >>
