Be a bit careful here. 128G is lots of memory, you may encounter very long garbage collection pauses. Just be aware that this may be happening later.
Best, Erick On Tue, Oct 22, 2013 at 5:04 PM, Tom Mortimer <tom.m.f...@gmail.com> wrote: > Just tried it with no other changes than upping the RAM to 128GB total, and > it's flying. I think that proves that RAM is good. =) Will implement > suggested changes later, though. > > cheers, > Tom > > > On 22 October 2013 09:04, Tom Mortimer <tom.m.f...@gmail.com> wrote: > > > Boogie, Shawn, > > > > Thanks for the replies. I'm going to try out some of your suggestions > > today. Although, without more RAM I'm not that optimistic.. > > > > Tom > > > > > > > > On 21 October 2013 18:40, Shawn Heisey <s...@elyograg.org> wrote: > > > >> On 10/21/2013 9:48 AM, Tom Mortimer wrote: > >> > >>> Hi everyone, > >>> > >>> I've been working on an installation recently which uses SolrCloud to > >>> index > >>> 45M documents into 8 shards on 2 VMs running 64-bit Ubuntu (with > another > >>> 2 > >>> identical VMs set up for replicas). The reason we're using so many > shards > >>> for a relatively small index is that there are complex filtering > >>> requirements at search time, to restrict users to items they are > licensed > >>> to view. Initial tests demonstrated that multiple shards would be > >>> required. > >>> > >>> The total size of the index is about 140GB, and each VM has 16GB RAM > >>> (32GB > >>> total) and 4 CPU units. I know this is far under what would normally be > >>> recommended for an index of this size, and I'm working on persuading > the > >>> customer to increase the RAM (basically, telling them it won't work > >>> otherwise.) Performance is currently pretty poor and I would expect > more > >>> RAM to improve things. However, there are a couple of other oddities > >>> which > >>> concern me, > >>> > >> > >> Running multiple shards like you are, where each operating system is > >> handling more than one shard, is only going to perform better if your > query > >> volume is low and you have lots of CPU cores. If your query volume is > high > >> or you only have 2-4 CPU cores on each VM, you might be better off with > >> fewer shards or not sharded at all. > >> > >> The way that I read this is that you've got two physical machines with > >> 32GB RAM, each running two VMs that have 16GB. Each VM houses 4 > shards, or > >> 70GB of index. > >> > >> There's a scenario that might be better if all of the following are > true: > >> 1) I'm right about how your hardware is provisioned. 2) You or the > client > >> owns the hardware. 3) You have an extremely low-end third machine > >> available - single CPU with 1GB of RAM would probably be enough. In > this > >> scenario, you run one Solr instance and one zookeeper instance on each > of > >> your two "big" machines, and use the third wimpy machine as a third > >> zookeeper node. No virtualization. For the rest of my reply, I'm > assuming > >> that you haven't taken this step, but it will probably apply either way. > >> > >> > >> The first is that I've been reindexing a fixed set of 500 docs to test > >>> indexing and commit performance (with soft commits within 60s). The > time > >>> taken to complete a hard commit after this is longer than I'd expect, > and > >>> highly variable - from 10s to 70s. This makes me wonder whether the SAN > >>> (which provides all the storage for these VMs and the customers several > >>> other VMs) is being saturated periodically. I grabbed some iostat > output > >>> on > >>> different occasions to (possibly) show the variability: > >>> > >>> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn > >>> sdb 64.50 0.00 2476.00 0 4952 > >>> ... > >>> sdb 8.90 0.00 348.00 0 6960 > >>> ... > >>> sdb 1.15 0.00 43.20 0 864 > >>> > >> > >> There are two likely possibilities for this. One or both of them might > >> be in play. 1) Because the OS disk cache is small, not much of the > index > >> can be cached. This can result in a lot of disk I/O for a commit, > slowing > >> things way down. Increasing the size of the OS disk cache is really the > >> only solution for that. 2) Cache autowarming, particularly the filter > >> cache. In the cache statistics, you can see how long each cache took to > >> warm up after the last searcher was opened. The solution for that is to > >> reduce the autowarmCount values. > >> > >> > >> The other thing that confuses me is that after a Solr restart or hard > >>> commit, search times average about 1.2s under light load. After > searching > >>> the same set of queries for 5-6 iterations this improves to 0.1s. > >>> However, > >>> in either case - cold or warm - iostat reports no device reads at all: > >>> > >>> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn > >>> sdb 0.40 0.00 8.00 0 160 > >>> ... > >>> sdb 0.30 0.00 10.40 0 104 > >>> > >>> (the writes are due to logging). This implies to me that the 'hot' > blocks > >>> are being completely cached in RAM - so why the variation in search > time > >>> and the number of iterations required to speed it up? > >>> > >> > >> Linux is pretty good about making limited OS disk cache resources work. > >> Sounds like the caching is working reasonably well for queries. It > might > >> not be working so well for updates or commits, though. > >> > >> Running multiple Solr JVMs per machine, virtual or not, causes more > >> problems than it solves. Solr has no limits on the number of cores > (shard > >> replicas) per instance, assuming there are enough system resources. > There > >> should be exactly one Solr JVM per operating system. Running more than > one > >> results in quite a lot of overhead, and your memory is precious. When > you > >> create a collection, you can give the collections API the > >> "maxShardsPerNode" parameter to create more than one shard per instance. > >> > >> > >> I don't have a great deal of experience in low-level performance > tuning, > >>> so > >>> please forgive any naivety. Any ideas of what to do next would be > greatly > >>> appreciated. I don't currently have details of the VM implementation > but > >>> can get hold of this if it's relevant. > >>> > >> > >> I don't think the virtualization details matter all that much. Please > >> feel free to ask questions or supply more info based on what I've told > you. > >> > >> Thanks, > >> Shawn > >> > >> > > >
