some basic tips. -try to create enough shards that you can get the size of each index portion on the shard closer to the amount of RAM you have on each node (e.g. if you are ~140GB index on 16GB nodes, try doing 12-16 shards)
-start with just the initial shards, add replicas later when you have dialed things in a bit more -try to leave some memory for the OS as well as the JVM -try starting with 1/2 of the total ram on each vm allocated to JVM as Xmx value -try setting the Xms in the range of .75 to 1.0 of Xmx -do all the normal JVM tuning, esp the part about capturing the gc events in a log such that you can see what is going on with java itself..this will probably lead you to adjust your GC type, etc -make sure you arent hammering your storage devices (or the interconnects between your servers and your storage)...the OS internal tools on the guest are helpful, but you probably want to look at the hypervisor and storage device layer directly as well. on vmware the built in perf graphs for datastore latency and network throughput are easily observed. esxtop is the cli tool which provides the same. -if you are using a SAN, you probably want to make sure you have some sort of MPIO in place (esp if you are using 1GB iscsi) ________________________________________ From: Tom Mortimer <tom.m.f...@gmail.com> Sent: Monday, October 21, 2013 08:48 To: solr-user@lucene.apache.org Subject: SolrCloud performance in VM environment Hi everyone, I've been working on an installation recently which uses SolrCloud to index 45M documents into 8 shards on 2 VMs running 64-bit Ubuntu (with another 2 identical VMs set up for replicas). The reason we're using so many shards for a relatively small index is that there are complex filtering requirements at search time, to restrict users to items they are licensed to view. Initial tests demonstrated that multiple shards would be required. The total size of the index is about 140GB, and each VM has 16GB RAM (32GB total) and 4 CPU units. I know this is far under what would normally be recommended for an index of this size, and I'm working on persuading the customer to increase the RAM (basically, telling them it won't work otherwise.) Performance is currently pretty poor and I would expect more RAM to improve things. However, there are a couple of other oddities which concern me, The first is that I've been reindexing a fixed set of 500 docs to test indexing and commit performance (with soft commits within 60s). The time taken to complete a hard commit after this is longer than I'd expect, and highly variable - from 10s to 70s. This makes me wonder whether the SAN (which provides all the storage for these VMs and the customers several other VMs) is being saturated periodically. I grabbed some iostat output on different occasions to (possibly) show the variability: Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sdb 64.50 0.00 2476.00 0 4952 ... sdb 8.90 0.00 348.00 0 6960 ... sdb 1.15 0.00 43.20 0 864 The other thing that confuses me is that after a Solr restart or hard commit, search times average about 1.2s under light load. After searching the same set of queries for 5-6 iterations this improves to 0.1s. However, in either case - cold or warm - iostat reports no device reads at all: Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sdb 0.40 0.00 8.00 0 160 ... sdb 0.30 0.00 10.40 0 104 (the writes are due to logging). This implies to me that the 'hot' blocks are being completely cached in RAM - so why the variation in search time and the number of iterations required to speed it up? The Solr caches are only being used lightly by these tests and there are no evictions. GC is not a significant overhead. Each Solr shard runs in a separate JVM with 1GB heap. I don't have a great deal of experience in low-level performance tuning, so please forgive any naivety. Any ideas of what to do next would be greatly appreciated. I don't currently have details of the VM implementation but can get hold of this if it's relevant. thanks, Tom
