Hi Greg, Sorry for the slow response. The general thinking is that you shouldn't worry about which nodes host leaders vs. replicas because A) that can change, and B) as you say, the additional responsibilities for leader nodes is quite minimal (mainly per-doc version management and then distributing updates to replicas). The segment merging all happens at the Lucene level, which has no knowledge of SolrCloud leaders / replicas. Since this is SolrCloud, all nodes pull the config from ZooKeeper so should be running the same settings. Can you diff the listings of the index data directories on a leader vs. replica? Might give us some insights to what files the leader has that the replicas don't have.
Cheers, Tim On Tue, Jun 3, 2014 at 8:32 PM, Greg Pendlebury <greg.pendleb...@gmail.com> wrote: > Hi all, > > We launched our new production instance of SolrCloud last week and since > then have noticed a trend with regards to disk usage. The non-leader > replicas all seem to be self-optimizing their index segments as expected, > but the leaders have (on average) around 33% more data on disk. My > assumption is that leader's are not self-optimising (or not to the same > extent)... but it is still early days of course. > > If it helps, there are 45 JVMs in the cloud, with 15 shards and 3 replicas > per shard. Each non-leader shard is sitting at between 59GB and 87GB on > their SSD, but the leaders are between 84GB and 116GB. > > We have pretty much constant read and write traffic 24x7, with just 'slow' > periods overnight when write traffic is < 1 document per second and > searches are between 1 and 2 per second. Is this light level of traffic > still too much for the leaders to self-optimise? > > I'd also be curious to hear about what others are doing in terms of > operating procedures. We load test before launch what would happen if we > turned off JVMs and forced recovery events. I know that these things all > work, just that customers will experience slower search responses whilst > they occur. For example, a restore from a leader to a replica under load > testing for us takes around 30 minutes and response times drop from around > 200-300ms average to 1.5s average. > > Bottleneck appears to be network I/O on the servers. We haven't explored > whether this is specific to the servers replicating, or saturation of the > of the infrastructure that all the servers share, because... > > This performance is acceptable for us, but I'm not sure if I'd like to > force that event to occur unless required... this is following the line of > reasoning proposed internally that we should periodically rotate leaders by > turning them off briefly. We aren't going to do that unless we have a > strong reason though. Does anyone try to manipulate production instances > that way? > > Vaguely related to this is leader distribution. We have 9 physical servers > and 5 JVMs running on each server. By virtue of the deployment procedures > the first 3 servers to come online are all running 5 leaders each. Is there > any merit in 'moving' these around (by reboots)? > > Our planning up to launch was based on lots of mailing list response we'd > seen that indicated leaders had no significant performance difference to > normal replicas, and all of our testing has agreed with that. The disk size > 'issue' (which we aren't worried about... yet. It hasn't been in prod long > enough to know for certain) may be the only thing we've seen so far. > > Ta, > Greg
