Thanks for the suggestions so for, all. 1) We are not using SolrJ on the client (not using Java at all) but I am working on writing a "smart" router so that we can always send to the correct node. I am certainly curious to see how that changes things. Nonetheless even with the overhead of extra routing hops, the observed behavior (no increase in performance with more nodes) doesn't make any sense to me.
2) Commits: we are using autoCommit with openSearcher=false (maxTime=60000) and autoSoftCommit (maxTime=15000). 3) Suggestions to batch documents certainly make sense for production code but in this case I am not real concerned with absolute performance; I just want to see the *relative* performance as we use more Solr nodes. So I don't think batching or not really matters. 4) "No, that won't affect indexing speed all that much. The way to increase indexing speed is to increase the number of processes or threads that are indexing at the same time. Instead of having one client sending update requests, try five of them." Can you elaborate on this some? I'm worried I might be misunderstanding something fundamental. A cluster of 3 shards over 3 Solr nodes *should* support a higher QPS than 2 shards over 2 Solr nodes, right? That's the whole idea behind sharding. Regarding your comment of "increase the number of processes or threads", note that for each value of K (number of Solr nodes) I measured with several different numbers of simulated users so that I could find a "saturation point". For example, take a look at my data for K=2: Num NodesNum UsersQPS214722517902102290215285022029002403210260320028032102 1003180 It's clear that once the load test client has ~40 simulated users, the Solr cluster is saturated. Creating more users just increases the average request latency, such that the total QPS remained (nearly) constant. So I feel pretty confident that a cluster of size 2 *maxes out* at ~3200 qps. The problem is that I am finding roughly this same "max point", no matter how many simulated users the load test client created, for any value of K (> 1). Cheers, - Ian On Thu, Oct 30, 2014 at 8:01 PM, Erick Erickson <erickerick...@gmail.com> wrote: > Your indexing client, if written in SolrJ, should use CloudSolrServer > which is, in Matt's terms "leader aware". It divides up the > documents to be indexed into packets that where each doc in > the packet belongs on the same shard, and then sends the packet > to the shard leader. This avoids a lot of re-routing and should > scale essentially linearly. You may have to add more clients > though, depending upon who hard the document-generator is > working. > > Also, make sure that you send batches of documents as Shawn > suggests, I use 1,000 as a starting point. > > Best, > Erick > > On Thu, Oct 30, 2014 at 2:10 PM, Shawn Heisey <apa...@elyograg.org> wrote: > > On 10/30/2014 2:56 PM, Ian Rose wrote: > >> I think this is true only for actual queries, right? I am not issuing > >> any queries, only writes (document inserts). In the case of writes, > >> increasing the number of shards should increase my throughput (in > >> ops/sec) more or less linearly, right? > > > > No, that won't affect indexing speed all that much. The way to increase > > indexing speed is to increase the number of processes or threads that > > are indexing at the same time. Instead of having one client sending > > update requests, try five of them. Also, index many documents with each > > update request. Sending one document at a time is very inefficient. > > > > You didn't say how you're doing commits, but those need to be as > > infrequent as you can manage. Ideally, you would use autoCommit with > > openSearcher=false on an interval of about five minutes, and send an > > explicit commit (with the default openSearcher=true) after all the > > indexing is done. > > > > You may have requirements regarding document visibility that this won't > > satisfy, but try to avoid doing commits with openSearcher=true (soft > > commits qualify for this) extremely frequently, like once a second. > > Once a minute is much more realistic. Opening a new searcher is an > > expensive operation, especially if you have cache warming configured. > > > > Thanks, > > Shawn > > >
