I didn't find any increase in indexing throughput by adding shards in the same machine.
However, I've managed to feed the index to Solr from more than one thread at a time. It can take up to 3 threads without affecting the indexing speed. Anything more than that, the CPU will hit 100%, and the indexing speed in all the threads will be reduced. Regards, Edwin On 18 September 2015 at 19:38, Gili Nachum <gilinac...@gmail.com> wrote: > If cpu is just 50% and adding a shard does increase indexing throughput > then check for disk bottleneck. > On Sep 17, 2015 18:19, "Zheng Lin Edwin Yeo" <edwinye...@gmail.com> wrote: > > > Thank you everyone for your reply. > > > > > How many CPUs on that machine? How many other requests using the > server? > > > > A) There's 8 CPU on the machine, and there is no other requests that's > > using the server. Only the indexing script is running. > > > > > A simple metric is to look at CPU usage on the machine: If it is near > > 100% when you index, you will need extra hardware to get more speed. > > If it is substantially less than 100%, then feed Solr from more than one > > thread at a time. > > > > A) So far from what I observe, the CPU usage is usually around 50% to > 70%. > > It haven't go up to 100% yet. But I'll probably try to do sharing on a > > different machine, as that is probably the case for the real production > > server. > > > > > > Regards, > > Edwin > > > > > > On 17 September 2015 at 19:55, Toke Eskildsen <t...@statsbiblioteket.dk> > > wrote: > > > > > On Thu, 2015-09-17 at 16:58 +0800, Zheng Lin Edwin Yeo wrote: > > > > > > > I was trying with 2 shards and 4 shards but all on the same machine, > > > > and they have the same performance (no improvement in performance) as > > > > the one with 1 shard. My machine has a 32GB RAM. > > > > > > As you are testing indexing speed, Shalin's post is spot-on: Sharding > on > > > the same machine won't help you. I just added my comment on search to > > > help build a complete picture. > > > > > > A simple metric is to look at CPU usage on the machine: If it is near > > > 100% when you index, you will need extra hardware to get more speed. > > > If it is substantially less than 100%, then feed Solr from more than > one > > > thread at a time. > > > > > > - Toke Eskildsen, State and University Library, Denmark > > > > > > > > > > > > > > >
