1 node having more load should be the leader (because of the extra work
of receiving and distributing updates, but my experiences show only a
bit more CPU usage, and no difference in disk IO).
A suggestion would be to hard commit much less often, ie every 10
minutes, and see if there is a change.
How much system RAM ? JVM Heap ? Enough space in RAM for system disk cache ?
What is the size of your documents ? A few KB, MB, ... ?
Ah, and what about network IO ? Could that be a limiting factor ?
André
On 2014-01-21 23:40, Software Dev wrote:
Any other suggestions?
On Mon, Jan 20, 2014 at 2:49 PM, Software Dev <static.void....@gmail.com>wrote:
4.6.0
On Mon, Jan 20, 2014 at 2:47 PM, Mark Miller <markrmil...@gmail.com>wrote:
What version are you running?
- Mark
On Jan 20, 2014, at 5:43 PM, Software Dev <static.void....@gmail.com>
wrote:
We also noticed that disk IO shoots up to 100% on 1 of the nodes. Do all
updates get sent to one machine or something?
On Mon, Jan 20, 2014 at 2:42 PM, Software Dev <
static.void....@gmail.com>wrote:
We commit have a soft commit every 5 seconds and hard commit every 30.
As
far as docs/second it would guess around 200/sec which doesn't seem
that
high.
On Mon, Jan 20, 2014 at 2:26 PM, Erick Erickson <
erickerick...@gmail.com>wrote:
Questions: How often do you commit your updates? What is your
indexing rate in docs/second?
In a SolrCloud setup, you should be using a CloudSolrServer. If the
server is having trouble keeping up with updates, switching to CUSS
probably wouldn't help.
So I suspect there's something not optimal about your setup that's
the culprit.
Best,
Erick
On Mon, Jan 20, 2014 at 4:00 PM, Software Dev <
static.void....@gmail.com>
wrote:
We are testing our shiny new Solr Cloud architecture but we are
experiencing some issues when doing bulk indexing.
We have 5 solr cloud machines running and 3 indexing machines
(separate
from the cloud servers). The indexing machines pull off ids from a
queue
then they index and ship over a document via a CloudSolrServer. It
appears
that the indexers are too fast because the load (particularly disk
io)
on
the solr cloud machines spikes through the roof making the entire
cluster
unusable. It's kind of odd because the total index size is not even
large..ie, < 10GB. Are there any optimization/enhancements I could
try
to
help alleviate these problems?
I should note that for the above collection we have only have 1 shard
thats
replicated across all machines so all machines have the full index.
Would we benefit from switching to a ConcurrentUpdateSolrServer where
all
updates get sent to 1 machine and 1 machine only? We could then
remove
this
machine from our cluster than that handles user requests.
Thanks for any input.
--
André Bois-Crettez
Software Architect
Search Developer
http://www.kelkoo.com/
Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris
Ce message et les pièces jointes sont confidentiels et établis à l'attention
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce
message, merci de le détruire et d'en avertir l'expéditeur.
