Yes, that's expected behavior. When you optimize, all segments are
copied over to new
segments(s). Since all changed/new segments are replicated to the slave,
you'll (temporarily) have twice the data on your disk.
You can stop optimizing, it's often not really very useful despite its name.
That
Optimization merges index to a single segment (one huge file), so entire index
will be copied on replication. So you really do need 2x disk in some cases
then.
Do you really need to optimize? We have a pretty big total index (about 200
million docs) and we never optimize. But we do have a
Thanks Robert.
We optimize less frequently than we used to. Down to twice a month from once
a day.
Without optimizing the search speed stays the same, however the index size
increases to 70+ GB.
Perhaps there is a different way to restrict disk usage.
Thanks,
Jason
Robert Stewart
Do you do a lot of deletes (or 'updates' of existing documents)?
Do you store lots of large fields? Maybe you can use compressed fields in that
case (we never have tried it so I cannot confirm how well it works or performs).
You can also turn off things like norms and vectors, etc. if you
Thanks Erick,
Will take a look at this article.
Cheers,
Jason
-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com]
Sent: Tuesday, November 01, 2011 8:05 AM
To: solr-user@lucene.apache.org
Subject: Re: Replicating Large Indexes
Yes, that's expected behavior. When
: We optimize less frequently than we used to. Down to twice a month from
once a day.
:
: Without optimizing the search speed stays the same, however the index size
increases to 70+ GB.
:
: Perhaps there is a different way to restrict disk usage.
Consider using the maxSegments option on
Hi Jason,
I'm very curious about how you build( rebuild ) such a big index efficiently?
Sorry that hijack this topic.
Floyd
2011/11/1 Jason Biggin jbig...@hipdigital.com:
Wondering if anyone has experience with replicating large indexes. We have a
Solr deployment with 1 master, 1
