Re: Replicating Large Indexes

2011-11-01 Thread Erick Erickson
Yes, that's expected behavior. When you optimize, all segments are copied over to new segments(s). Since all changed/new segments are replicated to the slave, you'll (temporarily) have twice the data on your disk. You can stop optimizing, it's often not really very useful despite its name. That

Re: Replicating Large Indexes

2011-11-01 Thread Robert Stewart
Optimization merges index to a single segment (one huge file), so entire index will be copied on replication. So you really do need 2x disk in some cases then. Do you really need to optimize? We have a pretty big total index (about 200 million docs) and we never optimize. But we do have a

Re: Replicating Large Indexes

2011-11-01 Thread Jason Biggin
Thanks Robert. We optimize less frequently than we used to. Down to twice a month from once a day. Without optimizing the search speed stays the same, however the index size increases to 70+ GB. Perhaps there is a different way to restrict disk usage. Thanks, Jason Robert Stewart

Re: Replicating Large Indexes

2011-11-01 Thread Robert Stewart
Do you do a lot of deletes (or 'updates' of existing documents)? Do you store lots of large fields? Maybe you can use compressed fields in that case (we never have tried it so I cannot confirm how well it works or performs). You can also turn off things like norms and vectors, etc. if you

RE: Replicating Large Indexes

2011-11-01 Thread Jason Biggin
Thanks Erick, Will take a look at this article. Cheers, Jason -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Tuesday, November 01, 2011 8:05 AM To: solr-user@lucene.apache.org Subject: Re: Replicating Large Indexes Yes, that's expected behavior. When

Re: Replicating Large Indexes

2011-11-01 Thread Chris Hostetter
: We optimize less frequently than we used to. Down to twice a month from once a day. : : Without optimizing the search speed stays the same, however the index size increases to 70+ GB. : : Perhaps there is a different way to restrict disk usage. Consider using the maxSegments option on

Re: Replicating Large Indexes

2011-10-31 Thread Floyd Wu
Hi Jason, I'm very curious about how you build( rebuild ) such a big index efficiently? Sorry that hijack this topic. Floyd 2011/11/1 Jason Biggin jbig...@hipdigital.com: Wondering if anyone has experience with replicating large indexes.  We have a Solr deployment with 1 master, 1