Re: index size with replication

2012-03-15 Thread Walter Underwood
No, the deleted files do not get replicated. Instead, the slaves do the same thing as the master, holding on to the deleted files after the new files are copied over. The optimize is obsoleting all of your index files, so maybe should quit doing that. Without an optimize, the deleted files will

Re: index size with replication

2012-03-15 Thread Mike Austin
The problem is that when replicating, the double-size index gets replicated to slaves. I am now doing a dummy commit with always the same document and it works fine.. After the optimize and dummy commit process I just end up with numDocs = x and maxDocs = x+1. I don't get the nice green checkmark

Re: index size with replication

2012-03-15 Thread Erick Erickson
Or just ignore it if you have the disk space. The files will be cleaned up eventually. I believe they'll magically disappear if you simply bounce the server (but work on *nix so can't personally guarantee it). And replication won't replicate the stale files, so that's not a problem either Best

Re: index size with replication

2012-03-14 Thread Mike Austin
Shawn, Thanks for the detailed answer! I will play around with this information in hand. Maybe a second optimize or just a dummy commit after the optimize will help get me past this. Both not the best options, but maybe it's a do it because it's running on windows work-around. If it is indeed a

Re: index size with replication

2012-03-14 Thread Shawn Heisey
On 3/14/2012 2:54 PM, Mike Austin wrote: The odd thing is that if I optimize the index it doubles in size.. If I then, add one more document to the index it goes back down to half size? Is there a way to force this without needing to wait until another document is added? Or do you have more info

Re: index size with replication

2012-03-14 Thread Mike Austin
ot > going to do anything helpful. > > James Dyer > E-Commerce Systems > Ingram Content Group > (615) 213-4311 > > > -Original Message- > From: Ahmet Arslan [mailto:iori...@yahoo.com] > Sent: Wednesday, March 14, 2012 4:25 PM > To: solr-user@lucene.apache.o

RE: index size with replication

2012-03-14 Thread Dyer, James
uot;, then optimize is (probably) not going to do anything helpful. James Dyer E-Commerce Systems Ingram Content Group (615) 213-4311 -Original Message- From: Ahmet Arslan [mailto:iori...@yahoo.com] Sent: Wednesday, March 14, 2012 4:25 PM To: solr-user@lucene.apache.org Subject: Re: i

Re: index size with replication

2012-03-14 Thread Ahmet Arslan
> Another note.. if I reload solr app > it goes back down in size. > > here is my replication settings on the master: > > class="solr.ReplicationHandler" > >         >           name="replicateAfter">startup >           name="replicateAfter">commit >           name="replicateAfter">optimize >  

Re: index size with replication

2012-03-14 Thread Mike Austin
Another note.. if I reload solr app it goes back down in size. here is my replication settings on the master: startup commit optimize 1 schema.xml,stopwords.txt,elevate.xml 00:00:30 On Wed, Mar 14, 2012 at 3:54 PM, Mike Aust

Re: index size with replication

2012-03-14 Thread Mike Austin
The odd thing is that if I optimize the index it doubles in size.. If I then, add one more document to the index it goes back down to half size? Is there a way to force this without needing to wait until another document is added? Or do you have more information on what you think is going on? I'm

Re: index size with replication

2012-03-13 Thread Li Li
optimize will generate new segments and delete old ones. if your master also provides searching service during indexing, the old files may be opened by old SolrIndexSearcher. they will be deleted later. So when indexing, the index size may double. But a moment later, old indexes will be deleted.

index size with replication

2012-03-13 Thread Mike Austin
I have a master with two slaves. For some reason on the master if I do an optimize after indexing on the master it double in size from 42meg to 90 meg.. however, when the slaves replicate they get the 42meg index.. Should the master and slaves always be the same size? Thanks, Mike