Thanks Shawn, that helps explain things.
So the issue there, with using maxSearchWarmers to try and prevent out
of control RAM/CPU usage from over-lapping on-deck, combined with
replication... is if you're still pulling down replications very
frequently but using maxSearchWarmers to prevent overlapping on-deck,
you'll save RAM/CPU but might trade that off to instead use a LOT of
disk space for multiple versions of index segment files, until a commit
finally goes through.
On 12/14/2010 11:38 AM, Shawn Heisey wrote:
On 12/14/2010 9:02 AM, Jonathan Rochkind wrote:
1. Will the existing index searcher have problems because the files
have been changed out from under it?
2. Will a future replication -- at which NO new files are available on
master -- still trigger a future commit on slave?
I'm not really sure of the answer to #2, but I believe I can answer #1.
Lucene is designed so that all files necessary for an index to work are
kept around after a commit until there is a new searcher to take over
all requests with the new files. If you are replicating only new
segments, the old files will still be there both before and after. If
you just optimized the master and therefore are copying an entire new
index, the old one will not be removed until there is a successful
commit and therefore a new searcher.
There is another thread on replication that I just replied to as well.
Solr actually seems a little too intent on keeping old files around -
see SOLR-1781.
Shawn
