OK let me rephrase. In solrconfig.xml there is a setting called mergeFactor. The default is usually 10. Practically it means there are 10 segments. If you are doing fast delta indexing (adding a couple documents, then committing), You will cycle through all 10 segments pretty fast.
It appears that if you do go past the 10 segments without replicating, the only recourse is for the replicator to do a full index replication instead of a delta index replication... Does that help? On 5/9/11 9:24 AM, "Ravi Solr" <ravis...@gmail.com> wrote: >Hello Mr. Bell, > Thank you very much for patiently responding to my >questions. We optimize once in every 2 days. Can you kindly rephrase >your answer, I could not understand - "if the amount of time if > 10 >segments, I believe that might also trigger a whole index, since you >cycled all the segments.In that case I think you might want to >increase the mergeFactor." > >The current index folder details and sizes are given below > >MASTER >-------------- > 5K search-data/spellchecker2 > 480M search-data/index > 5K search-data/spellchecker1 > 5K search-data/spellcheckerFile > 480M search-data > >SLAVE >---------- > 2K search-data/index.20110509103950 > 419M search-data/index > 2.3G search-data/index.20110429042508 ----> SLAVE is pointing to >this directory > 5K search-data/spellchecker1 > 5K search-data/spellchecker2 > 5K search-data/spellcheckerFile > 2.7G search-data > >Thanks, > >Ravi Kiran Bhaskar > >On Sat, May 7, 2011 at 11:49 PM, Bill Bell <billnb...@gmail.com> wrote: >> I did not see answers... I am not an authority, but will tell you what I >> think.... >> >> Did you get some answers? >> >> >> On 5/6/11 2:52 PM, "Ravi Solr" <ravis...@gmail.com> wrote: >> >>>Hello, >>> Pardon me if this has been already answered somewhere and I >>>apologize for a lengthy post. I was wondering if anybody could help me >>>understand Replication internals a bit more. We have a single >>>master-slave setup (solr 1.4.1) with the configurations as shown >>>below. Our environment is quite commit heavy (almost 100s of docs >>>every 5 minutes), and all indexing is done on Master and all searches >>>go to the Slave. We are seeing that the slave replication performance >>>gradually decreases and the speed decreases < 1kbps and ultimately >>>gets backed up. Once we reload the core on slave it will be work fine >>>for sometime and then it again gets backed up. We have mergeFactor set >>>to 10 and ramBufferSizeMB is set to 32MB and solr itself is running >>>with 2GB memory and locktype is simple on both master and slave. >> >> How big is your index? How many rows and GB ? >> >> Every time you replicate, there are several resets on caching. So if you >> are constantly >> Indexing, you need to be careful on how that performance impact will >>apply. >> >>> >>>I am hoping that the following questions might help me understand the >>>replication performance issue better (Replication Configuration is >>>given at the end of the email) >>> >>>1. Does the Slave get the whole index every time during replication or >>>just the delta since the last replication happened ? >> >> >> It depends. If you do an OPTIMIZE every time your index, then you will >>be >> sending the whole index down. >> If the amount of time if > 10 segments, I believe that might also >>trigger >> a whole index, since you cycled all the segments. >> In that case I think you might want to increase the mergeFactor. >> >> >>> >>>2. If there are huge number of queries being done on slave will it >>>affect the replication ? How can I improve the performance ? (see the >>>replications details at he bottom of the page) >> >> It seems that might be one way the you get the index.* directories. At >> least I see it more frequently when there is huge load and you are >>trying >> to replicate. >> You could replicate less frequently. >> >>> >>>3. Will the segment names be same be same on master and slave after >>>replication ? I see that they are different. Is this correct ? If it >>>is correct how does the slave know what to fetch the next time i.e. >>>the delta. >> >> Yes they better be. In the old days you could just rsync the data >> directory from master and slave and reload the core, that worked fine. >> >>> >>>4. When and why does the index.<TIMESTAMP> folder get created ? I see >>>this type of folder getting created only on slave and the slave >>>instance is pointing to it. >> >> I would love to know all the conditions... I believe it is supposed to >> replicate to index.*, then reload to point to it. But sometimes it gets >> stuck in index.* land and never goes back to straight index. >> >> There are several bug fixes for this in 3.1. >> >>> >>>5. Does replication process copy both the index and index.<TIMESTAMP> >>>folder ? >> >> I believe it is supposed to copy the segment or whole index/ from master >> to index.* on slave. >> >>> >>>6. what happens if the replication kicks off even before the previous >>>invocation has not completed ? will the 2nd invocation block or will >>>it go through causing more confusion ? >> >> That is not supposed to happen, if a replication is in process, it >>should >> not copy again until that one is complete. >> Try it, just delete the data/*, restart SOLR, and force a replication, >> while it is syncing, force it again. Does not seem to work for me. >>> >>>7. If I have to prep a new master-slave combination is it OK to copy >>>the respective contents into the new master-slave and start solr ? or >>>do I have have to wipe the new slave and let it replicate from its new >>>master ? >> >> If you shut down the slave, copy the data/* directory amd restart you >> should be fine. That is how we fix the data/ dir when >> there is corruption. >>> >>>8. Doing an 'ls | wc -l' on index folder of master and slave gave 194 >>>and 17968 respectively...I slave has lot of segments_xxx files. Is >>>this normal ? >> >> Several bugs fixed in 3.1 for this one. Not a good thing.... You are >> getting leftover segments or index.* directories. >>> >>>MASTER >>> >>><requestHandler name="/replication" class="solr. >>>ReplicationHandler" > >>> <lst name="master"> >>> <str name="replicateAfter">startup</str> >>> <str name="replicateAfter">commit</str> >>> <str name="replicateAfter">optimize</str> >>> >>> <str name="confFiles">schema.xml,stopwords.txt</str> >>> <str name="commitReserveDuration">00:00:10</str> >>> </lst> >>></requestHandler> >>> >>> >>>SLAVE >>> >>><requestHandler name="/replication" class="solr.ReplicationHandler" > >>> <lst name="slave"> >>> <str name="masterUrl">master core url</str> >>> <str name="pollInterval">00:03:00</str> >>> <str name="compression">internal</str> >>> <str name="httpConnTimeout">5000</str> >>> <str name="httpReadTimeout">10000</str> >>> </lst> >>></requestHandler> >>> >>> >>>REPLICATION DETAILS FROM PAGE >>> >>>Master master core url >>>Poll Interval 00:03:00 >>>Local Index Index Version: 1296217104577, Generation: 20190 >>> Location: /data/solr/core/search-data/index.20110429042508 >>> Size: 2.1 GB >>> Times Replicated Since Startup: 672 >>> Previous Replication Done At: Fri May 06 15:41:01 EDT 2011 >>> Config Files Replicated At: null >>> Config Files Replicated: null >>> Times Config Files Replicated Since Startup: null >>> Next Replication Cycle At: Fri May 06 15:44:00 EDT 2011 >>> Current Replication Status Start Time: Fri May 06 15:41:00 EDT >>>2011 >>> Files Downloaded: 43 / 197 >>> Downloaded: 477.08 KB / 588.82 MB [0.0%] >>> Downloading File: _hdm.prx, Downloaded: 9.3 KB / 9.3 KB [100.0%] >>> Time Elapsed: 967s, Estimated Time Remaining: 1221166s, Speed: 505 >>>bytes/s >>> >>> >>>Ravi Kiran Bhaskar >> >> >>
