Hi guys, I have a problem with Solr replication. I have 2 solr servers (Solr 4.0.0) 1 master and 1 slave (8 processors,16GB RAM ,Ubuntu 11, ext3, each). In every server, there are 2 independent instances of solr running (I tried also multicore config, but having independent instances has for me better performance), every instance having a differente collection. So, we have 2 masters in server 1, and 2 slaves in server 2.
Index size is currently (for the biggest collection) around 17 million documents, with a total size near 12 GB. The files transferred every replication cycle are typically not more than 100, with a total size not bigger than 50MB. The other collection is not that big, just around 1 million docs and not bigger than 2 GB and not a high update ratio. The big collection has a load around 200 queries per second (MoreLikeThis, RealTimeGetHandler , TermVectorComponent mainly), and for the small one it is below 50 queries per second Replication has been working for long time with any problem, but in the last weeks the replication cycles started to take long and long time for the big collection, even more than 2 minutes, some times even more. During that time, slaves are so overloaded, that many queries are timing out, despite the timeout in my clients is 30 seconds. The servers are in same LAN, gigabit ethernet, so the broadband should not be the bottleneck. Since the index is receiving frequents updates and deletes (update handler receives more than 200 request per second for the big collection, but not more than 5 per second for the small one), I tried to use the maxCommitsToKeep attribute, to ensure that no file was deleted during replication, but it has no effect. My solrconfig.xml in the big collection is like that: <?xml version="1.0" encoding="UTF-8" ?> <config> <luceneMatchVersion>LUCENE_40</luceneMatchVersion> <directoryFactory name="DirectoryFactory" class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/> <indexConfig> <mergeFactor>3</mergeFactor> <deletionPolicy class="solr.SolrDeletionPolicy"> <str name="maxCommitsToKeep">10</str> <str name="maxOptimizedCommitsToKeep">1</str> <str name="maxCommitAge">6HOUR</str> </deletionPolicy> </indexConfig> <jmx/> <updateHandler class="solr.DirectUpdateHandler2"> <autoCommit> <maxDocs>2000</maxDocs> <maxTime>30000</maxTime> </autoCommit> <autoSoftCommit> <maxTime>500</maxTime> </autoSoftCommit> <updateLog> <str name="dir">${solr.data.dir:}</str> </updateLog> </updateHandler> <query> <maxBooleanClauses>2048</maxBooleanClauses> <filterCache class="solr.FastLRUCache" size="2048" initialSize="1024" autowarmCount="1024"/> <queryResultCache class="solr.LRUCache" size="2048" initialSize="1024" autowarmCount="1024"/> <documentCache class="solr.LRUCache" size="2048" initialSize="1024" autowarmCount="1024"/> <enableLazyFieldLoading>true</enableLazyFieldLoading> <queryResultWindowSize>50</queryResultWindowSize> <queryResultMaxDocsCached>50</queryResultMaxDocsCached> <listener event="newSearcher" class="solr.QuerySenderListener"> <arr name="queries"> <lst> <str name="q">*:*</str> <str name="fq">date:[NOW/DAY-7DAY TO NOW/DAY+1DAY]</str> <str name="rows">1000</str> </lst> </arr> </listener> <listener event="firstSearcher" class="solr.QuerySenderListener"> <arr name="queries"> <lst> <str name="q">*:*</str> <str name="fq">date:[NOW/DAY-7DAY TO NOW/DAY+1DAY]</str> <str name="rows">1000</str> </lst> </arr> </listener> <useColdSearcher>true</useColdSearcher> <maxWarmingSearchers>4</maxWarmingSearchers> </query> <requestHandler name="/replication" class="solr.ReplicationHandler"> <lst name="master"> <str name="enable">${enable.master:false}</str> <str name="replicateAfter">commit</str> <str name="replicateAfter">startup</str> <str name="backupAfter">startup</str> <str name="confFiles">schema.xml,solrconfig.xml,stopwords_de.txt,stopwords_en.txt,mapping-FoldToASCII.txt,mapping-FoldToASCII_de.txt</str> </lst> <lst name="slave"> <str name="enable">${enable.slave:false}</str> <str name="masterUrl">http://${MASTER_HOST}:${MASTER_PORT}/solr/${MASTER_CORE}</str> <str name="pollInterval">05:00</str> <str name="httpBasicAuthUser">${MASTER_HTTP_USER}</str> <str name="httpBasicAuthPassword">${MASTER_HTTP_PWD}</str> </lst> </requestHandler> <admin> <defaultQuery>*:*</defaultQuery> </admin> </config> Poll interval is now set to 5 min, I tried to reduce it to 2 min and to increase it up to 10, with no effect, the replication is always taking so long., even with a poll time of 2 minutes, when there are only a few megas to replicate. Any idea suggestion about what could be the problem? Thanks in advance, Victor -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388.html Sent from the Solr - User mailing list archive at Nabble.com.