Re: Is optimize needed on slaves if it replicates from optimized master?
Walter, Thanks again. Can you specify the criteria based on which Solr optimizes/force merges segments automatically. Is this defined by the MergeFactor parameter - like if the mergefactor is 10, then merge happens for every 10 segments? Please explain. Thanks, Sudarshan -- View this message in context: http://lucene.472066.n3.nabble.com/Is-optimize-needed-on-slaves-if-it-replicates-from-optimized-master-tp3241604p3987086.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Merging Remote Solr Indexes?
Hi All, I'm new to Solr. I saw this post relating to Merging of indexes. I have a similar doubt. From the post, I understand that merging of indexes across different cores is possible only if the cores exist o a single machine. I want to merge indexes of different machines. Can you please explain me the different ways of doing this? Say I have N+1 Solr engines of which there are N different masters and the remaining 1 is meant for merging all N indexes together. How I have decided to merge N indexes to 1 is this. 1. Dynamically edit the solrconfig.xml file of the N+1st system to point as a slave to different master each time. Hence a total of N trials would be needed to cover all N masters. 2. During every trial I shall replicate the index of the master and store it in a different folder. Say index1 from master1, index2 from master2 . indexn from masterN. 3. After all indexes are replicated and moved/renamed to local directory, I shall perform a merge of all indexes. What problems will I have in implementing this? How efficient would be this? I believe all index folders will have to be available locally to perform merging. If not, please tell me how better can I do merge remote indexes. Another question I have is about MergeFactor. If I set the mergefactor as 5, will Solr automatically takes care of merging the segments to 1 if the number of segments reach 5? How this can be exploited? Your assistance is sincerely appreciated. Regards, Sudarshan -- View this message in context: http://lucene.472066.n3.nabble.com/Merging-Remote-Solr-Indexes-tp3434412p3987090.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: index merge
Hi All, I have a basic doubt about index merging in Solr. The setup that I have followed is as follows: Setup: I used the schema.xml that comes with the solr example. I had three cores - core0, core1 and core2. I tried merging the indexes of core 0 and core 1 to core2. I copied the same schema.xml from SOLR_HOME/example/solr/conf to core 0 and core 1 but changed the name field alone as core0 and core1 respectively. Operations: I indexed different files to core0 and core1. The search *:* in Solr showed 6 files and 9 files for core0 and core1 respectively. Then merged the indexes of core0 and core1 to core2. As expected the search *:* showed 15 files for core2. I added 2 new files to the index of core0 and 1 file to core1 and merged again to core2. This time to my surprise * showed the total number of files showed to be 33 = (15+18) instead of just 18. This duplication continued for each merge operation which is not efficient. Also the merged files were available for search only after restarting the Jetty server. Am I missing something or doing things wrongly? Is there a way to restart only a specific core to read the new index/reflect the merged changes? Please explain the merge operation. Thanks, Sudarshan -- View this message in context: http://lucene.472066.n3.nabble.com/index-merge-tp472904p3987121.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Is optimize needed on slaves if it replicates from optimized master?
Hi Walter, Thank you. Do you mean that optimize need not be used at all? If Solr merges segments (when needed as you said), is there a criteria during which Solr does this automatically. If I want the search to be faster and Solr does not optimize for quite a long time, would it not compromise my query processing rate? To All, I have another doubt. If I optimize and replicate, for the first time it would transfer all the segments from the master to slave irrespective of the modified segment(s). After first replication, how the transfer would be made - again all segments are replicated or only the modified segments are replicated? I believe after the first replication (master and slave in sync), only the modified segments would be transferred just like the non-optimized index transfer. Am I right? Regards, Sudarshan -- View this message in context: http://lucene.472066.n3.nabble.com/Is-optimize-needed-on-slaves-if-it-replicates-from-optimized-master-tp3241604p3986597.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Is optimize needed on slaves if it replicates from optimized master?
Hi All, I happen to see this message board just now. I want to clarify certain things. I'm new to Solr. I'm trying to combine Solr's index replication and optimization. I have some doubts about the working of replication in a master slave setup. From the post, I understand that if the index is not optimized, only the modified segments are transferred from the master to slave. I have two scenarios. 1. Optimizing the index only in the master and replicating the optimized index to the slave - from my understanding this would copy the whole index every time (pol interval). 2. Optimizing the index only in the slave. From my testing, I observed that slave replaces its optimized index with that of the masters non-optimized index during every replication (poll interval). So even if the index has not changed in the master, if the slave tries to optimize after every replication, sooner it will be replaced by the masters index - based on my observations. Questions: From my opinion, if at all I want to optimize, doing it in master and replicating optimized index to slaves would be more sensible. Am I right? 1.Is there a way to combine optimization along with replication? 2. I could not understand when merging of indexes would be useful. I believe that master and slave should always have a consistent view of the index which is what replication guarantees. So why should I merge index? 3. If I have to optimize either in the master or in the slave, will the entire index be copied always to slave? 4. During replication, I found that the size of the index and the number of files in the index are different in the master and slave. Still they were in sync. Do they have some internal meta data calculations to find the difference (number of files per index version) between the master and the slave to initiate replication? Your suggestions/guidance would me very helpful to get a clear understanding. Please help. Thanks, Sudarshan -- View this message in context: http://lucene.472066.n3.nabble.com/Is-optimize-needed-on-slaves-if-it-replicates-from-optimized-master-tp3241604p3986259.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Indexing files using multi-cores - could not fix after many retries
Thanks Gora i,t worked. -- View this message in context: http://lucene.472066.n3.nabble.com/Indexing-files-using-multi-cores-could-not-fix-after-many-retries-tp3985253p3985672.html Sent from the Solr - User mailing list archive at Nabble.com.
