Hi, Any inputs on this would be really helpful. Looking for suggestions/viewpoints from you guys.
Regards, Sourav -----Original Message----- From: souravm Sent: Saturday, December 06, 2008 9:41 PM To: solr-user@lucene.apache.org Subject: Limitations of Distributed Search .... Hi, We are planning to use Solr for processing large volume of application log files (around ~ 10 Billions documents of size 5-6 TB). One of the approach we are considering for the same is to use Distributed Search extensively. What we have in mind is distributing the log files in multiple boxes monthly or weekly basis - where at the weekly basis itself the volume can go to the level of 200 M of documents. And a search query can spread across all weeks (e.g. number of a given txn for 1st 6 months of a year) However, what we are not sure how well the distributed search would scale when we may use around 50-60 boxes to distribute indexed documents on weekly basis. The specific questions I have in mind are - a) How would be the impact on the performance when a query spreads over 50 boxes b) Is there any hard limit on the number of slaves which can be contacted from the master server? c) How much load will this type of approach create on master server for merging data, keeping the track whether a slave is down or not d) Any other manageability issues with so many slaves If anyone of you have deployed Solr in such a environment it would be great if you can share your experience on the same. Thanks in advance. Regards, Sourav **************** CAUTION - Disclaimer ***************** This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Infosys has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. Infosys reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Infosys e-mail system. ***INFOSYS******** End of Disclaimer ********INFOSYS***
