Re: Index updates blocking readers: To Multicore or not?
Thanks Yonik, I have more information... 1. We do indeed have large indexes: 40GB on disk, 30M documents - and is just a test server we have 8 of these in parallel. 2. The performance problem I was seeing followed replication, and first query on a new searcher. It turns out we didn't configure index warming queries very well so we removes the various solr rocks type queries to one that was better for our data - and had not improvement. The problem was that replication completed, a new searcher was created and registered but the first query qould take 10-20 seconds to complete. There after it took 200 milliseconds for similar non-cached queries. Profiler pointed us to building the FieldSortedHitQueue was taking all the time. Our warming query did not include a sort but our queries commonly do. Once we added the sort parameter our warming query started taking the 10-20 seconds prior to registering the searcher. After that the first query on the new searcher took the expected 200ms. LESSON LEARNED: warm your caches! And, if a sort is involved in your queries incorporate that sort in your warming query! Add a warming query for each kind of sort that you expect to do. Yonik Seeley wrote: On Mon, Oct 6, 2008 at 2:10 PM, Jim Murphy [EMAIL PROTECTED] wrote: We have a farm of several Master-Slave pairs all managing a single very large logical index sharded across the master-slaves. We notice on the slaves, after an rsync update, as the index is being committed that all queries are blocked sometimes resulting in unacceptable service times. I'm looking at ways we can manage these update burps. Updates should never block queries. What version of Solr are you using? Is it possible that your indexes are so big, opening a new index in the background causes enough of the old index to be flushed from OS cache, causing big slowdowns? -Yonik Question #1: Anything obvious I can tweak in the configuration to mitigate these multi-second blocking updates? Our Indexes are 40GB, 20M documents each. RSync updates are every 5 minutes several hundred KB per update. Question #2: I'm considering setting up each slave with multiple Solr cores. The 2 indexes per instance would be nearly identical copies but A would be read from while B is being updated, then they would swap. I'll have to figure out how to rsync these 2 indexes properly but if I can get the commits to happen to the offline index then I suspect my queries could proceed unblocked. Is this the wrong tree to be barking up? Any other thoughts? Thanks in advance, Jim -- View this message in context: http://www.nabble.com/Index-updates-blocking-readers%3A-To-Multicore-or-not--tp19843098p19843098.html Sent from the Solr - User mailing list archive at Nabble.com. -- View this message in context: http://www.nabble.com/Index-updates-blocking-readers%3A-To-Multicore-or-not--tp19843098p20112546.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Index updates blocking readers: To Multicore or not?
Jim, This is a off topic question. But for your 30M documents, did you fetch those from external web sites (Whole Web Search)? Or are they internal documents? If they are external what method did you use to fetch them and which spider? I am in the process of deciding between using Nutch for whole web indexing, Solr + Spider?, or Nutch + Solr, etc. Thank you in advance for your insight into this issue. -John On Oct 22, 2008, at 10:55 AM, Jim Murphy wrote: Thanks Yonik, I have more information... 1. We do indeed have large indexes: 40GB on disk, 30M documents - and is just a test server we have 8 of these in parallel. 2. The performance problem I was seeing followed replication, and first query on a new searcher. It turns out we didn't configure index warming queries very well so we removes the various solr rocks type queries to one that was better for our data - and had not improvement. The problem was that replication completed, a new searcher was created and registered but the first query qould take 10-20 seconds to complete. There after it took 200 milliseconds for similar non-cached queries. Profiler pointed us to building the FieldSortedHitQueue was taking all the time. Our warming query did not include a sort but our queries commonly do. Once we added the sort parameter our warming query started taking the 10-20 seconds prior to registering the searcher. After that the first query on the new searcher took the expected 200ms. LESSON LEARNED: warm your caches! And, if a sort is involved in your queries incorporate that sort in your warming query! Add a warming query for each kind of sort that you expect to do. Yonik Seeley wrote: On Mon, Oct 6, 2008 at 2:10 PM, Jim Murphy [EMAIL PROTECTED] wrote: We have a farm of several Master-Slave pairs all managing a single very large logical index sharded across the master-slaves. We notice on the slaves, after an rsync update, as the index is being committed that all queries are blocked sometimes resulting in unacceptable service times. I'm looking at ways we can manage these update burps. Updates should never block queries. What version of Solr are you using? Is it possible that your indexes are so big, opening a new index in the background causes enough of the old index to be flushed from OS cache, causing big slowdowns? -Yonik Question #1: Anything obvious I can tweak in the configuration to mitigate these multi-second blocking updates? Our Indexes are 40GB, 20M documents each. RSync updates are every 5 minutes several hundred KB per update. Question #2: I'm considering setting up each slave with multiple Solr cores. The 2 indexes per instance would be nearly identical copies but A would be read from while B is being updated, then they would swap. I'll have to figure out how to rsync these 2 indexes properly but if I can get the commits to happen to the offline index then I suspect my queries could proceed unblocked. Is this the wrong tree to be barking up? Any other thoughts? Thanks in advance, Jim -- View this message in context: http://www.nabble.com/Index-updates-blocking-readers%3A-To-Multicore-or-not--tp19843098p19843098.html Sent from the Solr - User mailing list archive at Nabble.com. -- View this message in context: http://www.nabble.com/Index-updates-blocking-readers%3A-To-Multicore-or-not--tp19843098p20112546.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Index updates blocking readers: To Multicore or not?
We index RSS content using our own home grown distributed spiders - not using Nutch. We use ruby processes do do the feed fetching and XML shreading, and Amazon SQS to queue up work packets to insert into our Solr cluster. Sorry can't be of more help. -- View this message in context: http://www.nabble.com/Index-updates-blocking-readers%3A-To-Multicore-or-not--tp19843098p20113143.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Index updates blocking readers: To Multicore or not?
Thank you that is good information, as that is kind of way that I am leaning. So when you fetch the content from RSS, does that get rendered to an XML document that Solr indexes? Also what where a couple of decision points for using Solr as opposed to using Nutch, or even straight Lucene? -John On Oct 22, 2008, at 11:22 AM, Jim Murphy wrote: We index RSS content using our own home grown distributed spiders - not using Nutch. We use ruby processes do do the feed fetching and XML shreading, and Amazon SQS to queue up work packets to insert into our Solr cluster. Sorry can't be of more help. -- View this message in context: http://www.nabble.com/Index-updates-blocking-readers%3A-To-Multicore-or-not--tp19843098p20113143.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Index updates blocking readers: To Multicore or not?
On Mon, Oct 6, 2008 at 2:10 PM, Jim Murphy [EMAIL PROTECTED] wrote: We have a farm of several Master-Slave pairs all managing a single very large logical index sharded across the master-slaves. We notice on the slaves, after an rsync update, as the index is being committed that all queries are blocked sometimes resulting in unacceptable service times. I'm looking at ways we can manage these update burps. Updates should never block queries. What version of Solr are you using? Is it possible that your indexes are so big, opening a new index in the background causes enough of the old index to be flushed from OS cache, causing big slowdowns? -Yonik Question #1: Anything obvious I can tweak in the configuration to mitigate these multi-second blocking updates? Our Indexes are 40GB, 20M documents each. RSync updates are every 5 minutes several hundred KB per update. Question #2: I'm considering setting up each slave with multiple Solr cores. The 2 indexes per instance would be nearly identical copies but A would be read from while B is being updated, then they would swap. I'll have to figure out how to rsync these 2 indexes properly but if I can get the commits to happen to the offline index then I suspect my queries could proceed unblocked. Is this the wrong tree to be barking up? Any other thoughts? Thanks in advance, Jim -- View this message in context: http://www.nabble.com/Index-updates-blocking-readers%3A-To-Multicore-or-not--tp19843098p19843098.html Sent from the Solr - User mailing list archive at Nabble.com.
Index updates blocking readers: To Multicore or not?
We have a farm of several Master-Slave pairs all managing a single very large logical index sharded across the master-slaves. We notice on the slaves, after an rsync update, as the index is being committed that all queries are blocked sometimes resulting in unacceptable service times. I'm looking at ways we can manage these update burps. Question #1: Anything obvious I can tweak in the configuration to mitigate these multi-second blocking updates? Our Indexes are 40GB, 20M documents each. RSync updates are every 5 minutes several hundred KB per update. Question #2: I'm considering setting up each slave with multiple Solr cores. The 2 indexes per instance would be nearly identical copies but A would be read from while B is being updated, then they would swap. I'll have to figure out how to rsync these 2 indexes properly but if I can get the commits to happen to the offline index then I suspect my queries could proceed unblocked. Is this the wrong tree to be barking up? Any other thoughts? Thanks in advance, Jim -- View this message in context: http://www.nabble.com/Index-updates-blocking-readers%3A-To-Multicore-or-not--tp19843098p19843098.html Sent from the Solr - User mailing list archive at Nabble.com.
