Yes, if you think about how blogs and Technorati work (new blog post -> ping -> ping server -> technorati -> index ==> searchable blog post), Adam is correct. Since Doug's implementation we (Technorati) have changed our clusters a LOT (think major rewrites). I can't talk about the details, obviously, but the first paragraph in Mark's email sounds like what we have.
Biggy will want to provide more details about how they want things to work (e.g. how quickly do changes need to be visible to searchers? Is this a write + read system, or are there deletes? ...) before others can make useful suggestions. Otis ----- Original Message ---- From: Adam Fleming <[EMAIL PROTECTED]> To: java-user@lucene.apache.org Sent: Thursday, December 28, 2006 2:33:37 AM Subject: RE: Clustering Lucene with 40 Servers Hello, I saw that Doug Cutting had an interesting solution for his Technorati website: http://www.mail-archive.com/lucene-user@jakarta.apache.org/msg12709.html It sounds like it's a single-writer, many readers type of system, but quite robust and efficient. Cheers, Adam ---------------------------------------- > Date: Wed, 27 Dec 2006 10:46:47 -0800 > From: [EMAIL PROTECTED] > To: java-user@lucene.apache.org > Subject: Re: Clustering Lucene with 40 Servers > > Some quick questions/points: > > What is the update rate? > > The number of nodes you described is no problem, the query rate would be > no problem too (because they use read locks and act independently). > > Do all nodes do updates or just 1? How often do these updates occur? > > Probably best thing to do is give it a try and run a few tests. > > Cheers, > Steve > > > On 12/27/06, Biggy <[EMAIL PROTECTED]> wrote: > > > > I'm currently investigating the best ways of clustering Lucene. > > I've heard of both Solr, Terracotta but do not know how well they scale. > > Their examples talk of a 4 node cluster. This is way too small for my needs. > > > > I have 30x JVMs each handling 3 requests/sec and each having their own > > Lucene index. The index changes are propagated to the cluster members using > > JGroups messages. This solution has more than reached its limit as JGroups > > has become unstable and a source of many JVMs crashes. Based on current > > traffic trends I anticipate needing to upgrade to 40x + JVMS very soon. > > > > Can anybody suggest a way to effectivily cluster / replicate document > > changes ? > > > > P.S.: > > JMS is not a possible solution as this was the prior JGroups solution. We > > had too many memory problems/queue full etc crashing the servers thereafter. > > > > -- > > View this message in context: > > http://www.nabble.com/Clustering-Lucene-with-40-Servers-tf2886546.html#a8064135 > > Sent from the Lucene - Java Users mailing list archive at Nabble.com. > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > _________________________________________________________________ Fixing up the home? Live Search can help. http://imagine-windowslive.com/search/kits/default.aspx?kit=improve&locale=en-US&source=wlmemailtaglinenov06 --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
