...but the index resides on disk doesn't it??? lol -----Original Message----- From: Otis Gospodnetic [mailto:otis_gospodne...@yahoo.com] Sent: Wednesday, March 09, 2011 9:06 AM To: solr-user@lucene.apache.org Subject: Re: True master-master fail-over without data gaps
Hi, ----- Original Message ---- > > I'd honestly think about buffer the incoming documents in some store that's >actually made for fail-over persistence reliability, maybe CouchDB or something. >And then that's taking care of not losing anything, and the problem becomes how >we make sure that our solr master indexes are kept in sync with the actual >persistent store; which I'm still not sure about, but I'm thinking it's a >simpler problem. The right tool for the right job, that kind of failover >persistence is not solr's specialty. > But check this! In some cases one is not allowed to save content to disk (think copyrights). I'm not making this up - we actually have a customer with this "cannot save to disk" (but can index) requirement. So buffering to disk is not an option, and buffering in memory is not practical because of the input document rate and their size. Otis ---- Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ > From: Otis Gospodnetic [otis_gospodne...@yahoo.com] > Sent: Tuesday, March 08, 2011 11:45 PM > To: solr-user@lucene.apache.org > Subject: True master-master fail-over without data gaps > > Hello, > > What are some common or good ways to handle indexing (master) fail-over? > Imagine you have a continuous stream of incoming documents that you have to > index without losing any of them (or with losing as few of them as possible). > How do you set up you masters? > In other words, you can't just have 2 masters where the secondary is the > Repeater (or Slave) of the primary master and replicates the index >periodically: > you need to have 2 masters that are in sync at all times! > How do you achieve that? > > * Do you just put N masters behind a LB VIP, configure them both to point to >the > index on some shared storage (e.g. SAN), and count on the LB to fail-over to >the > secondary master when the primary becomes unreachable? > If so, how do you deal with index locks? You use the Native lock and count on > it disappearing when the primary master goes down? That means you count on >the > whole JVM process dying, which may not be the case... > > * Or do you use tools like DRBD, Corosync, Pacemaker, etc. to keep 2 masters > with 2 separate indices in sync, while making sure you write to only 1 of them > via LB VIP or otherwise? > > * Or ... > > > This thread is on a similar topic, but is inconclusive: > http://search-lucene.com/m/aOsyN15f1qd1 > > Here is another similar thread, but this one doesn't cover how 2 masters are > kept in sync at all times: > http://search-lucene.com/m/aOsyN15f1qd1 > > Thanks, > Otis > ---- > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch > Lucene ecosystem search :: http://search-lucene.com/ > >
