On 27/06/2005, at 7:14 PM, Nader Henein wrote:
I implemented a JMS based solution about a year ago because I thought it would solve my atomicity problem and give me a centralized way of indexing, you'll have to use the pluggable persistence (if you use ActiveMQ) to be able to recover from a failure and you'll also need some way of maintaining which records have and haven't been indexed in a persistent store so you can run sanitization later on, assuming that every time you add a document to one of you clustered indecies everything goes well is a sure way of guaranteeing that you end up with n indecies with no two alike especially if you have a high volume indecies with high rates of change.
The rate of updates is actually quite low in our environment, so we are lucky, however we will be monitoring consistency closely. Each node exposes statistics about the last item indexed etc so we can tell when things are out of sync. Should a node consistently be out of sync we'll just remove it as a worker from Apache for searching purposes (still leaving it ready to be used as a hot spare).
We are fortunate that the volume of updates is relatively small (as fast as a human can upload things), so we are in a fortunate position in our case. Our guarantees are in the order of minutes rather than seconds.
Paul
