Hi Thanks for getting back to me. I have briefly looked at Hadoop. It looks interesting but I haven't thought about using it for indexing. But I think it might be on the plan. I am doing the initial investigation and technology frameworks. I would be very interested to get an idea of how you are designing your cluster indexing. I have been doing a documentation which I would be happy to show you.
Cheers Amin On Thu, Jun 11, 2009 at 5:29 PM, Tarandeep Singh <tarand...@gmail.com>wrote: > Hi Amin, > > I am also working on similar implementation. But right now I am working on > building Lucene Index in parallel using Hadoop. > > How are you building your Lucene index (or indexes) ? Have you tried > Hadoop? > If yes, we can share some design issues. > > Also, for serving Lucene indexes over cluster, you might be interested in > Katta project- > katta.sourceforge.net > > -Tarandeep > > On Thu, Jun 11, 2009 at 1:04 AM, Amin Mohammed-Coleman <ami...@gmail.com > >wrote: > > > Hi > > > > we are thinking of implementing a solution like this. All nodes on the > > cluster has a local index from which searches are performed. Once an > > update > > on the index needs to happen the work is sent to a queue where there is a > > master who is responsible for updating the central index. The master is > > then responsible for propagating/notifying other nodes in the cluster > that > > their local index is not in sync. The local index worker can perform an > > update against the master index. > > > > We haven't implemented this yet but its the approach we'll take inorder > to > > deal with clustering. If anyone has any ideas which could improve this > or > > a > > better approach your thoughts will be appreciated. > > > > Cheers > > Amin > > > > On Wed, Jun 10, 2009 at 10:15 AM, Ian Lea <ian....@gmail.com> wrote: > > > > > I'd recommend using your favourite queueing service to pass all > > > updates to a central process, the one and only process that updates > > > the index. If you don't already have a favourite queueing service, > > > > > > http://en.wikipedia.org/wiki/Java_Message_Service#Provider_implementations > > > lists several JMS implementations. > > > > > > > > > -- > > > Ian. > > > > > > On Tue, Jun 9, 2009 at 9:02 PM, Newman, Billy<billy.new...@itt.com> > > wrote: > > > > I am trying to figure out the best way to add to a lucene index > across > > a > > > clustered app server. I cannot grab an IndexWriter for each node in > the > > > cluster, because I would run into lock file problems. I am not sure if > I > > > can share one IndexWriter across the cluster because what happens when > > two > > > or more nodes update the same thing. Anyone have any insight? > > > > > > > > Thanks. > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > > > > >
