Hi I'm looking at Hadoop and Katta and I was wondering if some may be able clarify the following:
1) Is Katta replacing the Hadoop Lucene contribution 2) Are people still using Hadoop Lucene to perform indexing Cheers Amin On Sat, Jun 13, 2009 at 7:46 AM, Amin Mohammed-Coleman <ami...@gmail.com>wrote: > Hi > Thanks for getting back to me. I have briefly looked at Hadoop. It looks > interesting but I haven't thought about using it for indexing. But I think > it might be on the plan. I am doing the initial investigation and > technology frameworks. I would be very interested to get an idea of how you > are designing your cluster indexing. I have been doing a documentation > which I would be happy to show you. > > > Cheers > Amin > > > On Thu, Jun 11, 2009 at 5:29 PM, Tarandeep Singh <tarand...@gmail.com>wrote: > >> Hi Amin, >> >> I am also working on similar implementation. But right now I am working on >> building Lucene Index in parallel using Hadoop. >> >> How are you building your Lucene index (or indexes) ? Have you tried >> Hadoop? >> If yes, we can share some design issues. >> >> Also, for serving Lucene indexes over cluster, you might be interested in >> Katta project- >> katta.sourceforge.net >> >> -Tarandeep >> >> On Thu, Jun 11, 2009 at 1:04 AM, Amin Mohammed-Coleman <ami...@gmail.com >> >wrote: >> >> > Hi >> > >> > we are thinking of implementing a solution like this. All nodes on the >> > cluster has a local index from which searches are performed. Once an >> > update >> > on the index needs to happen the work is sent to a queue where there is >> a >> > master who is responsible for updating the central index. The master is >> > then responsible for propagating/notifying other nodes in the cluster >> that >> > their local index is not in sync. The local index worker can perform an >> > update against the master index. >> > >> > We haven't implemented this yet but its the approach we'll take inorder >> to >> > deal with clustering. If anyone has any ideas which could improve this >> or >> > a >> > better approach your thoughts will be appreciated. >> > >> > Cheers >> > Amin >> > >> > On Wed, Jun 10, 2009 at 10:15 AM, Ian Lea <ian....@gmail.com> wrote: >> > >> > > I'd recommend using your favourite queueing service to pass all >> > > updates to a central process, the one and only process that updates >> > > the index. If you don't already have a favourite queueing service, >> > > >> > >> http://en.wikipedia.org/wiki/Java_Message_Service#Provider_implementations >> > > lists several JMS implementations. >> > > >> > > >> > > -- >> > > Ian. >> > > >> > > On Tue, Jun 9, 2009 at 9:02 PM, Newman, Billy<billy.new...@itt.com> >> > wrote: >> > > > I am trying to figure out the best way to add to a lucene index >> across >> > a >> > > clustered app server. I cannot grab an IndexWriter for each node in >> the >> > > cluster, because I would run into lock file problems. I am not sure >> if I >> > > can share one IndexWriter across the cluster because what happens when >> > two >> > > or more nodes update the same thing. Anyone have any insight? >> > > > >> > > > Thanks. >> > > >> > > --------------------------------------------------------------------- >> > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> > > For additional commands, e-mail: java-user-h...@lucene.apache.org >> > > >> > > >> > >> > >
