I believe Hadoop RPC was originally built for distributed search for Nutch. Here's some core code I think Nutch still uses http://svn.apache.org/viewvc/lucene/nutch/trunk/src/java/org/apache/nutch/searcher/DistributedSearch.java?revision=619648&view=markup
One thing I wanted to add to the original email is if some of the core query and filter classes implemented java.io.Externalizable then there would be a speedup in serialization equivalent to using Writeable. It would also be backwards compatible with and enhance the existing distributed search using RMI. Classes that do not implement Externalizable would simply use the default reflection based serialization. On Fri, Jul 11, 2008 at 9:13 AM, Grant Ingersoll <[EMAIL PROTECTED]> wrote: > I believe there is a subproject over at Hadoop for doing distributed stuff > w/ Lucene, but I am not sure if they are doing search side, only indexing. > I was always under the impression that it was too slow for search side, as > I don't think Nutch even uses it for the search side of the equation, but I > don't know if that is still the case. > > > On Jul 10, 2008, at 10:16 PM, Jason Rutherglen wrote: > > Has anyone taken a look at using Hadoop RPC for enabling distributed >> Lucene? I am thinking it would implement the Searchable interface and use >> serialization to be compatible with the current RMI version. Somewhat >> defeats the purpose of using Hadoop RPC and serialization however Hadoop RPC >> scales far beyond what RMI can at the networking level. RMI uses a thread >> per socket and has reportedly has latency issues. Hadoop RPC uses NIO and >> is proven to scale to thousands of servers. Serialization unfortunately >> must be used with Lucene due to the Weight, Query and Filter classes. There >> could be an extended version of Searchable that allows passing Weight, >> Query, and Filter classes that implement Hadoop's Writeable interface if a >> user wants to bypass using serialization. >> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > >
