On Tue, 2008-09-02 at 17:26 +0200, Ryan McKinley wrote: > > > >> 3. Distributed crawling (i.e. give a bunch of links to it and some > >> distributed compute resources and have it go to town) > > > > If you mean hadoop style, no. However you can start various droids on > > different systems. > > > I'm looking into the same thing... (ideally sans-hadoop) > > I have not had time to look at implementing it, but design wise, (i > *think*) it would just take making a distributed Queue > implementation. perhaps this could be done with jmx or maybe > something akin to amazon simple queue service. That is a bunch of > workers can read and write "tasks" from a central queue.
That sounds swell and actually if someone knows a distributed Map implementation let me know and I will wrap something up. salu2 > > ... but i have not (yet) had time to investigate if this is a > relativly easy option > > ryan > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > -- Thorsten Scherler thorsten.at.apache.org Open Source Java consulting, training and solutions --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
