On Thu, 20 Sep 2007 09:53:46 -0400 "Yonik Seeley" <[EMAIL PROTECTED]> wrote:
> On 9/19/07, Norberto Meijome <[EMAIL PROTECTED]> wrote: > > Maybe I got this wrong...but isn't this what mapreduce is meant to deal > > with? > > Not really... you could force a *lot* of different problems into > map-reduce (that's sort of the point... being able to automatically > parallelize a lot of different problems). It really isn't the best > fit though, and would end up being much slower than a custom job. good point..i wondered whether the whole sorting/whatever wasn't going to make it far slower than something custom. I dont care about mapreduce in particular, but yes the effect - n indexers / searches all fulfilling their part of the overall search results. > Then there is the issue that the way map-reduce is implemented (like > hadoop) is also tuned for longer running batch jobs on huge data > (temporary files are used, external sorts, initial input, final output > is via files, etc). I see, didn't know this. > Check out the google map-reduce paper - they > don't use it for their search side either. yeah, need to :) > Things are already progressing in the distributed search area: > https://issues.apache.org/jira/browse/SOLR-303 > Hopefully I'll have time to dig into it more myself in a few weeks. excellent , thanks _________________________ {Beto|Norberto|Numard} Meijome "He uses statistics as a drunken man uses lamp-posts ... for support rather than illumination." Andrew Lang (1844-1912) I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.
