The difficulty with data transfer between tasks is handling synchronisation and failure. You may want to look at graph processing done on top of Hadoop (like Giraph). That's one way to do it but whether it is relevant or not to you will depend on your context.
Regards Bertrand On Wed, Sep 26, 2012 at 5:36 PM, Jane Wayne <[email protected]>wrote: > hi, > > i know that some algorithms cannot be parallelized and adapted to the > mapreduce paradigm. however, i have noticed that in most cases where i > find myself struggling to express an algorithm in mapreduce, the > problem is mainly due to no ability to cross-communicate between > mappers or reducers. > > one naive approach i've seen mentioned here and elsewhere, is to use a > database to store data for use by all the mappers. however, i have > seen many arguments (that i agree with largely) against this approach. > > in general, my question is this: has anyone tried to implement an > algorithm using mapreduce where mappers required cross-communications? > how did you solve this limitation of mapreduce? > > thanks, > > jane. > -- Bertrand Dechoux
