Check out the EM example in nltk:

http://code.google.com/p/nltk/source/browse/trunk/nltk/nltk_contrib/hadoop/EM/runStreaming.py

On Fri, Mar 27, 2009 at 5:19 PM, Sid123 <[email protected]> wrote:

>
> HI,
> I have to design an iterative algorithm, each iteration is a M-R cycle that
> calculates a parameter and has to feed it back to all the maps in the next
> iteration.
> Now the reduce procedure I need to just sum eveything from the Map
> procedure(Many similar size matrices) into a single matrix(of same size as
> each reduce ), irrespective of the key. This single matrix is the parameter
> I was taking about earlier.
> i want to know. PS This parameter MUST BE global to  all map processes.
>
> 1) How do I collect all the values into one single parameter? Do I need to
> write it to the File system or can i keep it in memory? I feel that I WILL
> have to write it to the HDFS somewhere...
> --
> View this message in context:
> http://www.nabble.com/Iterative-feedback-in-map-reduce....-tp22748317p22748317.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>


-- 
Peter N. Skomoroch
617.285.8348
http://www.datawrangling.com
http://delicious.com/pskomoroch
http://twitter.com/peteskomoroch

Reply via email to