iterative map-reduce
Hi, I want to call a map-reduce program recursively until some condition is met. How do I do that? Thanks, Shirley
Re: iterative map-reduce
if you are using java, just create job configure again and run it, otherwise you just need to write a iterative script. On Tue, Jul 29, 2008 at 9:57 AM, Shirley Cohen [EMAIL PROTECTED] wrote: Hi, I want to call a map-reduce program recursively until some condition is met. How do I do that? Thanks, Shirley
Re: iterative map-reduce
Thanks... would the iterative script be run outside of Hadoop? I was actually trying to figure out if the framework could handle iterations. Shirley On Jul 29, 2008, at 9:10 AM, Qin Gao wrote: if you are using java, just create job configure again and run it, otherwise you just need to write a iterative script. On Tue, Jul 29, 2008 at 9:57 AM, Shirley Cohen [EMAIL PROTECTED] wrote: Hi, I want to call a map-reduce program recursively until some condition is met. How do I do that? Thanks, Shirley
Re: iterative map-reduce
I think it is nothing to do with the framework, just treat the mapredcue as a batch process or a subroutine, and you may iteratively call them. If there are such interface, I am also interested to know. On Tue, Jul 29, 2008 at 10:31 AM, Shirley Cohen [EMAIL PROTECTED]wrote: Thanks... would the iterative script be run outside of Hadoop? I was actually trying to figure out if the framework could handle iterations. Shirley On Jul 29, 2008, at 9:10 AM, Qin Gao wrote: if you are using java, just create job configure again and run it, otherwise you just need to write a iterative script. On Tue, Jul 29, 2008 at 9:57 AM, Shirley Cohen [EMAIL PROTECTED] wrote: Hi, I want to call a map-reduce program recursively until some condition is met. How do I do that? Thanks, Shirley
Re: iterative map-reduce
Hi Shirley, I am basically doing as Qin suggested. I am running a job iteratively until some condition is met. My main looks something like:(in pseudo code) main: while (!converged): make new jobconf setup jobconf run jobconf check reporter for statistics decide if converged I use a custom reporter to check on the fitness of the solution in the reduce phase. If you need more(real java) code drop me a line. Cheers, Christian Qin Gao wrote: I think it is nothing to do with the framework, just treat the mapredcue as a batch process or a subroutine, and you may iteratively call them. If there are such interface, I am also interested to know. On Tue, Jul 29, 2008 at 10:31 AM, Shirley Cohen [EMAIL PROTECTED]wrote: Thanks... would the iterative script be run outside of Hadoop? I was actually trying to figure out if the framework could handle iterations. Shirley On Jul 29, 2008, at 9:10 AM, Qin Gao wrote: if you are using java, just create job configure again and run it, otherwise you just need to write a iterative script. On Tue, Jul 29, 2008 at 9:57 AM, Shirley Cohen [EMAIL PROTECTED] wrote: Hi, I want to call a map-reduce program recursively until some condition is met. How do I do that? Thanks, Shirley
Re: iterative map-reduce
A simple example of Hadoop application code which follows that pattern (iterate until condition). In the jyte section: http://code.google.com/p/ceteri-mapred/ Loop and condition test are in the same code which calls ToolRunner and JobClient. Best, Paco On Tue, Jul 29, 2008 at 10:03 AM, Christian Ulrik Søttrup [EMAIL PROTECTED] wrote: Hi Shirley, I am basically doing as Qin suggested. I am running a job iteratively until some condition is met. My main looks something like:(in pseudo code) main: while (!converged): make new jobconf setup jobconf run jobconf check reporter for statistics decide if converged I use a custom reporter to check on the fitness of the solution in the reduce phase. If you need more(real java) code drop me a line. Cheers, Christian
