[julia-users] Re: Trying to understand how to structure parallel computation
Maybe the pmap example from the doc can be useful, it feeds work to other processes: http://julia.readthedocs.org/en/latest/manual/parallel-computing/#synchronization-with-remote-references
Re: [julia-users] Re: Trying to understand how to structure parallel computation
I have not looked at your code, but I can suggest that you model your code in the following manner. - process 1 is the "driver" of the parrallel computations - it does not do any computations itself. - process 2 runs the "main" loop - process 3 runs subproblems - process 1 after starting the loop on process 2, waits on a remoteref (on pid 1) that is set by process 2 after all computations are complete - all remote refs that track task completion are only on process 1 - Now, when process 2 does an "isready", since the remoteref being tested is on process 1, it will not block. On Mon, Mar 24, 2014 at 4:14 AM, Iain Dunning wrote: > (talking to myself) isready seems to block, is that expected? > > Here is my code > https://gist.github.com/IainNZ/9730991 > > Is there a nonblocking isready or am I barking up the wrong tree? > > > > > On Sunday, March 23, 2014 6:10:59 PM UTC-4, Iain Dunning wrote: >> >> Digging more into the "standard library" part of the manual, is this a >> matter of using "isready" at the end of my main loop, which I know will be >> safe because no one else is going to be looking at that RemoteRef? >> >> On Sunday, March 23, 2014 5:58:27 PM UTC-4, Iain Dunning wrote: >>> >>> Hi all, >>> >>> I've never really used the parallel stuff for a "real" task and I'm >>> trying to understand the Julian way of structuring my computation. >>> >>> Heres the situation: >>> >>> - I have a "main" loop that is solving a series of problems >>> - After solving one of these prolems, I sometimes want to solve an >>> expensive subproblem that might improve the solution. >>> - I want to solve this subproblem in a separate process, and I don't >>> need the answer right away. >>> >>> Mentally I'm thinking of processor 1 running the main loop, and >>> processor 2 working on solving any subproblems I send its way (queueing >>> them up perhaps). >>> At the end/start of every iteration of the main loop on processor 1 I'd >>> "check" processor 2 to see if it has any solutions for me, and collect them >>> if it has. >>> >>> Can someone help me out with how I should be thinking of this? >>> >>> Cheers, >>> Iain >>> >>> >>> >>>
[julia-users] Re: Trying to understand how to structure parallel computation
(talking to myself) isready seems to block, is that expected? Here is my code https://gist.github.com/IainNZ/9730991 Is there a nonblocking isready or am I barking up the wrong tree? On Sunday, March 23, 2014 6:10:59 PM UTC-4, Iain Dunning wrote: > > Digging more into the "standard library" part of the manual, is this a > matter of using "isready" at the end of my main loop, which I know will be > safe because no one else is going to be looking at that RemoteRef? > > On Sunday, March 23, 2014 5:58:27 PM UTC-4, Iain Dunning wrote: >> >> Hi all, >> >> I've never really used the parallel stuff for a "real" task and I'm >> trying to understand the Julian way of structuring my computation. >> >> Heres the situation: >> >> - I have a "main" loop that is solving a series of problems >> - After solving one of these prolems, I sometimes want to solve an >> expensive subproblem that might improve the solution. >> - I want to solve this subproblem in a separate process, and I don't need >> the answer right away. >> >> Mentally I'm thinking of processor 1 running the main loop, and processor >> 2 working on solving any subproblems I send its way (queueing them up >> perhaps). >> At the end/start of every iteration of the main loop on processor 1 I'd >> "check" processor 2 to see if it has any solutions for me, and collect them >> if it has. >> >> Can someone help me out with how I should be thinking of this? >> >> Cheers, >> Iain >> >> >> >>
[julia-users] Re: Trying to understand how to structure parallel computation
Digging more into the "standard library" part of the manual, is this a matter of using "isready" at the end of my main loop, which I know will be safe because no one else is going to be looking at that RemoteRef? On Sunday, March 23, 2014 5:58:27 PM UTC-4, Iain Dunning wrote: > > Hi all, > > I've never really used the parallel stuff for a "real" task and I'm trying > to understand the Julian way of structuring my computation. > > Heres the situation: > > - I have a "main" loop that is solving a series of problems > - After solving one of these prolems, I sometimes want to solve an > expensive subproblem that might improve the solution. > - I want to solve this subproblem in a separate process, and I don't need > the answer right away. > > Mentally I'm thinking of processor 1 running the main loop, and processor > 2 working on solving any subproblems I send its way (queueing them up > perhaps). > At the end/start of every iteration of the main loop on processor 1 I'd > "check" processor 2 to see if it has any solutions for me, and collect them > if it has. > > Can someone help me out with how I should be thinking of this? > > Cheers, > Iain > > > >
