Michael That's right. With `sow` the mod.eval of the third argument gets bound to the second; where mod defaults to Main. Maybe someone could think of a cleaner way to make values available for later work, but it seems to do the trick. It seems best to avoid using the `sow` function heavily.
`reap` returns the mod.eval of the second argument with no assignment on each pid. Good luck! Matthew On Thursday, May 19, 2016 at 8:18:17 PM UTC+1, Michael Eastwood wrote: > > Hi Matthew, > > ClusterUtils.jl looks very useful. I will definitely try it out. Am I > correct in reading that the trick to moving input to the workers is here > <https://github.com/pearcemc/ClusterUtils.jl/blob/ac5eb73bd565b43d0b05b9d8af1c930cef4088b7/src/ClusterUtils.jl#L157-L160> > ? > > You're also correct that write_results_to_disk does actually depend on > myidx. I might have somewhat oversimplified the example. > > Thanks, > Michael > > On Thursday, May 19, 2016 at 7:41:49 AM UTC-7, Matthew Pearce wrote: >> >> Hi Michael >> >> Your current code looks like will pull back the `coefficients` across the >> network (500 gb transfer) and as you point out transfer `input` each time. >> >> I wrote a package ClusterUtils.jl >> <https://github.com/pearcemc/ClusterUtils.jl> to handle my own problems >> (MCMC sampling) which were somewhat similar. >> >> Roughly - given the available info - if I was trying to do something >> similar I'd do: >> >> ```julia >> using Compat >> using ClusterUtils >> >> sow(pids, :input, input) >> >> @everywhere function dostuff(input, myidxs) >> for myidx in myidxs >> coefficients = spherical_harmonic_transforms(input[myidx]) >> write_results_to_disk(coefficients) #needs myidx as arg too probably >> end >> end >> >> idxs = chunkit(limit, length(pids)) >> sow(pids, :work, :(Dict(zip($pids, $idxs)))) >> >> reap(pids, :(dostuff(input, $work[myid()]))) >> ``` >> >> This transfers `input` once, and writes something to disk from the remote >> process. >> >> >> >>
