My experiments with parallelism tend to occur in focused blocks, and I haven't done it in quite a while. So I doubt I can help much. But in general I suspect you're encountering these problems because much of the IPC goes through thunks, and so a lot of stuff gets reclaimed when execution is done.
If I were experimenting, I'd start by trying to create RemoteRef()s and put! ()ing my variables into them. Then perhaps you might be able to fetch them from other processes. Not sure that will work, but it seems to be worth a try. HTH, --Tim On Thursday, November 20, 2014 08:20:19 PM Madeleine Udell wrote: > I'm trying to use parallelism in julia for a task with a structure that I > think is quite pervasive. It looks like this: > > # broadcast lists of functions f and g to all processes so they're > available everywhere > # create shared arrays X,Y on all processes so they're available everywhere > for iteration=1:1000 > @parallel for i=1:size(X) > X[i] = f[i](Y) > end > @parallel for j=1:size(Y) > Y[j] = g[j](X) > end > end > > I'm having trouble making this work, and I'm not sure where to dig around > to find a solution. Here are the difficulties I've encountered: > > * @parallel doesn't allow me to create persistent variables on each > process; ie, the following results in an error. > > s = Base.shmem_rand(12,3) > @parallel for i=1:nprocs() m,n = size(s) end > @parallel for i=1:nprocs() println(m) end > > * @everywhere does allow me to create persistent variables on each process, > but doesn't send any data at all, including the variables I need in order > to define new variables. Eg the following is an error: s is a shared array, > but the variable (ie pointer to) s is apparently not shared. > s = Base.shmem_rand(12,3) > @everywhere m,n = size(s) > > Here are the kinds of questions I'd like to see protocode for: > * How can I broadcast a variable so that it is available and persistent on > every process? > * How can I create a reference to the same shared array "s" that is > accessible from every process? > * How can I send a command to be performed in parallel, specifying which > variables should be sent to the relevant processes and which should be > looked up in the local namespace? > > Note that everything I ask above is not specific to shared arrays; the same > constructs would also be extremely useful in the distributed case. > > ---------------------- > > An interesting partial solution is the following: > funcs! = Function[x->x[:] = x+k for k=1:3] > d = drand(3,12) > let funcs! = funcs! > @sync @parallel for k in 1:3 > funcs) > end > end > > Here, I'm not sure why the let statement is necessary to send funcs!, since > d is sent automatically. > > --------------------- > > Thanks! > Madeleine
