There are good synchronization primitives for Tasks, and a bit for threads, but not much for parallel processes. (One could use named system semaphores on Linux and Windows, but there's no Julia wrapper yet AFAIK.) I also found management of parallel processes confusing, and good nontrivial examples are not obvious. So here's my humble offering: https://gist.github.com/RalphAS/2a37b1c631923efa30ac4a7f02a2ee9d
(I just happen to be working on some synchronization problems this month; if a real expert has a better solution, let's hear it.) On Friday, October 14, 2016 at 3:19:25 PM UTC-4, Zachary Roth wrote: > > Thanks for the reply and suggestion, Ralph. I tried to get this working > with semaphores/mutexes/locks/etc. But I've not been having any luck. > > Here's a simplified, incomplete version of what I'm trying to do. I'm > hoping that someone can offer a suggestion if they see some sample code. > > function localfunction() > files = listfiles() > locks = [Threads.SpinLock() for _ in files] > ranges = getindexranges(length(files)) > > pmap(pairs(ranges)) do rows_and_cols > rows, cols = rows_and_cols > workerfunction(files, locks, rows, cols) > end > end > > function workerfunction(files, locks, rows, cols) > data = kindofexpensive(...) > pairs = pairs(rows, cols) > > @sync for idx in unique([rows; cols]) > @async begin > lock(locks[idx]) > updatefile(files[idx], data[idx]) > unlock(locks[idx]) > end > end > end > > This (obviously) does not work. I think that the problem is that the > locks are being copied when the function is spawned on each process. I've > tried wrapping the locks/semaphores in Futures/RemoteChannels, but that > also hasn't worked for me. > > I found that I could do the sort of coordination that I need by starting > Tasks on the local process. More specifically, each file would have an > associated Task to handle the coordination between processes. But this > only worked for me in a simplified situation with the Tasks being declared > globally. When I tried to implement this coordination within localfunction > above, I got an error (really a bunch of errors) that said that a running > Task cannot be serialized. > > Sorry for the long post, but I'm really hoping that someone can help me > out. I have a feeling that I'm missing something pretty simple. > > ---Zachary > > > > > On Tuesday, October 11, 2016 at 10:15:06 AM UTC-4, Ralph Smith wrote: >> >> You can do it with 2 (e.g. integer) channels per worker (requests and >> replies) and a task for each pair in the main process. That's so ugly I'd >> be tempted to write an >> interface to named system semaphores. Or just use a separate file for >> each worker. >> >> On Monday, October 10, 2016 at 11:09:39 AM UTC-4, Zachary Roth wrote: >>> >>> Hi, everyone, >>> >>> I'm trying to save to a single file from multiple worker processes, but >>> don't know of a nice way to coordinate this. When I don't coordinate, >>> saving works fine much of the time. But I sometimes get errors with >>> reading/writing of files, which I'm assuming is happening because multiple >>> processes are trying to use the same file simultaneously. >>> >>> I tried to coordinate this with a queue/channel of `Condition`s managed >>> by a task running in process 1, but this isn't working for me. I've tried >>> to simiplify this to track down the problem. At least part of the issue >>> seems to be writing to the channel from process 2. Specifically, when I >>> `put!` something onto a channel (or `push!` onto an array) from process 2, >>> the channel/array is still empty back on process 1. I feel like I'm >>> missing something simple. Is there an easier way to go about coordinating >>> multiple processes that are trying to access the same file? If not, does >>> anyone have any tips? >>> >>> Thanks for any help you can offer. >>> >>> Cheers, >>> ---Zachary >>> >>
