Re: Safely writing to the same file in parallel foreach loop

2012-11-15 Thread Jonathan M Davis
On Thursday, November 15, 2012 15:33:31 Joseph Rushton Wakeling wrote: > On 11/14/2012 10:17 PM, Jonathan M Davis wrote: > > I would point out though that given how expensive disk writes are, unless > > you're doing a lot of work within the parallel foreach loop, there's a > > good > > chance that

Re: Safely writing to the same file in parallel foreach loop

2012-11-15 Thread Vijay Nayar
I'm not a robot and didn't mean to spam, the page got stuck in this odd refresh loop and I wasn't sure what was going on. On Wednesday, 14 November 2012 at 17:45:35 UTC, Vijay Nayar wrote: On Wednesday, 14 November 2012 at 16:43:37 UTC, Joseph Rushton Wakeling wrote: On 11/14/2012 05:16 PM, H.

Re: Safely writing to the same file in parallel foreach loop

2012-11-15 Thread Joseph Rushton Wakeling
On 11/14/2012 10:17 PM, Jonathan M Davis wrote: I would point out though that given how expensive disk writes are, unless you're doing a lot of work within the parallel foreach loop, there's a good chance that it would be more efficient to use std.concurrency and pass the writes to another thread

Re: Safely writing to the same file in parallel foreach loop

2012-11-15 Thread Joseph Rushton Wakeling
On 11/15/2012 12:31 PM, Joseph Rushton Wakeling wrote: On 11/15/2012 01:55 AM, Joseph Rushton Wakeling wrote: An oddity here: although the correct results seem to come out of the calculation, at the end, the program containing the parallel foreach hangs -- it doesn't stop running, even though al

Re: Safely writing to the same file in parallel foreach loop

2012-11-15 Thread Joseph Rushton Wakeling
On 11/15/2012 01:55 AM, Joseph Rushton Wakeling wrote: An oddity here: although the correct results seem to come out of the calculation, at the end, the program containing the parallel foreach hangs -- it doesn't stop running, even though all the calculations are complete. Any thoughts as to why

Re: Safely writing to the same file in parallel foreach loop

2012-11-14 Thread Joseph Rushton Wakeling
On 11/15/2012 12:44 AM, Joseph Rushton Wakeling wrote: In the application I have in mind, there is a LOT of work that would be done within the parallel foreach loop -- we're talking at least 20 minutes' solid processing before the file write takes place, so this seems an appropriate approach give

Re: Safely writing to the same file in parallel foreach loop

2012-11-14 Thread Joseph Rushton Wakeling
On 11/14/2012 10:17 PM, Jonathan M Davis wrote: I would point out though that given how expensive disk writes are, unless you're doing a lot of work within the parallel foreach loop, there's a good chance that it would be more efficient to use std.concurrency and pass the writes to another thread

Re: Safely writing to the same file in parallel foreach loop

2012-11-14 Thread Jonathan M Davis
On Wednesday, November 14, 2012 18:59:29 Joseph Rushton Wakeling wrote: > On 11/14/2012 06:49 PM, Vijay Nayar wrote: > > Could you put the file access in a synchronized block? > > > > http://dlang.org/statement.html#SynchronizedStatement > > Oh, good call -- seems to work. I would point out thou

Re: Safely writing to the same file in parallel foreach loop

2012-11-14 Thread Vijay Nayar
I think this is what you want around the file access section: http://dlang.org/statement.html#SynchronizedStatement - Vijay On Wednesday, 14 November 2012 at 16:43:37 UTC, Joseph Rushton Wakeling wrote: I take it there's no more "native-to-D" way of implementing a file lock? :-(

Re: Safely writing to the same file in parallel foreach loop

2012-11-14 Thread Joseph Rushton Wakeling
On 11/14/2012 06:49 PM, Vijay Nayar wrote: Could you put the file access in a synchronized block? http://dlang.org/statement.html#SynchronizedStatement Oh, good call -- seems to work. If you try and run the parallel code without it, there's a pretty nasty-looking error: /tmp/.rdmd-1000/

Re: Safely writing to the same file in parallel foreach loop

2012-11-14 Thread Vijay Nayar
On Wednesday, 14 November 2012 at 16:43:37 UTC, Joseph Rushton Wakeling wrote: On 11/14/2012 05:16 PM, H. S. Teoh wrote: I take it there's no more "native-to-D" way of implementing a file lock? :-( Could you put the file access in a synchronized block? http://dlang.org/statement.html#Synchron

Re: Safely writing to the same file in parallel foreach loop

2012-11-14 Thread Vijay Nayar
On Wednesday, 14 November 2012 at 16:43:37 UTC, Joseph Rushton Wakeling wrote: On 11/14/2012 05:16 PM, H. S. Teoh wrote: I take it there's no more "native-to-D" way of implementing a file lock? :-( Could you put the file access in a synchronized block? http://dlang.org/statement.html#Synchron

Re: Safely writing to the same file in parallel foreach loop

2012-11-14 Thread Joseph Rushton Wakeling
On 11/14/2012 05:16 PM, H. S. Teoh wrote: If you're on Posix, you can use file locks to ensure atomic writes to the file (all threads have to use it though: it's only an advisory lock, not a mandatory lock): see the manpage for fcntl, look for F_GETLK. I take it there's no more "native-to-D" wa

Re: Safely writing to the same file in parallel foreach loop

2012-11-14 Thread H. S. Teoh
On Wed, Nov 14, 2012 at 04:56:53PM +0100, Joseph Rushton Wakeling wrote: > Suppose that I've got a foreach loop in which I write output to a file: > > auto f = File("test.txt", "w"); f.close(); // to start with a blank file > foreach(i; iota(0, 100)) > { > f = File("test.txt",

Safely writing to the same file in parallel foreach loop

2012-11-14 Thread Joseph Rushton Wakeling
Suppose that I've got a foreach loop in which I write output to a file: auto f = File("test.txt", "w"); f.close(); // to start with a blank file foreach(i; iota(0, 100)) { f = File("test.txt", "a"); f.writeln(i); f.close(); } I'm guessing it is at least p