Re: Concurrent writes to the same file
> > https://bitbucket.org/cameron_simpson/css/src/374f650025f156554a986fb3fd472003d2a2519a/lib/python/cs/fileutils.py?at=default#cl-408 > > That looks like it will do the trick for me, thank you Cameron. -- http://mail.python.org/mailman/listinfo/python-list
Re: Concurrent writes to the same file
Dave Angel wrote: > On 07/11/2013 12:57 AM, Jason Friedman wrote: >> Other than using a database, what are my options for allowing two processes >> to edit the same file at the same time? When I say same time, I can accept >> delays. I considered lock files, but I cannot conceive of how I avoid race >> conditions. >> > > In general, no. That's what a database is for. > > Now, you presumably have some reason to avoid database, but in its stead > you have to specify some other limitations. To start with, what do you > mean by "the same time"? If each process can modify the entire file, > then there's no point in one process reading the file at all until it > has the lock. So the mechanism would be >1) wait till you can acquire the lock >2) open the file, read it, modify it, flush and close >3) release the lock > > To come up with an appropriate lock, it'd be nice to start by specifying > the entire environment. Which Python, which OS? Are the two processes > on the same CPU, what's the file system, and is it locally mounted? > > > > You can use a seperate server process to do file I/O, taking multiple inputs from clients. Like syslogd. -- http://mail.python.org/mailman/listinfo/python-list
Re: Concurrent writes to the same file
On Wed, 10 Jul 2013 22:57:09 -0600, Jason Friedman wrote: > Other than using a database, what are my options for allowing two processes > to edit the same file at the same time? When I say same time, I can accept > delays. What do you mean by "edit"? Overwriting bytes and appending bytes are simple enough, but inserting or deleting bytes such that subsequent bytes change their offsets is harder. > I considered lock files, Well, you shouldn't have, unless you're targeting a platform which doesn't support file locks (are there any left?). > but I cannot conceive of how I avoid race conditions. By using locks. E.g. fcntl.lockf() or msvcrt.locking(). -- http://mail.python.org/mailman/listinfo/python-list
Re: Concurrent writes to the same file
On 10Jul2013 22:57, Jason Friedman wrote: | Other than using a database, what are my options for allowing two processes | to edit the same file at the same time? When I say same time, I can accept | delays. I considered lock files, but I cannot conceive of how I avoid race | conditions. Sure. (Assuming UNIX. Windows probably excludes both processes having the file open at the one time, but that's not a show stopper.) You need to use a lock file to ensure only one process accesses the file at a time. While a process holds the lock, access the file. There are two basic approaches here: take lock open file for write, modify the file, close it release lock both processes have the file open for update take lock modify file, flush buffers release lock The code I use to take a lockfile is my "lockfile" context manager: https://bitbucket.org/cameron_simpson/css/src/374f650025f156554a986fb3fd472003d2a2519a/lib/python/cs/fileutils.py?at=default#cl-408 An example caller goes: with lockfile(self.csvpath): backup = "%s.bak-%s" % (self.csvpath, datetime.datetime.now().isoformat()) copyfile(self.csvpath, backup) write_csv_file(self.csvpath, self.nodedb.nodedata()) if not self.keep_backups: os.remove(backup) as shown here: https://bitbucket.org/cameron_simpson/css/src/374f650025f156554a986fb3fd472003d2a2519a/lib/python/cs/nodedb/csvdb.py?at=default#cl-171 Simplify as necessary; you just want: with lockfile(mainfile): modify(mainfile) if you have no wish for backups in case of a failed modify. If both processes need to see each other's changes then there's some more logic needed to monitor the file for changes. I've got such a scheme for CSV files in beta at present for a personal project. It won't suit all use cases; mine is well defined. Cheers, -- Cameron Simpson Is it true, Sen. Bedfellow, that your wife rides with bikers? - Milo Bloom -- http://mail.python.org/mailman/listinfo/python-list
Re: Concurrent writes to the same file
On 07/11/2013 12:57 AM, Jason Friedman wrote: Other than using a database, what are my options for allowing two processes to edit the same file at the same time? When I say same time, I can accept delays. I considered lock files, but I cannot conceive of how I avoid race conditions. In general, no. That's what a database is for. Now, you presumably have some reason to avoid database, but in its stead you have to specify some other limitations. To start with, what do you mean by "the same time"? If each process can modify the entire file, then there's no point in one process reading the file at all until it has the lock. So the mechanism would be 1) wait till you can acquire the lock 2) open the file, read it, modify it, flush and close 3) release the lock To come up with an appropriate lock, it'd be nice to start by specifying the entire environment. Which Python, which OS? Are the two processes on the same CPU, what's the file system, and is it locally mounted? -- DaveA -- http://mail.python.org/mailman/listinfo/python-list
Concurrent writes to the same file
Other than using a database, what are my options for allowing two processes to edit the same file at the same time? When I say same time, I can accept delays. I considered lock files, but I cannot conceive of how I avoid race conditions. -- http://mail.python.org/mailman/listinfo/python-list
