esult in a given client never being served the files it's
trying to fetch.
Potential Soln:
1) Invoke a form of file locking, with each client processes waiting
until it gets its lock.
2) Invoke some form of round-robin process, where the master process
puts files in different dirs, so each
Hi Bruce,
you can do it like Maildir [1] you move (os.rename()) file or directories.
Maybe something like this: You have three directories: "todo", "in-process" and
"done".
A process tries to os.rename from todo to in-process. If it fails, some other
process has done it before. If the process is
In message , bruce
wrote:
> my concern about a "gatekeeper" wasn't so much related to performance, as
> the possibility of race conditions...
Which is what the gatekeeper will prevent. It serializes the granting of
locks, and that means no race conditions.
--
http://mail.python.org/mailman/li
python-list-bounces+bedouglas=earthlink@python.org
[mailto:python-list-bounces+bedouglas=earthlink@python.org]on Behalf
Of Lawrence D'Oliveiro
Sent: Monday, March 02, 2009 12:29 AM
To: python-list@python.org
Subject: RE: file locking...
In message , bruce
wrote:
> using any kind of file lo
In message , bruce
wrote:
> using any kind of file locking process requires that i essentially have a
> gatekeeper, allowing a single process to enter, access the files at a
> time...
The gatekeeper doesn't need to do any more than ensure that any particular
filename is on
In message , Nigel
Rantor wrote:
> In other words, why would you rely on a scheme that limits some
> processes to certain parts of the data?
That could be part of the original requirements, it's not clear from the
description so far.
--
http://mail.python.org/mailman/listinfo/python-list
zugnush wrote:
You could do something like this so that every process will know if
the file "belongs" to it without prior coordination, it means a lot
of redundant hashing though.
In [36]: import md5
In [37]: pool = 11
In [38]: process = 5
In [39]: [f for f in glob.glob('*') if int(md5.md5(
You could do something like this so that every process will know if
the file "belongs" to it without prior coordination, it means a lot
of redundant hashing though.
In [36]: import md5
In [37]: pool = 11
In [38]: process = 5
In [39]: [f for f in glob.glob('*') if int(md5.md5(f).hexdigest(),16
En Sun, 01 Mar 2009 16:11:41 -0200, Nigel Rantor
escribió:
bruce wrote:
i can easily setup a file read/write lock process where a client app
gets/locks a file, and then copies/moves the required files from the
initial
dir to a tmp dir. after the move/copy, the lock is released, and the
c
Hi Bruce,
Excuse me if I'm a little blunt below. I'm ill grumpy...
bruce wrote:
hi nigel...
using any kind of file locking process requires that i essentially have a
gatekeeper, allowing a single process to enter, access the files at a
time...
I don't beleive this is a nece
.org]on Behalf
Of Gabriel Genellina
Sent: Sunday, March 01, 2009 9:50 AM
To: python-list@python.org
Subject: Re: file locking...
En Sun, 01 Mar 2009 15:39:56 -0200, bruce
escribió:
> the issue that i'm looking at is analogous to a FIFO, where i have lots
> of
> files being shoved in a dir
En Sun, 01 Mar 2009 15:39:56 -0200, bruce
escribió:
the issue that i'm looking at is analogous to a FIFO, where i have lots
of
files being shoved in a dir from different processes.. on the other end,
i
want to allow mutiple client processes to access unique groups of these
files as fast a
hi nigel...
using any kind of file locking process requires that i essentially have a
gatekeeper, allowing a single process to enter, access the files at a
time...
i can easily setup a file read/write lock process where a client app
gets/locks a file, and then copies/moves the required files
-list-bounces+bedouglas=earthlink@python.org
[mailto:python-list-bounces+bedouglas=earthlink@python.org]on Behalf
Of MRAB
Sent: Sunday, March 01, 2009 8:04 AM
To: python-list@python.org
Subject: Re: file locking...
bruce wrote:
> Hi.
>
> Got a bit of a question/issue that I'
bruce wrote:
Hi.
Got a bit of a question/issue that I'm trying to resolve. I'm asking this of
a few groups so bear with me.
I'm considering a situation where I have multiple processes running, and
each process is going to access a number of files in a dir. Each process
accesses a unique group o
koranthala wrote:
On Mar 1, 2:28 pm, Nigel Rantor wrote:
bruce wrote:
Hi.
Got a bit of a question/issue that I'm trying to resolve. I'm asking
this of a few groups so bear with me.
I'm considering a situation where I have multiple processes running,
and each process is going to access a number
On Mar 1, 2:28 pm, Nigel Rantor wrote:
> bruce wrote:
> > Hi.
>
> > Got a bit of a question/issue that I'm trying to resolve. I'm asking
> > this of a few groups so bear with me.
>
> > I'm considering a situation where I have multiple processes running,
> > and each process is going to access a nu
bruce wrote:
Hi.
Got a bit of a question/issue that I'm trying to resolve. I'm asking
this of a few groups so bear with me.
I'm considering a situation where I have multiple processes running,
and each process is going to access a number of files in a dir. Each
process accesses a unique group o
Hi.
Got a bit of a question/issue that I'm trying to resolve. I'm asking this of
a few groups so bear with me.
I'm considering a situation where I have multiple processes running, and
each process is going to access a number of files in a dir. Each process
accesses a unique group of files, and th
Windows, can I
> > have it so that the same file can be open in read mode by more than
> > one person or could Window's file locking system get in the way?
>
> Assuming your question is: can processes A, B & C read
> from the same file at the same time, then: Yes.
could Window's file locking system get in the way?
Assuming your question is: can processes A, B & C read
from the same file at the same time, then: Yes. (You
can try it out yourself fairly easily if you want. Just
open a clutch of interpreter windows and do some
open ("abc.t
x27;s file locking system get in the way?
Thanks,
Sam
--
http://mail.python.org/mailman/listinfo/python-list
Marc 'BlackJack' Rintsch wrote:
> In <[EMAIL PROTECTED]>, Dr. Who wrote:
>
>
>>Please let me know if I'm missing something since they seem like
>>normal file operations that I would hope Python would abstract away.
>
>
> I don't think of them as "normal" operations. They are "extra" stuff that
In <[EMAIL PROTECTED]>, Dr. Who wrote:
> Please let me know if I'm missing something since they seem like
> normal file operations that I would hope Python would abstract away.
I don't think of them as "normal" operations. They are "extra" stuff that
is not only dependent on the operating system
I'm always disappointed when I find something that Python doesn't
handle in a platform independent way. It seems to me that file
locking is in that boat.
1. I don't see a way to atomically open a file for writing if and only
if it doesn't exist without resorting to os.
Thanks for the reply; I somehow missed this entire thread in
python-list. I'm going to give it a whirl, after digging a bit into
the status quo of Linux' NFSv3 implementation.
--
http://mail.python.org/mailman/listinfo/python-list
Fredrik Lundh wrote:
>os.link(tempfile, lockfile) # atomic!
Fredrik, thanks for replying - I monitored python-list but didn't see
anything. Gotta get a proper Usenet feed...
Are you sure os.link() will be atomic over NFS?
--
http://mail.python.org/mailman/listinfo/python-list
Fredrik Lundh wrote:
> 5) check the number of links to each file
>
> n = os.stat(tempfile)[3]
> m = os.stat(lockfile)[3]
aw, forget that. I shouldn't trust google over my own code. here's the
correct algorithm:
f = open(tempfile, "w")
f.close()
n = os.stat(tempfile)[3]
Christopher DeMarco wrote:
> 2. open(2) is atomic on a local FS, I've read discussions that imply
> that link(2) is atomic over NFS (is it?), so I can link from local
> lockfile to remote target atomically. I don't grok this; open(2) +
> link(2) + stat(2) == 3 calls on my fingers. HTH is this s
Christopher DeMarco wrote:
>Hi all...
>
>...I've got a Python script running on a bunch of boxen sharing some
>common NFS-exported space. I need (not want :) to lock files for
>writing, and I need (not want :) to do it safely (i.e. atomically).
>I'm doing this in Linux; NFS4 is available. As I u
Hi all...
...I've got a Python script running on a bunch of boxen sharing some
common NFS-exported space. I need (not want :) to lock files for
writing, and I need (not want :) to do it safely (i.e. atomically).
I'm doing this in Linux; NFS4 is available. As I understand it, my
options are:
1.
Sure it will do if one of the processes needs read access only.
Scenario when you need shared rw acces with locking:
In the file you have "records" say 30 bytes long, 2 processes are
reading/writing these records by: lock-read-unlock or lock-write-unlock
. Both processes have to open the file with
Sure it will do if one of the processes needs read access only.
Scenario when you need shared rw acces with locking:
In the file you have "records" say 30 bytes long, 2 processes are
reading/writing these records by: lock-read-unlock or lock-write-unlock
. Both processes have to open the file with
Hi everybody:
I played with the class Flock and changed the line
win32con.FILE_SHARE_READ|win32con.FILE_SHARE_WRITE,\
to
win32con.FILE_SHARE_READ,\
and now I cannot copy the file over which suits me.
When file is NOT locked I get:
E:\>copy d:\log.txt .
Overwrite .\log.txt? (Yes/No/All): y
1
I just have written the program in "C", which does the same. It behaves
almost the way you described.
Tthe copy command gives such diagnostic:
The process cannot access the file because
another process has locked a portion of the file.
0 file(s) copied.
BUT THE FILE IS ACTUALLY OVERWRITTEN..
I'
I get
The process cannot access the file because another process has locked a
portion of the file.
0 file(s) copied.
on both Win2k and WinXP. (Python 2.4, Pywin32 build 203)
Are you sure the log.txt you're copying to is actually the right one ? You
should at least get a
prompt to confir
Pekka Niiranen wrote:
> Is there a way around this? How can I stop file being COPIED OVER while
> it is being open?
you could rename the file before you start updating it, and rename it back when
done.
--
http://mail.python.org/mailman/listinfo/python-list
Hi,
I have used the following example from win32 extensions:
-SCRIPT STARTS
import win32file
import win32con
import win32security
import pywintypes
class Flock:
def __init__(self,file):
self.file=file
secur_att = win32security.SECURITY_ATTRIBUTES()
Kamus of Kadizhar <[EMAIL PROTECTED]> wrote in message news:<[EMAIL
PROTECTED]>...
> Thanks to Robert Brewer, I got enough insight into logging to make it work
>
> Now I have another issue: file locking. Sorry if this is a very basic
> question, but I can't fi
Thanks to Robert Brewer, I got enough insight into logging to make it work
Now I have another issue: file locking. Sorry if this is a very basic
question, but I can't find a handy reference anywhere that mentions this.
When a logger opens a log file for append, is it automatically l
40 matches
Mail list logo
