Re: Locking problem with mod_python
Daniel Poelzleithner wrote: > Yes i know. There would be other solutions like shm, or global mutexes, In fact those two don't seem to work for this situation anyway. I was digging this whole field over a month ago and remember that the main problem is that to use anything shared in memory you should control forking of your processes. But here a web server does this before you have a chance to execute your code. I also vaguely remember a discussion in the mod_python list about providing some API to control these things in Apache. > However, on a apache > with worker mpm, thread locks are definitive the best choice and allow > easy usage of Semaphores etc, too. I'm not sure but I always thought that even under worker mpm Apache still can use several children with threads, not only one. So any thread-level locks wouldn't work here also. But this is just my speculation I didn't check properly. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users -~--~~~~--~~--~--~---
Re: Locking problem with mod_python
On Thu, 2006-08-17 at 04:15 +0200, Daniel Poelzleithner wrote: [...] > I think django should have a Lock class and maybe Semaphores, too, which > check which type of server the app is running and choose the best > locking method. I think this is really beyond the scope of Django. If you need inter-process locking, it is going to be for specific circumstances (e.g. is it an exclusive lock, a counted lock, a read-write lock, a re-entrant lock, ...?) and it is more appropriate to pick a solution based on the surrounding infrastructure you have (web server + OS + process/thread structure). Otherwise we end up having to maintain something that is about as complex as the python threading + fcntl modules put together for all possible platforms. Asking the Django maintainers to shoulder the burden so that somebody with a precise requirement like that doesn't have to do a little bit of work for their own situation is probably too much. "Batteries included" can be taken too far: you already have the batteries in the Python libraries and should use them as appropriate. I'm not sure how you detect what mode you are running in for all the different web server configurations, either. For those who think this shouldn't be that hard, consider implementing a Semaphore-like class using filesystem behaviour in a secure and robust fashion. It is unbelievably hard. Compounded in difficulty by the fact that many systems only implement a portion of the POSIX interfaces, so you end up with a lot of non-portable, system-specific code. A simple exclusive lock are easy (os.mkdir() is the most portable way to do this that I know of, although there are still places it won't work perfectly), but it rapidly gets more fiddly once you go beyond that. All that being said, we take patches. So code something up and let's see how it goes. > Django already has a RWLock class under utils/sync.py, which is simply > not working on all servers. We need to document that more clearly. In core, it's only used for the in-process, multi-threaded local memory caching backend (the "locmem" cache), so it's working as advertised in that case. Users do need to be aware of the limitations there, though. Anything based on Python's threading module isn't going to be able to work cross-process. Cheers, Malcolm --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users -~--~~~~--~~--~--~---
Re: Locking problem with mod_python
Ivan Sagalaev wrote: > As Ian has pointed you can use file system as a device for locking > between separate processes. This is not really something specific to Django. Yes i know. There would be other solutions like shm, or global mutexes, but filelocks seem to be the best of the worst. However, on a apache with worker mpm, thread locks are definitive the best choice and allow easy usage of Semaphores etc, too. I think django should have a Lock class and maybe Semaphores, too, which check which type of server the app is running and choose the best locking method. Django already has a RWLock class under utils/sync.py, which is simply not working on all servers. If you use apache 1.3, or apache2.x with mpm-prefork or the experimental threadpool, or worker with more then one instance, thread locks are useless. I think providing a easy to use Locking class that works regardless the server used is something important for a web framework. > On Windows, if I remember correctly, you don't use fcntl but use some > exclusive locking options for open(). I don't use open, but python2.4 docs suggest using fp.lock() as a platform independent function. kindly regards Daniel --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users -~--~~~~--~~--~--~---
Re: Locking problem with mod_python
Daniel Poelzleithner wrote:
> After further investigation, i found out that locking of any kind
> doesn't work with apache in prefolk mode, and more or less in the
> threaded mode. I haven't found a nice and clean solution yet to do
> locking on requests, which worries me a little bit. Locking can be a
> important part of apps and I think django should provide a way to have
> working locks regardless which server backend is used.
As Ian has pointed you can use file system as a device for locking
between separate processes. This is not really something specific to Django.
On unix it can look like this:
from fcntl import flock, LOCK_EX
def lock():
f = open('somefile.lock', 'w')
flock(f, LOCK_EX)
return f
def unlock(f):
f.close()
On Windows, if I remember correctly, you don't use fcntl but use some
exclusive locking options for open().
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-users
-~--~~~~--~~--~--~---
Re: Locking problem with mod_python
Ian Holsman wrote: > aren't semaphores inter-process (not cross process)? > > try using a file handle instead.. After further investigation, i found out that locking of any kind doesn't work with apache in prefolk mode, and more or less in the threaded mode. I haven't found a nice and clean solution yet to do locking on requests, which worries me a little bit. Locking can be a important part of apps and I think django should provide a way to have working locks regardless which server backend is used. kindly regards Daniel --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users -~--~~~~--~~--~--~---
Re: Locking problem with mod_python
aren't semaphores inter-process (not cross process)?try using a file handle instead..regardsIan.On 16/08/2006, at 11:49 AM, Daniel Poelzleithner wrote:Hi,I have a function that generates a tile for a google map overlay andwrites it to disc so later requests can simply use the file. Due thenature of the map, generating the tiles previous is not a option so ithas to be done on demand, which works nice on the developer server butnot on apache. There are many requests for tiles and the calculation isexpensive, so I want to limit apache to only generate one tile at the time.I used something like this:from threading import Semaphore# create Basemap instance. Use 'crude' resolution coastlines.GENSEM = Semaphore(settings.TOPO_MAX_TILE_THREADS)def get_topoimg(request): ... def gen(): print "Lock GENSEM" GENSEM.acquire() topoimg(x, y, zoom, typ, name, rstate) GENSEM.release() print "Unlock GENSEM" return True rv = return_cached_file(name, gen, rebuild=False) ...TOPO_MAX_TILE_THREADS is 1In developing mode and apache2 -X the Semaphore seems to work. Iftopoimg raises a exception, the next request hangs. But in normal apachemode the semaphore doesn't seem to have any affect, causing very highload on the machine.What am I doing wrong ?kindly regards Daniel --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users -~--~~~~--~~--~--~---
Locking problem with mod_python
Hi, I have a function that generates a tile for a google map overlay and writes it to disc so later requests can simply use the file. Due the nature of the map, generating the tiles previous is not a option so it has to be done on demand, which works nice on the developer server but not on apache. There are many requests for tiles and the calculation is expensive, so I want to limit apache to only generate one tile at the time. I used something like this: from threading import Semaphore # create Basemap instance. Use 'crude' resolution coastlines. GENSEM = Semaphore(settings.TOPO_MAX_TILE_THREADS) def get_topoimg(request): ... def gen(): print "Lock GENSEM" GENSEM.acquire() topoimg(x, y, zoom, typ, name, rstate) GENSEM.release() print "Unlock GENSEM" return True rv = return_cached_file(name, gen, rebuild=False) ... TOPO_MAX_TILE_THREADS is 1 In developing mode and apache2 -X the Semaphore seems to work. If topoimg raises a exception, the next request hangs. But in normal apache mode the semaphore doesn't seem to have any affect, causing very high load on the machine. What am I doing wrong ? kindly regards Daniel --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users -~--~~~~--~~--~--~---
