Nikolaus Rath wrote: > I need to synchronize the access to a couple of hundred-thousand > files[1]. It seems to me that creating one lock object for each of the > files is a waste of resources, but I cannot use a global lock for all > of them either (since the locked operations go over the network, this > would make the whole application essentially single-threaded even > though most operations act on different files).
Just wondering, but at what time do you know what files are needed? If you know that rather early, you could simply 'check out' the required files, do whatever you want with them and then release them again. If one of the requested files is marked as already in use, you simply wait (without reserving the others) until someone releases files and then try again. You could also wait for that precise file to be available, but that would require that you already reserve the other files, which might unnecessarily block other accesses. Note that this idea requires that each access locks one set of files at the beginning and releases them at the end, i.e. no attempts to lock files in between, which would otherwise easily lead to deadlocks. Further, you could distinguish between read-only and read-write access as an optimisation. > My idea is therefore to create and destroy per-file locks "on-demand" > and to protect the creation and destruction by a global lock > (self.global_lock). For that, I add a "usage counter" > (wlock.user_count) to each lock, and destroy the lock when it reaches > zero. [...code...] > - Does that look like a proper solution, or does anyone have a better > one? This should work, at least the idea is not flawed. However, I'd say there are too many locks involved. Rather, you just need a simple flag and the global lock. Further, you need a condition/event that tells waiting threads that you released some of the files so that it should see again if the ones it wants are available. > - Did I overlook any deadlock possibilities? The normal deadlock possibilities when multiple locks are involved apply, you must make sure that they are always acquired in an order that prevents two threads waiting for a resource held by the other. Uli -- Sator Laser GmbH Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932 -- http://mail.python.org/mailman/listinfo/python-list
