Hi all, I have a peculiar problem with a multithreaded program of mine (actually I've sort of inherited it). Before i show you the error, here's a litle background. Its a program to check email addresses are valid, and its main task is to verify the domain names.
Here's the basic functionality: * The prog has a list of domains it has seen before which is read into memory (the 'rollover'). * A new list of emails is read-in from a file (to a queue) and is checked against the rollover. * If we've seen the domain before then update the existing entry. * If we've not seen the domain before, add it. The program is multithreaded to speed up the processing...there are input and output Queues. Now, each domain entry is an class object containing various bits of info. Each domain class also has its own lock, so that only one thread can modify each domain at a time. I'm load-testing the program with a sample of 1 million email addresses and when i hit about the 500,000 mark i get a locking error... sem_init: No space left on device Exception in thread Thread-2: Traceback (most recent call last): File "/usr/local/lib/python2.4/threading.py", line 442, in __bootstrap self.run() File "/usr/local/lib/python2.4/threading.py", line 422, in run self.__target(*self.__args, **self.__kwargs) File "jess.py", line 250, in worker record.result = function( id, record.value ) File "jess.py", line 291, in action found_domain = domains.addNewDomain( domain_name ) File "jess.py", line 123, in addNewDomain self.domain_store.append( self.Domain( name = name ) ) File "jess.py", line 46, in __init__ self.lock = Lock() error: can't allocate lock Googling for this sort of error doesn't yield any results, and i can't find any information about limits to the number of locks you can have in Python. The 'No space left on device' message indicates a memory issue, however i doubt this since its running on a linux server with 4 cores and 16GB ram. It seems more like an internal Python limit has been hit (sem_init - semaphore initialisation?). Does anyone know more about threading internals and any internal limits? For the timebeing, I've implemented locking at a higher level which will reduce performance but keep the number of lock objects to a minumum. Thanks -- http://mail.python.org/mailman/listinfo/python-list