Thanks a lot Danny, That certainly does make sense. I'll look into implementing the Queue approach in my program tomorrow. I remember you recommending me this module as well not long ago, although in a different discussion (where I suspected problem with file access from multiple thread, but I guess it's pretty much a similar problem, isn't it? :-)
Thanks again Bernard On 1/19/06, Danny Yoo <[EMAIL PROTECTED]> wrote: > > > On Thu, 19 Jan 2006, Kent Johnson wrote: > > > > In your original desing were you sharing a connection between threads? > > That could cause trouble. But if each connection has its own thread and > > you are using transactions and isolation levels appropriately, they > > shouldn't stomp on each other. > > Hi Kent and Bernard, > > It's possible that MySQLdb was linked against the non-thread-safe > mysqlclient (rather than mysqlclient_r) library; that would probably cause > havoc. > > > > > So the solution was to start some sort of queue server in a separate > > > thread. This queue would consist of a list, and each time the program > > > would want to perform a MySQL operation, it would add it to the queue. > > We may want to use the Queue module here instead of a list; the way the > program is described now sounds like the server is busy-spinning when it > looks for new things to do. The thing that makes busy-spinning slightly > not-nice is that it consumes CPU regardless if the system's doing anything > or not. > > But a more serious problem than busy-waiting is one of thread-safety: the > object that's used to communicate between two threads --- a shared list > --- might be unreliable if the list methods aren't thread-safe. And I'm > not certain that all the list methods are thread-safe. > > That is, the problem with mutual exclusion may have just been pushed up > the program's structure, from the MySQL queries up to the shared queue > list access. *grin* > > > The synchronized Queue described in: > > http://www.python.org/doc/lib/module-Queue.html > > is designed to be a reliable communication medium between threads, and I > strongly recommend you look at it, especially because you've seen > first-hand the weird things that can happen with threads. *grin* > > > Here's a small example that shows what a different Queue can make: > > ############################################ > from threading import Thread > > class Server: > def __init__(self): > self.queue = [] > > def acceptJob(self, query): > self.queue.append((query,)) > > def shutdownOnIdle(self): > self.queue.append("QUIT!") > > def jobLoop(self): > while True: > print "looping" > if len(self.queue) > 0: > nextJob = self.queue.pop(0) > if nextJob == "QUIT!": > return > print "I should do", nextJob > > def startServer(self): > Thread(target=self.jobLoop).start() > > if __name__ == '__main__': > server = Server() > server.startServer() > server.acceptJob("a") > server.acceptJob("b") > server.acceptJob("c") > server.shutdownOnIdle() > ############################################ > > > Running this will show just how much work a busy-waiting thread does. > > > But if we change a few methods to use Queues instead of lists: > > ###### > from threading import Thread > from Queue import Queue > > class Server: > def __init__(self): > self.queue = Queue() > > def acceptJob(self, query): > self.queue.put((query,)) > > def shutdownOnIdle(self): > self.queue.put("QUIT!") > > def jobLoop(self): > while True: > print "looping" > nextJob = self.queue.get() > if nextJob == "QUIT!": > return > print "I should do", nextJob > > def startServer(self): > Thread(target=self.jobLoop).start() > > if __name__ == '__main__': > server = Server() > server.startServer() > server.acceptJob("a") > server.acceptJob("b") > server.acceptJob("c") > server.shutdownOnIdle() > ###### > > and compare the output of this to the original implementation, it should > be clearer why Queues are cool. *grin* > > > Does this make sense? I hope this helps! > > _______________________________________________ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
