Hello there question: I'm looking for information / patterns / recipe's for implementing a multi threaded program (multiple producers / consumers) that will manage transactions and maintain sequence across the different threads.
background: I'm busy writing an adapter taking messages from ibm websphere mq using pymqi and inserting these messages into a database (after some processing, the messages come in as xml). If I comment out the database .execute bit of the code, i process hundreds of messages per second, the database slows the whole process down to around 60 - 100 messages per second. I'm hoping to speed this up some by having 2 threads reading the messages from mq, and another populating the database - at least then the database bit will never have to wait for me to fetch another message. (i'm thinking of later adding a second thread for the databse as well). The challenge is that these messages need to be processed in exact sequence and transactionally. (ie. the thread that fetched a message will have to wait for confirmation that the message has successfully been inserted into the database before committing the transaction, in mq terms the message will only really be deleted then.) I've started implementing a kind of state machine using dictionary's to facilitate this, but I'm now concerned that - there is a better way - i might get bitten with dictionary processing updating / setting of values not necessarily being thread safe. I've looked at this recipe http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/302997 as a possible replacement for dictionary's but I'm not even sure if I'm using it correctly. thank you jaco -- http://mail.python.org/mailman/listinfo/python-list