>-----Original Message----- >From: Gavin Towey [mailto:gto...@ffn.com] >Sent: Tuesday, September 08, 2009 1:41 PM >To: Allen Fowler; mysql@lists.mysql.com >Subject: RE: Queue / FIFO in MySQL? > >You can add a LIMIT n to your update clause. > >Regards, >Gavin Towey > >-----Original Message----- >From: Allen Fowler [mailto:allen.fow...@yahoo.com] >Sent: Monday, September 07, 2009 5:18 PM >To: mysql@lists.mysql.com >Subject: Queue / FIFO in MySQL? > >Hello, > >I need to create a system where records are generated by a "producer" process >and processed by several "worker" processes. > >I was thinking about something like: > >Producer: >1) Producer INSERTs new records with "state" = "new" & "worker" = "null" >2) Producer sleeps and loops back to step #1 > >Worker(s): >1) Worker UPDATEs all records with "worker" = "pid" & "state" = "working" >where >"state" == "new" >2) Worker SELECTs all records where "worker" = "pid" & "state" = "working" >3) For each record that is done, worker updates record with "state" = "done" >4) Worker loops back to step #1 > >Note: In this scheme the worker winds up with all "new" records generated >since >the last worker claimed any. Not sure how else to guarantee atomicity. I >would >prefer "only n records per request". Ideas? > >I am sure something like this must have been before.... Can anyone point me >to >example code, libraries, and/or refinements on the scheme? (preferably using >python...) > >Thank you, >:) > [JS] Based on my (somewhat distant) experience, using a database as a FIFO can turn into a performance nightmare. I don't know how many "workers" you plan on having, nor how many incoming jobs you're going to have, but unless your sleep times can be long relative to the time it takes to insert or update the records things can get out of control very quickly.
Isn't there some alternative available, such as (in *NIX) a FIFO? As for atomicity, I would think you'd want to wrap the UPDATE and SELECT in a single transaction. Regards, Jerry Schwartz The Infoshop by Global Information Incorporated 195 Farmington Ave. Farmington, CT 06032 860.674.8796 / FAX: 860.674.8341 www.the-infoshop.com -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/mysql?unsub=arch...@jab.org
