[sqlalchemy] Re: Use with multiple Processes
entirely different machines, then youd have to partition out rows from the table yourself. you'd select some range of rows using LIMIT/ OFFSET on each machine to be processed. the exact count would depend on the total rows in the table and the total number of machines. On Feb 25, 2009, at 8:50 AM, Daniel wrote: > > Hi, > > your suggestion sounds really reasonable but the point is that these > processes are running on different machines and I don't want to put a > lot of effort in synchronizing these threads. > > Isn't there any easy solution for having multiple processes working on > the same database table? > > Thanks. > > Cheers > > Am Dienstag, den 24.02.2009, 07:46 -0800 schrieb Michael Bayer: >> I would select out the total set of rows and then hand off groups of >> those rows, converted into serializable objects first, using the >> "imap" function of a multiprocessing.Pool object. It would be best >> if the rows are returned via a ResultProxy so that work can begin on >> results before all results are fetched. >> >> See >> http://docs.python.org/library/multiprocessing.html#module-multiprocessing.pool >> . Its critical that you call create_engine() within each >> subprocess if additional queries are to be issued in workers, and >> don't pass any other database resources like connections or cursors >> across. Pool allows initialization activities via the "initializer" >> argument. >> >> On Feb 24, 7:05 am, dagooglaa wrote: >>> Hi, >>> >>> I am using SQLAlchemy 0.4.8 with Python 2.4. >>> >>> I have a process which is selecting some rows, doing something with >>> the data of these rows and then writes a result into the database. >>> >>> How can I achieve a solution where multiple of these processes are >>> running without having them selecting the same rows and doing the >>> same >>> stuff? >>> >>> Normally I would suggest s.th. like Select for update or an trigger >>> which sets a timestamp after a select... >>> >>> Do you have any suggestions for having multiple processes working on >>> the same table but not on the same rows? >>> >>> Thanks >> > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Use with multiple Processes
Hi, your suggestion sounds really reasonable but the point is that these processes are running on different machines and I don't want to put a lot of effort in synchronizing these threads. Isn't there any easy solution for having multiple processes working on the same database table? Thanks. Cheers Am Dienstag, den 24.02.2009, 07:46 -0800 schrieb Michael Bayer: > I would select out the total set of rows and then hand off groups of > those rows, converted into serializable objects first, using the > "imap" function of a multiprocessing.Pool object. It would be best > if the rows are returned via a ResultProxy so that work can begin on > results before all results are fetched. > > See > http://docs.python.org/library/multiprocessing.html#module-multiprocessing.pool > . Its critical that you call create_engine() within each > subprocess if additional queries are to be issued in workers, and > don't pass any other database resources like connections or cursors > across. Pool allows initialization activities via the "initializer" > argument. > > On Feb 24, 7:05 am, dagooglaa wrote: > > Hi, > > > > I am using SQLAlchemy 0.4.8 with Python 2.4. > > > > I have a process which is selecting some rows, doing something with > > the data of these rows and then writes a result into the database. > > > > How can I achieve a solution where multiple of these processes are > > running without having them selecting the same rows and doing the same > > stuff? > > > > Normally I would suggest s.th. like Select for update or an trigger > > which sets a timestamp after a select... > > > > Do you have any suggestions for having multiple processes working on > > the same table but not on the same rows? > > > > Thanks > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Use with multiple Processes
I would select out the total set of rows and then hand off groups of those rows, converted into serializable objects first, using the "imap" function of a multiprocessing.Pool object. It would be best if the rows are returned via a ResultProxy so that work can begin on results before all results are fetched. See http://docs.python.org/library/multiprocessing.html#module-multiprocessing.pool . Its critical that you call create_engine() within each subprocess if additional queries are to be issued in workers, and don't pass any other database resources like connections or cursors across. Pool allows initialization activities via the "initializer" argument. On Feb 24, 7:05 am, dagooglaa wrote: > Hi, > > I am using SQLAlchemy 0.4.8 with Python 2.4. > > I have a process which is selecting some rows, doing something with > the data of these rows and then writes a result into the database. > > How can I achieve a solution where multiple of these processes are > running without having them selecting the same rows and doing the same > stuff? > > Normally I would suggest s.th. like Select for update or an trigger > which sets a timestamp after a select... > > Do you have any suggestions for having multiple processes working on > the same table but not on the same rows? > > Thanks --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---