On Wednesday, October 14, 2015 at 4:25:52 PM UTC-4, Michael Bayer wrote: > > What am I missing that one wouldn't use say multiprocessing.Pool() to do > this kind of thing in the general sense? If we're only talking about > 5-10 runners they could just as well be local forked processes.
That would totally work too. I just *swear* I saw a SqlAlchemy recipe for this once, where I migration script would partition the SELECT based on the number of workers. It would probably be better too, because I would avoid taking such a hit on memory (this imports a Pyramid app that weighs 150mb - all loaded per worker). The one benefit of doing this as multiple invocations of the same script through a broker, is that you can tweak the number of workers as needed. I've got a console window monitoring memory/cpu and another one displaying how many records were migrated in the past minute. -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
