So this is working great so far. This is a bit postgres specific, but this is how I did it:
1. Create "migration ids" table manually CREATE TABLE record_migration ( id INT NOT NULL PRIMARY KEY, is_migrated BOOLEAN DEFAULT NULL ); INSERT INTO record_migration (id) SELECT id FROM record_table; 2. In SqlAlchemy Task runner can get a batch of IDS & lock them: UPDATE record_migration SET is_migrated = False FROM ( SELECT id from record_migration WHERE is_migrated IS NULL ORDER BY id ASC LIMIT 100 ) subq WHERE record_migration.id = subq.id RETURNING record_migration.id; Then just loop the ids for migration using sqlalchemy And when done, mark the record as migrated. UPDATE record_migration SET is_migrated = True WHERE id = %s; Each taskrunner has it's own batch and "locks" them with a `False` flag. After everything is done, I can go in and look for any records that are still `False` (most likely because of an uncaught exception in that batch) and set them to NULL for another migration run. -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.