Using `bulk_insert_mappings` is much more performant than using the ORM and
custom collections. Right now you are bypassing ORM object creation and
state management. Your operations might be 10x slower with a collection.
IIRC, `bulk_insert_mappings` will iterate over the payload like this:
INSERT INTO foo VALUES (bar);
INSERT INTO foo VALUES (bar);
INSERT INTO foo VALUES (bar);
and you may be able to increase performance by batching the payload into
groups of 100+ and constructing a sqlalchemy core statement with the form
of:
INSERT INTO foo VALUES (bar), (bar), (bar)...;
There is a lot of discussion on this list about "high volume inserts",
different ways to approach it for performance, and the overhead of using
the ORM. At 4k entries, people avoid the ORM. At 400k, you really want to
avoid the ORM
--
SQLAlchemy -
The Python SQL Toolkit and Object Relational Mapper
http://www.sqlalchemy.org/
To post example code, please provide an MCVE: Minimal, Complete, and Verifiable
Example. See http://stackoverflow.com/help/mcve for a full description.
---
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
