I wrote a full post regarding this topic on stackoverflow at http://stackoverflow.com/questions/11769366/why-is-sqlalchemy-insert-with-sqlite-25-times-slower-than-using-sqlite3-directly/11769768#11769768 . If you start with this, I can answer more specific questions.
On Nov 6, 2013, at 10:28 AM, Achim Domma <do...@procoders.net> wrote: > Hi, > > I want to speed up my SqlAlchemy bulk inserting code and yes, I'm aware that > this is not the main purpose of SqlAlchemy and all databases have faster low > level import tools. > > The background is the following: We import data from various sources and > apply various mappings. Currently I'm willing to trade raw runtime for much > simpler code, which is much easier to maintain. But I still want my code to > run as fast as it's possible with those assumptions. > > There are two scenarios which I want to optimize: > > 1. Flat inserts without relations, but with unique ids generated inside the > database: In that case, SqlAlchemy retrieves the unique ids from the > database, but those ids are never used in my import process. I thought about > generating an insert statement out of an object. Obviously SqlAlchemy has to > do that too, so there might be some existing tool for that? The other option > would be, to tell SqlAlchemy to ignore the generated id an to not retrieve it > from the database. Is that possible? > > 2. Inserts using relations and unique ids generated in the database: I think > SqlAlchemy is already quite good at optimizing this and sending objects as > batches. If there are any tweeks and tricks to speed up inserts having > relations, I would be happy to hear them. > > cheers, > Achim > > -- > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sqlalchemy+unsubscr...@googlegroups.com. > To post to this group, send email to sqlalchemy@googlegroups.com. > Visit this group at http://groups.google.com/group/sqlalchemy. > For more options, visit https://groups.google.com/groups/opt_out.
signature.asc
Description: Message signed with OpenPGP using GPGMail
