Re: [sqlalchemy] insert performance vs executemany and max_allowed_packet limit

On 05/29/2017 11:34 PM, Михаил Доронин wrote: Umm, what I've meant is how to use postgresql on_conflict_do_update in such a way that sqlalchemy would use executemany behind the scenes. In examples it usage looks like this. stmt = insert(table, values) stmt =

Re: [sqlalchemy] insert performance vs executemany and max_allowed_packet limit

Umm, what I've meant is how to use postgresql on_conflict_do_update in such a way that sqlalchemy would use executemany behind the scenes. In examples it usage looks like this. stmt = insert(table, values) stmt = stmt.on_conflict_do_update(set_=dict(a=stmt.excluded.a)) excluded is generated

Re: [sqlalchemy] insert performance vs executemany and max_allowed_packet limit

On 05/29/2017 08:54 AM, Михаил Доронин wrote: I have a questions. If the only way to issue executemany is to do a conn.execute(table.insert(), list_of_dict). How users supposed to use postgres on_conflict and mysql on duplicate key update with this? Seems like there would be no way to

Re: [sqlalchemy] insert performance vs executemany and max_allowed_packet limit

I have a questions. If the only way to issue executemany is to do a conn.execute(table.insert(), list_of_dict). How users supposed to use postgres on_conflict and mysql on duplicate key update with this? Seems like there would be no way to access .updated in postgres and corresponding .values

Re: [sqlalchemy] insert performance vs executemany and max_allowed_packet limit

I have a questions. If the only way to issue executemany is to do a conn.execute(table.insert(), list_of_dict). How users supposed to use postgres on_conflict and mysql on duplicate key update with this? Seems like there would be no way to provide access .updated in postgres and corresponding

Re: [sqlalchemy] insert performance vs executemany and max_allowed_packet limit

Wow, I've never saw this thing about multiple statements. Thanks! RTFM On Thursday, 25 May 2017 22:43:55 UTC+3, Mike Bayer wrote: > > > > On 05/25/2017 01:44 PM, Михаил Доронин wrote: > >> SQLAlchemy batches inserts in the ORM as

Re: [sqlalchemy] insert performance vs executemany and max_allowed_packet limit

On 05/25/2017 01:44 PM, Михаил Доронин wrote: SQLAlchemy batches inserts in the ORM as is possible and with Core you do this explicitly, both make use of cursor.executemany() which is then determined by how the DBAPI handles it. Ummm. If that is true, why this line is used when I pass a

Re: [sqlalchemy] insert performance vs executemany and max_allowed_packet limit

For Christ sake how to post a new message here and not just reply? And how to edit the old post? Arg! -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example.

Re: [sqlalchemy] insert performance vs executemany and max_allowed_packet limit

> SQLAlchemy batches inserts in the ORM as is possible and with Core you > do this explicitly, both make use of cursor.executemany() which is then > determined by how the DBAPI handles it. Ummm. If that is true, why this line is used when I pass a list of values (its from visit_insert)?

Re: [sqlalchemy] insert performance vs executemany and max_allowed_packet limit

On 05/25/2017 12:02 PM, Михаил Доронин wrote: Sorry I can't post a benchmark here, because I'm not in the office. Maybe I will do that tomorrow. I use vmprof and can just share a link to the uploaded profile. Will that suite you? I would much prefer if you can send a traditional cProfile

Re: [sqlalchemy] insert performance vs executemany and max_allowed_packet limit

Sorry I can't post a benchmark here, because I'm not in the office. Maybe I will do that tomorrow. I use vmprof and can just share a link to the uploaded profile. Will that suite you? But I can say right now that I've passed only integers, not strings. Also mysqlclient encoding values if

Re: [sqlalchemy] insert performance vs executemany and max_allowed_packet limit

the topic of performance is one that many years of effort have gone into. For background on this topic particularly inserts, see: http://docs.sqlalchemy.org/en/latest/faq/performance.html#i-m-inserting-400-000-rows-with-the-orm-and-it-s-really-slow for a detailed test suite that will

[sqlalchemy] insert performance vs executemany and max_allowed_packet limit

I've tried to benchmark alchemy performance when inserting a lot of data. The results wasn't that good for sqlalchemy. The difference was up to three times in median values. First of all the more elements inserted the more the difference between sqlalchemy and executemany (mysqlclient). I've