[sqlalchemy] Working with large IN lists

2012-02-21 Thread Vlad K.
Hi all, I have to read thousands of rows from a table and compile some data, but in certain conditions update those rows, all with same value. The ratio of reads and writes here is widest possible. Sometimes no rows, sometimes few, and sometimes all rows that are read have to be updated.

[sqlalchemy] Re: Working with large IN lists

2012-02-21 Thread Manav Goel
This depends upon the execution plan of the query and is more really a postgresql question. Google postgresql IN performance and you will get a good idea of it. By the look of your code, Second option would obviously be faster as it hits database once whereas first one flush after every change.

Re: [sqlalchemy] Re: Working with large IN lists

2012-02-21 Thread Simon King
On Tue, Feb 21, 2012 at 3:24 PM, Manav Goel manav.goe...@gmail.com wrote: This depends upon the execution plan of the query and is more really a postgresql question. Google postgresql IN performance and you will get a good idea of it. By the look of your code, Second option would obviously be

[sqlalchemy] alembic questions/comments

2012-02-21 Thread Chris Withers
Hi Mike, I've been reading the Alembic docs, and the following popped out: - The partial guid approach is nice for machines, but a nightmare for humans, looking at a folder full of these can't be fun. Since it's the link in the file that's important, could the files be given friendly names

Re: [sqlalchemy] alembic questions/comments

2012-02-21 Thread Michael Bayer
On Feb 21, 2012, at 12:57 PM, Chris Withers wrote: Hi Mike, I've been reading the Alembic docs, and the following popped out: - The partial guid approach is nice for machines, but a nightmare for humans, looking at a folder full of these can't be fun. Since it's the link in the file

Re: [sqlalchemy] Re: Working with large IN lists

2012-02-21 Thread Vlad K.
Thanks for your replies. Using the IN list definitely speeds up the process, but I hate the resulting query which uses bound variables for each and every element of the list. But I have another problem with this, there's a massive memory leak somewhere. Take a look at this model: class