Re: [sqlalchemy] Concurrent raw sql execution in a multi-threaded application: Connection is busy with results for another hstmt error

2014-05-12 Thread Sylvester Steele
the code here isn’t really showing me the nature of the two separate connections. They will be different unless you’re using “threadlocal” on the engine or pool and both are in the same thread. A simple comparison of the “raw” DBAPI connections to see if they are different objects will

[sqlalchemy] engine.execute() slow with parameterized queries?

2014-05-12 Thread Seth P
After tracking down some extreme slowness in loading a one-to-many relationship (e.g. myobject.foobars), I seem to have isolated the issue to engine.execute() being much slower with parameterized queries than with explicit queries. The following is actual code and output for loading 10,971

[sqlalchemy] Re: engine.execute() slow with parameterized queries?

2014-05-12 Thread Seth P
Forgot to mention that I'm running SQLAlchemy 0.9.4 on 64-bit Python 3.4.0 on Windows. On Monday, May 12, 2014 3:48:44 PM UTC-4, Seth P wrote: After tracking down some extreme slowness in loading a one-to-many relationship (e.g. myobject.foobars), I seem to have isolated the issue to

Re: [sqlalchemy] Concurrent raw sql execution in a multi-threaded application: Connection is busy with results for another hstmt error

2014-05-12 Thread Michael Bayer
On May 12, 2014, at 2:30 PM, Sylvester Steele sylvesterste...@gmail.com wrote: the code here isn't really showing me the nature of the two separate connections. They will be different unless you're using threadlocal on the engine or pool and both are in the same thread. A simple

Re: [sqlalchemy] engine.execute() slow with parameterized queries?

2014-05-12 Thread Michael Bayer
well there's only one parameter being processed here so there is clearly negligible difference in time spent within Python as far as getting the statement ready to execute and then executing it. So the time is either in what SQL Server spends fetching the rows, or the number of rows being

Re: [sqlalchemy] Concurrent raw sql execution in a multi-threaded application: Connection is busy with results for another hstmt error

2014-05-12 Thread Sylvester Steele
On Mon, May 12, 2014 at 4:53 PM, Michael Bayer mike...@zzzcomputing.comwrote: On May 12, 2014, at 2:30 PM, Sylvester Steele sylvesterste...@gmail.com wrote: the code here isn’t really showing me the nature of the two separate connections. They will be different unless you’re using

[sqlalchemy] Oracle index not used on SQLAlchemy prepared statement

2014-05-12 Thread Thierry Florac
Hi, I use Python and SQLAlchemy to access an Oracle 11 database. As far as I think, SQLAlchemy always use prepared statements. On a huge table (4 millions records), correctly indexed, SQLAlchemy filters queries doesn't use the index, so doing a full table scan is very slow ; using the same SQL

Re: [sqlalchemy] engine.execute() slow with parameterized queries?

2014-05-12 Thread Seth P
Is it possible that the (primary key index (which is a composite index that begins with gvkey, and is the only index on the table) isn't being used because the the gvkey parameter is somehow passed as an integer rather than as a string? The first EXEC below is pretty much instantaneous, whereas

Re: [sqlalchemy] Oracle index not used on SQLAlchemy prepared statement

2014-05-12 Thread Michael Bayer
On May 12, 2014, at 6:12 PM, Thierry Florac tflo...@gmail.com wrote: Hi, I use Python and SQLAlchemy to access an Oracle 11 database. As far as I think, SQLAlchemy always use prepared statements. SQLAlchemy has no control over this as the DBAPI has no prepared statement system exposed

Re: [sqlalchemy] engine.execute() slow with parameterized queries?

2014-05-12 Thread Michael Bayer
On May 12, 2014, at 6:33 PM, Seth P spadow...@gmail.com wrote: Is it possible that the (primary key index (which is a composite index that begins with gvkey, and is the only index on the table) isn't being used because the the gvkey parameter is somehow passed as an integer rather than as

Re: [sqlalchemy] engine.execute() slow with parameterized queries?

2014-05-12 Thread Seth P
Yep, it's not a SQLAlchemy issue. The following code demonstrates the problem with direct pyodbc access. import pyodbc import time def print_timing(func): def wrapper(*arg): t1 = time.time() rows = func(*arg) t2 = time.time() print(%30s() len=%d, last=%s,

Re: [sqlalchemy] engine.execute() slow with parameterized queries?

2014-05-12 Thread Seth P
Looks like other people have encountered similar problems with indices being ignored by prepared sql statements: http://www.postgresql.org/message-id/43250afa.7010...@arbash-meinel.com. (If the diagnosis there is correct, then I'm guessing the server would use a unique index where all the

Re: [sqlalchemy] engine.execute() slow with parameterized queries?

2014-05-12 Thread Michael Bayer
On May 12, 2014, at 7:35 PM, Seth P spadow...@gmail.com wrote: Looks like other people have encountered similar problems with indices being ignored by prepared sql statements: http://www.postgresql.org/message-id/43250afa.7010...@arbash-meinel.com. (If the diagnosis there is correct, then

Re: [sqlalchemy] engine.execute() slow with parameterized queries?

2014-05-12 Thread Seth P
Fair enough. I'll take a look at pymssql, though I suspect it may be a SQL Server rather than a driver issue. On Monday, May 12, 2014 7:50:03 PM UTC-4, Michael Bayer wrote: On May 12, 2014, at 7:35 PM, Seth P spad...@gmail.com javascript: wrote: Looks like other people have encountered

Re: [sqlalchemy] engine.execute() slow with parameterized queries?

2014-05-12 Thread Seth P
pymssql produces the same results as pyodbc. So it looks like a SQL Server issue. On Monday, May 12, 2014 8:06:08 PM UTC-4, Seth P wrote: Fair enough. I'll take a look at pymssql, though I suspect it may be a SQL Server rather than a driver issue. On Monday, May 12, 2014 7:50:03 PM UTC-4,

Re: [sqlalchemy] engine.execute() slow with parameterized queries?

2014-05-12 Thread Michael Bayer
well or a FreeTDS issue, more likely, if that's what you're using. the SQL compiler has a parameter called literal_binds that will make it render a bound parameter as an inline string, but it only supports a few very basic types.As far as getting this parameter set for a general class of