Re: [sqlalchemy] session flush inserts performing VERY slow for MS Sql Server

2016-03-09 Thread Jonathan Vanasco
On Wednesday, March 9, 2016 at 3:17:26 PM UTC-5, jbrow...@clearcosthealth.com wrote: > > Like I said I am not terribly network savvy but what I meant was that the > particular vpn client that I am using says that my "connection speed" is > ~540Mpbs. > That's most-likely a speed to your router

Re: [sqlalchemy] session flush inserts performing VERY slow for MS Sql Server

2016-03-09 Thread jbrownstein
Like I said I am not terribly network savvy but what I meant was that the particular vpn client that I am using says that my "connection speed" is ~540Mpbs. I know comparing sql alchemy to raw sql is kind of a no-no but I was just illustrating that the there isn't obvious evidence of a slow net

Re: [sqlalchemy] session flush inserts performing VERY slow for MS Sql Server

2016-03-09 Thread Jonathan Vanasco
Those aren't the same inserts as SqlAlchemy The SQL you noted above (DECLARE / BEGIN/ END) is one packet of text that is sent over the wire to the database server and executed. (a few hundred bytes) A more appropriate comparison would be sending 1000 separate "INSERT INTO " statements ove

Re: [sqlalchemy] session flush inserts performing VERY slow for MS Sql Server

2016-03-09 Thread Mike Bayer
On 03/09/2016 11:56 AM, jbrownst...@clearcosthealth.com wrote: This is all extremely helpful. After some hair pulling I was able to get the local sql server express running and I ran the same test on a local instance. 10100.1260.0000.1260.000 {method 'execute' of 'pyodbc

Re: [sqlalchemy] session flush inserts performing VERY slow for MS Sql Server

2016-03-09 Thread jbrownstein
This is all extremely helpful. After some hair pulling I was able to get the local sql server express running and I ran the same test on a local instance. 10100.1260.0000.1260.000 {method 'execute' of 'pyodbc.Cursor' objects} 10100.0600.0000.0600.000 {m

Re: [sqlalchemy] session flush inserts performing VERY slow for MS Sql Server

2016-03-08 Thread Mike Bayer
so the profile shows pretty clearly that the time is spent on the DB side. 1006 SQL statements are clocking at .09 seconds each (!) to take a total of 93 seconds: 10060.0120.000 93.8730.093 C:\Users\jbrownstein\Documents\GitHub\ETL\ETL\env\lib\site-packages\sqlalchemy\engine\b

Re: [sqlalchemy] session flush inserts performing VERY slow for MS Sql Server

2016-03-08 Thread jbrownstein
I'm on sqla 1.0.12 / MS SQL Server 2014 / python 64bit 2.7 (sorry should have included that in the first post) Here is an extract from the profile using the code from http://docs.sqlalchemy.org/en/rel_0_8/faq.html#code-profiling I am reading explanations in the FAQ now but any guidance would be

Re: [sqlalchemy] session flush inserts performing VERY slow for MS Sql Server

2016-03-07 Thread Jason Brownstein
Thanks Mike - will the full profile ASAP. On Monday, March 7, 2016, Mike Bayer wrote: > What will make inserts slow is network as well as indexes. If you aren't > comparing your script to some other backend then there's probably a whole > lot of other things that can get in the way like unicod

Re: [sqlalchemy] session flush inserts performing VERY slow for MS Sql Server

2016-03-07 Thread Mike Bayer
What will make inserts slow is network as well as indexes. If you aren't comparing your script to some other backend then there's probably a whole lot of other things that can get in the way like unicode encoding, large columns or other surprises in the model. If you run the full profile ver

[sqlalchemy] session flush inserts performing VERY slow for MS Sql Server

2016-03-07 Thread jbrownstein
I've run sqlalchemy on MySQL dbs and never had a problem with performance, but when I started writing an app that connected to Microsoft SQL Server 2014 via pyodbc I ran into a serious performance problem. I'm running the app on my laptop and the database server IS in a VPN but my internet is f