[sqlalchemy] Re: pandas.DataFrame.to_sql method: how to speed up exporting to Microsoft SQL Server (6 minutes for 11 MB!)

2015-04-28 Thread Sam Mourad
Joe - I'm having the same issue. Dataframe.to_sql performance is fine with sqllite and mysql but SQLAlchemy engine it too slow. After a few days of research and trial and error, I was able to improve performance by using pymssql. However, the performance is still way below standards. 35000 rows

[sqlalchemy] Re: pandas.DataFrame.to_sql method: how to speed up exporting to Microsoft SQL Server (6 minutes for 11 MB!)

2015-04-22 Thread Jonathan Vanasco
I've ran into similar issues like this before -- but on different databases. I think it's bad to think of the problem as transmitting 31k/s -- which assumes a particular issue; and much better to think of it as processing 31k/s, which gives a lot more room for interpretation. Looking on

[sqlalchemy] Re: pandas.DataFrame.to_sql method: how to speed up exporting to Microsoft SQL Server (6 minutes for 11 MB!)

2015-04-22 Thread John Doe
On Wednesday, April 22, 2015 at 7:19:08 PM UTC+1, Jonathan Vanasco wrote: [...] I'm guessing that this issue is with the driver. Here's a semi-related thread: * http://stackoverflow.com/questions/5693885/pyodbc-very-slow-bulk-insert-speed