[sqlalchemy] Re: executemany + postgresql

2009-11-07 Thread Jon Nelson
On Sat, Nov 7, 2009 at 3:02 PM, Michael Bayer wrote: > > > On Nov 7, 2009, at 1:30 PM, Jon Nelson wrote: > >>  File "pg8000/protocol.py", line 121, in serialize >>    val = struct.pack("!i", len(val) + 4) + val >> UnicodeDecodeError: 'ascii' codec can't decode byte 0x8d in position >> 3: ordinal

[sqlalchemy] Re: executemany + postgresql

2009-11-07 Thread Michael Bayer
On Nov 7, 2009, at 1:30 PM, Jon Nelson wrote: > File "pg8000/protocol.py", line 121, in serialize >val = struct.pack("!i", len(val) + 4) + val > UnicodeDecodeError: 'ascii' codec can't decode byte 0x8d in position > 3: ordinal not in range(128) make sure you're on the latest tip of pg8000,

[sqlalchemy] Re: executemany + postgresql

2009-11-07 Thread Jon Nelson
On Sat, Nov 7, 2009 at 11:58 AM, Michael Bayer wrote: > > On Nov 7, 2009, at 12:53 PM, Jon Nelson wrote: > > have you asked about this on the psycopg2 mailing list ?   its at > http://mail.python.org/mailman/listinfo/python-list > >  .   Let me know if you do, because I'll get out the popcorn...

[sqlalchemy] Re: executemany + postgresql

2009-11-07 Thread Michael Bayer
On Nov 7, 2009, at 12:53 PM, Jon Nelson wrote: >> >> have you asked about this on the psycopg2 mailing list ? its at >> http://mail.python.org/mailman/listinfo/python-list >> . Let me know if you do, because I'll get out the popcorn... :) > > That's the python list. oops: http://lists.ini

[sqlalchemy] Re: executemany + postgresql

2009-11-07 Thread Jon Nelson
On Fri, Nov 6, 2009 at 9:57 AM, Michael Bayer wrote: >> Before I even posted I resorted to strace. strace immediately >> confirmed my suspicion - when using psycopg2 I don't see one big fat >> INSERT with lots of binds, I see one INSERT per bind, and it's this >> that is ultimately killing the pe

[sqlalchemy] Re: executemany + postgresql

2009-11-06 Thread Adrian von Bidder
Heyho! On Friday 06 November 2009 02.46:11 Jon Nelson wrote: > ... was performing an individual > INSERT for every single row. Don't know sqlalchemy good enough, but for big bulk imports on the SQL side, shouldn't "COPY" be used? Which is as far as I know pg-specific / non-SQL standard. chee

[sqlalchemy] Re: executemany + postgresql

2009-11-05 Thread Jon
On Nov 5, 8:40 pm, Michael Bayer wrote: > On Nov 5, 2009, at 8:46 PM, Jon Nelson wrote: > > > > > > > I recently ran into an issue today where batched (inside a > > transaction) I was able to achieve not more than about 9000 > > inserts/second to a postgresql database (same machine running the >

[sqlalchemy] Re: executemany + postgresql

2009-11-05 Thread Michael Bayer
On Nov 5, 2009, at 8:46 PM, Jon Nelson wrote: > > I recently ran into an issue today where batched (inside a > transaction) I was able to achieve not more than about 9000 > inserts/second to a postgresql database (same machine running the > test). > > With everything exactly the same, I was able