[sqlalchemy] Re: Inserting Entry Fastest way

2013-03-11 Thread Russ
On Monday, March 11, 2013 4:56:11 PM UTC-4, Arkilic, Arman wrote:

  Hi,
 I am working on a database design that I am required to use lots of tables 
 with one-to-many relationship. As a consequence of the design, I need to 
 insert thousands of entries. I tried session.add(), session.merge, however 
 none of them is fast enough for me to meet the requirements. I was 
 wondering if you can suggest me an efficient way through either ORM or 
 ORM+Core.
 Thanks!


I recently did a presentation structured around getting fast bulk inserts 
with SQLAlchemy.  You may find it useful:

https://speakerdeck.com/rwarren/a-brief-intro-to-profiling-in-python

Please note that the focus was on profiling, and not on SQLAlchemy.   The 
SQLAlchemy example just worked out well (with a contrived step or two) as a 
vehicle for showing different profiling steps/gotchas.  Since the focus was 
on profiling, the example is quite simple (a single user table)... but you 
can easily extend on it for your one-to-many tables.

I also didn't 100% scrub the SQLAlchemy code (I threw this together in a 
hurry), so no yelling at me for bad code. :)

Russ

-- 
You received this message because you are subscribed to the Google Groups 
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.




Re: [sqlalchemy] Re: Inserting Entry Fastest way

2013-03-11 Thread Warwick Prince
Thanks Russ - took a look and found it very interesting indeed.

Cheers
Warwick

 On Monday, March 11, 2013 4:56:11 PM UTC-4, Arkilic, Arman wrote:
 Hi,
 I am working on a database design that I am required to use lots of tables 
 with one-to-many relationship. As a consequence of the design, I need to 
 insert thousands of entries. I tried session.add(), session.merge, however 
 none of them is fast enough for me to meet the requirements. I was wondering 
 if you can suggest me an efficient way through either ORM or ORM+Core.
 Thanks!
 
 I recently did a presentation structured around getting fast bulk inserts 
 with SQLAlchemy.  You may find it useful:
 
 https://speakerdeck.com/rwarren/a-brief-intro-to-profiling-in-python
 
 Please note that the focus was on profiling, and not on SQLAlchemy.   The 
 SQLAlchemy example just worked out well (with a contrived step or two) as a 
 vehicle for showing different profiling steps/gotchas.  Since the focus was 
 on profiling, the example is quite simple (a single user table)... but you 
 can easily extend on it for your one-to-many tables.
 
 I also didn't 100% scrub the SQLAlchemy code (I threw this together in a 
 hurry), so no yelling at me for bad code. :)
 
 Russ
 
 
 -- 
 You received this message because you are subscribed to the Google Groups 
 sqlalchemy group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to sqlalchemy+unsubscr...@googlegroups.com.
 To post to this group, send email to sqlalchemy@googlegroups.com.
 Visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
 For more options, visit https://groups.google.com/groups/opt_out.
  
  

-- 
You received this message because you are subscribed to the Google Groups 
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.