Using a scoped session with a session generator and I didn't want
expire_on_commit to be False for everything, so setting it using the
Session constructor wouldn't work properly. If a session was created prior
to the one that needed that flag, it'd give me a ProtocolError since it
couldn't
The application I'm working on operates over extremely large datasets, so
I'm using the query windowing from here
(https://bitbucket.org/zzzeek/sqlalchemy/wiki/UsageRecipes/WindowedRangeQuery)
to break it into manageable chunks. The query window is usually around 10k
rows, after which it
A couple of questions:
I'm writing an application using concurrent.futures (by process). The
processes themselves are fairly involved - not simple functions. I'm using
scoped_sessions and a context manager like so:
# db.py
engine = create_engine(sqlalchemy_url)
Session =
I wasn't going to bother, but I had a look at doing this just out of
curiosity, and these were the results:
executemany():
Inserting 424 entries: 0.3362s
Inserting 20,000 segments: 14.01s
COPY:
Inserting 425 entries: 0.04s
Inserting 20,000 segments: 0.3s
So a pretty massive boost. Thanks :)
spent in Python. That said, if you have
any tips for improvements I'd be all ears.
Thanks for the help!
On Monday, 24 March 2014 09:19:25 UTC+8, Michael Bayer wrote:
On Mar 23, 2014, at 11:33 AM, James Meneghello
muro...@gmail.comjavascript:
wrote:
I'm having a few issues with unique
2014 14:40:52 UTC+8, James Meneghello wrote:
Thanks for the quick reply!
This seems to work pretty well. I took out the batching (as it's already
batched at a higher level) and modified it to suit the insertion of
children as well (and reducded the unique to a single field
That's effectively what I'm doing now. I'm not sure there's much I can
speed up at this point - the SELECTs take about 0.05s, it's just the
INSERTs taking a bulk of the time - 11-15s depending on the number of rows.
That said, I'm still running on development and there'll be a significant
I'm having a few issues with unique constraints and bulk inserts. The
software I'm writing takes data from an external source (a lot of it,
anywhere from 1,000 rows per minute to 100-200k+), crunches it down into
its hierarchy and saves it to the DB, to be aggregated in the background.
The
