Hi Mike,
sorry for the late reply, but it took a considerable amount of time to
sort the non SQLA related issues out.. Will elaborate in a reply to
Jonathan's answer a bit.
On Donnerstag, 26. Juni 2014 17:34:58 Mike Bayer wrote:
> On 6/26/14, 5:18 PM, Hans-Peter Jansen wrote:
> >
> > Still, attributes is highest in profiles. I guess, this is, what you
> > called overkill in your first response. A helper for dealing with this
> > situation sounds interesting, though.
> >
> > Thu Jun 26 20:41:50 2014 srelay.pstats
> >
> > 55993702 function calls (54767154 primitive calls) in 533.479
> > seconds
> >
> > Ordered by: internal time
> > List reduced from 1277 to 30 due to restriction <30>
> >
> > ncalls tottime percall cumtime percall filename:lineno(function)
> >
> > 42160 292.536 0.007 292.910 0.007
> > attributes.py:1321(from_collection) 63209 39.185 0.001 39.294
> > 0.001 attributes.py:1255(as_state)
> from_collection is essentially returning a structure that describes
> which members of a collection were removed or added. if the collection
> is totally empty or is unloaded, this method should be much faster. try
> not do to this:
>
> enos.values.append(ev)
>
> that's what's causing that latency to occur. The one-to-many of
> enos->ev is mirrored by a many to one from ev->enos - set enos as a
> parent of ev instead, or even faster, just set the foreign key column on
> ev to that of the primary key of enos.
Okay, I did that already, but didn't realize, that it's better to just add the
record to the non auto flushing session. With auto flushing disabled, some
other issues arose with related db operations in that session. I finally solved
that with creating an additional session besides the scoped one, and setting
that one up with autoflush = False, as you suggested.
Although, I haven't given up the declarative layer, about the whole work is
done in the database layer, which is a great achievement for my POV.
Sun Jun 29 17:17:16 2014 profiles/srelay_20140629_171715.pstats
9740100 function calls (9425633 primitive calls) in 141.169 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 141.170 141.170 cProfile.py:146(runcall)
1 0.169 0.169 141.170 141.170 srelay.py:24(run)
32579 0.094 0.000 93.955 0.003 fetch_srelay.py:62(fetch_srelay)
39 0.000 0.000 91.449 2.345 fetch_srelay.py:49(recv)
78 0.002 0.000 91.449 1.172 fetch_srelay.py:34(recvblock)
167 91.446 0.548 91.446 0.548 {method 'recv' of
'_socket.socket' objects}
46 0.001 0.000 18.571 0.404 session.py:687(commit)
95/51 0.001 0.000 18.570 0.364 session.py:351(commit)
137 0.057 0.000 18.437 0.135 session.py:1790(flush)
95/51 0.000 0.000 18.414 0.361 session.py:320(_prepare_impl)
49 0.073 0.001 18.372 0.375 session.py:1841(_flush)
49 0.001 0.000 17.175 0.351 unitofwork.py:350(execute)
82 0.015 0.000 16.777 0.205 unitofwork.py:522(execute)
82 0.001 0.000 16.762 0.204 persistence.py:25(save_obj)
22 0.365 0.017 15.898 0.723
persistence.py:523(_emit_insert_statements)
25139 0.102 0.000 14.937 0.001 base.py:597(execute)
25139 0.202 0.000 14.835 0.001
base.py:727(_execute_clauseelement)
25139 0.276 0.000 14.466 0.001 base.py:812(_execute_context)
25139 0.056 0.000 11.485 0.000 default.py:323(do_execute)
25184 0.310 0.000 11.444 0.000 cursors.py:164(execute)
25184 0.046 0.000 10.213 0.000 cursors.py:353(_query)
25184 0.117 0.000 10.117 0.000 cursors.py:315(_do_query)
25184 9.660 0.000 9.660 0.000 {method 'query' of
'_mysql.connection' objects}
32573 0.445 0.000 3.688 0.000 srelay.py:88(store_enos_rec)
25139 0.675 0.000 1.729 0.000 default.py:391(_init_compiled)
Without profiling, I'm down to 90 secs for fetching > 33000 records from a
raspi,
and pushing them in a mysql database. All in pure Python. Fun!
Thanks again, Mike, for this great lib and your even GREATER support.
Cheers,
Pete
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
