Hi,
I am using sqlalchemy like this:
entries = session.query(User)
for entry in entries:
entry.check_it_is_all_right() # includes changing values of
columns if necessary
session.flush()
As you might have noticed, I am iterating over all rows in the
database. Since there are like thousands rows, each containing like
10kB of data (and I _do_ need all of the data for each entry), it is
unconvenient for me how sqlalchemy treats this:
it loads _all_ objects into memory - this takes approx. 30 seconds on
a dedicated machine with 100% CPU usage, taking away tens of MB of
RAM. (it is my virtual testing machine, it's fine there (although who
wants to wait 30 seconds to only realize that the processing fails
with exception on first entry :-)), but I most probably can't use such
a resource-eater on a production server).
It would however suffice that sqlalchemy made the access to the result
sequentially, without loading more than is really needed at the
moment. I thought the Python iterator protocol (__iter__ and next
methods) are exactly tailored for this efficient access (e.g.
iterating over lines of file objects (for line in open("file.txt",
"r")) works exactly efficiently how I would want sqlalchemy to in my
case :-) -
Is there a way for efficient "on-demand" iteration over all rows when
using orm object mapping, or do I have to use a more low-level
protocol (like sqlalchemy without orm (fetchone()), or even Python DB
API itself?)
Thanks for your suggestions,
Boris Duek
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~----------~----~----~----~------~----~------~--~---
