All:
I'm trying to process a large set of records in batches to control
memory usage, but after each batch,
the memory usage keeps going up. I suspect I am not freeing-up the
processed mapped objects correctly.
Can anyone here enlighten me as to what I'm doing wrong? Is there
another function I should be calling other
than session.expunge() once I have processed a record and am ready to
discard it? Out of the entire set,
only a small number will need to be updated so I can discard the
majority once I have determined that they
are not eligible to be changed.
The complete record set can be quite huge so I desperately need a way
to control Python's memory usage
while iterating through result sets. :(
Kindest regards,
Peter Buschman
session = objectstore.get_session()
batchsize = 100
query = select( [records.c.record_id])
results = query.execute()
record_ids = [ row[0] for row in results.fetchall() ]
record_count = len(record_ids)
batches = [backup_ids[i:i+batchsize] for i in range(0,
len(backup_ids), batchsize)]
batchnum = 0
debug('')
for batch in batches:
batchnum += 1
debug('[ Batch %d ] (%d)' % (batchnum, len(batch)))
records =
RecordLogMsg.mapper.select(recordlog.c.record_id.in_(*batch))
for record in records:
debug(' %s' % (record.record_id))
session.expunge(record)
-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Sqlalchemy-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/sqlalchemy-users
