On Sunday, July 17, 2016 at 8:47:11 AM UTC+2, Martijn van Oosterhout wrote: > > > I'll play a bit and see what I can get to work. Thanks again. > > So, I have a chance to play and got something that actually works quite nicely, see below. Two things:
- I switched to referencing the primary key of the original object directly, because some of our relationships are a bit more complex. - Chained relationships don't work. But in any case, even this improves performance greatly. from itertools import groupby, islice from sqlalchemy.orm import attributes, object_session from sqlalchemy import tuple_ def yielded_load(query, attrs, N=1000): # Note: query must return only a single object (for now anyway) main_query = query.yield_per(N) main_res = iter(main_query) while True: # Fetch block of results from query objs = list(islice(main_res, N)) if not objs: break for attr in attrs: target = attr.prop.mapper pk = attr.prop.parent.primary_key # Generate query that joins against original table child_q = object_session(objs[0]).query(target, *pk).order_by(* pk) if attr.prop.order_by: child_q = child_q.order_by(*attr.prop.order_by) keys = [[getattr(obj, col.key) for col in pk] for obj in objs] child_q = child_q.join(attr).filter(tuple_(*pk).in_(keys)) collections = dict((k, [r[0] for r in v]) for k, v in groupby( child_q, lambda x:tuple([getattr(x, c.key) for c in pk]) )) for obj in objs: attributes.set_committed_value( obj, attr.key, collections.get( tuple(getattr(obj, c.key) for c in pk), ()) ) for obj in objs: yield obj -- Martijn -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.