On 07/19/2016 12:33 PM, Martijn van Oosterhout wrote:
On Sunday, July 17, 2016 at 8:47:11 AM UTC+2, Martijn van Oosterhout wrote:
I'll play a bit and see what I can get to work. Thanks again.
So, I have a chance to play and got something that actually works quite
nicely, see below. Two things:
- I switched to referencing the primary key of the original object
directly, because some of our relationships are a bit more complex.
- Chained relationships don't work.
But in any case, even this improves performance greatly.
this is adequate for a new recipe if you are interested in adding it.
Also I think sqlalchemy-utils provides a feature along these lines but
I'm not sure how well it works or handles those harder cases like
chained relationships.
|
fromitertools importgroupby,islice
fromsqlalchemy.orm importattributes,object_session
fromsqlalchemy importtuple_
defyielded_load(query,attrs,N=1000):
# Note: query must return only a single object (for now anyway)
main_query =query.yield_per(N)
main_res =iter(main_query)
whileTrue:
# Fetch block of results from query
objs =list(islice(main_res,N))
ifnotobjs:
break
forattr inattrs:
target =attr.prop.mapper
pk =attr.prop.parent.primary_key
# Generate query that joins against original table
child_q =object_session(objs[0]).query(target,*pk).order_by(*pk)
ifattr.prop.order_by:
child_q =child_q.order_by(*attr.prop.order_by)
keys =[[getattr(obj,col.key)forcol inpk]forobj inobjs]
child_q =child_q.join(attr).filter(tuple_(*pk).in_(keys))
collections =dict((k,[r[0]forr inv])fork,v ingroupby(
child_q,
lambdax:tuple([getattr(x,c.key)forc inpk])
))
forobj inobjs:
attributes.set_committed_value(
obj,
attr.key,
collections.get(
tuple(getattr(obj,c.key)forc inpk),
())
)
forobj inobjs:
yieldobj
|
--
Martijn
--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to sqlalchemy+unsubscr...@googlegroups.com
<mailto:sqlalchemy+unsubscr...@googlegroups.com>.
To post to this group, send email to sqlalchemy@googlegroups.com
<mailto:sqlalchemy@googlegroups.com>.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
