On Apr 27, 2008, at 5:04 PM, Kyle Schaffrick wrote:
> partial = partial + list(...) > > The bits inside that list() calls up to Query.instances for each shard > in the for loop, which seems to return an iterator. > > Assuming the desired ordering can be figured out from that spot, and a > python comparison function generated from it, creating a generator > that > performs a merge would be just the thing, no? > > For one thing, the entire result set wouldn't be sucked into RAM at > once, as it appears to be doing here. And if the various shard queries > emit sorted results, you just need a merge, which can be done > incrementally with a generator. It would have a RAM savings in that each shard could be dealt with individually, although in the usual case (unless the yield_per() option is used) each individual iterate_instances() call fully fetches before yielding results. Also, lots of DBAPIs like psycopg2 pre- buffer result sets by default. But yes its probably how that section should be done anyway so that ShardedQuery so that the iterative framework provided by iterate_instances() (this method would need to be used instead of instances()). --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~----------~----~----~----~------~----~------~--~---
