> On Jan 24, 2017, at 12:48 PM, Craig Rodrigues <rodr...@crodrigues.org> wrote: > > > > On Mon, Jan 23, 2017 at 5:10 PM, Jean-Paul Calderone > <exar...@twistedmatrix.com <mailto:exar...@twistedmatrix.com>> wrote: > On Mon, Jan 23, 2017 at 7:08 PM, Craig Rodrigues <rodr...@crodrigues.org > <mailto:rodr...@crodrigues.org>> wrote: > > > > I did some more debugging. The callstack is quite deep. :/ > I used this command to trigger the error: > > trial > buildbot.test.unit.test_process_buildrequestdistributor.TestMaybeStartBuilds.test_slow_db > > I found in this function in buildbot's BuildRequestsEndpoint.get() function ( > https://github.com/buildbot/buildbot/blob/master/master/buildbot/data/buildrequests.py#L146 > > <https://github.com/buildbot/buildbot/blob/master/master/buildbot/data/buildrequests.py#L146> > ) there is this line: > > defer.returnValue( > [(yield self.db2data(br)) for br in buildrequests]) > > On Python 2, this line returns: > an object list > each list entry is a dictionary of name/value pairs that looks like: > [{'buildrequestid': 10, 'complete': False, 'waited_for': False, 'claimed_at': > None, 'results': -1, 'claimed': False, 'buildsetid': 11, 'complete_at': None, > 'submitted_at': datetime.datetime(1970, 1, 2, 12, 6, 40, tzinfo=tzutc()), > 'builderid': 77, 'claimed_by_masterid': None, 'priority': 0}, > {'buildrequestid': 11, 'complete': False, 'waited_for': False, 'claimed_at': > None, 'results': -1, 'claimed': False, 'buildsetid': 11, 'complete_at': None, > 'submitted_at': datetime.datetime(1970, 1, 2, 13, 30, tzinfo=tzutc()), > 'builderid': 77, 'claimed_by_masterid': None, 'priority': 0}] > > > On Python 3, this returns: > an object <generator object BuildRequestsEndpoint.get.<locals>.<listcomp> > of type <class 'generator'> > > > Yep. > > On Python 2, [(yield self.db2data(br)) for br in buildrequests] is a list > comprehension. It will have len(buildrequests) elements and each will be the > value sent back in to the generator via the yield expression. > > On Python 3, the same expression is a list of one element which is a > generator expression. > > This form is much clearer, I think, and behaves as intended on both versions > of Python: > > results = [] > for br in buildrequests: > results.append((yield self.db2data(br))) > defer.returnValue(results) > > > > Wow, thanks! That fixed it. I submitted those fixes upstream to buildbot. > > I'm not so familiar with generator expressions and Deferred's so have been > playing > around with things to try to understand. > > I went back to the problem code, and changed: > > defer.returnValue( > [(yield self.db2data(br)) for br in buildrequests]) > > to: > > defer.returnValue( > list([(yield self.db2data(br)) for br in buildrequests]) > > and ran the code under Python 3 and got a return value of: > > [ Deferred ] > > instead of: > [ {......} ] > > where the Deferred.results value was the dictionary. > > How does Python 2 directly go to returning the dictionary out of the Deferred, > while Python 3 returns the Deferred inside the list? self.db2data() returns > a Deferred.
I've encountered this before and quickly worked around it, but I think this might actually be a bug in python 3, or at least its documentation. The language docs officially say that a "list display" (which is what I believe we're looking at here) "yields a new list object". But that is not what I see here: >>> [(yield 1) for x in range(10)] <generator object <listcomp> at 0x10cd210f8> That is a comprehension enclosed in square brackets, but I'm getting a generator object out rather than a list. Is there a PEP for this that just hasn't updated the language reference? -glyph
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
