On Tue, Jan 24, 2017 at 12:48 PM, Craig Rodrigues <rodr...@crodrigues.org> wrote:
> > > On Mon, Jan 23, 2017 at 5:10 PM, Jean-Paul Calderone < > exar...@twistedmatrix.com> wrote: > >> On Mon, Jan 23, 2017 at 7:08 PM, Craig Rodrigues <rodr...@crodrigues.org> >> wrote: >> >>> >>> >>> >>> I did some more debugging. The callstack is quite deep. :/ >>> I used this command to trigger the error: >>> >>> trial buildbot.test.unit.test_process_buildrequestdistributor.Tes >>> tMaybeStartBuilds.test_slow_db >>> >>> I found in this function in buildbot's BuildRequestsEndpoint.get() >>> function ( https://github.com/buildbot/buildbot/blob/master/master/buil >>> dbot/data/buildrequests.py#L146 ) there is this line: >>> >>> >>> >>> * defer.returnValue( [(yield self.db2data(br)) for br >>> in buildrequests])* >>> On Python 2, this line returns: >>> an object list >>> each list entry is a dictionary of name/value pairs that looks like: >>> >>> [{'buildrequestid': 10, 'complete': False, 'waited_for': False, >>> 'claimed_at': None, 'results': -1, 'claimed': False, 'buildsetid': 11, >>> 'complete_at': None, 'submitted_at': datetime.datetime(1970, 1, 2, 12, 6, >>> 40, tzinfo=tzutc()), 'builderid': 77, 'claimed_by_masterid': None, >>> 'priority': 0}, {'buildrequestid': 11, 'complete': False, 'waited_for': >>> False, 'claimed_at': None, 'results': -1, 'claimed': False, 'buildsetid': >>> 11, 'complete_at': None, 'submitted_at': datetime.datetime(1970, 1, 2, 13, >>> 30, tzinfo=tzutc()), 'builderid': 77, 'claimed_by_masterid': None, >>> 'priority': 0}] >>> >>> On Python 3, this returns: >>> an object <generator object BuildRequestsEndpoint.get.<loc >>> als>.<listcomp> >>> of type <class 'generator'> >>> >>> >> Yep. >> >> On Python 2, [(yield self.db2data(br)) for br in buildrequests] is a list >> comprehension. It will have len(buildrequests) elements and each will be >> the value sent back in to the generator via the yield expression. >> >> On Python 3, the same expression is a list of one element which is a >> generator expression. >> >> This form is much clearer, I think, and behaves as intended on both >> versions of Python: >> >> results = [] >> for br in buildrequests: >> results.append((yield self.db2data(br))) >> defer.returnValue(results) >> >> >> > Wow, thanks! That fixed it. I submitted those fixes upstream to buildbot. > > I'm not so familiar with generator expressions and Deferred's so have > been playing > around with things to try to understand. > > I went back to the problem code, and changed: > > > * defer.returnValue( [(yield self.db2data(br)) for br in > buildrequests])* > > to: > > * defer.returnValue(* > * list([**(yield self.db2data(br)) for br in buildrequests])* > > and ran the code under Python 3 and got a return value of: > > [ Deferred ] > > instead of: > [ {......} ] > > where the Deferred.results value was the dictionary. > > How does Python 2 directly go to returning the dictionary out of the > Deferred, > while Python 3 returns the Deferred inside the list? self.db2data() > returns a Deferred. > > Thanks. > > -- > Craig > On the python-dev mailing list, Joe Jevnik gave a really excellent explanation of what is going on here, down to the Python bytecode level: https://mail.python.org/pipermail/python-dev/2017-January/147244.html -- Craig
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python