Re: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__]
2014-03-08 1:14 GMT+01:00 Jim Jewett :
>>> Could you clarify what the problem actually is?
>
>> Please see:
>> http://bugs.python.org/file33238/never_deleted.py
>
> I would not expect it to be cleared at least until go runs ... and reading
> the ticket, it sounds like it is cleared then.
Attached script: never_deleted2.py, it's almost the same but it
explains better the problem. The script creates MyObject and Future
objects which are never deleted. Calling gc.collect() does *not* break
the reference cycle (between the future, the exception, traceback and
frames). Stopping the event loop does not remove Future nor MyObject
objects. Only exiting Python does remove the Future object.
The Future destructor calls code between this code fails between
Python is exiting and so most symbols are set to None. I'm testing
with Python 3.4 default, so with Antoine's PEP 442 and Serhiy's
changes on Python shutdown (restore builtins at exit).
The Future destructor is used in asyncio to notice the developer that
the application has a bug, the exception was not handled which may be
a major bug.
And MyObject is not destroyed which is an obvious memory leak, beause
there is no more explicit reference to it.
Script output:
---
gc.collect
stop!
exit
--- Logging error ---
Traceback (most recent call last):
--- Logging error ---
Traceback (most recent call last):
Exception ignored in: >
Traceback (most recent call last):
File "Lib/asyncio/futures.py", line 184, in __del__
File "Lib/asyncio/base_events.py", line 704, in call_exception_handler
File "Lib/logging/__init__.py", line 1280, in error
File "Lib/logging/__init__.py", line 1386, in _log
File "Lib/logging/__init__.py", line 1396, in handle
File "Lib/logging/__init__.py", line 1466, in callHandlers
File "Lib/logging/__init__.py", line 837, in handle
File "Lib/logging/__init__.py", line 961, in emit
File "Lib/logging/__init__.py", line 890, in handleError
File "Lib/traceback.py", line 169, in print_exception
File "Lib/traceback.py", line 153, in _format_exception_iter
File "Lib/traceback.py", line 18, in _format_list_iter
File "Lib/traceback.py", line 65, in _extract_tb_or_stack_iter
File "Lib/linecache.py", line 15, in getline
File "Lib/linecache.py", line 41, in getlines
File "Lib/linecache.py", line 126, in updatecache
File "Lib/tokenize.py", line 437, in open
AttributeError: 'module' object has no attribute 'open'
DELETE OBJET
---
Victor
import asyncio
import gc
import sys
class MyObject:
def __del__(self):
print("DELETE OBJECT")
@asyncio.coroutine
def func(fut):
obj = MyObject()
try:
raise ValueError()
except Exception as err:
fut.set_exception(err)
def collect():
print("gc.collect")
gc.collect()
def stop():
print("stop!")
loop.stop()
fut = asyncio.Future()
asyncio.Task(func(fut))
loop = asyncio.get_event_loop()
loop.call_later(2, collect)
loop.call_later(4, stop)
loop.run_forever()
loop.close()
print("exit")
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] for Python + Java devs - real-world importance of the security model
Hello everybody, I'm getting back into some Java game programming using the (excellent) libgdx library. It's been a couple years since I've written Java classes from scratch and it's got me thinking. The Java code I'm going through has lots 'final' and 'static' variable declarations, along with public vs private vs protected. For those of you who are Java devs in the real world ... is this security model really important? Is it protection against other developers somehow modifying state in your program via another package somewhere in the system? I've been doing Ruby code for a couple years now, and Python for a bit longer, and while I'm not a senior level dev by any means, I have yet to see where this security model is actually useful. Why not just keep it simple? For now I am writing my classes with very little protection keywords ... with as little keywords as possible actually. But I'm just getting started and I need to read up more. Any Java devs out there care to chime in? I'm not trying to troll, just interested in a response from those who appreciate KISS as a general principle. Thanks! -- A musician must make music, an artist must paint, a poet must write, if he is to be ultimately at peace with himself. - Abraham Maslow ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__]
On Sat, Mar 8, 2014 at 9:06 PM, Victor Stinner wrote: > And MyObject is not destroyed which is an obvious memory leak, beause > there is no more explicit reference to it. And it doesn't seem to be getting put into gc.garbage, either, which is probably worth mentioning. You have __del__ in there, so it's entirely possible it would get dropped into the garbage rather than actually disposed of, but I added a print(gc.garbage) after gc.collect() and it showed empty. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] for Python + Java devs - real-world importance of the security model
On 8 March 2014 20:27, Sean Felipe Wolfe wrote: > Hello everybody, > > I'm getting back into some Java game programming using the (excellent) > libgdx library. It's been a couple years since I've written Java > classes from scratch and it's got me thinking. Sean, did you mean to send this to core Python list? It is not relevant to CPython development (it's really off topic even for python-ideas). A more appropriate venue for this kind of general programming question is the "Programmers" subsite of Stack Exchange. For example, you can see some discussion of the "private" variable question at https://programmers.stackexchange.com/questions/143736/why-do-we-need-private-variables Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__]
On Sat, 8 Mar 2014 11:06:54 +0100 Victor Stinner wrote: > > Attached script: never_deleted2.py, it's almost the same but it > explains better the problem. The script creates MyObject and Future > objects which are never deleted. Calling gc.collect() does *not* break > the reference cycle (between the future, the exception, traceback and > frames). Stopping the event loop does not remove Future nor MyObject > objects. Only exiting Python does remove the Future object. So clearly the coroutine must be kept alive by something. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__]
2014-03-08 12:45 GMT+01:00 Antoine Pitrou :
>> Attached script: never_deleted2.py, it's almost the same but it
>> explains better the problem. The script creates MyObject and Future
>> objects which are never deleted. Calling gc.collect() does *not* break
>> the reference cycle (between the future, the exception, traceback and
>> frames). Stopping the event loop does not remove Future nor MyObject
>> objects. Only exiting Python does remove the Future object.
>
> So clearly the coroutine must be kept alive by something.
It's a reference cycle. Something like that:
Future -> Exception -> Traceback -> Frames -> Local variables ->
{Future, MyObject}
Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__]
On 8 March 2014 23:01, Victor Stinner wrote:
> 2014-03-08 12:45 GMT+01:00 Antoine Pitrou :
>>> Attached script: never_deleted2.py, it's almost the same but it
>>> explains better the problem. The script creates MyObject and Future
>>> objects which are never deleted. Calling gc.collect() does *not* break
>>> the reference cycle (between the future, the exception, traceback and
>>> frames). Stopping the event loop does not remove Future nor MyObject
>>> objects. Only exiting Python does remove the Future object.
>>
>> So clearly the coroutine must be kept alive by something.
>
> It's a reference cycle. Something like that:
>
> Future -> Exception -> Traceback -> Frames -> Local variables ->
> {Future, MyObject}
It seems unlikely we could have an uncollectable reference cycle that
doesn't end up in gc.garbage. Are you sure there are no external
references to one of the objects in the cycle, thus keeping the whole
thing alive?
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__]
On Sat, 8 Mar 2014 23:16:07 +1000
Nick Coghlan wrote:
> On 8 March 2014 23:01, Victor Stinner wrote:
> > 2014-03-08 12:45 GMT+01:00 Antoine Pitrou :
> >>> Attached script: never_deleted2.py, it's almost the same but it
> >>> explains better the problem. The script creates MyObject and Future
> >>> objects which are never deleted. Calling gc.collect() does *not* break
> >>> the reference cycle (between the future, the exception, traceback and
> >>> frames). Stopping the event loop does not remove Future nor MyObject
> >>> objects. Only exiting Python does remove the Future object.
> >>
> >> So clearly the coroutine must be kept alive by something.
> >
> > It's a reference cycle. Something like that:
> >
> > Future -> Exception -> Traceback -> Frames -> Local variables ->
> > {Future, MyObject}
>
> It seems unlikely we could have an uncollectable reference cycle that
> doesn't end up in gc.garbage. Are you sure there are no external
> references to one of the objects in the cycle, thus keeping the whole
> thing alive?
Ok, it's actually quite trivial. The whole chain is kept alive by the
"fut" global variable. If you arrange for it to be disposed of:
fut = asyncio.Future()
asyncio.Task(func(fut))
del fut
[etc.]
then the problem disappears: as soon as gc.collect() happens, the
MyObject instance is destroyed, the future is collected, and the
future's traceback is printed out.
Regards
Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__]
2014-03-08 14:33 GMT+01:00 Antoine Pitrou : > Ok, it's actually quite trivial. The whole chain is kept alive by the > "fut" global variable. If you arrange for it to be disposed of: > > fut = asyncio.Future() > asyncio.Task(func(fut)) > del fut > [etc.] > > then the problem disappears: as soon as gc.collect() happens, the > MyObject instance is destroyed, the future is collected, and the > future's traceback is printed out. Well, the problem is more general than this specific example. I would like to implement a general solution which would not hold references to local variables, to destroy objects when Python exits the except block. It looks like a "exception summary" containing only data to format the traceback would fit asyncio needs. If you don't want it in the traceback module, I will try to implement it in asyncio. It would be nice to provide an "exception summary" in the traceback module, because it looks like reference cycles related to exception and/or traceback is a common issue (see the list of links I gave in a previous email). Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__]
On Sat, Mar 8, 2014 at 5:14 PM, Victor Stinner wrote: > 2014-03-08 14:33 GMT+01:00 Antoine Pitrou : >> Ok, it's actually quite trivial. The whole chain is kept alive by the >> "fut" global variable. If you arrange for it to be disposed of: >> >> fut = asyncio.Future() >> asyncio.Task(func(fut)) >> del fut >> [etc.] >> >> then the problem disappears: as soon as gc.collect() happens, the >> MyObject instance is destroyed, the future is collected, and the >> future's traceback is printed out. > > Well, the problem is more general than this specific example. I would > like to implement a general solution which would not hold references > to local variables, to destroy objects when Python exits the except > block. > > It looks like a "exception summary" containing only data to format the > traceback would fit asyncio needs. If you don't want it in the > traceback module, I will try to implement it in asyncio. > > It would be nice to provide an "exception summary" in the traceback > module, because it looks like reference cycles related to exception > and/or traceback is a common issue (see the list of links I gave in a > previous email). > > Victor How about fixing cyclic gc to deal with __del__ instead? That sounds like an awful change to the semantics. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] unittest.TestCase.assert* methods calling TestCase.fail method directly.
Hi All, This is my first time on any mailing list... Please point out any mistakes.. I had a suggestion about the implementation of unittest.TestCase.assert* methods. They all call failureException separately. This is also what the unittest.TestCase.fail method does. Is there any specific reason it is so? If not, then is there any reason stopping them from calling the unittest.TestCase.fail method? As then we have one point where all the tests fail meaning that one can reimplement the unittest.TestCase.fail method for handling test failures in their own custom way. Regards, Pradyun ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__]
On Sat, 8 Mar 2014 16:14:23 +0100 Victor Stinner wrote: > 2014-03-08 14:33 GMT+01:00 Antoine Pitrou : > > Ok, it's actually quite trivial. The whole chain is kept alive by the > > "fut" global variable. If you arrange for it to be disposed of: > > > > fut = asyncio.Future() > > asyncio.Task(func(fut)) > > del fut > > [etc.] > > > > then the problem disappears: as soon as gc.collect() happens, the > > MyObject instance is destroyed, the future is collected, and the > > future's traceback is printed out. > > Well, the problem is more general than this specific example. I would > like to implement a general solution which would not hold references > to local variables, to destroy objects when Python exits the except > block. How about the following patch: diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py --- a/Lib/asyncio/futures.py +++ b/Lib/asyncio/futures.py @@ -315,6 +315,7 @@ class Future: self._schedule_callbacks() if _PY34: self._log_traceback = True +self._loop.call_soon(traceback.clear_frames, exception.__traceback__) else: self._tb_logger = _TracebackLogger(exception, self._loop) # Arrange for the logger to be activated after all callbacks That said, I agree an automated mechanism would be useful. It is overwhelmingly rare to want to analyze local variables in a traceback, yet Python always keeps a reference to those. Perhaps tracebacks could have a __debug__ attribute which, when sent to False, would prevent the locals from being kept alive (but how?). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] for Python + Java devs - real-world importance of the security model
Oops sorry I meant to send it elsewhere. Stupid auto complete :P On Sat, Mar 8, 2014 at 3:25 AM, Nick Coghlan wrote: > On 8 March 2014 20:27, Sean Felipe Wolfe wrote: >> Hello everybody, >> >> I'm getting back into some Java game programming using the (excellent) >> libgdx library. It's been a couple years since I've written Java >> classes from scratch and it's got me thinking. > > Sean, did you mean to send this to core Python list? It is not > relevant to CPython development (it's really off topic even for > python-ideas). > > A more appropriate venue for this kind of general programming question > is the "Programmers" subsite of Stack Exchange. For example, you can > see some discussion of the "private" variable question at > https://programmers.stackexchange.com/questions/143736/why-do-we-need-private-variables > > Regards, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia -- A musician must make music, an artist must paint, a poet must write, if he is to be ultimately at peace with himself. - Abraham Maslow ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
