On Wed Mar 5 17:37:12 CET 2014, Victor Stinner wrote:
> Python 3 now stores the traceback object in Exception.__traceback__
> and exceptions can be chained through Exception.__context__. It's
> convenient but it introduced tricky reference cycles if the exception
> object is used out of the except block.
> ... see Future.set_exception() of the ayncio module.
> ... frame.clear() raises an RuntimeError if the frame is still
> running. And it doesn't break all reference cycles.
> An obvious workaround is to store the traceback as text, but this
> operation is "expensive" especially if the traceback is only needed in
> rare cases.
> I tried to write "views" of the traceback (and frames), but
> Exception.__traceback__ rejects types other than traceback and
> traceback instances cannot be created. It's possible to store the
> traceback somewhere else and set Exception.__traceback__ to None, but
> there is still the problem with chained exceptions.
> Any idea for a generic fix to such problem?
Could you clarify what the problem actually is? I can imagine
any of the following:
(A) Exceptions take a lot of memory, because of all the related
details.
+ But sometimes the details are needed, so there is no good solution.
(B) Exceptions take a lot of memory, because of all the related
details. There is a common use case that knows it will never
need certain types of details, and releasing just those details
would save a lot of memory. But frame.clear() picks the wrong
details to release, at least for this case.
+ So write another function (or even a method) that does work, and
have your framework call it. (Also see (F))
+ Instead of saving the original exception, could you instead
create and store a new (copied?) one, which obviously won't
(yet) be referenced by the traceback you assign to it?
(C) Exceptions take a lot of memory, because of all the related
details. There is a common use case that knows it can make do
with a summary of certain types of details, and releasing just
those details would save a lot of memory. But generating the
summary is expensive.
+ It would help to have the summarize method available.
+ It would help to have feedback from gc saying when there is enough
memory pressure to make this call worthwhile.
(D) Exceptions are not released until cyclic gc, and so they
eat a lot of memory for a long time prior to that.
+ This may be like case B
+ Are there references that can be replaced by weak references?
+ Are there references that you can replace with weak references
when your framework stores the exception? (Also see (F))
(E) Exceptions are not released even during cyclic gc, because
of ambiguity over which __del__ to run first.
+ This may be like case B or case E
+ This may be a concrete use case for the __close__ protocol.
__close__ is similar to __del__, except that it promises not
to care about order of finalization, and it is run eagerly.
As soon as an instance is known to be in a garbage cycle,
__close__ should be run without worrying about whether other
objects also have __close__ or __del__ methods. Hopefully,
this will break the cycle, or at least reduce the number of
objects with __del__ methods. (Whether to require that
__close__ be idempotent, or to guarantee that it is run
only once/instance -- that would be part of the bikeshedding.)
(F) You know what to delete (or turn into weakrefs), but can't
actually do it without changing a type.
(F1) Why does Exception.__traceback__ reject other objects
which are neither tracebacks nor None?
+ Can that restriction be relaxed?
+ Can you create a mixin subtype of Exception, which relaxes
the constraint, and gets used by your framework?
+ Can the restriction on creating tracebacks be relaxed?
+ Can traceback's restriction on frames' types be relaxed?
(F2) Do you need the original Exception? (see (B))
(F3) Do you care about frame.clear() raising a runtime
exception? Could you suppress it (or, better, get clear()
to raise something more specific, and suppress that)?
It would still have released what memory it reasonably
could.
-jJ
--
If there are still threading problems with my replies, please
email me with details, so that I can try to resolve them. -jJ
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
