New submission from Jesús Cea Avión <[email protected]>:
Try this in a terminal:
"""
import gc
import concurrent.futures
executor = concurrent.futures.ThreadPoolExecutor(999)
def a():
1/0
future=executor.submit(a)
future.result()
# An exception is raised here. That is normal
gc.set_debug(gc.DEBUG_SAVEALL)
gc.collect()
gc.garbage
"""
You will see (python 3.7) that 23 objects are collected when cleaning the cycle.
The problem is the attribute "future._exception". If the exception provided by
the "future" is raised somewhere else, we will have reference cycles because we
have the same exception/traceback in two different places in the traceback
framestack.
I commonly do this in my code:
"""
try:
future.result() # This will raise an exception if the future did it
except Exception:
... some clean up ...
raise # Propagate the "future" exception
"""
This approach will create reference cycles. They will eventually cleaned up,
but I noticed this issue because the cycle clean up phase was touching big
objects with many references but unused for a long time, so they were living in
the SWAP. The cycle collection was hugely slow because of this and the
interpreter is completely stopped until done.
Not sure about what to do about this. I am currently doing something like:
"""
try:
future.result() # This will raise an exception if the future did it
except Exception:
if future.done():
del future._exception
raise # Propagate the exception
"""
I am breaking the cycle manually. I do not use "future.set_exception(None)
because side effects like notifying waiters.
I think this is a bug to be solved. Not sure how to do it cleanly.
What do you think? Ideas?.
----------
messages: 335022
nosy: jcea
priority: normal
severity: normal
status: open
title: Raising an exception raised in a "future" instance will create reference
cycles
versions: Python 3.6, Python 3.7
_______________________________________
Python tracker <[email protected]>
<https://bugs.python.org/issue35930>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
