[issue35930] Raising an exception raised in a "future" instance will create reference cycles

[Jesús Cea Avión]

Jesús Cea Avión  added the comment:

I retired the backporting request. Thanks, Ned.

--
versions:  -Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35930] Raising an exception raised in a "future" instance will create reference cycles

[Ned Deily]

Ned Deily  added the comment:

"Those releases are in their"

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35930] Raising an exception raised in a "future" instance will create reference cycles

[Ned Deily]

Ned Deily  added the comment:

@jcea, I see you have created backports for 3.7 and 3.6 as well. Those release 
as in their security-fix-only phase of their life cycles. This doesn't seem 
like a security issue but am I missing something?

--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35930] Raising an exception raised in a "future" instance will create reference cycles

[Jesús Cea Avión]

Change by Jesús Cea Avión :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35930] Raising an exception raised in a "future" instance will create reference cycles

[Jesús Cea Avión]

Jesús Cea Avión  added the comment:


New changeset d914813a7a9cee3b42e9c91f91ac491f3bbfe118 by Miss Islington (bot) 
in branch '3.9':
bpo-35930: Raising an exception raised in a "future" instance will create 
reference cycles (GH-24995) (#25070)
https://github.com/python/cpython/commit/d914813a7a9cee3b42e9c91f91ac491f3bbfe118


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35930] Raising an exception raised in a "future" instance will create reference cycles

[Jesús Cea Avión]

Jesús Cea Avión  added the comment:


New changeset dae1963cf38f730291126b7dadfda89ffb21cefd by Miss Islington (bot) 
in branch '3.8':
bpo-35930: Raising an exception raised in a "future" instance will create 
reference cycles (GH-24995) (#25071)
https://github.com/python/cpython/commit/dae1963cf38f730291126b7dadfda89ffb21cefd


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35930] Raising an exception raised in a "future" instance will create reference cycles

[miss-islington]

Change by miss-islington :


--
pull_requests: +23821
pull_request: https://github.com/python/cpython/pull/25071

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35930] Raising an exception raised in a "future" instance will create reference cycles

[miss-islington]

Change by miss-islington :


--
pull_requests: +23823
pull_request: https://github.com/python/cpython/pull/25073

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35930] Raising an exception raised in a "future" instance will create reference cycles

[Jesús Cea Avión]

Jesús Cea Avión  added the comment:


New changeset 32430aadadf6e012e39167d3c18a24e49fb84874 by Jesús Cea in branch 
'master':
bpo-35930: Raising an exception raised in a "future" instance will create 
reference cycles (#24995)
https://github.com/python/cpython/commit/32430aadadf6e012e39167d3c18a24e49fb84874


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35930] Raising an exception raised in a "future" instance will create reference cycles

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 1.0 -> 2.0
pull_requests: +23820
pull_request: https://github.com/python/cpython/pull/25070

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35930] Raising an exception raised in a "future" instance will create reference cycles

[miss-islington]

Change by miss-islington :


--
pull_requests: +23822
pull_request: https://github.com/python/cpython/pull/25072

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35930] Raising an exception raised in a "future" instance will create reference cycles

[Jesús Cea Avión]

Change by Jesús Cea Avión :


--
versions: +Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35930] Raising an exception raised in a "future" instance will create reference cycles

[Jesús Cea Avión]

Change by Jesús Cea Avión :


--
assignee:  -> jcea

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35930] Raising an exception raised in a "future" instance will create reference cycles

[Jesús Cea Avión]

Change by Jesús Cea Avión :


--
keywords: +patch
pull_requests: +23753
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/24995

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35930] Raising an exception raised in a "future" instance will create reference cycles

[Jesús Cea Avión]

Change by Jesús Cea Avión :


--
versions: +Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35930] Raising an exception raised in a "future" instance will create reference cycles

[Jesús Cea Avión]

Jesús Cea Avión  added the comment:

Even more reproductible case, now 100%:

"""
import gc
import concurrent.futures
executor = concurrent.futures.ThreadPoolExecutor(999)

def a():
  1/0

future=executor.submit(a)
future.result()
# An exception is raised here. That is normal
del future  # Variable vanish, but data is still there because the cycle
1/0  # Raises another exception drop references to the future one
gc.set_debug(gc.DEBUG_SAVEALL)
gc.collect()
gc.garbage
"""

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35930] Raising an exception raised in a "future" instance will create reference cycles

[Jesús Cea Avión]

Jesús Cea Avión  added the comment:

The corrected test case in the terminal: (a "del" was missing)

"""
import gc
import concurrent.futures
executor = concurrent.futures.ThreadPoolExecutor(999)

def a():
  1/0

future=executor.submit(a)
future.result()
# An exception is raised here. That is normal
del future  # Variable vanish, but data is still there because the cycle
gc.set_debug(gc.DEBUG_SAVEALL)
gc.collect()
gc.garbage
"""

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35930] Raising an exception raised in a "future" instance will create reference cycles

[Jesús Cea Avión]

New submission from Jesús Cea Avión :

Try this in a terminal:

"""
import gc
import concurrent.futures
executor = concurrent.futures.ThreadPoolExecutor(999)

def a():
  1/0

future=executor.submit(a)
future.result()
# An exception is raised here. That is normal
gc.set_debug(gc.DEBUG_SAVEALL)
gc.collect()
gc.garbage
"""

You will see (python 3.7) that 23 objects are collected when cleaning the cycle.

The problem is the attribute "future._exception". If the exception provided by 
the "future" is raised somewhere else, we will have reference cycles because we 
have the same exception/traceback in two different places in the traceback 
framestack.

I commonly do this in my code:

"""
try:
  future.result()  # This will raise an exception if the future did it
except Exception:
   ... some clean up ...
  raise  # Propagate the "future" exception
"""

This approach will create reference cycles. They will eventually cleaned up, 
but I noticed this issue because the cycle clean up phase was touching big 
objects with many references but unused for a long time, so they were living in 
the SWAP. The cycle collection was hugely slow because of this and the 
interpreter is completely stopped until done.

Not sure about what to do about this. I am currently doing something like:

"""
try:
  future.result()  # This will raise an exception if the future did it
except Exception:
   if future.done():
   del future._exception
  raise  # Propagate the exception
"""

I am breaking the cycle manually. I do not use "future.set_exception(None) 
because side effects like notifying waiters.

I think this is a bug to be solved. Not sure how to do it cleanly.

What do you think? Ideas?.

--
messages: 335022
nosy: jcea
priority: normal
severity: normal
status: open
title: Raising an exception raised in a "future" instance will create reference 
cycles
versions: Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com