[issue22427] TemporaryDirectory attempts to clean up twice
Roundup Robot added the comment: New changeset 7ea2153eae87 by Serhiy Storchaka in branch '3.4': Issue #22427: TemporaryDirectory no longer attempts to clean up twice when https://hg.python.org/cpython/rev/7ea2153eae87 New changeset e9d4288c32de by Serhiy Storchaka in branch 'default': Issue #22427: TemporaryDirectory no longer attempts to clean up twice when https://hg.python.org/cpython/rev/e9d4288c32de -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22427 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22427] TemporaryDirectory attempts to clean up twice
Serhiy Storchaka added the comment: Committed without this test. Thank you Victor and Yury for your comments. -- resolution: - fixed stage: patch review - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22427 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22427] TemporaryDirectory attempts to clean up twice
STINNER Victor added the comment: Cool, the final code is simpler than before! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22427 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22427] TemporaryDirectory attempts to clean up twice
Changes by Serhiy Storchaka storch...@gmail.com: -- assignee: - serhiy.storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22427 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22427] TemporaryDirectory attempts to clean up twice
Serhiy Storchaka added the comment: Fixed a bug in the test and partially addressed Victor's and Yury's comments. Antoine, could your pleas answer following question? Is it safe to remove the self._finalizer is not None check in cleanup()? I.e. is it possible that self._finalizer becomes None at garbage collection? -- Added file: http://bugs.python.org/file36658/tempfile_exit_on_shutdown3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22427 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22427] TemporaryDirectory attempts to clean up twice
Antoine Pitrou added the comment: self._finalizer can be None if an exception was raised during __init__(). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22427 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22427] TemporaryDirectory attempts to clean up twice
Serhiy Storchaka added the comment: But in this case cleanup() can't be called. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22427 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22427] TemporaryDirectory attempts to clean up twice
Antoine Pitrou added the comment: Ah... you are right. It seems the None test has been superfluous all the time. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22427 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22427] TemporaryDirectory attempts to clean up twice
Changes by Jakub Wilk jw...@jwilk.net: -- nosy: +jwilk ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22427 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22427] TemporaryDirectory attempts to clean up twice
Changes by Josh Rosenberg shadowranger+pyt...@gmail.com: -- nosy: +josh.rosenberg ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22427 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22427] TemporaryDirectory attempts to clean up twice
Changes by STINNER Victor victor.stin...@gmail.com: -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22427 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22427] TemporaryDirectory attempts to clean up twice
STINNER Victor added the comment: On Python 3.5 compiled in debug mode (or probably with python -Wd when compiled in release mode), I got this warning: /home/haypo/prog/python/default/Lib/tempfile.py:708: ResourceWarning: Implicitly cleaning up TemporaryDirectory '/tmp/tmpcr8u3m8v' _warnings.warn(warn_message, ResourceWarning) The script is really a corner case: the generator is garbage collected whereas it is not done. At the same time, the TemporaryDirectory object is also garbage collected. I guess that the exact order of object deletion is not reliable: the generator may be deleted before or after the TemporaryDirectory. TemporaryDirectory._cleanup() is called by the finalizer (_weakref.finalize) of the TemporaryDirectory, which means that the TemporaryDirectory object is garbage collected. TemporaryDirectory.cleanup() is called indirectly by TemporaryDirectory.__exit__(). I'm suprised that deleting a generator calls __exit__(). The following script has a more reliable behaviour: TemporaryDirectory.__exit__() is called when the generator is deleted, and TemporaryDirectory.cleanup() is not called. --- import tempfile import gc def generator(): with tempfile.TemporaryDirectory(): print(before yield) yield print(after yield) g = generator() next(g) g = None gc.collect() --- -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22427 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22427] TemporaryDirectory attempts to clean up twice
Jack O'Connor added the comment: My example script is definitely a corner case, but where this actually came up for me was in asyncio. Using a TemporaryDirectory inside a coroutine creates a similar situation. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22427 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22427] TemporaryDirectory attempts to clean up twice
Serhiy Storchaka added the comment: Here is a patch which may fix this issue. -- keywords: +patch nosy: +pitrou, serhiy.storchaka stage: - patch review versions: +Python 3.5 Added file: http://bugs.python.org/file36644/tempfile_exit_on_shutdown.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22427 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22427] TemporaryDirectory attempts to clean up twice
Serhiy Storchaka added the comment: Here is alternative patch. It simplifies TemporaryDirectory instead of complicate it. -- Added file: http://bugs.python.org/file36645/tempfile_exit_on_shutdown2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22427 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22427] TemporaryDirectory attempts to clean up twice
New submission from Jack O'Connor: The following little script prints (but ignores) a FileNotFoundError. import tempfile def generator(): with tempfile.TemporaryDirectory(): yield g = generator() next(g) Exception ignored in: generator object generator at 0x7fb319fe2c60 Traceback (most recent call last): File gen.py, line 6, in generator File /usr/lib/python3.4/tempfile.py, line 691, in __exit__ File /usr/lib/python3.4/tempfile.py, line 697, in cleanup File /usr/lib/python3.4/shutil.py, line 454, in rmtree File /usr/lib/python3.4/shutil.py, line 452, in rmtree FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp7wek4xhy' Putting print statements in the TemporaryDirectory class shows what's happening (confirming Guido's theory from https://groups.google.com/forum/#!topic/python-tulip/QXgWH32P2uM): As the program exits, the TemporaryDirectory object is finalized. This actually rmtree's the directory. After that, the generator is finalized, which raises a GeneratorExit inside of it. That exception causes the with statement to call __exit__ on the already-finalized TemporaryDirectory, which tries to rmtree again and throws the FileNotFoundError. The main downside of this bug is just garbage on stderr. I suppose in exceptionally unlikely circumstances, a new temp dir by the same name could be created between the two calls, and we might actually delete something we shouldn't. The simple fix would be to store a _was_cleaned flag or something on the object. Is there any black magic in finalizers or multithreading that demands something more complicated? -- components: Library (Lib) messages: 226979 nosy: oconnor663 priority: normal severity: normal status: open title: TemporaryDirectory attempts to clean up twice type: behavior versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22427 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
