On 8 August 2018 at 08:48, Nathaniel Smith <[email protected]> wrote:
> On Tue, Aug 7, 2018 at 11:14 PM, Ken Hilton <[email protected]> wrote:
>>
>> Now, let's take a look at the following scenario:
>>
>> def read_multiple(*filenames):
>> for filename in filenames:
>> with open(filename) as f:
>> yield f.read()
>>
>> Can you spot the problem? The "with open(filename)" statement is supposed to
>> ensure that the file object is disposed of properly. However, the "yield
>> f.read()" statement suspends execution within the with block, so if this
>> happened:
>>
>> for contents in read_multiple('chunk1', 'chunk2', 'chunk3'):
>> if contents == 'hello':
>> break
>>
>> and the contents of "chunk2" were "hello" then the loop would exit, and
>> "chunk2" would never be closed! Yielding inside a with block, therefore,
>> doesn't make sense and can only lead to obscure bugs.
>
> This is a real problem. (Well, technically the 'with' block's __exit__
> function *will* eventually close the file, when the generator is
> garbage-collected – see PEP 342 for details – but this is not exactly
> satisfying, because the whole purpose of the 'with' block is to close
> the file *without* relying on the garbage collector.)
PEP 342 guarantees that *if* a generator is garbage collected its
.close() method will be called which would in turn trigger __exit__()
for any active context managers in the generator. However PEP 342 does
not guarantee that the generator would be garbage collected. As you
noted in PEP 533:
In Python implementations that do not use reference counting
(e.g. PyPy, Jython), calls to __del__ may be arbitrarily delayed...
However this statement does not go far enough. It is not guaranteed
that __del__ will be called *at all* under other implementations. An
example to test this is:
$ cat gencm.py
class CM:
def __enter__(self):
print("Entering")
return self
def __exit__(self, *args):
print("Exiting")
def generator():
with CM():
yield 1
yield 2
yield 3
g = generator()
def f():
for x in g:
break # Or return
f()
print("End of program")
$ python3 gencm.py
Entering
End of program
Exiting
$ pypy --version
Python 2.7.2 (1.8+dfsg-2, Feb 19 2012, 19:18:08)
[PyPy 1.8.0 with GCC 4.6.2]
$ pypy gencm.py
Entering
End of program
(I don't actually have pypy to hand right now so I'm copying this from here:
https://www.mail-archive.com/[email protected]/msg70961.html)
What the above shows is that for this example:
1) Under CPython __exit__ is called by __del__ at process exit after
every line of Python code has finished.
2) Under PyPy __exit__ was not called at any point
That's not a bug in PyPy: Python the language makes very few
guarantees about garbage collection.
> Unfortunately, your proposal for solving it is a non-starter -- there
> are lots of cases where 'yield' inside a 'with' block is not only
> used, but is clearly the right thing to do. A notable one is when
> defining a next contextmanager in terms of a pre-existing
> contextmanager:
>
> @contextmanager
> def tempfile():
> # This is an insecure way of making a temp file but good enough
> for an example
> tempname = pick_random_filename()
> with open(tempname, "w") as f:
> yield f
This is the only example I can think of where yielding from a with
statement is the right thing to do. Really this comes from the way
that the contextmanager is abusing syntax though. It takes a bunch of
strange machinery to make this work as it does:
https://github.com/python/cpython/blob/3.7/Lib/contextlib.py#L116
> One small step that might be doable would be to start issuing
> ResourceWarning whenever a generator that was suspended inside a
> 'with' or 'try' block is GC'ed.
This seems like a great idea to me.
--
Oscar
_______________________________________________
Python-ideas mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
