On Tue, Aug 7, 2018 at 11:14 PM, Ken Hilton <kenlhil...@gmail.com> wrote: > This mostly springs off of a comment I saw in some thread. > > The point of a with statement is that it ensures that some resource will be > disposed of, yes? For example, this: > > with open(filename) as f: > contents = f.read() > > is better than this: > > contents = open(filename).read() > > because the former definitely closes the file while the latter relies on > garbage collection? > > The point of a yield expression is to suspend execution. This is nice for > efficient looping because instead of having to hold all results in memory, > each result can be consumed immediately, yes? Therefore this: > > def five_to_one(): > for i in range(4): > yield 5 - i > > is better than this: > > def five_to_one(): > result = [] > for i in range(4): > result.append(5 - i) > return result > > because the former suspends execution of "five_to_one" while the latter > holds all five results in memory? > > Now, let's take a look at the following scenario: > > def read_multiple(*filenames): > for filename in filenames: > with open(filename) as f: > yield f.read() > > Can you spot the problem? The "with open(filename)" statement is supposed to > ensure that the file object is disposed of properly. However, the "yield > f.read()" statement suspends execution within the with block, so if this > happened: > > for contents in read_multiple('chunk1', 'chunk2', 'chunk3'): > if contents == 'hello': > break > > and the contents of "chunk2" were "hello" then the loop would exit, and > "chunk2" would never be closed! Yielding inside a with block, therefore, > doesn't make sense and can only lead to obscure bugs.
This is a real problem. (Well, technically the 'with' block's __exit__ function *will* eventually close the file, when the generator is garbage-collected – see PEP 342 for details – but this is not exactly satisfying, because the whole purpose of the 'with' block is to close the file *without* relying on the garbage collector.) Unfortunately, your proposal for solving it is a non-starter -- there are lots of cases where 'yield' inside a 'with' block is not only used, but is clearly the right thing to do. A notable one is when defining a next contextmanager in terms of a pre-existing contextmanager: @contextmanager def tempfile(): # This is an insecure way of making a temp file but good enough for an example tempname = pick_random_filename() with open(tempname, "w") as f: yield f Here are some links for previous discussions around these kinds of issues, none of which have really gone anywhere but might help you get a sense of the landscape of options: https://www.python.org/dev/peps/pep-0533/ https://www.python.org/dev/peps/pep-0521/ https://www.python.org/dev/peps/pep-0568/ https://github.com/python-trio/trio/issues/264 One small step that might be doable would be to start issuing ResourceWarning whenever a generator that was suspended inside a 'with' or 'try' block is GC'ed. -n -- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/