Good idea, +1 from me. On Sun, May 19, 2019, 3:17 AM Yonatan Zunger <zun...@humu.com> wrote:
> Hi everyone, > > I'd like to bounce this proposal off everyone and see if it's worth > formulating as a PEP. I haven't found any prior discussion of it, but as we > all know, searches can easily miss things, so if this is old hat please LMK. > > *Summary: *The construction > > with expr1 as var1, expr2 as var2, ...: > body > > fails (with an AttributeError) unless each expression returns a value > satisfying the context manager protocol. Instead, we should permit any > expression to be used. If a value does not expose an __enter__ method, it > should behave as though its __enter__ method is return self; if it does > not have an __exit__ method, it should behave as though that method is return > False. > > *Rationale: *The with statement has proven to be a valued extension to > Python. In addition to providing improved readability for block scoping, it > has strongly encouraged the use of scoped cleanups for objects which > require them, such as files and mutices, in the process eliminating a lot > of annoying bugs. I would argue that at present, whenever dealing with an > object which requires such cleanup at a known time, with should be the > default way of doing it, and *not* doing so is the sort of thing one > should be explaining in a code comment. However, the current syntax makes a > few common patterns harder to implement than they should be. > > For example, this is a good pattern: > > with functionReturningFile(...) as input: > doSomething(input) > > There are many cases where an Optional[file] makes sense as a parameter, > as well; for example, an optional debug output stream, or an input source > which may either be a file (if provided) or some non-file source (by > default). Likewise, there are many cases where a function may naturally > return an Optional[file], e.g. "open the file if the user has provided the > filename." However, the following is *not* valid Python: > > with functionReturningOptionalFile(...) as input: > doSomething(input) > > To handle this case, one has a few options. One may only use the 'with' in > the known safe cases: > > inputFile = functionReturningOptionalFile(...) > if inputFile: > with inputFile as input: > doSomething(input) > else: > doSomething(None) > > (NB that this requires factoring the with statement body into its own > function, which may separately reduce readability and/or introduce > overhead); one may dispense with the 'with' clause and do it in the > pre-PEP343 way: > > try: > input = functionReturningOptionalFile(...) > doSomething(input) > finally: > if input: > input.close() > > (This sacrifices all the benefits of the with statement, and requires the > caller to explicitly call the cleanup methods, increasing error-proneness); > or one may construct an explicit 'dev-null' class and return it instead of > the file: > > class DevNullFile(object): > .... implement the entire File API, including a context manager ... > > (This can only be described as god-awful, especially for complex API's > like files) > > One obvious option would be to allow None to act as a context manager as > well. We might contrast this with PEP 336 > <https://www.python.org/dev/peps/pep-0336/>, "Make None Callable." This > was rejected (rightly, I think) because "it is considered a feature that > None raises an error if called." For example, it means that if a function > variable has been nulled, attempting to call it later raises an error, as > this usually indicates a code mistake. In the case where that is not > correct, it is easy to assign a noop lambda to the function variable > instead of None, thus allowing the error-checking and the > function-deactivating behaviors to both persist, and in a clear and easily > understandable way. > > In this case, OTOH, the AttributeError raised if None is passed to a with > statement has significantly lower value. As the example above illustrates, > there are many cases where None is an entirely legitimate value to want to > pass, and unlike in the other situation, there is no equally easy way to > pass it. Furthermore, if the passing of None *is* an error in some case, > it is more useful to see that error at the site where the variable is > actually used in the with statement body -- the thing for which it does not > make sense to use None -- rather than at a structural declaration which > essentially defines a variable scope. > > This is also the reason why such a change would impact relatively little > existing code: code already has to be structured to prevent this from > happening. If the assigned expression in the with statement could only > return None as a result of a code bug, and a piece of existing code is > relying on the with statement to catch it, it would instead fall through > and be caught by their own body code, presumably giving a more coherent > error anyway. This is a nonzero change in behavior, but it's well within > the scope of behavior changes which normally occur from version to version. > > One alternative to this proposal would be to have only None allowed to act > as a context manager. However, None is not particularly special in this > regard; the logic above applies to any function which might return a Union > type. Furthermore, allowing it for any type would permit the following > construction as well: > > with var1 as expr1, var2 as expr2, ... > .... body ... > > where the common factor between the variables is no longer their need for > a guaranteed cleanup operation, but simply that they are semantically all > tied to a single scope of the code. This improves code clarity, as it > allows the syntax to follow the intent more closely, and also eliminates > one other ugliness. In present Python, the required syntax for the above > would be > > var1 = expr1 > var3 = expr3 > with var2 as expr2, var4 as expr4: > ... body ... > > where the variables in the 'with' statement are those which satisfy the > context manager protocol, and the ones above it are those which do not > satisfy the protocol. The split between the two is entirely tied to a > nonlocal fact about the code, namely the implementation of the return > values of each of the expressions, making it nonobvious which is which. > Worse, if the expressions depend on each other in sequence, this may have > to be broken up into > > var1 = expr1 > with var2 as expr2: > var3 = expr3(var1, var2) > with var4 as expr4(var3, ...): > .... body ... > > This seems to lose on every measure of clarity and maintainability > relative to the single compound 'with' statement above. > > Finally, one may ask if an (effective) default implementation of a > protocol is ever a good idea. "Hidden defaults" are a great way to trigger > surprising behavior, after all. However, in this case I would argue that > the proposed default behavior is sufficiently obvious that there is no > risk. Someone seeing a compound 'with' statement of the above form would > naturally assume that its consequence is (a) to set each varN to the > corresponding exprN, and (b) to execute any scope-initializers tied to > exprN. Likewise, someone would naturally assume that nothing at all happens > at scope exit, which is exactly the behavior of __exit__ being 'return > False'. In fact, this *increases* local code clarity, since the > counter-case -- where the implementation of each defaults (effectively) to > raising an AttributeError -- is nonobvious and so requires that "nonlocal > knowledge" of the code to assemble with statements. > > *Specific implementation proposal: *Actually defining __enter__ and > __exit__ methods for each object would be a lot of overhead for no good > value. Instead, we can easily implement this as a change to the specified > behavior of the 'with' statement, simply by changing the error-handling > behavior in the SETUP_WITH > <https://github.com/python/cpython/blob/d5d9e81ce9a7efc5bc14a5c21398d1ef6f626884/Python/ceval.c#L3119> > and WITH_CLEANUP_START > <https://github.com/python/cpython/blob/d5d9e81ce9a7efc5bc14a5c21398d1ef6f626884/Python/ceval.c#L3119> > cases in ceval.c. If this does proceed to the PEP stage, I'll put together > a changelist, but it's very straightforward. Null values for enter and exit > are no longer errors; if enter is null, then instead of decrementing the > refcount of mgr and calling enter, we leave the mgr refcount alone and push > it onto the stack in place of the result. If exit is null, we simply push > it onto the stack just like we would normally, and ignore it in > WITH_CLEANUP_START. > _______________________________________________ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ >
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
