Re: [Python-ideas] PEP draft: context variables
On Mon, Oct 9, 2017 at 8:37 PM, Koos Zevenhoven wrote: [..] >> Another thing: suppose someone calls >> 'context_var.assign().__enter__()' manually, without calling >> '__exit__()'. You will have unbound growth of the context values >> stack. > > > You can cause unbound growth in PEP 550 too. All you have to do is nest an > unbounded number of generators. You can only nest up to 'sys.get_recursion_limit()' number of generators. With PEP 555 you can do: while True: context_var.assign(42).__enter__() > In PEP 555, nesting generators doesn't do > anything really, unless you actually assign to context arguments in the > generators. Only those who use it will pay. Same for 550. If a generator doesn't set context variables, its LC will be an empty mapping (or NULL if you want to micro-optimize things). Nodes for the chain will come from a freelist. The effective overhead for generators is a couple operations on pointers, and thus visible only in microbenchmarks. > > But seriously, you will always end up in a weird situation if you call an > unbounded number of contextmanager.__enter__() methods without calling > __exit__(). Nothing new about that. But entering a handful of assignment > contexts and leaving them open until a script ends is not the end of the > world. I don't think anyone should do that though. > > >> >> You'll say that it's not how the API is supposed to be used, >> and we say that we want to convert things like decimal and numpy to >> use the new mechanism. That question was also hand-waved by you: >> numpy and decimal will have to come up with new/better APIs to use PEP >> 555. Well, that's just not good enough. > > > What part of my explanation of this are you unhappy with? For instance, the > 12th (I think) email in this thread, which is my response to Nathaniel. > Could you reply to that and tell us your concern? I'm sorry, I'm not going to find some 12th email in some thread. I stated in this thread the following: not being able to use PEP 555 to fix *existing* decimal & numpy APIs is not good enough. And decimal & numpy is only one example, there's tons of code out there that can benefit from its APIs to be fixed to support for async code in Python 3.7. > >> >> >> And the key problem is that you still haven't directly highlighted >> differences in semantics between PEP 550 and PEP 555. This is the >> most annoying part, because almost no one (including me) knows the >> complete answer here. Maybe you know, but you refuse to include that >> in the PEP for some reason. > > > I don't refuse to > . I just haven't prioritized it. But I've probably made the mistake of > mentioning *similarities* between 550 and 555 > . > One major difference is that there is no .set(value) in PEP 555, so one > shouldn't try to map PEP 550 uses directly to PEP 555. This is not a "major difference". You might feel that it is, but it is a simple API design choice. As I illustrated in a few emails before, as long as users can call 'context_var.assign(..).__enter__()' manually, your PEP *does* allow to effectively do ".set(value)". If using a context manager instead of 'set' method is the only difference you can highlight, then why bother writing a PEP? The idea of using context managers to set values is very straightforward and is easy to be incorporated to PEP 550. In fact, it will be added to the next version of the PEP (after discussing it with Guido on the lang summit). > >> > Some kind of >> > chained-lookup-like thing is inevitable if you want the state not to >> > leak >> > though yields out of the generator: >> >> No, it's not "inevitable". In PEP 550 v1, generators captured the >> context when they are created and there was always only one level of >> context. This means that: >> >> 1. Context changes in generators aren't visible to the outside world. >> 2. Changes to the context in the outside world are not visible to >> running generators. > > > Sure, if you make generators completely isolated from the outside world, > then you can avoid chaining-like things too. But that would just sweep it > under the carpet. What do you mean by "just sweep it under the carpet"? Capturing the context at the moment of generators creation is a design choice with some consequences (that I illustrated in my previous email). There are cons and pros of doing that. Yury ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP draft: context variables
On Tue, Oct 10, 2017 at 1:55 AM, Yury Selivanov wrote: > On Mon, Oct 9, 2017 at 4:39 PM, Koos Zevenhoven wrote: > > On Mon, Oct 9, 2017 at 6:24 PM, Guido van Rossum > wrote: > [..] > >> I'm not sure I agree on the usefulness. Certainly a lot of the > complexity > >> of PEP 550 exists just to cater to Nathaniel's desire to influence what > a > >> generator sees via the context of the send()/next() call. I'm still not > sure > >> that's worth it. In 550 v1 there's no need for chained lookups. > > > > > > We do need some sort of chained lookups, though, at least in terms of > > semantics. But it is possible to optimize that away in PEP 555. > > You keep using the "optimize away" terminology. I assume that you > mean that ContextVar.get() will have a cache (so it does in PEP 550 > btw). What else do you plan to "optimize away"? Where's a detailed > implementation spec? What you have in the PEP is still vague and > leaves many important implementation details to the imagination of the > reader. > > I'm hesitant to call it a cache, because the "cache" sort of automatically builds itself. I think I'll need to draw a diagram to explain it. The implementation is somewhat simpler than its explanation. I can go more into detail regarding the implementation, but I feel that semantics is more important at this point. > The fact is that the datastructure choice in PEP 555 is plain weird. > You want to use a sequence of values to represent a mapping. And then > you hand-waved all questions about what will happen in pathological > cases, saying that "we'll have a cache and applications won't have to > many context values anyways". I don't think I've heard of any pathological cases... What do you mean? But your design means that in the worst > case, the uncached path requires you to potentially traverse all > values in the context. It is in fact possible to implement it in a way that this never happens, but the best thing might actually be an implementation, where this *almost* never happens. (Off-topic: It would be kind of cool, if you could do the same thing with MROs, so OOP method access will speed up. But that might be somewhat more difficult to implement, because there are more moving parts there.) > Another thing: suppose someone calls > 'context_var.assign().__enter__()' manually, without calling > '__exit__()'. You will have unbound growth of the context values > stack. You can cause unbound growth in PEP 550 too. All you have to do is nest an unbounded number of generators. In PEP 555, nesting generators doesn't do anything really, unless you actually assign to context arguments in the generators. Only those who use it will pay. But seriously, you will always end up in a weird situation if you call an unbounded number of contextmanager.__enter__() methods without calling __exit__(). Nothing new about that. But entering a handful of assignment contexts and leaving them open until a script ends is not the end of the world. I don't think anyone should do that though. > You'll say that it's not how the API is supposed to be used, > and we say that we want to convert things like decimal and numpy to > use the new mechanism. That question was also hand-waved by you: > numpy and decimal will have to come up with new/better APIs to use PEP > 555. Well, that's just not good enough. > What part of my explanation of this are you unhappy with? For instance, the 12th (I think) email in this thread, which is my response to Nathaniel. Could you reply to that and tell us your concern? > > And the key problem is that you still haven't directly highlighted > differences in semantics between PEP 550 and PEP 555. This is the > most annoying part, because almost no one (including me) knows the > complete answer here. Maybe you know, but you refuse to include that > in the PEP for some reason. > I don't refuse to . I just haven't prioritized it. But I've probably made the mistake of mentioning *similarities* between 550 and 555 . One major difference is that there is no .set(value) in PEP 555, so one shouldn't try to map PEP 550 uses directly to PEP 555. > Some kind of > > chained-lookup-like thing is inevitable if you want the state not to leak > > though yields out of the generator: > > No, it's not "inevitable". In PEP 550 v1, generators captured the > context when they are created and there was always only one level of > context. This means that: > > 1. Context changes in generators aren't visible to the outside world. > 2. Changes to the context in the outside world are not visible to > running generators. > Sure, if you make generators completely isolated from the outside world, then you can avoid chaining-like things too. But that would just sweep it under the carpet. > PEP 550 v1 was the simplest thing possible with a very efficient > implementation. It had the following "issues" that led us to v2+ > semantics of chained lookup: > > 1. Refactoring. > > with some_context
Re: [Python-ideas] PEP draft: context variables
On Mon, Oct 9, 2017 at 4:39 PM, Koos Zevenhoven wrote: > On Mon, Oct 9, 2017 at 6:24 PM, Guido van Rossum wrote: [..] >> I'm not sure I agree on the usefulness. Certainly a lot of the complexity >> of PEP 550 exists just to cater to Nathaniel's desire to influence what a >> generator sees via the context of the send()/next() call. I'm still not sure >> that's worth it. In 550 v1 there's no need for chained lookups. > > > We do need some sort of chained lookups, though, at least in terms of > semantics. But it is possible to optimize that away in PEP 555. You keep using the "optimize away" terminology. I assume that you mean that ContextVar.get() will have a cache (so it does in PEP 550 btw). What else do you plan to "optimize away"? Where's a detailed implementation spec? What you have in the PEP is still vague and leaves many important implementation details to the imagination of the reader. The fact is that the datastructure choice in PEP 555 is plain weird. You want to use a sequence of values to represent a mapping. And then you hand-waved all questions about what will happen in pathological cases, saying that "we'll have a cache and applications won't have to many context values anyways". But your design means that in the worst case, the uncached path requires you to potentially traverse all values in the context. Another thing: suppose someone calls 'context_var.assign().__enter__()' manually, without calling '__exit__()'. You will have unbound growth of the context values stack. You'll say that it's not how the API is supposed to be used, and we say that we want to convert things like decimal and numpy to use the new mechanism. That question was also hand-waved by you: numpy and decimal will have to come up with new/better APIs to use PEP 555. Well, that's just not good enough. And the key problem is that you still haven't directly highlighted differences in semantics between PEP 550 and PEP 555. This is the most annoying part, because almost no one (including me) knows the complete answer here. Maybe you know, but you refuse to include that in the PEP for some reason. > Some kind of > chained-lookup-like thing is inevitable if you want the state not to leak > though yields out of the generator: No, it's not "inevitable". In PEP 550 v1, generators captured the context when they are created and there was always only one level of context. This means that: 1. Context changes in generators aren't visible to the outside world. 2. Changes to the context in the outside world are not visible to running generators. PEP 550 v1 was the simplest thing possible with a very efficient implementation. It had the following "issues" that led us to v2+ semantics of chained lookup: 1. Refactoring. with some_context(): for i in gen(): pass would not be equivalent to: g = gen() with some_context(): for i in g: pass 2. Restricting generators to only see context at the point of their creation feels artificial. We know there are better solutions here (albeit more complex) and we try to see if they are worth it. 3. Nathaniel't use case in Trio. Yury ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP draft: context variables
On Mon, Oct 9, 2017 at 6:24 PM, Guido van Rossum wrote: > On Sun, Oct 8, 2017 at 11:46 PM, Nick Coghlan wrote: > >> On 8 October 2017 at 08:40, Koos Zevenhoven wrote: >> >>> On Sun, Oct 8, 2017 at 12:16 AM, Nathaniel Smith wrote: >>> On Oct 7, 2017 12:20, "Koos Zevenhoven" wrote: Unfortunately, we actually need a third kind of generator semantics, something like this: @contextvars.caller_context def genfunc(): assert cvar.value is the_value yield assert cvar.value is the_value with cvar.assign(the_value): gen = genfunc() next(gen) with cvar.assign(1234567890): try: next(gen) except StopIteration: pass Nick, Yury and I (and Nathaniel, Guido, Jim, ...?) somehow just narrowly missed the reasons for this in discussions related to PEP 550. Perhaps because we had mostly been looking at it from an async angle. That's certainly a semantics that one can write down (and it's what the very first version of PEP 550 did), >>> >>> I do remember Yury mentioning that the first draft of PEP 550 captured >>> something when the generator function was called. I think I started reading >>> the discussions after that had already been removed, so I don't know >>> exactly what it was. But I doubt that it was *exactly* the above, because >>> PEP 550 uses set and get operations instead of "assignment contexts" like >>> PEP 555 (this one) does. >>> >> >> We didn't forget it, we just don't think it's very useful. >> > > I'm not sure I agree on the usefulness. Certainly a lot of the complexity > of PEP 550 exists just to cater to Nathaniel's desire to influence what a > generator sees via the context of the send()/next() call. I'm still not > sure that's worth it. In 550 v1 there's no need for chained lookups. > We do need some sort of chained lookups, though, at least in terms of semantics. But it is possible to optimize that away in PEP 555. Some kind of chained-lookup-like thing is inevitable if you want the state not to leak though yields out of the generator: with cvar.assign(a_value): # don't leak `a_value` to outer context yield some_stuff() ––Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP draft: context variables
On Mon, Oct 9, 2017 at 9:46 AM, Nick Coghlan wrote: > On 8 October 2017 at 08:40, Koos Zevenhoven wrote: > >> On Sun, Oct 8, 2017 at 12:16 AM, Nathaniel Smith wrote: >> >>> On Oct 7, 2017 12:20, "Koos Zevenhoven" wrote: >>> >>> >>> Unfortunately, we actually need a third kind of generator semantics, >>> something like this: >>> >>> @contextvars.caller_context >>> def genfunc(): >>> assert cvar.value is the_value >>> yield >>> assert cvar.value is the_value >>> >>> with cvar.assign(the_value): >>> gen = genfunc() >>> >>> next(gen) >>> >>> with cvar.assign(1234567890): >>> try: >>> next(gen) >>> except StopIteration: >>> pass >>> >>> Nick, Yury and I (and Nathaniel, Guido, Jim, ...?) somehow just narrowly >>> missed the reasons for this in discussions related to PEP 550. Perhaps >>> because we had mostly been looking at it from an async angle. >>> >>> >>> That's certainly a semantics that one can write down (and it's what the >>> very first version of PEP 550 did), >>> >> >> I do remember Yury mentioning that the first draft of PEP 550 captured >> something when the generator function was called. I think I started reading >> the discussions after that had already been removed, so I don't know >> exactly what it was. But I doubt that it was *exactly* the above, because >> PEP 550 uses set and get operations instead of "assignment contexts" like >> PEP 555 (this one) does. >> > > We didn't forget it, we just don't think it's very useful. > Yeah, I'm not surprised you remember that :). But while none of us saw a good enough reason for it at that moment, I have come to think we absolutely need it. We need both the forest and the trees. Sure, if you think of next() as being a simple function call that does something that involves state, then you might want the other semantics (with PEP 555, that situation would look like): def do_stuff_with_side_effects(): with cvar.assign(value): return next(global_gen_containing_state) Now stuff happens within next(..), and whatever happens in next(..) is expected to see the cvar assignment. However, probably much more often, one just thinks of next(..) as "getting the next value", although some computations happen under the hood that one doesn't need to care about. As we all know, in the real world, the use case is usually just to generate the Fibonacci sequence ;). And when you call fibonacci(), the whole sequence should already be determined. You just evaluate the sequence lazily by calling next() each time you want a new number. It may not even be obvious when the computations are made: fib_cache = [0, 1] def fibonacci(): for i in itertools.counter(): if i < len(fib_cache): yield fib_cache[i] else: # not calculated before new = sum(fib_cache[-2:]) fib_cache.append(new) yield new # (function above is thread-unsafe, for clarity) (Important:) So in *any* situation, where you want the outer context to affect the stuff inside the generator through next(), like in the `do_stuff_with_side_effects` example, *the author of the generator function needs to know about it*. And then it is ok to require that the author uses a decorator on the generator function. But when you just generate a pre-determined set of numbers (like fibonacci), the implementation of the generator function should not matter, but if the outer context leaks in through next(..), the internal implementation does matters, and the abstraction is leaky. I don't want to give the leaky things by default. ––Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP draft: context variables
On Sun, Oct 8, 2017 at 11:46 PM, Nick Coghlan wrote: > On 8 October 2017 at 08:40, Koos Zevenhoven wrote: > >> On Sun, Oct 8, 2017 at 12:16 AM, Nathaniel Smith wrote: >> >>> On Oct 7, 2017 12:20, "Koos Zevenhoven" wrote: >>> >>> >>> Unfortunately, we actually need a third kind of generator semantics, >>> something like this: >>> >>> @contextvars.caller_context >>> def genfunc(): >>> assert cvar.value is the_value >>> yield >>> assert cvar.value is the_value >>> >>> with cvar.assign(the_value): >>> gen = genfunc() >>> >>> next(gen) >>> >>> with cvar.assign(1234567890): >>> try: >>> next(gen) >>> except StopIteration: >>> pass >>> >>> Nick, Yury and I (and Nathaniel, Guido, Jim, ...?) somehow just narrowly >>> missed the reasons for this in discussions related to PEP 550. Perhaps >>> because we had mostly been looking at it from an async angle. >>> >>> >>> That's certainly a semantics that one can write down (and it's what the >>> very first version of PEP 550 did), >>> >> >> I do remember Yury mentioning that the first draft of PEP 550 captured >> something when the generator function was called. I think I started reading >> the discussions after that had already been removed, so I don't know >> exactly what it was. But I doubt that it was *exactly* the above, because >> PEP 550 uses set and get operations instead of "assignment contexts" like >> PEP 555 (this one) does. >> > > We didn't forget it, we just don't think it's very useful. > I'm not sure I agree on the usefulness. Certainly a lot of the complexity of PEP 550 exists just to cater to Nathaniel's desire to influence what a generator sees via the context of the send()/next() call. I'm still not sure that's worth it. In 550 v1 there's no need for chained lookups. -- --Guido van Rossum (python.org/~guido) ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Fwd: Fwd: A PEP to define basical metric which allows to guarantee minimal code quality
Hi, After some reflexion on this full thread, with all your arguments and discussion with my team, i have finally a better understanding on PEP finality. I saw that PEP 8 & 20 i used as example are "specials" PEP. So i let my idea here, and eventually, as previously suggested, i'll contact PYCQA. Thank you very much everybody for your help and your attention :) Le mardi 26 septembre 2017 04:54:45 UTC+2, Nick Coghlan a écrit : > > Forwarding my reply, since Google Groups still can't get the Reply-To > headers for the mailing list right, and we still don't know how to > categorically prohibit posting from there. > > -- Forwarded message -- > From: Nick Coghlan > > Date: 26 September 2017 at 12:51 > Subject: Re: [Python-ideas] Fwd: A PEP to define basical metric which > allows to guarantee minimal code quality > To: Alexandre GALODE > > Cc: python-ideas > > > > On 25 September 2017 at 21:49, > > wrote: > > Hi, > > > > Sorry from being late, i was in professional trip to Pycon FR. > > > > I see that the subject is divising advises. > > > > Reading responses, i have impression that my proposal has been saw as > > mandatory, that i don't want of course. As previously said, i see this > "PEP" > > as an informational PEP. So it's a guideline, not a mandatory. Each > > developer will have right to ignore it, as each developer can choose to > > ignore PEP8 or PEP20. > > > > Perfect solution does not exist, i know it, but i think this "PEP" > could, > > partially, be a good guideline. > > Your question is essentially "Are python-dev prepared to offer generic > code quality assessment advice to Python developers?" > > The answer is "No, we're not". It's not our role, and it's not a role > we're the least bit interested in taking on. Just because we're the > ones making the software equivalent of hammers and saws doesn't mean > we're also the ones that should be drafting or signing off on people's > building codes :) > > Python's use cases are too broad, and what's appropriate for my ad hoc > script to download desktop wallpaper backgrounds, isn't going to be > what's appropriate for writing an Ansible module, which in turn isn't > going to be the same as what's appropriate for writing a highly > scalable web service or a complex data analysis job. > > So the question of "What does 'good enough for my purposes' actually > mean?" is something for end users to tackle for themselves, either > individually or collaboratively, without seeking specific language > designer endorsement of their chosen criteria. > > However, as mentioned earlier in the thread, it would be *entirely* > appropriate for the folks participating in PyCQA to decide to either > take on this work themselves, or else endorse somebody else taking it > on. I'd see such an effort as being similar to the way that > packaging.python.org originally started as an independent PyPA project > hosted at python-packaging-user-guide.readthedocs.io, with a fair bit > of content already being added before we later requested and received > the python.org subdomain. > > Cheers, > Nick. > > -- > Nick Coghlan | ncog...@gmail.com| Brisbane, > Australia > > > -- > Nick Coghlan | ncog...@gmail.com| Brisbane, > Australia > ___ > Python-ideas mailing list > python...@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
