Re: [Python-Dev] Discussion about the proposed ignore_modules argument for traceback functions
On Jan 3, 2018 18:38, "Dmitry Kazakov" wrote: Hello! I'd like to draw some attention to the feature(s) proposed in the issue 31299. https://bugs.python.org/issue31299 It's a dependency of the other issue, it still needs discussion, and it hasn't received any comments from committers since last September. Personally, I think that a general traceback cleaning facility proposed by Antoine should be accompanied by a similar non-destructive feature in a traceback module. If it's decided that the latter makes sense implementing, I'm willing to revive (and update) the PR I've closed earlier in time for the approaching Python 3.7 feature code cutoff. Regarding the ability to mutate a traceback to add/remove frames, there's a PR here, which I think is enough to do what Antoine was talking about: https://github.com/python/cpython/pull/4793 (It's been sitting for 24 days, maybe someone could review it?) If you want a general way to mark certain frames in tracebacks as being hidden, then I think the big question is what the actual API for marking them should look like. Being able to mark a whole module as never showing up in tracebacks is pretty crude -- should it be per-function? What would it look like, a decorator? That's probably difficult to implement without first implementing bpo-12857 (which would be a good idea anyway!). If an invisible function calls another function that's neither marked as visible or invisible, should the default be that it's visible, or that it inherits its parent's visibility? -n ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 (contextvars) idea: really implement the current context
2018-01-04 3:15 GMT+01:00 Nathaniel Smith : > The problem with such an API is that it doesn't work (or at the very > least creates a lot of complications) in a potential future PEP 550 > world, (...) Hum, it's annoying that many design choices of the PEP 567 are motivated by the hypothetical acceptance of the PEP 550, whereas this PEP is not scheduled for Python 3.7. What if the PEP 550 is rejected because it's too complicated? We might have a nicer PEP 567 API without the big shadow of the PEP 550. Is it possible to run a generator in a context explicitly using PEP 567 Context.run() API? -- I never looked seriously at the PEP 550, because I'm scared by its length and its complexity :-) While I easily see use cases for asyncio web frameworks with the PEP 567, it's harder for me to see the use cases of the PEP 550. I never had to keep a context in a generator. I guess that developers who have to do that learnt how to workaround this Python limitation, maybe passing manually the context to the generator and reapply it at each step? Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Discussion about the proposed ignore_modules argument for traceback functions
Hello! I'd like to draw some attention to the feature(s) proposed in the issue 31299. https://bugs.python.org/issue31299 It's a dependency of the other issue, it still needs discussion, and it hasn't received any comments from committers since last September. Personally, I think that a general traceback cleaning facility proposed by Antoine should be accompanied by a similar non-destructive feature in a traceback module. If it's decided that the latter makes sense implementing, I'm willing to revive (and update) the PR I've closed earlier in time for the approaching Python 3.7 feature code cutoff. Cheers ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 v2
Guido van Rossum wrote: contextvars.copy_context().run(func, ) If contexts are immutable, why is there something called copy_context? -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 v2
Guido van Rossum wrote: There needs to be some way to introspect a Context, for debugging. It could have a read-only introspection interface without having to be a full Mapping. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 (contextvars) idea: really implement the current context
On Wed, Jan 3, 2018 at 5:42 PM, Victor Stinner wrote: > Hi, > > It seems like many people, including myself, are confused by the lack > of concrete current context in the PEP 567 (contextvars). But it isn't > difficult to implement the current context (I implemented it, see > below). The problem with such an API is that it doesn't work (or at the very least creates a lot of complications) in a potential future PEP 550 world, where the "current context" becomes something like a ChainMap-of-Contexts instead of just the last Context that had run() called on it. This isn't a big problem for contextvars.get_context(), which returns a snapshot of the current context -- in a PEP 550 world it would return a snapshot of the current "effective" (flattened) context. Maybe it would help a little to rename get_context() to something like snapshot_context()? -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 (contextvars) idea: really implement the current context
Victor:
> I modified Context mapping API to use the context variables from the
> current thread state if it's the current thread state.
Oops, I mean: "if it's the current context".
Nathaniel:
"""
- BUT it doesn't allow mutation through the MutableMapping interface;
instead, the only way to mutate it is by calling Context.run and then
ContextVar.set(). Funneling all 'set' operations through a single
place makes it easier to do clever caching tricks, and it lets us
avoid dealing with operations that we don't want here (like 'del')
just because they happen to be in the MutableMapping interface.
"""
If a context knows that it's the current context, Context.set() can
delegate the change to ContextVar.set(), since Context access directly
the thread local storage in this case (with my suggested changes), and
so the optimization is kept.
If the context is not the current context, the cache doesn't have to
be invalidated. Moreover, PyContext_Enter() and PyContext_Exit()
already increases ts->contextvars_stack_ver which indirectly
invalidates cached values.
Nathaniel: Would you be ok to implement the MutableMapping API if the
optimization is kept? "del context[var]" would raise a
TypeError('Context' object doesn't support item deletion) exception,
as it does currently.
Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 567 (contextvars) idea: really implement the current context
Hi,
It seems like many people, including myself, are confused by the lack
of concrete current context in the PEP 567 (contextvars). But it isn't
difficult to implement the current context (I implemented it, see
below). It might have a *minor* impact on performance, but Context
mapping API is supposed to only be used for introspection according to
Yury.
With contextvars.get_context(), it becomes possible to introspect the
current context, rather than a copy, and it becomes even more obvious
than a context is mutable ;-)
vstinner@apu$ ./python
>>> import contextvars
>>> ctx=contextvars.get_context()
>>> name=contextvars.ContextVar('name', default='victor')
>>> print(list(ctx.items()))
[]
>>> name.set('yury')
>>> print(list(ctx.items()))
[(, 'yury')]
With my changes, the running context remains up to date in
Context.run(). Example:
---
import contextvars
name = contextvars.ContextVar('name', default='victor')
def func():
name.set('yury')
print(f"context[name]: {context.get(name)}")
print(f"name in context: {name in context}")
context = contextvars.copy_context()
context.run(func)
---
Output:
---
context[name]: yury
name in context: True
---
Compare it to the output without my changes:
---
context[name]: None
name in context: False
---
If we have contextvars.get_context(), maybe contextvars.copy_context()
can be removed and add a new Context.copy() method instead:
new_context = contextvars.get_context().copy()
***
I implemented contextvars.get_context() which returns the current context:
https://github.com/vstinner/cpython/commit/1e5ee71c15e2b1387c888d6eca2b08ef14595130
from https://github.com/vstinner/cpython/commits/current_context
I added a context thread local storage (TLS):
class PyThreadState:
context: Context # can be None
context_data: _ContextData
I modified Context mapping API to use the context variables from the
current thread state if it's the current thread state.
PyContext_Enter() now not only sets PyThreadState.context_data, but
also PyThreadState.context to itself.
Pseudo-code for Context.get():
---
class Context(collections.abc.Mapping):
def __init__(self):
self._data = _ContextData()
def _get_data(self):
ts : PyThreadState = PyThreadState_Get()
if ts.context is self:
return ts.context_data
else:
return self._data
def get(self, var): # FIXME: implement default
data = self._get_data()
return data.get(var)
---
And Context.run() is modified to set also the context TLS:
---
def run(self, callable, *args, **kwargs):
ts : PyThreadState = PyThreadState_Get()
saved_context : Optional[Context] = ts.context # NEW
saved_data : _ContextData = ts.context_data
try:
ts.context_data = self._data
return callable(*args, **kwargs)
finally:
self._data = ts.context_data
ts.context = saved_context # NEW
ts.context_data = saved_data
---
Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 v2
On Wed, Jan 3, 2018 at 3:44 PM, Victor Stinner wrote: > Ok, I finally got access to a computer and I was able to test the PEP > 567 implementation: see my code snippet below. > > The behaviour is more tricky than what I expected. While running > context.run(), the context object is out of sync of the "current > context". It's only synchronized again at run() exit. So > ContextVar.set() doesn't immediately modifies the "current context" > object (set by Context.run()). To me this sounds like a oversight (= bug), not intended behavior. At the conceptual level, I think what we want is: - Context is a mutable object representing a mapping - BUT it doesn't allow mutation through the MutableMapping interface; instead, the only way to mutate it is by calling Context.run and then ContextVar.set(). Funneling all 'set' operations through a single place makes it easier to do clever caching tricks, and it lets us avoid dealing with operations that we don't want here (like 'del') just because they happen to be in the MutableMapping interface. - OTOH we do implement the (read-only) Mapping interface because there's no harm in it and it's probably useful for debuggers. (Note that I didn't say anything about HAMTs here, because that's orthogonal implementation detail. It would make perfect sense to have Context be an opaque wrapper around a regular dict; it would just give different performance trade-offs.) -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 'continue'/'break'/'return' inside 'finally' clause
On 2018-01-03, Guido van Rossum wrote: > I'm sorry, I don't think more research can convince me either way. > I want all three of return/break/continue to work inside finally > clauses, despite there being few use cases. That's fine. The history of 'continue' inside 'finally' is interesting. The restriction dates back to at least when Jeremy committed the AST-based compiler (I have fond memories of hacking on it with Armin Rigo and others at a Python core sprint). Going further back, I looked at 1.5.2 and there is the comment in compile.c: TO DO: ... XXX Allow 'continue' inside try-finally So if we allow 'continue' we will be knocking off a nearly 20 year old todo item. ;-) For giggles, I unpacked a Python 0.9.1 tarball. The source code is all under 'src' in that version. There doesn't seem to be a restriction on 'continue' but only because the grammar doesn't include it! Without doing more research, I think the restriction could be as old as the 'continue' keyword. BTW, the bytecode structure for try/except shown in the compile.c comments is very simlar to what is currently generated. It is quite remarkable how well your initial design and implementation have stood the test of time. Thank you for making it open source. Regards, Neil ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 v2
2018-01-04 0:44 GMT+01:00 Victor Stinner : > The behaviour is more tricky than what I expected. While running > context.run(), the context object is out of sync of the "current > context". It's only synchronized again at run() exit. So > ContextVar.set() doesn't immediately modifies the "current context" > object (set by Context.run()). A better description of Context.run() behaviour would be: "Create a copy of the context and sets it as the current context. Once the function completes, updates the context from the copy." This description explains why run() doesn't immediately updates the context object. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible bug in base64.decode: linebreaks are not ignored
I opened https://bugs.python.org/issue32491 so this wouldn't be lost under
the assumption that you'll eventually resolve your bugtracker account
situation. :)
I don't know what the correct behavior *should* be but agree that it seems
odd for decode to behave different than decodebytes.
-gps
On Wed, Jan 3, 2018 at 8:00 AM Oleg Sivokon wrote:
> Hello,
>
> I've tried reading various RFCs around Base64 encoding, but I couldn't
> make the ends meet. Yet there is an inconsistency between
> base64.decodebytes() and base64.decode() in that how they handle linebreaks
> that were used to collate the encoded text. Below is an example of what
> I'm talking about:
>
> >>> import base64
> >>> foo = base64.encodebytes(b'123456789')
> >>> foo
> b'MTIzNDU2Nzg5\n'
> >>> foo = b'MTIzND\n' + b'U2Nzg5\n'
> >>> foo
> b'MTIzND\nU2Nzg5\n'
> >>> base64.decodebytes(foo)
> b'123456789'
> >>> from io import BytesIO
> >>> bytes_in = BytesIO(foo)
> >>> bytes_out = BytesIO()
> >>> bytes_in.seek(0)
> 0
> >>> base64.decode(bytes_in, bytes_out)
> Traceback (most recent call last):
> File "", line 1, in
> File "/somewhere/lib/python3.6/base64.py", line 512, in decode
> s = binascii.a2b_base64(line)
> binascii.Error: Incorrect padding
> >>> bytes_in = BytesIO(base64.encodebytes(b'123456789'))
> >>> bytes_in.seek(0)
> 0
> >>> base64.decode(bytes_in, bytes_out)
> >>> bytes_out.getvalue()
> b'123456789'
>
> Obviously, I'd expect encodebytes() and encode both to either accept or to
> reject the same input.
>
> Thanks.
>
> Oleg
>
> PS. I couldn't register to the bug-tracker (never received an email
> confirmation, not even in a spam folder), this is why I'm sending it here.
> This communication and all information contained in or attached to it is
> confidential, intended solely for the addressee, may be legally privileged
> and is the intellectual property of one of the companies of NEX Group plc
> ("NEX") or third parties. If you are not the intended addressee or receive
> this message in error, please immediately delete all copies of it and
> notify the sender. We have taken precautions to minimise the risk of
> transmitting software viruses, but we advise you to carry out your own
> virus checks on any attachments. We do not accept liability for any loss or
> damage caused by software viruses. NEX reserves the right to monitor all
> communications. We do not accept any legal responsibility for the content
> of communications, and no communication shall be considered legally
> binding. Furthermore, if the content of this communication is personal or
> unconnected with our business, we accept no liability or responsibility for
> it. NEX Group plc is a public limited company regi
> stered in England and Wales under number 10013770 and certain of its
> affiliates are authorised and regulated by regulatory authorities. For
> further regulatory information please see www.NEX.com.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 v2
That was actually Yury's original design, but I insisted on a Mapping so
you can introspect it. (There needs to be some way to introspect a Context,
for debugging.) Again, more later. :-(
On Wed, Jan 3, 2018 at 4:44 PM, Victor Stinner
wrote:
> Ok, I finally got access to a computer and I was able to test the PEP
> 567 implementation: see my code snippet below.
>
> The behaviour is more tricky than what I expected. While running
> context.run(), the context object is out of sync of the "current
> context". It's only synchronized again at run() exit. So
> ContextVar.set() doesn't immediately modifies the "current context"
> object (set by Context.run()).
>
> Ok, and now something completely different! What if Context looses its
> whole mapping API and becomes a "blackbox" object only with a run()
> method and no attribute? It can be documented as an object mapping
> context variables to their values, but don't give access to these
> values directly. It would avoid to have explain the weird run()
> behaviour (context out of sync). It would avoid to have to decide if
> it's a mutable or immutable mapping. It would avoid to have to explain
> the internal O(1) copy using HAMT.
>
> Code testing Context.run():
> ---
> import contextvars
>
> name = contextvars.ContextVar('name', default='name')
>
> def assertNotIn(var, context):
> try:
> context[var]
> except LookupError:
> pass
> else:
> raise AssertionError("name is set is context")
>
>
> def func1():
> name.set('yury')
> assert name.get() == 'yury'
> assertNotIn(name, context)
>
> def func2():
> assert name.get() == 'yury'
> assert context[name] == 'yury'
>
> context = contextvars.copy_context()
>
> assert name.get() == 'name'
> assertNotIn(name, context)
>
> context.run(func1)
>
> assert name.get() == 'name'
> assert context[name] == 'yury'
>
> context.run(func2)
>
> assert name.get() == 'name'
> assert context[name] == 'yury'
> ---
>
> Victor
>
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 v2
Oh, dang, I forgot about this. ContextVar.set() modifies the current Context in-place using a private API. In the PEP, asyncio copies the Context once and then calls run() repeatedly (for each _step call). So run() isn't stateless, it just saves and restores the notion of the current context (in the thread state). I don't have time right now to respond in more detail, sorry for shedding darkness. :-( Hopefully I'll have time Friday. On Wed, Jan 3, 2018 at 4:25 PM, Victor Stinner wrote: > 2018-01-03 23:01 GMT+01:00 Guido van Rossum : > > Heh, you're right, I forgot about that. It should be more like this: > > > > def run(self, func, *args, **kwds): > > old = _get_current_context() > > _set_current_context(self) # <--- changed line > > try: > > return func(*args, **kwds) > > finally: > > _set_current_context(old) > > > > This version, like the PEP, assumes that the Context object is truly > > immutable (not just in name) and that you should call it like this: > > > > contextvars.copy_context().run(func, ) > > I don't see how asyncio would use Context.run() to keep the state > (variables values) between callbacks and tasks, if run() is > "stateless": forgets everything at exit. > > I asked if it would be possible to modify run() to return a new > context object with the new state, but Yury confirmed that it's not > doable: > > Yury: > > [Context.run()] can't return a new context because the callable you're > running can raise an exception. In which case you'd lose modifications > prior to the error. > > Guido: > > Yury strongly favors an immutable Context, and that's what his reference > implementation has (https://github.com/python/cpython/pull/5027). His > reasoning is that in the future we *might* want to support automatic > context management for generators by default (like described in his > original PEP 550), and then it's essential to use the immutable version so > that "copying" the context when a generator is created or resumed is super > fast (and in particular O(1)). > > To get acceptable performances, PEP 550 and 567 require O(1) cost when > copying a context, since the API requires to copy contexts frequently > (in asyncio, each task has its own private context, creating a task > copies the current context). Yury proposed to use "Hash Array Mapped > Tries (HAMT)" to get O(1) copy. > > Each ContextVar.set() change creates a *new* HAMT. Extract of the PEP 567: > --- > def set(self, value): > ts : PyThreadState = PyThreadState_Get() > data : _ContextData = ts.context_data > > try: > old_value = data.get(self) > except KeyError: > old_value = Token.MISSING > > ts.context_data = data.set(self, value) > return Token(self, old_value) > --- > > The link between ContextVar, Context and HAMT (called "context data" > in the PEP 567) is non obvious: > > * ContextVar.set() creates a new HAMT from > PyThreadState_Get().context_data and writes the new one into > PyThreadState_Get().context_data -- technically, it works on a thread > local storage (TLS) > * Context.run() doesn't set the "current context": in practice, it > sets its internal "context data" as the current context data, and then > save the *new* context data in its own context data > > PEP 567: > --- > def run(self, callable, *args, **kwargs): > ts : PyThreadState = PyThreadState_Get() > saved_data : _ContextData = ts.context_data > > try: > ts.context_data = self._data > return callable(*args, **kwargs) > finally: > self._data = ts.context_data > ts.context_data = saved_data > --- > > The main key of the PEP 567 implementation is that there is no > "current context" in practice. There is only a private *current* > context data. > > Not having get_current_contex() allows the trick of context data > handled by a TLS. Otherwise, I'm not sure that it would be possible to > synchronize a Context object with a TLS variable. > > From the user point of view, Context.run() does modify the context. > After the call, variables values changed. A second run() call gives > you the updated context. > > I don't think that a mutable context would have an impact in > performance, since copying "context data" will still have a cost of > O(1). IMHO it's just a matter of taste for the API. > > Or maybe I missed something. > > Victor > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 v2
Ok, I finally got access to a computer and I was able to test the PEP
567 implementation: see my code snippet below.
The behaviour is more tricky than what I expected. While running
context.run(), the context object is out of sync of the "current
context". It's only synchronized again at run() exit. So
ContextVar.set() doesn't immediately modifies the "current context"
object (set by Context.run()).
Ok, and now something completely different! What if Context looses its
whole mapping API and becomes a "blackbox" object only with a run()
method and no attribute? It can be documented as an object mapping
context variables to their values, but don't give access to these
values directly. It would avoid to have explain the weird run()
behaviour (context out of sync). It would avoid to have to decide if
it's a mutable or immutable mapping. It would avoid to have to explain
the internal O(1) copy using HAMT.
Code testing Context.run():
---
import contextvars
name = contextvars.ContextVar('name', default='name')
def assertNotIn(var, context):
try:
context[var]
except LookupError:
pass
else:
raise AssertionError("name is set is context")
def func1():
name.set('yury')
assert name.get() == 'yury'
assertNotIn(name, context)
def func2():
assert name.get() == 'yury'
assert context[name] == 'yury'
context = contextvars.copy_context()
assert name.get() == 'name'
assertNotIn(name, context)
context.run(func1)
assert name.get() == 'name'
assert context[name] == 'yury'
context.run(func2)
assert name.get() == 'name'
assert context[name] == 'yury'
---
Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 v2
2018-01-03 23:01 GMT+01:00 Guido van Rossum : > Heh, you're right, I forgot about that. It should be more like this: > > def run(self, func, *args, **kwds): > old = _get_current_context() > _set_current_context(self) # <--- changed line > try: > return func(*args, **kwds) > finally: > _set_current_context(old) > > This version, like the PEP, assumes that the Context object is truly > immutable (not just in name) and that you should call it like this: > > contextvars.copy_context().run(func, ) I don't see how asyncio would use Context.run() to keep the state (variables values) between callbacks and tasks, if run() is "stateless": forgets everything at exit. I asked if it would be possible to modify run() to return a new context object with the new state, but Yury confirmed that it's not doable: Yury: > [Context.run()] can't return a new context because the callable you're > running can raise an exception. In which case you'd lose modifications prior > to the error. Guido: > Yury strongly favors an immutable Context, and that's what his reference > implementation has (https://github.com/python/cpython/pull/5027). His > reasoning is that in the future we *might* want to support automatic context > management for generators by default (like described in his original PEP > 550), and then it's essential to use the immutable version so that "copying" > the context when a generator is created or resumed is super fast (and in > particular O(1)). To get acceptable performances, PEP 550 and 567 require O(1) cost when copying a context, since the API requires to copy contexts frequently (in asyncio, each task has its own private context, creating a task copies the current context). Yury proposed to use "Hash Array Mapped Tries (HAMT)" to get O(1) copy. Each ContextVar.set() change creates a *new* HAMT. Extract of the PEP 567: --- def set(self, value): ts : PyThreadState = PyThreadState_Get() data : _ContextData = ts.context_data try: old_value = data.get(self) except KeyError: old_value = Token.MISSING ts.context_data = data.set(self, value) return Token(self, old_value) --- The link between ContextVar, Context and HAMT (called "context data" in the PEP 567) is non obvious: * ContextVar.set() creates a new HAMT from PyThreadState_Get().context_data and writes the new one into PyThreadState_Get().context_data -- technically, it works on a thread local storage (TLS) * Context.run() doesn't set the "current context": in practice, it sets its internal "context data" as the current context data, and then save the *new* context data in its own context data PEP 567: --- def run(self, callable, *args, **kwargs): ts : PyThreadState = PyThreadState_Get() saved_data : _ContextData = ts.context_data try: ts.context_data = self._data return callable(*args, **kwargs) finally: self._data = ts.context_data ts.context_data = saved_data --- The main key of the PEP 567 implementation is that there is no "current context" in practice. There is only a private *current* context data. Not having get_current_contex() allows the trick of context data handled by a TLS. Otherwise, I'm not sure that it would be possible to synchronize a Context object with a TLS variable. >From the user point of view, Context.run() does modify the context. After the call, variables values changed. A second run() call gives you the updated context. I don't think that a mutable context would have an impact in performance, since copying "context data" will still have a cost of O(1). IMHO it's just a matter of taste for the API. Or maybe I missed something. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 'continue'/'break'/'return' inside 'finally' clause
I'm sorry, I don't think more research can convince me either way. I want all three of return/break/continue to work inside finally clauses, despite there being few use cases. On Wed, Jan 3, 2018 at 2:30 PM, Neil Schemenauer wrote: > On 2018-01-03, Serhiy Storchaka wrote: > > I haven't found 'finally' clauses in > > https://github.com/gevent/gevent/blob/master/src/gevent/ > libev/corecffi.py. > > Perhaps this code was changed in recent versions. > > Yes, I was looking at was git revision bcf4f65e. I reran my AST > checker and found this: > > ./src/gevent/_ffi/loop.py: 181: return inside finally > > > In any case we now know that this combination is occurred (but > > very rarely) in the wild. > > Looks like it. If we do want to seriously consider changing the > grammar, I will download more packages of PyPI and check them. > > BTW, ./src/gevent/threadpool.py doesn't compile with 3.7 because it > uses 'async' as a variable name. So either they didn't notice the > deprecation warnings or they didn't care to update their code. > > Regards, > > Neil > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 v2
On Wed, Jan 3, 2018 at 12:32 PM, Glenn Linderman wrote: > On 1/3/2018 11:16 AM, Guido van Rossum wrote: > > Maybe I should clarify again what run() does. Here's how I think of it in > pseudo code: > > def run(self, func, *args, **kwds): > old = _get_current_context() > new = old.copy() > _set_current_context(new) > try: > return func(*args, **kwds) > finally: > _set_current_context(old) > > > > I find it interesting that self isn't used in the above pseudo-code. I > thought that Context.run() would run the function in the "context" of self, > not in the context of a copy of "current context". > > > Heh, you're right, I forgot about that. It should be more like this: def run(self, func, *args, **kwds): old = _get_current_context() _set_current_context(self) # <--- changed line try: return func(*args, **kwds) finally: _set_current_context(old) This version, like the PEP, assumes that the Context object is truly immutable (not just in name) and that you should call it like this: contextvars.copy_context().run(func, ) The PEP definitely needs to be clearer about this -- right now one has to skip back and forth between the specification and the implementation (and sometimes the introduction) to figure out how things really work. Help is wanted! -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 'continue'/'break'/'return' inside 'finally' clause
On 2018-01-03, Serhiy Storchaka wrote: > I haven't found 'finally' clauses in > https://github.com/gevent/gevent/blob/master/src/gevent/libev/corecffi.py. > Perhaps this code was changed in recent versions. Yes, I was looking at was git revision bcf4f65e. I reran my AST checker and found this: ./src/gevent/_ffi/loop.py: 181: return inside finally > In any case we now know that this combination is occurred (but > very rarely) in the wild. Looks like it. If we do want to seriously consider changing the grammar, I will download more packages of PyPI and check them. BTW, ./src/gevent/threadpool.py doesn't compile with 3.7 because it uses 'async' as a variable name. So either they didn't notice the deprecation warnings or they didn't care to update their code. Regards, Neil ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 v2
On 1/3/2018 11:16 AM, Guido van Rossum wrote: Maybe I should clarify again what run() does. Here's how I think of it in pseudo code: def run(self, func, *args, **kwds): old = _get_current_context() new = old.copy() _set_current_context(new) try: return func(*args, **kwds) finally: _set_current_context(old) I find it interesting that self isn't used in the above pseudo-code. I thought that Context.run() would run the function in the "context" of self, not in the context of a copy of "current context". ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 'continue'/'break'/'return' inside 'finally' clause
02.01.18 22:31, Neil Schemenauer пише: Serhiy Storchaka wrote: Currently 'break' and 'return' are never used inside 'finally' clause in the stdlib. See the _recv_bytes() function: Lib/multiprocessing/connection.py: 316 Thank you Neil! I missed this case because ran only fast tests, without enabling network tests. I would want to see a third-party code that uses them. These are the only ones I found so far: ../gevent/src/gevent/libev/corecffi.py: 147 I haven't found 'finally' clauses in https://github.com/gevent/gevent/blob/master/src/gevent/libev/corecffi.py. Perhaps this code was changed in recent versions. In any case we now know that this combination is occurred (but very rarely) in the wild. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 v2
I think decimal is a bit of a red herring, because the entire decimal context is stored as a single thread-local variable (_local.__decimal_context__ in _pydecimal.py, where _local is a global threading.local instance; using "___DECIMAL_CTX__" as a key in the C-level dict of thread-locals in the C version). Nevertheless there is the issue of whether it's better to make contextvars.Context a MutableMapping or to make it an (immutable) Mapping. >From the POV of explaining the implementation, a MutableMapping is simpler. E.g. ContextVar.set() just does `get_context()[self] = value`, and ContextVar.get() is `return get_context()[self]` (some details left out relating to defaults). Token and reset() are still useful because they make it simpler to write a context manager that restores the previous value, regardless of whether a value was set or not. Context.run() makes a copy of the current context, sets that as the current context, runs the function, and then restores the previous context (the one that it copied). With a truly immutable Context offering only the (immutable) Mapping interface (plus an internal API that returns a new Context that has a different value for one key), ContextVar.set() is a bit more complicated because it has to use set_context() (actually an internal thing that updates the current context in the thread state) and similar for ContextVar.reset(token). (An alternative design is possible where a Context is an immutable-looking wrapper around a plain dict, with private APIs that mutate that dict, but apart from having different invariants about the identities of Context objects it works out about the same from a user's POV.) Anyway, the differences between these are user-visible so we can't make this an implementation detail: We have to choose. Should Context be a MutableMapping or not? Yury strongly favors an immutable Context, and that's what his reference implementation has (https://github.com/python/cpython/pull/5027). His reasoning is that in the future we *might* want to support automatic context management for generators by default (like described in his original PEP 550), and then it's essential to use the immutable version so that "copying" the context when a generator is created or resumed is super fast (and in particular O(1)). I am personally not sure that we'll ever need it but at the same time I'm also not confident that we won't, so I give Yury the benefit of the doubt here -- after all he has spent an enormous amount of time thinking this design through so I value his intuition. In addition I agree that for most users the basic interface will be ContextVar, not Context, and the needs of framework authors are easily met by Context.run(). So I think that Yury's desire for an immutable Context will not harm anyone, and hence I support the current design of the PEP. (Though I want some of the details to be written up clearer -- everyone seems to agree with that. :-) Maybe I should clarify again what run() does. Here's how I think of it in pseudo code: def run(self, func, *args, **kwds): old = _get_current_context() new = old.copy() _set_current_context(new) try: return func(*args, **kwds) finally: _set_current_context(old) If you look carefully at the version in the PEP you'll see that it's the same thing, but the PEP inlines the implementations of _get_current_context() and _set_current_context() (which I made up -- these won't be real APIs) through manipulation of the current thread state. I hope this clarifies everything. --Guido On Wed, Jan 3, 2018 at 2:26 AM, Victor Stinner wrote: > Le 3 janv. 2018 06:05, "Yury Selivanov" a > écrit : > > tuples in Python are immutable, but you can have a tuple with a dict as > its single element. The tuple is immutable, the dict is mutable. > > At the C level we have APIs that can mutate a tuple though. > > Now, tuple is not a direct analogy to Context, but there are some > parallels. Context is a container like tuple, with some additional APIs on > top. > > > Sorry, I don't think that it's a good analogy. Context.run() is a public > method accessible in Python which allows to modify the context. A tuple > doesn't have such method. > > While it's technically possible to modify a tuple or a str at C level, > it's a bad practice leading to complex bugs when it's not done carefully: > see https://bugs.python.org/issue30156 property_descr_get() optimization > was fixed twice but still has a bug. I proposed a PR to remove the hack. > > Why Context could not inherit from MutableMapping? (Allow ctx.set(var, >> value) and ctx [var] = value.) Is it just to keep the API small: changes >> should only be made using var.set()? >> > > Because that would be confusing to end users. > > ctx = copy_context() > ctx[var] = something > > What did we just do? Did we modify the 'var' in the code that is > currently executing? No, you still need to call Context.run to see the new > value for var. > > >
Re: [Python-Dev] Concerns about method overriding and subclassing with dataclasses
I’ll open an issue after I have time to read this thread and comment on it. -- Eric. > On Jan 3, 2018, at 12:31 PM, Ivan Levkivskyi wrote: > > I like the Guido's proposal, i.e. > > if '__repr__' not in cls.__dict__: > ... # generate the method > > etc. I didn't find an issue to track this. Maybe we should open one? > > -- > Ivan > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 v2
> Do you have any use case for modifying a variable inside some context?
> numpy, decimal, or some sort of tracing for http requests or async
frameworks like asyncio do not need that.
Maybe I misunderstood how contextvars is supposed to be used. So let me
give you an example.
I understand that decimal.py will declare its context variable like this:
---
contextvar = contextvars.ContextVar('decimal', default=Context(...))
---
Later if I would like to run an asyncio callback with a different decimal
context, I would like to write:
---
cb_context = contextvars.copy_context()
decimal_context = cb_context[decimal.contextvar].copy()
decimal_context.prec = 100
cb_context[decimal.contextvar] = decimal_context # <--- HERE
loop.call_soon(func, context=cb_context)
The overall code would behaves as:
---
with localcontext() as ctx:
ctx.prec = 100
loop.call_soon(func)
---
I don't know if the two code snippets have exactly the same behaviour.
I don't want to modify func() to run it with a different decimal context.
So I would prefer to not have to call decimal.contextvar.set() or
decimal.setcontext() in func().
But I would need contextvars.Context[var]=value to support such use case.
Decimal contexts are mutable, so modifying directly the decimal context
object would impact all contexts which isn't my intent here.
Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 v2
Victor: > ContextVar would be simpler if the default would be mandatory as well :-) If we modify ContextVar to use None as the default value, the 'default' parameter of ContextVar.get() becomes useless, ContextVar.get() and context[var] behaviour becomes obvious. Token.MISSING could also be removed. Since it's not possible to delete a variable, would it be crazy to always initialize variables with a value, None by default? Decimal context is given as an example of contextvars user. Decimal already has a default context. I don't know numpy.errstate: would it make sense to initialize it to None? If someone really needs the crazy case of "uninitialized" variable, a custom "UNINITIALIZED = object()" can be used, no? With these proposed changes, there is no more need to worry if a variable is set or not. It's unclear to me if context.items() contains variables which weren't explicit set, and so are set to their default value. Same question for "var in context" test. If variables always have a value, I would expect that "var in context" is always true but I don't understand if it's technically possible to implement it or even if it makes sense :-) Maybe my whole proposed change doesn't make sense? Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Concerns about method overriding and subclassing with dataclasses
I like the Guido's proposal, i.e. if '__repr__' not in cls.__dict__: ... # generate the method etc. I didn't find an issue to track this. Maybe we should open one? -- Ivan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 'continue'/'break'/'return' inside 'finally' clause
Generally I think programming language implementers don't get to decide how the language works. You just have to implement it as specified, inconvenient as that might be. However, from a languge design prespective, I think there is a good argument that this is a corner of the language we should consider changing. First, I analyzed over one million lines of Python code with my AST walker and only found this construct being used in four different places. It seems to be extremely rare. Second, the existance of a pylint warning for it suggests that it is confusing. I did a little more searching using the pylint warning and found these pages: https://stackoverflow.com/questions/35505624/break-statement-in-finally-block-swallows-exception http://thegreyblog.blogspot.ca/2011/02/do-not-return-in-finally-block-return.html So, given the above and that the implementation (both compiler and bytecode evaluator) is pretty complicated, I vote that we should disallow it. Regards, Neil ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Possible bug in base64.decode: linebreaks are not ignored
Hello,
I've tried reading various RFCs around Base64 encoding, but I couldn't make the
ends meet. Yet there is an inconsistency between base64.decodebytes() and
base64.decode() in that how they handle linebreaks that were used to collate
the encoded text. Below is an example of what I'm talking about:
>>> import base64
>>> foo = base64.encodebytes(b'123456789')
>>> foo
b'MTIzNDU2Nzg5\n'
>>> foo = b'MTIzND\n' + b'U2Nzg5\n'
>>> foo
b'MTIzND\nU2Nzg5\n'
>>> base64.decodebytes(foo)
b'123456789'
>>> from io import BytesIO
>>> bytes_in = BytesIO(foo)
>>> bytes_out = BytesIO()
>>> bytes_in.seek(0)
0
>>> base64.decode(bytes_in, bytes_out)
Traceback (most recent call last):
File "", line 1, in
File "/somewhere/lib/python3.6/base64.py", line 512, in decode
s = binascii.a2b_base64(line)
binascii.Error: Incorrect padding
>>> bytes_in = BytesIO(base64.encodebytes(b'123456789'))
>>> bytes_in.seek(0)
0
>>> base64.decode(bytes_in, bytes_out)
>>> bytes_out.getvalue()
b'123456789'
Obviously, I'd expect encodebytes() and encode both to either accept or to
reject the same input.
Thanks.
Oleg
PS. I couldn't register to the bug-tracker (never received an email
confirmation, not even in a spam folder), this is why I'm sending it here.
This communication and all information contained in or attached to it is
confidential, intended solely for the addressee, may be legally privileged and
is the intellectual property of one of the companies of NEX Group plc ("NEX")
or third parties. If you are not the intended addressee or receive this message
in error, please immediately delete all copies of it and notify the sender. We
have taken precautions to minimise the risk of transmitting software viruses,
but we advise you to carry out your own virus checks on any attachments. We do
not accept liability for any loss or damage caused by software viruses. NEX
reserves the right to monitor all communications. We do not accept any legal
responsibility for the content of communications, and no communication shall be
considered legally binding. Furthermore, if the content of this communication
is personal or unconnected with our business, we accept no liability or
responsibility for it. NEX Group plc is a public limited company regi
stered in England and Wales under number 10013770 and certain of its
affiliates are authorised and regulated by regulatory authorities. For further
regulatory information please see www.NEX.com.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 v2
On 28 December 2017 at 06:08, Yury Selivanov wrote: > This is a second version of PEP 567. Overall, I like the proposal. It's relatively straightforward to follow, and makes sense. One thing I *don't* see in the PEP is an example of how code using thread-local storage should be modified to use context variables. My impression is that it's simply a matter of replacing the TLS calls with equivalent ContextVar calls, but an example might be helpful. Some detail points below. > Rationale > = > > Thread-local variables are insufficient for asynchronous tasks that > execute concurrently in the same OS thread. Any context manager that > saves and restores a context value using ``threading.local()`` will > have its context values bleed to other code unexpectedly when used > in async/await code. I understand how this could happen, having followed the discussions here, but a (simple) example of the issue might be useful. > A few examples where having a working context local storage for > asynchronous code is desirable: > > * Context managers like ``decimal`` contexts and ``numpy.errstate``. > > * Request-related data, such as security tokens and request > data in web applications, language context for ``gettext``, etc. > > * Profiling, tracing, and logging in large code bases. > > > Introduction > > > The PEP proposes a new mechanism for managing context variables. > The key classes involved in this mechanism are ``contextvars.Context`` > and ``contextvars.ContextVar``. The PEP also proposes some policies > for using the mechanism around asynchronous tasks. > > The proposed mechanism for accessing context variables uses the > ``ContextVar`` class. A module (such as ``decimal``) that wishes to > store a context variable should: > > * declare a module-global variable holding a ``ContextVar`` to > serve as a key; > > * access the current value via the ``get()`` method on the > key variable; > > * modify the current value via the ``set()`` method on the > key variable. > > The notion of "current value" deserves special consideration: > different asynchronous tasks that exist and execute concurrently > may have different values for the same key. This idea is well-known > from thread-local storage but in this case the locality of the value is > not necessarily bound to a thread. Instead, there is the notion of the > "current ``Context``" which is stored in thread-local storage, and > is accessed via ``contextvars.copy_context()`` function. Accessed by copying it? That seems weird to me. I'd expect either that you'd be able to access the current Context directly, *or* that you'd say that the current Context is not directly accessible by the user, but that a copy can be obtained using copy_context. But given that the Context is immutable, why the need tp copy it? Also, the references to threads in the above are confusing. It says that this is a well-known concept in terms of thread-local storage, but this case is different. It then goes on to say that the current Context is stored in thread local storage, which gives me the impression that the new idea *is* related to thread local storage... I think that the fact that a Context is held in thread-local storage is an implementation detail. Assuming I'm right, don't bother mentioning it - simply say that there's a notion of a current Context and leave it at that. > Manipulation of the current ``Context`` is the responsibility of the > task framework, e.g. asyncio. > > A ``Context`` is conceptually a read-only mapping, implemented using > an immutable dictionary. The ``ContextVar.get()`` method does a > lookup in the current ``Context`` with ``self`` as a key, raising a > ``LookupError`` or returning a default value specified in > the constructor. > > The ``ContextVar.set(value)`` method clones the current ``Context``, > assigns the ``value`` to it with ``self`` as a key, and sets the > new ``Context`` as the new current ``Context``. > On first reading, this confused me because I didn't spot that you're saying a *Context* is read-only, but a *ContextVar* has get and set methods. Maybe reword this to say that a Context is a read-only mapping from ContextVars to values. A ContextVar has a get method that looks up its value in the current Context, and a set method that replaces the current Context with a new one that associates the specified value with this ContextVar. (The current version feels confusing to me because it goes into too much detail on how the implementation does this, rather than sticking to the high-level specification) > Specification > = > > A new standard library module ``contextvars`` is added with the > following APIs: > > 1. ``copy_context() -> Context`` function is used to get a copy of >the current ``Context`` object for the current OS thread. > > 2. ``ContextVar`` class to declare and access context variables. > > 3. ``Context`` class encapsulates context state. Every OS thread >store
Re: [Python-Dev] PEP 567 v2
I think we can expose the default property. If it's not set we can return MISSING. Yury Sent from my iPhone > On Jan 3, 2018, at 1:04 PM, Victor Stinner wrote: > > Le 3 janv. 2018 06:38, "Guido van Rossum" a écrit : > But is there a common use case? For var.get() I'd rather just pass the > default or catch the exception if the flow is different. Using ctx[var] is > rare (mostly for printing contexts, and perhaps for explaining var.get()). > > I don't think that it would be a common use case. Maybe we don't need > is_set(), I'm fine with catching an exception. > > But for introspection at least, it would help to expose the default as a > read-only attribute, no? > > Another example of a mapping with default value: > > https://docs.python.org/dev/library/collections.html#collections.defaultdict > > And defaultdict has a default_factory attribute. The difference here is that > default_factory is mandatory. ContextVar would be simpler if the default > would be mandatory as well :-) > > Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 v2
Le 3 janv. 2018 06:38, "Guido van Rossum" a écrit : But is there a common use case? For var.get() I'd rather just pass the default or catch the exception if the flow is different. Using ctx[var] is rare (mostly for printing contexts, and perhaps for explaining var.get()). I don't think that it would be a common use case. Maybe we don't need is_set(), I'm fine with catching an exception. But for introspection at least, it would help to expose the default as a read-only attribute, no? Another example of a mapping with default value: https://docs.python.org/dev/library/collections.html#collections.defaultdict And defaultdict has a default_factory attribute. The difference here is that default_factory is mandatory. ContextVar would be simpler if the default would be mandatory as well :-) Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 v2
Le 3 janv. 2018 06:34, "Guido van Rossum" a écrit :
I think the issue here is a bit different than Yury's response suggests --
it's more like how a variable containing an immutable value (e.g. a string)
can be modified, e.g.
x = 'a'
x += 'b'
In our case the *variable* is the current thread state (in particular the
slot therein that holds the context -- this slot can be modified by the C
API). The *value* is the Context object. It is a collections.Mapping (or
typing.Mapping) which does not have mutating methods. (The mutable type is
called MutableMapping.)
I can see a parallel with a Python namespace, like globals and locals
arguments of exec():
ns = globals().copy() # ctx = copy_context()
exec("x = 'a'", ns, ns) # ctx.run(...)
ns['x'] += 'b' # Context ???
print(ns ['x']) # print(ctx[x])
The *reason* for doing it this way is that Yury doesn't want Context to
implement __delitem__, since it would complicate the specification of
chained lookups by a future PEP, and chained lookups look to be the best
option to extend the Context machinery for generators.
Again, why not just raise an exception on "del ctx[var]"?
Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 v2
> On Jan 3, 2018, at 12:26 PM, Victor Stinner wrote:
>
> Le 3 janv. 2018 06:05, "Yury Selivanov" a écrit :
> tuples in Python are immutable, but you can have a tuple with a dict as its
> single element. The tuple is immutable, the dict is mutable.
>
> At the C level we have APIs that can mutate a tuple though.
>
> Now, tuple is not a direct analogy to Context, but there are some parallels.
> Context is a container like tuple, with some additional APIs on top.
>
> Sorry, I don't think that it's a good analogy. Context.run() is a public
> method accessible in Python which allows to modify the context. A tuple
> doesn't have such method.
>
> While it's technically possible to modify a tuple or a str at C level, it's a
> bad practice leading to complex bugs when it's not done carefully: see
> https://bugs.python.org/issue30156 property_descr_get() optimization was
> fixed twice but still has a bug. I proposed a PR to remove the hack.
>
>> Why Context could not inherit from MutableMapping? (Allow ctx.set(var,
>> value) and ctx [var] = value.) Is it just to keep the API small: changes
>> should only be made using var.set()?
>
> Because that would be confusing to end users.
>
> ctx = copy_context()
> ctx[var] = something
>
> What did we just do? Did we modify the 'var' in the code that is currently
> executing? No, you still need to call Context.run to see the new value for
> var.
>
> IMHO it's easy to understand that modifying a *copy* of the current context
> doesn't impact the current context. It's one the first thing to learn when
> learning Python:
>
> a = [1, 2]
> b = a.copy()
> b.append(3)
> assert a == [1, 2]
> assert b == [1, 2, 3]
>
> Another problem is that MutableMapping defines a __delitem__ method, which i
> don't want the Context to implement.
>
> I wouldn't be shocked if "del ctx [var]" would raise an exception.
>
> I almost never use del anyway. I prefer to assign a variable to None, since
> "del var" looks like C++ destructor whereas it's more complex than a direct
> call to the destructor.
>
> But it's annoying to have to call a function with Context.run() whereas
> context is just a mutable mapping. It seems overkill to me to have to call
> run() to modify a context variable:
Do you have any use case for modifying a variable inside some context?
numpy, decimal, or some sort of tracing for http requests or async frameworks
like asyncio do not need that.
> run() changes temporarely the context and requires to use the indirect
> ContextVar API, while I know that ContextVar.set() modifies the context.
>
> Except of del corner case, I don't see any technical reason to prevent direct
> modification of a context.
>
> contextvars isn't new, it extends what we already have: decimal context. And
> decimal quick start documentation shows how to modify a context and then set
> it as the current context:
>
I think you are confusing context in decimal and pep 567.
Decimal context is a mutable object. We use threading.local to store it. With
pep 567 you will use a context variable behind the scenes to store it.
I think it's incorrect to compare decimal contexts to pep567 in any way.
Yury
> >>> myothercontext = Context(prec=60, rounding=ROUND_HALF_DOWN)
> >>> setcontext(myothercontext)
> >>> Decimal(1) / Decimal(7)
> Decimal('0.142857142857142857142857142857142857142857142857142857142857')
>
> https://docs.python.org/dev/library/decimal.html
>
> Well, technically it doesn't modify a context. An example closer to
> contextvars would be:
>
> >>> mycontext = getcontext().copy()
> >>> mycontext.prec = 60
> >>> setcontext(mycontext)
> >>> Decimal(1) / Decimal(7)
> Decimal('0.142857142857142857142857142857142857142857142857142857142857')
>
> Note: "getcontext().prec = 6" does modify the decimal context directly, and
> it's the *first* example in the doc. But here contextvars is different since
> there is no API to get the current API. The lack of API to access directly
> the current contextvars context is the main difference with decimal context,
> and I'm fine with that.
>
> It's easy to see a parallel since decimal context can be copied using
> Context.copy(), it has also multiple (builtin) "variables", it's just that
> the API is different (decimal context variables are modified as attributes),
> and it's possible to set a context using decimal.setcontext().
>
> Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 v2
Le 3 janv. 2018 06:05, "Yury Selivanov" a écrit :
tuples in Python are immutable, but you can have a tuple with a dict as its
single element. The tuple is immutable, the dict is mutable.
At the C level we have APIs that can mutate a tuple though.
Now, tuple is not a direct analogy to Context, but there are some
parallels. Context is a container like tuple, with some additional APIs on
top.
Sorry, I don't think that it's a good analogy. Context.run() is a public
method accessible in Python which allows to modify the context. A tuple
doesn't have such method.
While it's technically possible to modify a tuple or a str at C level, it's
a bad practice leading to complex bugs when it's not done carefully: see
https://bugs.python.org/issue30156 property_descr_get() optimization was
fixed twice but still has a bug. I proposed a PR to remove the hack.
Why Context could not inherit from MutableMapping? (Allow ctx.set(var,
> value) and ctx [var] = value.) Is it just to keep the API small: changes
> should only be made using var.set()?
>
Because that would be confusing to end users.
ctx = copy_context()
ctx[var] = something
What did we just do? Did we modify the 'var' in the code that is currently
executing? No, you still need to call Context.run to see the new value for
var.
IMHO it's easy to understand that modifying a *copy* of the current context
doesn't impact the current context. It's one the first thing to learn when
learning Python:
a = [1, 2]
b = a.copy()
b.append(3)
assert a == [1, 2]
assert b == [1, 2, 3]
Another problem is that MutableMapping defines a __delitem__ method, which
i don't want the Context to implement.
I wouldn't be shocked if "del ctx [var]" would raise an exception.
I almost never use del anyway. I prefer to assign a variable to None, since
"del var" looks like C++ destructor whereas it's more complex than a direct
call to the destructor.
But it's annoying to have to call a function with Context.run() whereas
context is just a mutable mapping. It seems overkill to me to have to call
run() to modify a context variable: run() changes temporarely the context
and requires to use the indirect ContextVar API, while I know that
ContextVar.set() modifies the context.
Except of del corner case, I don't see any technical reason to prevent
direct modification of a context.
contextvars isn't new, it extends what we already have: decimal context.
And decimal quick start documentation shows how to modify a context and
then set it as the current context:
>>> myothercontext = Context(prec=60, rounding=ROUND_HALF_DOWN)
>>> setcontext(myothercontext)
>>> Decimal(1) / Decimal(7)
Decimal('0.142857142857142857142857142857142857142857142857142857142857')
https://docs.python.org/dev/library/decimal.html
Well, technically it doesn't modify a context. An example closer to
contextvars would be:
>>> mycontext = getcontext().copy()
>>> mycontext.prec = 60
>>> setcontext(mycontext)
>>> Decimal(1) / Decimal(7)
Decimal('0.142857142857142857142857142857142857142857142857142857142857')
Note: "getcontext().prec = 6" does modify the decimal context directly, and
it's the *first* example in the doc. But here contextvars is different
since there is no API to get the current API. The lack of API to access
directly the current contextvars context is the main difference with
decimal context, and I'm fine with that.
It's easy to see a parallel since decimal context can be copied using
Context.copy(), it has also multiple (builtin) "variables", it's just that
the API is different (decimal context variables are modified as
attributes), and it's possible to set a context using decimal.setcontext().
Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
