[Python-ideas] Re: Stack-scoped variables

[Guido van Rossum]

Oh, I like that. It does feel like a property of the variable. (But can you
efficiently enumerate all context vars when creating a thread?)

I imagine that thread pools add some complication, because you don’t want
to inherit these accidentally between tasks run by the same worker.

On Thu, Aug 19, 2021 at 14:28 Paul Prescod  wrote:

> On Thu, Aug 19, 2021 at 8:43 AM Guido van Rossum  wrote:
>
>> Perhaps we need a library for creating/managing threads that inherits all
>> current context values?
>>
>
> Or is it a "kind of context variable that is shared among threads?" That
> was more the direction my mind was going.
>
> Context variables that hold explicitly or implicitly immutable values
> are safe to share. Context variables that hold values intended to be
> mutated: seldom.
>
> What if it were some kind of ContextVar subclass or flag where you would
> say you did or didn't want something shared?
>
> And for legacy modules like decimal, you could turn on the flag somehow to
> get the sharing behaviour.
>
> Sometimes threads are made by third-party libraries and those libraries
> don't know anything about the context.
>
>  In my case, the pattern was:
>
> 1. my code set context variable
> 2. my code configured a third party library with a callback
> 3. the third party library used threading for perf reasons
> 4. it called my callback, which failed due to the context variable failing
> to be inherited
>
> I believe (after only thinking about it for a day or so) that one can
> generally determine the right "sharing rule" at the point of definition of
> a ContextVar, based on the semantics and type of the Var.
>
> On Thu, Aug 19, 2021 at 8:43 AM Guido van Rossum  wrote:
>
>> Perhaps we need a library for creating/managing threads that inherits all
>> current context values?
>>
>> On Thu, Aug 19, 2021 at 7:48 AM Paul Prescod  wrote:
>>
>>> There are certain values that are managed independent of any specific
>>> code-accessible object. The decimal precision is a good example:
>>>
>>> https://docs.python.org/3/library/decimal.html
>>>
>>> We can call these context variables. As your code shows, the true "home"
>>> of the "prec" value is not some object you manage in your code, but rather
>>> some background object called the Context.
>>>
>>> The Context is not inherited by sub-threads.
>>>
>>> Your code is fine, because you did not use threads. Replace the map with
>>> concurrent.futures code (as in my example).
>>>
>>> Your code will not work properly anymore. This is the issue.
>>> ___
>>> Python-ideas mailing list -- python-ideas@python.org
>>> To unsubscribe send an email to python-ideas-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/python-ideas@python.org/message/6YU3RZ6WDWVJI7QEJKICJZN72B5FYOEV/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> *Pronouns: he/him **(why is my pronoun here?)*
>> 
>>
> --
--Guido (mobile)
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/KWDN3T7QUEBLGL3YBHDIARHDEAQJMW43/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Stack-scoped variables

[Paul Prescod]

On Thu, Aug 19, 2021 at 8:43 AM Guido van Rossum  wrote:

> Perhaps we need a library for creating/managing threads that inherits all
> current context values?
>

Or is it a "kind of context variable that is shared among threads?" That
was more the direction my mind was going.

Context variables that hold explicitly or implicitly immutable values
are safe to share. Context variables that hold values intended to be
mutated: seldom.

What if it were some kind of ContextVar subclass or flag where you would
say you did or didn't want something shared?

And for legacy modules like decimal, you could turn on the flag somehow to
get the sharing behaviour.

Sometimes threads are made by third-party libraries and those libraries
don't know anything about the context.

 In my case, the pattern was:

1. my code set context variable
2. my code configured a third party library with a callback
3. the third party library used threading for perf reasons
4. it called my callback, which failed due to the context variable failing
to be inherited

I believe (after only thinking about it for a day or so) that one can
generally determine the right "sharing rule" at the point of definition of
a ContextVar, based on the semantics and type of the Var.

On Thu, Aug 19, 2021 at 8:43 AM Guido van Rossum  wrote:

> Perhaps we need a library for creating/managing threads that inherits all
> current context values?
>
> On Thu, Aug 19, 2021 at 7:48 AM Paul Prescod  wrote:
>
>> There are certain values that are managed independent of any specific
>> code-accessible object. The decimal precision is a good example:
>>
>> https://docs.python.org/3/library/decimal.html
>>
>> We can call these context variables. As your code shows, the true "home"
>> of the "prec" value is not some object you manage in your code, but rather
>> some background object called the Context.
>>
>> The Context is not inherited by sub-threads.
>>
>> Your code is fine, because you did not use threads. Replace the map with
>> concurrent.futures code (as in my example).
>>
>> Your code will not work properly anymore. This is the issue.
>> ___
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-ideas@python.org/message/6YU3RZ6WDWVJI7QEJKICJZN72B5FYOEV/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/UK2PY4XEKHE652X6QGVSLFFYQ75BVTQQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Stack-scoped variables

[Thomas Grainger]

asyncio.to_thread creates threads that inherit the current context, according 
to https://www.python.org/dev/peps/pep-0567/#rationale the decimal module 
should use contextvars for this too
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/IWP3BYSQW43GYZIABQUBJJEHWJTI2ITI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Stack-scoped variables

[Guido van Rossum]

Perhaps we need a library for creating/managing threads that inherits all
current context values?

On Thu, Aug 19, 2021 at 7:48 AM Paul Prescod  wrote:

> There are certain values that are managed independent of any specific
> code-accessible object. The decimal precision is a good example:
>
> https://docs.python.org/3/library/decimal.html
>
> We can call these context variables. As your code shows, the true "home"
> of the "prec" value is not some object you manage in your code, but rather
> some background object called the Context.
>
> The Context is not inherited by sub-threads.
>
> Your code is fine, because you did not use threads. Replace the map with
> concurrent.futures code (as in my example).
>
> Your code will not work properly anymore. This is the issue.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/6YU3RZ6WDWVJI7QEJKICJZN72B5FYOEV/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/ZSOM5DJ3JFCXJDANRWTRVF3LHYNJ2EDJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Stack-scoped variables

[Paul Prescod]

There are certain values that are managed independent of any specific 
code-accessible object. The decimal precision is a good example:

https://docs.python.org/3/library/decimal.html

We can call these context variables. As your code shows, the true "home" of the 
"prec" value is not some object you manage in your code, but rather some 
background object called the Context.

The Context is not inherited by sub-threads.

Your code is fine, because you did not use threads. Replace the map with 
concurrent.futures code (as in my example).

Your code will not work properly anymore. This is the issue.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/6YU3RZ6WDWVJI7QEJKICJZN72B5FYOEV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Stack-scoped variables

[Cameron Simpson]

On 18Aug2021 16:30, Paul Prescod  wrote:
>Let's imagine I have an algorithm which depends on a context variable.
>
>I write an algorithm (elided below for space) which depends on it. Then I 
>realize that I can improve the performance of my algorithm by using 
>concurrent.futures. But my algorithm will change its behaviour because it does 
>not inherit the context. Simply trying to parallelize an "embarrassingly 
>parallel" algorithm changes its behaviour.
>
>What I really want is a stack-scoped variable: a variable which retains its 
>value for all child scopes whether in the same thread or not, unless it is 
>overwritten in a child scope (whether in the same thread or not).

Do you mean a facility for setting an attribute on an object and 
reversing that when you exit the scope where you set it?

I've got a stackattrs context manager I use for that. Example use:

# this is on PyPI
from cs.context import stackattrs

with stackattrs(getcontext(), prec=8):
# .prec=8
vals = map(my_algorithm, range(0, 10))
# .prec is whatever it used to be, including absent

Otherwise it is not clear to me what you are after. If this isn't what 
you want, can you describe what's different?

Cheers,
Cameron Simpson 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/76G7B7DICTKHZRB6QOOAXYYKYDCSSMQS/
Code of Conduct: http://python.org/psf/codeofconduct/