[Python-ideas] Add .cache and compile_commands.json to .gitignote

[Stephen J. Turnbull]

Jack DeVries writes:

 > What does everyone think? Can we add these two items to the .gitignore:
 > 
 > - `.cache`
 > - `compile_commands.json`

I don't see any cost to this -- .cache is uncomfortably generic, but
given the semantics of "cache" gitignoring it seems a good idea.

ISTM rather than have every project add these to the project-specific
gitignore this is really something every clang user should have in
their personal gitignore for best effect.  As long as they don't
though, having the project do it will avoid inadvertantly git-adding
these objects.

Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/E2SH5Z27SRMQCLOZA72LJZM5HSQEDEJF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Add .cache and compile_commands.json to .gitignote

[Jack DeVries]

Hi All,

I use clangd. Clangd creates a .cache directory, and there are also
tools for generating a compile_commands.json file which tells clangd how
to behave. I did a quick check, and there are no naming collisions with
other files in the cpython project.

What does everyone think? Can we add these two items to the .gitignore:

- `.cache`
- `compile_commands.json`

-- 
Thanks!
Jack DeVries
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/HYWJIRZBBORBOWSRFDQMCSOKTP3UBAMO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Stack-scoped variables

[Guido van Rossum]

Oh, I like that. It does feel like a property of the variable. (But can you
efficiently enumerate all context vars when creating a thread?)

I imagine that thread pools add some complication, because you don’t want
to inherit these accidentally between tasks run by the same worker.

On Thu, Aug 19, 2021 at 14:28 Paul Prescod  wrote:

> On Thu, Aug 19, 2021 at 8:43 AM Guido van Rossum  wrote:
>
>> Perhaps we need a library for creating/managing threads that inherits all
>> current context values?
>>
>
> Or is it a "kind of context variable that is shared among threads?" That
> was more the direction my mind was going.
>
> Context variables that hold explicitly or implicitly immutable values
> are safe to share. Context variables that hold values intended to be
> mutated: seldom.
>
> What if it were some kind of ContextVar subclass or flag where you would
> say you did or didn't want something shared?
>
> And for legacy modules like decimal, you could turn on the flag somehow to
> get the sharing behaviour.
>
> Sometimes threads are made by third-party libraries and those libraries
> don't know anything about the context.
>
>  In my case, the pattern was:
>
> 1. my code set context variable
> 2. my code configured a third party library with a callback
> 3. the third party library used threading for perf reasons
> 4. it called my callback, which failed due to the context variable failing
> to be inherited
>
> I believe (after only thinking about it for a day or so) that one can
> generally determine the right "sharing rule" at the point of definition of
> a ContextVar, based on the semantics and type of the Var.
>
> On Thu, Aug 19, 2021 at 8:43 AM Guido van Rossum  wrote:
>
>> Perhaps we need a library for creating/managing threads that inherits all
>> current context values?
>>
>> On Thu, Aug 19, 2021 at 7:48 AM Paul Prescod  wrote:
>>
>>> There are certain values that are managed independent of any specific
>>> code-accessible object. The decimal precision is a good example:
>>>
>>> https://docs.python.org/3/library/decimal.html
>>>
>>> We can call these context variables. As your code shows, the true "home"
>>> of the "prec" value is not some object you manage in your code, but rather
>>> some background object called the Context.
>>>
>>> The Context is not inherited by sub-threads.
>>>
>>> Your code is fine, because you did not use threads. Replace the map with
>>> concurrent.futures code (as in my example).
>>>
>>> Your code will not work properly anymore. This is the issue.
>>> ___
>>> Python-ideas mailing list -- python-ideas@python.org
>>> To unsubscribe send an email to python-ideas-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/python-ideas@python.org/message/6YU3RZ6WDWVJI7QEJKICJZN72B5FYOEV/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> *Pronouns: he/him **(why is my pronoun here?)*
>> 
>>
> --
--Guido (mobile)
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/KWDN3T7QUEBLGL3YBHDIARHDEAQJMW43/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Stack-scoped variables

[Paul Prescod]

On Thu, Aug 19, 2021 at 8:43 AM Guido van Rossum  wrote:

> Perhaps we need a library for creating/managing threads that inherits all
> current context values?
>

Or is it a "kind of context variable that is shared among threads?" That
was more the direction my mind was going.

Context variables that hold explicitly or implicitly immutable values
are safe to share. Context variables that hold values intended to be
mutated: seldom.

What if it were some kind of ContextVar subclass or flag where you would
say you did or didn't want something shared?

And for legacy modules like decimal, you could turn on the flag somehow to
get the sharing behaviour.

Sometimes threads are made by third-party libraries and those libraries
don't know anything about the context.

 In my case, the pattern was:

1. my code set context variable
2. my code configured a third party library with a callback
3. the third party library used threading for perf reasons
4. it called my callback, which failed due to the context variable failing
to be inherited

I believe (after only thinking about it for a day or so) that one can
generally determine the right "sharing rule" at the point of definition of
a ContextVar, based on the semantics and type of the Var.

On Thu, Aug 19, 2021 at 8:43 AM Guido van Rossum  wrote:

> Perhaps we need a library for creating/managing threads that inherits all
> current context values?
>
> On Thu, Aug 19, 2021 at 7:48 AM Paul Prescod  wrote:
>
>> There are certain values that are managed independent of any specific
>> code-accessible object. The decimal precision is a good example:
>>
>> https://docs.python.org/3/library/decimal.html
>>
>> We can call these context variables. As your code shows, the true "home"
>> of the "prec" value is not some object you manage in your code, but rather
>> some background object called the Context.
>>
>> The Context is not inherited by sub-threads.
>>
>> Your code is fine, because you did not use threads. Replace the map with
>> concurrent.futures code (as in my example).
>>
>> Your code will not work properly anymore. This is the issue.
>> ___
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-ideas@python.org/message/6YU3RZ6WDWVJI7QEJKICJZN72B5FYOEV/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/UK2PY4XEKHE652X6QGVSLFFYQ75BVTQQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Stack-scoped variables

[Thomas Grainger]

asyncio.to_thread creates threads that inherit the current context, according 
to https://www.python.org/dev/peps/pep-0567/#rationale the decimal module 
should use contextvars for this too
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/IWP3BYSQW43GYZIABQUBJJEHWJTI2ITI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Stack-scoped variables

[Guido van Rossum]

Perhaps we need a library for creating/managing threads that inherits all
current context values?

On Thu, Aug 19, 2021 at 7:48 AM Paul Prescod  wrote:

> There are certain values that are managed independent of any specific
> code-accessible object. The decimal precision is a good example:
>
> https://docs.python.org/3/library/decimal.html
>
> We can call these context variables. As your code shows, the true "home"
> of the "prec" value is not some object you manage in your code, but rather
> some background object called the Context.
>
> The Context is not inherited by sub-threads.
>
> Your code is fine, because you did not use threads. Replace the map with
> concurrent.futures code (as in my example).
>
> Your code will not work properly anymore. This is the issue.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/6YU3RZ6WDWVJI7QEJKICJZN72B5FYOEV/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/ZSOM5DJ3JFCXJDANRWTRVF3LHYNJ2EDJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Stack-scoped variables

[Paul Prescod]

There are certain values that are managed independent of any specific 
code-accessible object. The decimal precision is a good example:

https://docs.python.org/3/library/decimal.html

We can call these context variables. As your code shows, the true "home" of the 
"prec" value is not some object you manage in your code, but rather some 
background object called the Context.

The Context is not inherited by sub-threads.

Your code is fine, because you did not use threads. Replace the map with 
concurrent.futures code (as in my example).

Your code will not work properly anymore. This is the issue.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/6YU3RZ6WDWVJI7QEJKICJZN72B5FYOEV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Notation for subscripts.

[Stephen J. Turnbull]

Matsuoka Takuo writes:

 > >>> *(1,2),
 > (1, 2)

Yes, this works, and now that I see you just want that to work in
"a[*(1,2),]", I agree, I don't know why that is a syntax error.  This
works, of course:

t = *(1,2),
a[t]

(in the sense that if a is a sequence you get a TypeError because the
index isn't an integer or slice, but if it's a mapping you'll get the
value corresponding to (1,2) or a KeyError because it's not in the
mapping).  I think it's not good that "a[*(1,2),]" gives a SyntaxError,
but I don't know why it was made that way in the first place.

 > I find it unfortunate SyntaxError you got with
 > 
 > >>> *(1,2)
 > 
 > says "can't use starred expression here" since "*(1,2)" is not really
 > a "starred expression" as defined in the Language Reference.

I don't have time to go into that tonight but if the appropriate
thread gets takeup I'll comment there.  You could promote the issue by
submitting a merge request.

Steve

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XPBMDKG3AUM42EBO2CAKCAYY42FMREFG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: multiprocessing: hybrid CPUs

[Stephen J. Turnbull]

Thomas Grainger writes:

 > Would a work stealing approach work better for you here? Then the only
 > signalling overhead would be when a core runs out of work

Not sure what you're talking about with "work stealing".  It sounds
conceptually more complex than the queue + worker pool approach,
which is already implemented in both the threading and multiprocessing
modules.

The overhead of creating hundreds of multiprocessing tasks is going to
be barely human-perceptible.  The other "overhead" is the programmer
effort in assembling the finished product (assuming order matters, or
there are interdependencies between chunks that require keeping
per-chunk state).  But I don't see how such programmer effort would be
much greater for the "many chunks in a queue" approach vs. the
chunk-per-core approach.  So it seems to me that multiprocessing with
a worker pool is a low programmer effort, very high efficiency gain
approach to this problem.

The remaining question is "how many chunks?"  If that's relevant, ISTM
a few simple experiments will show where the sweet spot is.  Try a
queue of 64 chunks, then 128 chunks, and refine guesses from there.

I may be missing something, but that's the thinking that led to my
previous post.

Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/YUEBSYCWGU4ZJIJOE6XXMIA6N3KHK75I/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Notation for subscripts.

[Chris Angelico]

On Thu, Aug 19, 2021 at 11:27 PM Matsuoka Takuo  wrote:
>
> Dear Steve,
>
> Thank you for your detailed explanation.
>
> >  > (i.e., it can be mistyped as "s[1,2,]" but without SyntaxError this
> >  > time). It would at least be consistent if we got SyntaxError in
> >  > both cases (namely, the syntax allowed only a single Python
> >  > expression here), but if we don't want SyntaxError in the latter
> >  > case, how may we want it in the former case?
> >
> > I defy you to explain how the compiler or interpreter can distinguish
> > any of those stipulated programmer errors from intended programs.
>
> Sure, if you ever want any error raised in those cases, that's going
> to be SyntaxError to be always raised regardless of the
> programmer's intention. That's fine. But then if you don't want
> SyntaxError in the "latter" case, it in fact means you never want
> any error raised since SyntaxError is the only thing you could
> hope for anyway. So I could equivalently have asked "if we never
> want any error raised in the latter case, how may we want some
> error raised at least sometimes in the former case?". This may
> have been clearer to you, but it's an equivalent question.
>
> Anyway, I assume everything you say about "*(1,2)" is right, but it's
> not "*(1,2)" that I wanted to create
> an object, but, as I have indicated, it's things like "*iterable , "
> (with comma!), where the iterable may be "(1,2)". Just as in "1,2",
> the comma indicates the intention to create an object. For example,
>
> >>> *(1,2),
> (1, 2)
>
> and another example:
>
> >>> *(), *(1,2)
> (1, 2)
>
> I find it unfortunate SyntaxError you got with
>
> >>> *(1,2)
>
> says "can't use starred expression here" since "*(1,2)" is not really
> a "starred expression" as defined in the Language Reference. I think
> this should be fixed, so have started a new thread on that, titled
> '''SyntaxError says: "can't use starred expression here", but its not
> the same "starred expression" defined in the Language Reference''',
> You are invited to comment on that!

Is the issue just one of wording in the exception message, or are you
asking for a change of functionality? In your final example, you've
never said that you're creating a tuple, so it's correct to get a
SyntaxError. I'm confused as to whether you're asking for this to have
meaning, or if instead you are simply proposing a change to wording.

Changes to wording can definitely be done, and aren't a major backward
or forward compatibility concern.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/YCT4EQAABSDAHIQ65NXQZI3SSC6FKNV7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Notation for subscripts.

[Matsuoka Takuo]

Dear Steve,

Thank you for your detailed explanation.

>  > (i.e., it can be mistyped as "s[1,2,]" but without SyntaxError this
>  > time). It would at least be consistent if we got SyntaxError in
>  > both cases (namely, the syntax allowed only a single Python
>  > expression here), but if we don't want SyntaxError in the latter
>  > case, how may we want it in the former case?
>
> I defy you to explain how the compiler or interpreter can distinguish
> any of those stipulated programmer errors from intended programs.

Sure, if you ever want any error raised in those cases, that's going
to be SyntaxError to be always raised regardless of the
programmer's intention. That's fine. But then if you don't want
SyntaxError in the "latter" case, it in fact means you never want
any error raised since SyntaxError is the only thing you could
hope for anyway. So I could equivalently have asked "if we never
want any error raised in the latter case, how may we want some
error raised at least sometimes in the former case?". This may
have been clearer to you, but it's an equivalent question.

Anyway, I assume everything you say about "*(1,2)" is right, but it's
not "*(1,2)" that I wanted to create
an object, but, as I have indicated, it's things like "*iterable , "
(with comma!), where the iterable may be "(1,2)". Just as in "1,2",
the comma indicates the intention to create an object. For example,

>>> *(1,2),
(1, 2)

and another example:

>>> *(), *(1,2)
(1, 2)

I find it unfortunate SyntaxError you got with

>>> *(1,2)

says "can't use starred expression here" since "*(1,2)" is not really
a "starred expression" as defined in the Language Reference. I think
this should be fixed, so have started a new thread on that, titled
'''SyntaxError says: "can't use starred expression here", but its not
the same "starred expression" defined in the Language Reference''',
You are invited to comment on that!

Best regards,
Takuo
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/J7WDSY7JTC64WQZ6YWB5HWLADXPK2KQZ/
Code of Conduct: http://python.org/psf/codeofconduct/