[Python-Dev] Re: PEP 467 feedback from the Steering Council

[Antoine Pitrou]

On Sun, 22 Aug 2021 16:08:56 -0700
Guido van Rossum  wrote:
> Hm, I don’t think the major use for bchr() will be with a constant.

What would be the major use for bchr()?  I don't think I've ever
regretted its absence.

Regards

Antoine.


> 
> On Sun, Aug 22, 2021 at 14:48 Gregory P. Smith  wrote:
> 
> >
> > On Tue, Aug 10, 2021 at 3:48 PM Christopher Barker 
> > wrote:
> >  
> >> On Tue, Aug 10, 2021 at 3:00 PM  wrote:
> >>  
> >>> The history of bytes/bytearray is a dual-purpose view.  It can be used
> >>> in a string-like way to emulate Python 2 string handling (hence all the
> >>> usual string methods and a repr that displays in a string-like fashion).
> >>> It can also be used as an array of numbers, 0 to 255 (hence the list
> >>> methods and having an iterator of ints).  ISTM that the authors of this 
> >>> PEP
> >>> reject or want to discourage the latter use cases.
> >>>  
> >>
> >> I didn't read it that way, but if so, please no, I"d rather see the
> >> former use cases discouraged. ISTM that the Py2 string handling is still
> >> needed for working with mixed binary / text data -- but that should be a
> >> pretty specialized use case. spelling the way to create a byte, byte() sure
> >> makes more sense in any other context.
> >>
> >>  
> >>> ... anything where a C programmer would an array of unsigned chars).
> >>>  
> >>
> >> or any programmer would use an array of unsigned 8bit integers :-) numpy
> >> spells it: `np.uint8`, and the the type in the C99 stdint.h is `uint8_t`.
> >> My point is that for anyone not an "old time" C programmer, or even a
> >> Python2 programmer, the  "character is an unsigned 8 bit int" concept is
> >> alien and confusing, not a helpful mnemonic.
> >>
> >>  
> >>> For example, creating a single byte with bytes([0x1f]) isn't pleasant,
> >>> obvious, or fast.
> >>>  
> >>
> >> no, though bytes([31]) isn't horrible ;-)   (despite coding for over four
> >> decades, I'm still not comfortable with hex notation)
> >>
> >> I say it's not horrible, because bytes is a Sequence of bytes (or integer
> >> values between 0 and 255), initializing it with an iterable seems pretty
> >> reasonable, that's how we initialize most (all?) other sequences after all.
> >> And compatible with array.array and numpy arrays.
> >>  
> >
> > I consider bytes([31]) notation to be horrible API design because a simple
> > easy to make typo of omitting the [] or using () and forgetting the
> > tupleizing comma turns it into a different valid call with an entirely
> > different meaning.  bytes([31]) vs bytes((31)) vs bytes(31).
> >
> > It's also ugly to anyone who thinks about what bytecode is generated and
> > executed in order to do it.  an entire new list object with a single
> > element referring to a tiny int is created and destroyed just to create a
> > b'\037' object?  An optimizer pass to fix that up at the bytecode level
> > isn't easy as it can only be done when it can prove that `bytes` has not
> > been reassigned to something other than the builtin.  Near impossible in a
> > lot of code.  bytes.fromint(31) isn't much better in the bytecode regard,
> > but at least a temporary list is not being created.
> >
> > As much as I think that bytes(size: int) was a bad idea to have as an API
> > - bytearray(size: int) is fine and useful as it is mutable - that ship
> > sailed and getting rid of it would break some odd code.  It doesn't have
> > much use, so adding fromsize(size: int) methods don't sound very compelling
> > as it just adds yet another way to do the same thing.  we should just live
> > with that specific wart.
> >
> > `bchr` as a builtin... I'm with the others on saying no to any new builtin
> > that isn't expected to see frequent use.  bchr won't see frequent use.
> >
> > `bytes.fromint` seems fine.  others are proposing `bytes.byte` for that.
> > I don't *like* to argue over names (the last stage of anything) but I do
> > need to point out how that sounds to read.  It falls victim to API
> > stuttering.  "bytes dot byte" or "bytes byte" doesn't convey much to a
> > reader in English as the difference is a subtle "s".  "bytes dot from int"
> > or "bytes from int" is quite clear.  (avoiding stuttering in API design was
> > popularized by golang - it's a good thing to strive for in any language)
> > It's times like this that i wish Python had chosen consistent camelCase,
> > CapWords, or snake_case in all API names as conjoinedwords aren't great.
> > But they are sadly consistent with our past sins.
> >
> > One thing never mentioned in the PEP.  If you expect a primary use of the
> > fromint (aka bchr builtin that isn't going to happen) to be called on
> > constant values often.  Why are we adding name lookups and function calls
> > to this?  Why not address the elephant in the room and allow for decimal
> > values to be written as an escape sequence within bytes literals?
> >
> > b'\d31' for example to say "decimal byte 31".  Proposal: Only values 0-255
> > with no leading zero should

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Mark Shannon]

Hi Nick,

On 22/08/2021 4:51 am, Nick Coghlan wrote:



On Sun, 22 Aug 2021, 10:47 am Guido van Rossum, > wrote:


Hopefully anyone is still reading python-dev.

I'm going to try to summarize the differences between the two
proposals, even though Mark already did so in his PEP. But I'd like
to start by calling out the key point of contention.

Everything here is about locals() and f_locals in *function scope*.
(I use f_locals to refer to the f_locals field of frame objects as
seen from Python code.) And in particular, it is about what I'll
call "extra variables": the current CPython feature that you can add
*new* variables to f_locals that don't exist in the frame, for example:

def foo():
     x = 1
     locals()["y"] = 2  # or sys._getframe()["y"] = 2

My first reaction was to propose to drop this feature, but I realize
it's kind of important for debuggers to be able to execute arbitrary
code in function code -- assignments to locals should affect the
frame, but it should also be possible to create new variables (e.g.
temporaries). So I agree we should keep this.


I actually tried taking this feature out in one of the PEP 558 drafts, 
but actually doing so breaks the pdb test suite.





So apparently the key difference of opinion between Mark and Nick is
about f_locals, and what to do with extras. In Nick's proposal when
you reference f.f_locals twice in a row (for the same frame object
f), you get the same proxy object, whereas in Mark's proposal you
get a different object each time, but it doesn't matter, because the
proxy has no state other than a reference to the frame.


If PEP 558 is still giving that impression, I need to fix the wording - 
the proxy objects are ephemeral in both PEPs (the 558 text is slightly 
behind the implementation on that point, as the fast refs mapping is now 
stored on the frame object, so it only needs to be built once)


In Mark's proposal, if you assign a value to an extra variable, it
gets stored in a hidden dict field on the frame, and when you read
the proxy, the contents of that hidden dict field gets included.
This hidden dict lazily created on the first store to an extra
variable. (Mark shows pseudo-code to clarify this; the hidden dict
is stored as _extra_locals on the frame.)


PEP 558 works essentially the same way, the difference is that it uses 
the existing locals dict storage rather than adding new storage just for 
optimised frames.


In Nick's proposal, there's a cache on the frame that stores both
the extras and the proper variables. This cache can get out of sync
with the contents of the proper variables when some bytecode is
executed (for performance reasons we don't want the bytecode to keep
the cache up to date on every store), so there's an operation to
sync the frame cache (sync_frame_cache(), it's not defined in which
namespace this exists -- is it a builtin or in sys?).


It's an extra method on the proxy objects. You only need it if you keep 
an old proxy object around - if you always retrieve a new proxy object 
after executing Python code, that proxy will refresh the cache when it 
needs to.




Frankly the description in Nick's PEP is hard to follow -- I am not
100% sure what is meant by "the dynamic snapshot", and it's not
quite clear whether proper variables are copied into the cache (and
if so, why).


Aye, Mark was a bit quicker with his PEP than I anticipated, so I've 
incorporated the implementation improvements arising from his last round 
of comments, but the PEP text hasn't been updated yet.



Personally, I find Mark's proposed semantics for f_locals simpler --
there's no cache, only storage for extras, so there's nothing that
can get out of sync.


The wording in PEP 667 undersells the cost of that simplification:

"Code that uses PyEval_GetLocals() will continue to operate safely, but 
will need to be changed to use PyEval_Locals() to restore functionality."



Code that uses PyEval_GetLocals() will NOT continue to operate safely 
under PEP 667: all such code will raise an exception at runtime, and 
need to be rewritten to use a new API with different refcounting 
semantics. That's essentially all code that accesses the frame locals 
from C, since we don't offer supported APIs for that other than 
PyEval_GetLocals() (directly accessing the f_locals field on the frame 
object is only "supported" in a very loose sense of the word, although 
PEP 558 mostly keeps that working, too)


This means the real key difference between the two PEPs is that Mark is 
proposing a gratuitous compatibility break for PyEval_GetLocals() that 
also means that the algorithmic complexity characteristics of the proxy 
implementation will be completely off from those of a regular dict (e.g. 
len(proxy) will be O(n) in the number of variables defined on the fra

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Guido van Rossum]

On Mon, Aug 23, 2021 at 4:38 AM Mark Shannon  wrote:

> Hi Nick,
>
> On 22/08/2021 4:51 am, Nick Coghlan wrote:
>
> > If Mark's claim that PyEval_GetLocals() could not be fixed was true then
> > I would be more sympathetic to his proposal, but I know it isn't true,
> > because it still works fine in the PEP 558 implementation (it even
> > immediately sees changes made via proxies, and proxies see changes to
> > extra variables). The only truly unfixable public API is
> > PyFrame_LocalsToFast().
>
> You are making claims that seem inconsistent with each other.
> Namely, you are claiming that:
>
> 1. That the result of locals() is ephemeral.
> 2. That PyEval_GetLocals() returns a borrowed reference.
>
> This seems impossible, as you can't return a borrowed reference to
> an emphemeral object. That's just a pointer to freed memory.
>
> Do `locals()` and `PyEval_GetLocals()` behave differently?
>

That is my understanding, yes. in PEP 558 locals() returns a snapshot dict,
the Python-level f_locals property returns a fresh proxy that has no state
except a pointer to the frame, and PyEval_GetLocals() returns a borrowed
reference to the dict that's stored on the frame's C-level f_locals
attribute.

(In my "crazy" proposal all that is the same.)


> Is the result of `PyEval_GetLocals()` cached, but `locals()` not?
>

I wouldn't call it a cache -- deleting it would affect the semantics, not
just the performance. But yes, it returns a reference to an object that is
owned by the frame, just as it does in 3.10 and before.


> If that were the case, then it is a bit confusing, but could work.
>

Yes, see my "crazy" proposal.


> Would PyEval_GetLocals() be defined as something like this?
>
> (add _locals_cache attribute to the frame which is initialized to NULL).
>
> def PyEval_GetLocals():
>  frame._locals_cache attribute = locals()
>  return borrow(frame._locals_cache attribute)
>

Nah, the dict returned by PyEval_GetLocals() is stored in the frame's
C-level f_locals attribute, which is consulted by the Python-level f_locals
proxy -- primarily to store "extra" variables, but IIUC in Nick's latest
version it is also still used to cache by that proxy. Nick's locals() just
returns dict(sys._getframe().f_locals).


> None of this is clear (at least not to me) from PEP 558.
>

One problem with PEP 558 is that it's got too many words, and it's lacking
a section that crisply describes the semantics of the proposed
implementation. I've suggested to Nick that he add a section with
pseudo-code for the implementation, like you did in yours.

(PS, did you read my PS about what locals() should do in class scope when
__prepare__ returns a non-dict?)

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/54HKYGCFC4M76ZCWYVMG56BOGJNVACFM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Mark Shannon]

Hi Guido,

On 23/08/2021 3:53 pm, Guido van Rossum wrote:
On Mon, Aug 23, 2021 at 4:38 AM Mark Shannon > wrote:


Hi Nick,

On 22/08/2021 4:51 am, Nick Coghlan wrote:

 > If Mark's claim that PyEval_GetLocals() could not be fixed was
true then
 > I would be more sympathetic to his proposal, but I know it isn't
true,
 > because it still works fine in the PEP 558 implementation (it even
 > immediately sees changes made via proxies, and proxies see
changes to
 > extra variables). The only truly unfixable public API is
 > PyFrame_LocalsToFast().

You are making claims that seem inconsistent with each other.
Namely, you are claiming that:

1. That the result of locals() is ephemeral.
2. That PyEval_GetLocals() returns a borrowed reference.

This seems impossible, as you can't return a borrowed reference to
an emphemeral object. That's just a pointer to freed memory.

Do `locals()` and `PyEval_GetLocals()` behave differently?


That is my understanding, yes. in PEP 558 locals() returns a snapshot 
dict, the Python-level f_locals property returns a fresh proxy that has 
no state except a pointer to the frame, and PyEval_GetLocals() returns a 
borrowed reference to the dict that's stored on the frame's C-level 
f_locals attribute


Can we avoid describing the C structs in any of these PEPs?

It confuses readers having Python attributes and "C-level attributes"
(C struct fields?).
It also restricts the implementation unnecessarily.

(E.g. the PyFrameObject doesn't have a `f_locals` field in 3.11: 
https://github.com/python/cpython/blob/main/Include/cpython/frameobject.h#L7)




(In my "crazy" proposal all that is the same.)




Is the result of `PyEval_GetLocals()` cached, but `locals()` not?


I wouldn't call it a cache -- deleting it would affect the semantics, 
not just the performance. But yes, it returns a reference to an object 
that is owned by the frame, just as it does in 3.10 and before.


If that were the case, then it is a bit confusing, but could work.


Yes, see my "crazy" proposal.

Would PyEval_GetLocals() be defined as something like this?

(add _locals_cache attribute to the frame which is initialized to NULL).

def PyEval_GetLocals():
      frame._locals_cache attribute = locals()
      return borrow(frame._locals_cache attribute)


Nah, the dict returned by PyEval_GetLocals() is stored in the frame's 
C-level f_locals attribute, which is consulted by the Python-level 
f_locals proxy -- primarily to store "extra" variables, but IIUC in 
Nick's latest version it is also still used to cache by that proxy. 
Nick's locals() just returns dict(sys._getframe().f_locals).


The "extra" variables must be distinct from the result of locals() as
that includes both extras and "proper" variables.
If we want to cache the locals(), it needs to be distinct from the extra 
variables.


A debugger setting extra variables in a function that that is also 
accessed by a C call to PyEval_GetLocals() is going to be incredibly 
rare. Let's not worry about efficiency here.




None of this is clear (at least not to me) from PEP 558.


One problem with PEP 558 is that it's got too many words, and it's 
lacking a section that crisply describes the semantics of the proposed 
implementation. I've suggested to Nick that he add a section with 
pseudo-code for the implementation, like you did in yours.


(PS, did you read my PS about what locals() should do in class scope 
when __prepare__ returns a non-dict?)


Yes, but no harm in a reminder :)
I'll update my PEP to fix the semantics of locals().

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PA3ELKA4QLUV5VKEKUHZHRGT4P64EHGC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Guido van Rossum]

On Mon, Aug 23, 2021 at 8:46 AM Mark Shannon  wrote:

> Hi Guido,
>
> On 23/08/2021 3:53 pm, Guido van Rossum wrote:
> > On Mon, Aug 23, 2021 at 4:38 AM Mark Shannon  > > wrote:
> >
> > Hi Nick,
> >
> > On 22/08/2021 4:51 am, Nick Coghlan wrote:
> >
> >  > If Mark's claim that PyEval_GetLocals() could not be fixed was
> > true then
> >  > I would be more sympathetic to his proposal, but I know it isn't
> > true,
> >  > because it still works fine in the PEP 558 implementation (it even
> >  > immediately sees changes made via proxies, and proxies see
> > changes to
> >  > extra variables). The only truly unfixable public API is
> >  > PyFrame_LocalsToFast().
> >
> > You are making claims that seem inconsistent with each other.
> > Namely, you are claiming that:
> >
> > 1. That the result of locals() is ephemeral.
> > 2. That PyEval_GetLocals() returns a borrowed reference.
> >
> > This seems impossible, as you can't return a borrowed reference to
> > an emphemeral object. That's just a pointer to freed memory.
> >
> > Do `locals()` and `PyEval_GetLocals()` behave differently?
> >
> >
> > That is my understanding, yes. in PEP 558 locals() returns a snapshot
> > dict, the Python-level f_locals property returns a fresh proxy that has
> > no state except a pointer to the frame, and PyEval_GetLocals() returns a
> > borrowed reference to the dict that's stored on the frame's C-level
> > f_locals attribute
>
> Can we avoid describing the C structs in any of these PEPs?
>
> It confuses readers having Python attributes and "C-level attributes"
> (C struct fields?).
> It also restricts the implementation unnecessarily.
>
> (E.g. the PyFrameObject doesn't have a `f_locals` field in 3.11:
>
> https://github.com/python/cpython/blob/main/Include/cpython/frameobject.h#L7
> )
>

I'd be happy to. Nick's PEP still references it (and indeed it is very
confusing) and I took it from him. And honestly it would be nice to have a
specific short name for it, rather than circumscribing it with "an internal
dynamic snapshot stored on the frame object " :-)


> >
> > (In my "crazy" proposal all that is the same.)
>
> >
> > Is the result of `PyEval_GetLocals()` cached, but `locals()` not?
> >
> >
> > I wouldn't call it a cache -- deleting it would affect the semantics,
> > not just the performance. But yes, it returns a reference to an object
> > that is owned by the frame, just as it does in 3.10 and before.
> >
> > If that were the case, then it is a bit confusing, but could work.
> >
> >
> > Yes, see my "crazy" proposal.
> >
> > Would PyEval_GetLocals() be defined as something like this?
> >
> > (add _locals_cache attribute to the frame which is initialized to
> NULL).
> >
> > def PyEval_GetLocals():
> >   frame._locals_cache attribute = locals()
> >   return borrow(frame._locals_cache attribute)
> >
> >
> > Nah, the dict returned by PyEval_GetLocals() is stored in the frame's
> > C-level f_locals attribute, which is consulted by the Python-level
> > f_locals proxy -- primarily to store "extra" variables, but IIUC in
> > Nick's latest version it is also still used to cache by that proxy.
> > Nick's locals() just returns dict(sys._getframe().f_locals).
>
> The "extra" variables must be distinct from the result of locals() as
> that includes both extras and "proper" variables.
> If we want to cache the locals(), it needs to be distinct from the extra
> variables.
>

I don't care that much about caching locals(), but it seems we're bound to
cache any non-NULL result from PyEval_GetLocals(), since it returns a
borrowed reference. So they may be different things, with different
semantics, if we don't cache locals().


> A debugger setting extra variables in a function that that is also
> accessed by a C call to PyEval_GetLocals() is going to be incredibly
> rare. Let's not worry about efficiency here.
>

Agreed.

>
> > None of this is clear (at least not to me) from PEP 558.
> >
> >
> > One problem with PEP 558 is that it's got too many words, and it's
> > lacking a section that crisply describes the semantics of the proposed
> > implementation. I've suggested to Nick that he add a section with
> > pseudo-code for the implementation, like you did in yours.
> >
> > (PS, did you read my PS about what locals() should do in class scope
> > when __prepare__ returns a non-dict?)
>
> Yes, but no harm in a reminder :)
> I'll update my PEP to fix the semantics of locals().
>

I'm in suspense as to what semantics you chose. :-)

PS. Another point where you seem to have missed some detail is in the
mapping from (proper) variable names to "frame locals" -- you use
co_varnames, but that (currently, at least) doesn't include cells.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

[Python-Dev] New moderator

[Ethan Furman]

Greetings, all!

After petitioning the Steering Council I have been added as a moderator for the Python Dev mailing list.  I was already 
the moderator of four other Python lists.


My goal as a moderator is to have our forums be civil and within the CoC.  I do rely heavily on users to let me know if 
someone may be crossing the line.


For the record, I do feel I can be impartial even when I am the target of attacks -- I don't think I deserve any special 
treatment, so such things will be handled the same as if any other community member were the target.


--
~Ethan~
Moderator
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UXDQLENNZRPPTJ2AAVUUP2DPT2KRIEPG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Suspension: Marco Sulla

[Ethan Furman]

After reviewing the "Problems with dict subclassing performance" thread, I am suspending Marco Sulla from Python-Dev for 
a period of three months.  His behavior became blatantly inappropriate, and rather than apologize he continued taunting 
those that were trying to help him.


Marco took exception to Steven D'Aprano's post, but I see nothing in Steven's post in either style or substance that 
warrants any moderator attention.


--
~Ethan~
Moderator
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RHB4HHNARDVBYZFTSXGMFDIVED5SAYAH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: New moderator

[Terry Reedy]

On 8/23/2021 4:02 PM, Ethan Furman wrote:

After petitioning the Steering Council I have been added as a moderator 
for the Python Dev mailing list.  I was already the moderator of four 
other Python lists.


Thank you Ethan.  I think you have done better than I would have with 
python-list.


My goal as a moderator is to have our forums be civil and within the 
CoC.  I do rely heavily on users to let me know if someone may be 
crossing the line.


--
Terry Jan Reedy


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/F4ALR5JRKUY3SJ6ZHB4KC7YZGAWJ67X6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Gregory P. Smith]

Just adding a datapoint, searching our internal codebase at work, I only
found two things that I'd consider to be uses of PyEval_GetLocals() outside
of CPython itself:

https://github.com/earwig/mwparserfromhell/blob/develop/src/mwparserfromhell/parser/ctokenizer/tokenizer.c#L220
https://github.com/numba/numba/blob/master/numba/_dispatcher.cpp#L664

The bulk of uses are naturally with CPython's own ceval.c and similar.

-gps

On Mon, Aug 23, 2021 at 9:02 AM Guido van Rossum  wrote:

> On Mon, Aug 23, 2021 at 8:46 AM Mark Shannon  wrote:
>
>> Hi Guido,
>>
>> On 23/08/2021 3:53 pm, Guido van Rossum wrote:
>> > On Mon, Aug 23, 2021 at 4:38 AM Mark Shannon > > > wrote:
>> >
>> > Hi Nick,
>> >
>> > On 22/08/2021 4:51 am, Nick Coghlan wrote:
>> >
>> >  > If Mark's claim that PyEval_GetLocals() could not be fixed was
>> > true then
>> >  > I would be more sympathetic to his proposal, but I know it isn't
>> > true,
>> >  > because it still works fine in the PEP 558 implementation (it
>> even
>> >  > immediately sees changes made via proxies, and proxies see
>> > changes to
>> >  > extra variables). The only truly unfixable public API is
>> >  > PyFrame_LocalsToFast().
>> >
>> > You are making claims that seem inconsistent with each other.
>> > Namely, you are claiming that:
>> >
>> > 1. That the result of locals() is ephemeral.
>> > 2. That PyEval_GetLocals() returns a borrowed reference.
>> >
>> > This seems impossible, as you can't return a borrowed reference to
>> > an emphemeral object. That's just a pointer to freed memory.
>> >
>> > Do `locals()` and `PyEval_GetLocals()` behave differently?
>> >
>> >
>> > That is my understanding, yes. in PEP 558 locals() returns a snapshot
>> > dict, the Python-level f_locals property returns a fresh proxy that has
>> > no state except a pointer to the frame, and PyEval_GetLocals() returns
>> a
>> > borrowed reference to the dict that's stored on the frame's C-level
>> > f_locals attribute
>>
>> Can we avoid describing the C structs in any of these PEPs?
>>
>> It confuses readers having Python attributes and "C-level attributes"
>> (C struct fields?).
>> It also restricts the implementation unnecessarily.
>>
>> (E.g. the PyFrameObject doesn't have a `f_locals` field in 3.11:
>>
>> https://github.com/python/cpython/blob/main/Include/cpython/frameobject.h#L7
>> )
>>
>
> I'd be happy to. Nick's PEP still references it (and indeed it is very
> confusing) and I took it from him. And honestly it would be nice to have a
> specific short name for it, rather than circumscribing it with "an internal
> dynamic snapshot stored on the frame object " :-)
>
>
>> >
>> > (In my "crazy" proposal all that is the same.)
>>
>> >
>> > Is the result of `PyEval_GetLocals()` cached, but `locals()` not?
>> >
>> >
>> > I wouldn't call it a cache -- deleting it would affect the semantics,
>> > not just the performance. But yes, it returns a reference to an object
>> > that is owned by the frame, just as it does in 3.10 and before.
>> >
>> > If that were the case, then it is a bit confusing, but could work.
>> >
>> >
>> > Yes, see my "crazy" proposal.
>> >
>> > Would PyEval_GetLocals() be defined as something like this?
>> >
>> > (add _locals_cache attribute to the frame which is initialized to
>> NULL).
>> >
>> > def PyEval_GetLocals():
>> >   frame._locals_cache attribute = locals()
>> >   return borrow(frame._locals_cache attribute)
>> >
>> >
>> > Nah, the dict returned by PyEval_GetLocals() is stored in the frame's
>> > C-level f_locals attribute, which is consulted by the Python-level
>> > f_locals proxy -- primarily to store "extra" variables, but IIUC in
>> > Nick's latest version it is also still used to cache by that proxy.
>> > Nick's locals() just returns dict(sys._getframe().f_locals).
>>
>> The "extra" variables must be distinct from the result of locals() as
>> that includes both extras and "proper" variables.
>> If we want to cache the locals(), it needs to be distinct from the extra
>> variables.
>>
>
> I don't care that much about caching locals(), but it seems we're bound to
> cache any non-NULL result from PyEval_GetLocals(), since it returns a
> borrowed reference. So they may be different things, with different
> semantics, if we don't cache locals().
>
>
>> A debugger setting extra variables in a function that that is also
>> accessed by a C call to PyEval_GetLocals() is going to be incredibly
>> rare. Let's not worry about efficiency here.
>>
>
> Agreed.
>
> >
>> > None of this is clear (at least not to me) from PEP 558.
>> >
>> >
>> > One problem with PEP 558 is that it's got too many words, and it's
>> > lacking a section that crisply describes the semantics of the proposed
>> > implementation. I've suggested to Nick that he add a section with
>> > pseudo-code for the implementation, like you did in yours.
>> >
>> > (PS, did you read my PS about

[Python-Dev] Is anyone relying on new-bugs-announce/python-bugs-list/bugs.python.org summaries

[Ammar Askar]

Hey everyone,

As part of PEP 588, migrating bugs.python.org issues to Github, there
are two current mailing list related items that need a replacement or
need to be turned down.

1. Weekly summary emails with bug counts and issues from the week,
example: 
https://mail.python.org/archives/list/python-dev@python.org/thread/JRFJ4QH7TR35HFRQWOYPPCGOYRFAXK24/

2. Emails sent to the new-bugs-announce and python-bugs-list for new
issues and comments to existing issues.

I wanted to quickly gauge if anyone is relying on these mailing
lists/emails or if it's a convenient part of your workflow. Feel free
to chime in here or on the migration issues directly with your
use-case:

1. https://github.com/psf/gh-migration/issues/6
2. https://github.com/psf/gh-migration/issues/7

Thanks,
Ammar
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LEQ5G64OVOFGY63563XDPF52R5XAVAIY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Is anyone relying on new-bugs-announce/python-bugs-list/bugs.python.org summaries

[Terry Reedy]

On 8/23/2021 6:11 PM, Ammar Askar wrote:

Hey everyone,

As part of PEP 588, migrating bugs.python.org issues to Github, there
are two current mailing list related items that need a replacement or
need to be turned down.

1. Weekly summary emails with bug counts and issues from the week,
example: 
https://mail.python.org/archives/list/python-dev@python.org/thread/JRFJ4QH7TR35HFRQWOYPPCGOYRFAXK24/


Essential for my current non-IDLE contributions.  I get this every 
Friday about noon US Eastern and usually use it for a few hours of 
tracker triage later that day and maybe into Saterday.  I usually close 
at least one issue each week, as either rejected or fixed.



2. Emails sent to the new-bugs-announce and python-bugs-list for new
issues and comments to existing issues.

I wanted to quickly gauge if anyone is relying on these mailing
lists/emails or if it's a convenient part of your workflow. Feel free
to chime in here or on the migration issues directly with your
use-case:

1. https://github.com/psf/gh-migration/issues/6
2. https://github.com/psf/gh-migration/issues/7


--
Terry Jan Reedy

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3BPOSYMQQ52RA5KMWUZBHJV6DS7CCGWP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Is anyone relying on new-bugs-announce/python-bugs-list/bugs.python.org summaries

[Steven D'Aprano]

I rely heavily on the new-bugs-announce list, and the email sent by 
Roundup to people on the Nosy list of a task.

I don't care about the weekly summary emails.

I don't know what the python-bugs-list is used for.


-- 
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6J3AQMMRBTDMN6JVFOS3BCBQ52EKSE6Q/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Steven D'Aprano]

On Sat, Aug 21, 2021 at 05:46:52PM -0700, Guido van Rossum wrote:
> Hopefully anyone is still reading python-dev.

I am :-)

[...]
> Everything here is about locals() and f_locals in *function scope*. (I use
> f_locals to refer to the f_locals field of frame objects as seen from
> Python code.) And in particular, it is about what I'll call "extra
> variables": the current CPython feature that you can add *new* variables to
> f_locals that don't exist in the frame, for example:
> 
> def foo():
> x = 1
> locals()["y"] = 2  # or sys._getframe()["y"] = 2

I'm confused. I don't think it currently works, at least not in the 
sense that I understand "works" to mean. Sure, you can add a new key to 
the dict, but that doesn't add a new local variable:


>>> def spam():
... if False: y = 0  # Fool the compiler into treating y as a local.
... x = 0
... locals()['y'] = 1
... print(sys._getframe().f_locals)
... print(y)
... 
>>> spam()
{'x': 0}
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 6, in spam
UnboundLocalError: local variable 'y' referenced before assignment


Am I missing something? The above is in 3.9, has something changed in 
3.10 or am I just misunderstanding what you mean by "works"?

Using f_locals instead of locals() doesn't change the UnboundLocalError 
in my testing.

If you are not referring to new local variables, then I don't understand 
what these "extra variables" are or where they live in the current 
implementation.

A recent thread on Discuss is maybe relevant:

https://discuss.python.org/t/how-can-i-use-exec-in-functions-in-python3-6-similar-to-python2-7/10191

Would either of these two proposals re-enable exec to work inside 
functions as it used to?

I think Nick's PEP 558 does not, it wants to make it explicit that 
changes to locals will not be reflected in the local variables. Although 
Nick does refer to a "write back" strategy that apparently works *now*. 
That suggests:

- there currently is a trick to writing new local variables in the 
function namespace;

- and Nick's PEP will break it.

Am I correct?

Mark's PEP 667 says "However f.f_locals == f.f_locals will be True, and 
all changes to the underlying variables, by any means, will be always be 
visible" so I think that means it will allow exec to work as in 
Python2.x, at least if you pass sys._getframe().f_locals to exec instead 
of locals().

Perhaps we need an informational PEP to explain how local variables 
inside functions work now :-)



-- 
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CREZXKZWVSICQY4WHKUPXGJU67UUPQZ2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Steven D'Aprano]

On Fri, Aug 20, 2021 at 04:22:56PM +0100, Mark Shannon wrote:
> Hi all,
> 
> I have submitted PEP 667 as an alternative to PEP 558.
> https://www.python.org/dev/peps/pep-0667


Specification has a code snippet:

def test():
x = 1
l()['x'] = 2
l()['y'] = 4
l()['z'] = 5
y
print(locals(), x)

https://www.python.org/dev/peps/pep-0667/#id10

Wouldn't that attempt to resolve global y, rather than local y? Unless 
there is a change to the current behaviour of the compiler, I think you 
need to fool the compiler:

   if False: y = 0  # anywhere inside the function is okay


Open Issues says:

"there would be backwards compatibility issues when locals is assigned 
to a local variable or when passed to eval."

https://www.python.org/dev/peps/pep-0667/#id24

Is that eval meant to be exec? Or both eval and exec?


-- 
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DZMWW3ROSXSNWONOHAARPDHWLBLVLEFH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Guido van Rossum]

I apologize. This does indeed not work. What does work, however, is setting
a variable from a tracing function. There is magic in the tracing machinery
that writes back the variables to the frame when the tracing function
returns. This causes the bug referenced as [1] in both PEPs, and both
propose to fix the bug by  not writing things back that way, instead
writing back whenever a key in the proxy is set. The discussion is about
subtler differences between the proposals.

—Guido

On Mon, Aug 23, 2021 at 22:19 Steven D'Aprano  wrote:

> On Sat, Aug 21, 2021 at 05:46:52PM -0700, Guido van Rossum wrote:
> > Hopefully anyone is still reading python-dev.
>
> I am :-)
>
> [...]
> > Everything here is about locals() and f_locals in *function scope*. (I
> use
> > f_locals to refer to the f_locals field of frame objects as seen from
> > Python code.) And in particular, it is about what I'll call "extra
> > variables": the current CPython feature that you can add *new* variables
> to
> > f_locals that don't exist in the frame, for example:
> >
> > def foo():
> > x = 1
> > locals()["y"] = 2  # or sys._getframe()["y"] = 2
>
> I'm confused. I don't think it currently works, at least not in the
> sense that I understand "works" to mean. Sure, you can add a new key to
> the dict, but that doesn't add a new local variable:
>
>
> >>> def spam():
> ... if False: y = 0  # Fool the compiler into treating y as a
> local.
> ... x = 0
> ... locals()['y'] = 1
> ... print(sys._getframe().f_locals)
> ... print(y)
> ...
> >>> spam()
> {'x': 0}
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "", line 6, in spam
> UnboundLocalError: local variable 'y' referenced before assignment
>
>
> Am I missing something? The above is in 3.9, has something changed in
> 3.10 or am I just misunderstanding what you mean by "works"?
>
> Using f_locals instead of locals() doesn't change the UnboundLocalError
> in my testing.
>
> If you are not referring to new local variables, then I don't understand
> what these "extra variables" are or where they live in the current
> implementation.
>
> A recent thread on Discuss is maybe relevant:
>
>
> https://discuss.python.org/t/how-can-i-use-exec-in-functions-in-python3-6-similar-to-python2-7/10191
>
> Would either of these two proposals re-enable exec to work inside
> functions as it used to?
>
> I think Nick's PEP 558 does not, it wants to make it explicit that
> changes to locals will not be reflected in the local variables. Although
> Nick does refer to a "write back" strategy that apparently works *now*.
> That suggests:
>
> - there currently is a trick to writing new local variables in the
> function namespace;
>
> - and Nick's PEP will break it.
>
> Am I correct?
>
> Mark's PEP 667 says "However f.f_locals == f.f_locals will be True, and
> all changes to the underlying variables, by any means, will be always be
> visible" so I think that means it will allow exec to work as in
> Python2.x, at least if you pass sys._getframe().f_locals to exec instead
> of locals().
>
> Perhaps we need an informational PEP to explain how local variables
> inside functions work now :-)
>
>
>
> --
> Steve
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/CREZXKZWVSICQY4WHKUPXGJU67UUPQZ2/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-- 
--Guido (mobile)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WHK32PFNLULIRQMHPFIC3SETT2O6PMKV/
Code of Conduct: http://python.org/psf/codeofconduct/