[issue35105] Document that CPython accepts "invalid" identifiers

[Andrei Kulakov]

Andrei Kulakov  added the comment:

In the last message I've said that according to __dict__ docs, anything in 
__dict__ is an attribute of respective obj. That's a bit too-strongly worded, 
the docs can be understood in the sense that anything that ends up in __dict__ 
via other mechanisms, such as dotted notation or setattr(), is an attribute.

Since direct manipulation of __dict__ is not prohibited, and no limits are set, 
AFAIK, on keys that can be used for __dict__, the more natural reading of the 
docs is that anything that can be directly set in __dict__ is also an attribute.

The only thing that would make a user doubt this reading is if he or she finds 
that getattr() cannot get non-string attrs, and going by its name, user would 
assume you can get any valid attrs using getattr().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[Andrei Kulakov]

Andrei Kulakov  added the comment:

It seems like the documentation is lacking and perhaps misleading in regard to 
attributes.

- anything hashable can be used as a key in an obj.__dict__: an int, a tuple, 
etc. According to __dict__ docs, all of those are attributes. There's no 
warning that this isn't recommended. Various tooling may break, for example 
dir() will break if keys are not comparable. inspect.getmembers() will break 
because it relies on dir(). Probably other tooling will break in some way -- 
this is just the two first things I tried.

Is the reason for allowing that, - only performance (obviously that's a strong 
enough reason in this case)? Or can this be useful in some other corner cases?

- setattr() requires a string but a string can be '1 2', '(1,2)', etc. The docs 
for setattr strongly imply that it's the same as dotted notation, but it's not. 
The non-identifier string attrs don't break dir() or anything in inspect module 
AFAICT. Should the docs discourage this usage? Should they suggest some use 
cases where this is useful?

In addition to just being confusing, I think this can create an impression for 
users that setattr() allows you to set 'private' or 'hidden' attrs, and setting 
attrs via __dict__ allows you to set even more 'private', 'top secret' attrs.

Since attributes are such a core concept in Python, it might be good to have a 
section that lays out all of these corner cases and reasons for them, so that 
it can be linked from docs for setattr(), __dict__, dir(), 
inspect.getmembers(), etc.

--
nosy: +andrei.avk

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[Roy Smith]

Roy Smith  added the comment:

Just as another edge case, type() can do the same thing:

Foo = type("Foo", (object,), {"a b": 1})
f = Foo()

for example, will create a class attribute named "a b".  Maybe this actually 
calls setattr() under the covers, but if it's going to be documented, it should 
be noted in both places.

--
nosy: +roysmith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[Windson Yang]

Windson Yang  added the comment:

I agreed with @Raymond Hettinger, I will update the PR from your suggestion if 
no other ideas in next week.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[Raymond Hettinger]

Raymond Hettinger  added the comment:

FWIW, the only restriction added by setattr() is that *name* must be a string.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[Raymond Hettinger]

Raymond Hettinger  added the comment:

I don't think we can mark this as an implementation detail for setattr(). The 
details are downstream and determined by the target object, not by setattr() 
itself.

Suggested wording:

'''
Note, setattr() attempts to update the object with the given attr/value pair.
Whether this succeeds and what its affect is is determined by the target object.
If an object's class defines `__slots__`, the attribute may not be writeable.
If an object's class defines property with a setter method, the *setattr()*
will trigger the setter method which may or may not actually write the 
attribute.
For objects that have a regular dictionary (which is the typical case), the
*setattr()* call can make any string keyed update allowed by the dictionary
including keys that aren't valid identifiers -- for example setattr(a, '1', 
'one')
will be the equivalent of vars()['1'] ='one'.
This issue has little to do with setattr() and is more related to the fact that 
instance dictionaries can hold any valid key. In a way, it is no different than 
a user writing a.__dict__['1'] = 'one'. That has always been allowed and the 
__dict__ attribute is documented as writeable, so a user is also allowed to 
write `a.dict = {'1': 'one'}.
'''

In short, we can talk about this in the setattr() docs but it isn't really a 
setattr() issue. Also, the behavior is effectively guaranteed by the other 
things users are allowed to do, so there is no merit in marking this as an 
implementation detail. Non-identifier keys can make it into an instance 
dictionary via multiple paths that are guaranteed to work.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[Windson Yang]

Change by Windson Yang :


--
keywords: +patch
pull_requests: +10494
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[Steven D'Aprano]

Steven D'Aprano  added the comment:

> Any ideas? Or I will create a PR in a week without 'CPython implementation 
> detail'

I don't think we want to give any stability guarantees for this. Perhaps 
we should explicitly state that this is not guaranteed behaviour and may 
change in the future.

I would be happy for it to be stated as an CPython implementation 
detail. If PyPy or any other implementation happen to duplicate it, 
we're not responsible for documenting that fact.

Please go ahead and make a PR.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[Windson Yang]

Windson Yang  added the comment:

Any ideas? Or I will create a PR in a week without 'CPython implementation 
detail'

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[Windson Yang]

Windson Yang  added the comment:

I try to create a PR for it. Should we add 'CPython implementation detail' at 
the document? Because this happens at cpython as well as pypy. BTW, where 
should we add the document? I have two choices.

* https://docs.python.org/3/reference/datamodel.html#object.__setattr__
* https://docs.python.org/3/library/functions.html#setattr

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[orlnub123]

orlnub123  added the comment:

I take back my previous suggestion, I agree that documenting it in setattr() 
(and **kwargs) is the way to go. It's obvious that you can assign anything to 
the __dict__, since it represents a dict, but setattr() is more ambiguous.
'Anything' was the key word for me here. For example you can assign ints to 
__dict__ and it won't complain but try to do the same with setattr()/getattr() 
and it results in an error.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Documenting something as an 'implementation detail' denies that it is a 
language feature and does not offer stability guarantees.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[Chris Jerdonek]

Chris Jerdonek  added the comment:

> In the pydev thread, Guido said "My feeling is that limiting it to strings is 
> fine, but checking those strings for resembling identifiers is pointless and 
> wasteful."

But in a later message, after additional discussion, he acknowledged there 
could be reasons to change and said, "we needn't rush this."

So if the docs do describe the current implementation, I think it should warn 
people that this behavior might not be subject to the same backwards 
compatibility guarantees as other documented behavior.

--
nosy: +chris.jerdonek

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[Ned Batchelder]

Ned Batchelder  added the comment:

This seems like a confusion of two things: identifiers are lexical elements of 
the language.  Attributes are not limited to identifiers.

We could add to the docs for setattr: "The attribute name does not have to be a 
valid identifier."   I don't know what the language guarantees about what 
strings are valid as attribute names.

--
nosy: +nedbat

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[STINNER Victor]

STINNER Victor  added the comment:

> I don't know the proper markup for these.

It's ".. impl-detail::". See for example:
https://docs.python.org/dev/library/codecs.html#standard-encodings

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[Terry J. Reedy]

Terry J. Reedy  added the comment:

It is an implementation detail that some people need to know, and that is very 
unlikely to change.  In the pydev thread, Guido said
"
My feeling is that limiting it to strings is fine, but checking those
strings for resembling identifiers is pointless and wasteful."

We occasionally document such things in a 'CPython implementation detail' note. 
 I don't know the proper markup for these.  At present, I think the note should 
be in setattr and **kwargs docs.

--
nosy: +terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[orlnub123]

orlnub123  added the comment:

The customizing attribute access section of the data model might be a suitable 
place.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[Windson Yang]

Windson Yang  added the comment:

I agreed we should document it, it' not obvious to me at least.

--
nosy: +Windson Yang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[STINNER Victor]

STINNER Victor  added the comment:

> I'd argue that it's an implementation detail. Documenting it might be nice as 
> some projects such as pytest do use it but I don't think it would make sense 
> in setattr() or getattr() since all they do (at least in this case) is 
> assign/retrieve from the __dict__. One thing to note is that __slots__ 
> doesn't accept them.

Maybe it's obvious to you, but the question is asked again and again, at least 
once per year. So it seems like we need to document it somewhere.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[orlnub123]

orlnub123  added the comment:

I'd argue that it's an implementation detail. Documenting it might be nice as 
some projects such as pytest do use it but I don't think it would make sense in 
setattr() or getattr() since all they do (at least in this case) is 
assign/retrieve from the __dict__. One thing to note is that __slots__ doesn't 
accept them.

--
nosy: +orlnub123

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
nosy: +pablogsal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
nosy: +xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35105] Document that CPython accepts "invalid" identifiers

[STINNER Victor]

New submission from STINNER Victor :

The Python 3 language has a strict definition for an identifier:
https://docs.python.org/dev/reference/lexical_analysis.html#identifiers

... but in practice, it's possible to create "invalid" identifiers using 
setattr().

Example with PyPy:

$ pypy
Python 2.7.13 (0e7ea4fe15e82d5124e805e2e4a37cae1a402d4b, Apr 12 2018, 14:50:12)
 class A: pass
 
 a=A()
 setattr(a, "1", 2)
 getattr(a, "1")
2

 vars(a)
{'1': 2}
 a.__dict__
{'1': 2}

 a.1
  File "", line 1
a.1
^
SyntaxError: invalid syntax


The exact definition of "identifiers" is a common question. Recent examples:

* bpo-25205
* [Python-Dev] Arbitrary non-identifier string keys when using **kwargs
  https://mail.python.org/pipermail/python-dev/2018-October/155435.html

It would be nice to document the answer. Maybe in the Langage Specification, 
maybe in the setattr() documentation, maybe in a FAQ, maybe everywhere?

--
assignee: docs@python
components: Documentation
messages: 328816
nosy: docs@python, serhiy.storchaka, steven.daprano, vstinner
priority: normal
severity: normal
status: open
title: Document that CPython accepts "invalid" identifiers
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com