[Python-Dev] Re: importing does not dispatch to __builtins__.__getitem__ to lookup __import__

[Brett Cannon]

On Wed, Jul 14, 2021 at 10:17 AM Patrick Reader 
wrote:

> Hi there,
>
> I've noticed that, when a frame's __builtins__ is a subclass of dict with
> an overridden __getitem__ method, this overriden method is not used by the
> IMPORT_NAME instruction to lookup __import__ in the dictionary; it uses the
> lookup function of normal dictionaries (via _PyDict_GetItemIdWithError).
> This is contrary to the behaviour of the similar LOAD_BUILD_CLASS, as well
> as the typical name lookup semantics of LOAD_GLOBAL/LOAD_NAME, which all
> use PyDict_CheckExact for a "fast path" before defaulting to
> PyObject_GetItem, which is unexpected.
>
> Perhaps more seriously, if __builtins__ is not a dict at all, then it gets
> erroneously passed to some internal dict functions resulting in a
> mysterious SystemError ("Objects/dictobject.c:1440: bad argument to
> internal function") which, to me, indicates fragile behaviour that isn't
> supposed to happen.
>
> I'm not sure if this intended, so I didn't want to open an issue yet. It
> also seems a highly specific use case and changing it would probably cause
> a bit of a slow-down in module importing so is perhaps not worth fixing. I
> just wanted to ask here in case this issue had been documented anywhere
> before, and to check if it might actually be supposed to happen before
> opening a bug report.
>

At this point it's basically a feature request since it's been like that
for so long.


> I cannot find evidence that this behaviour has changed at all in recent
> history and it seems to be the same on the main branch as in 3.9.6.
>

I believe it's been like that for as long as I have worked on importlib
which is over a decade. The lookup also skips over any global definition of
__import__ and goes straight to __builtins__.


> A short demo of these things is attached.
>
> Links to relevant CPython code in v3.9.6:
>
> IMPORT_NAME:
> https://github.com/python/cpython/blob/v3.9.6/Python/ceval.c#L5179
>
> BUILD_CLASS:
> https://github.com/python/cpython/blob/v3.9.6/Python/ceval.c#L2316
>
> LOAD_NAME:
> https://github.com/python/cpython/blob/v3.9.6/Python/ceval.c#L2488
>
> LOAD_GLOBAL:
> https://github.com/python/cpython/blob/v3.9.6/Python/ceval.c#L2546
>
> Thanks,
>
> Patrick Reader
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZQMF6XC76J4APJPB3X6PGATG6CV5NN44/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LFF7RUZPYWPKQWB243IQLBIUEI3UKIK3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python3 OSF/1 support?

[Brett Cannon]

On Wed, Jul 14, 2021 at 9:36 AM Chris Johns  wrote:

> As someone who has ported Python 3 to another "less commonly used"
> system (RISC OS) I just do semi-regular updates rather than trying to
> keep it it totally up to date with "main".
>
> I have some vague recollection of there being talk about a "second tier"
> of systems where there is at least a maintainer and build bot, but I'm
> not sure what, it anything, came from that.
>

Nothing as of yet. People mentioned clarifying and cleaning up how we
define platform support, but no one has taken the time to try and tackle
that issue.

-Brett


>
> Cheers
>
> Chris
>
> On 14/07/2021 15:44, Senthil Kumaran wrote:
> > On Wed, Jul 14, 2021 at 04:30:33AM +, Jay K wrote:
> >> Hi. I have an Alpha/OSF machine up and running.
> >> It is little slow, but it works ok.
> >>
> >> I would like to run Python3 on it.
> >>
> >> I have it "compiling and working". It was easy enough so far.
> >>https://github.com/python/cpython/pull/27063
> >>
> >> Admitted, I haven't figured out what is happening with the
> posixsubprocess
> >> module.
> >>
> >> Yes? No? Maybe?
> >>
> >> I can possibly provide ongoing "support" (I don't expect it will be
> much) and a
> >> buildbot (if really required),
> >> if I can get it working a bit more, if this is approved, etc. I haven't
> looked
> > It is difficult to maintain support for less commonly used systems. If
> > you maintain it personally (like using a cron) and look at the results
> > over time, you could see the difficulty in maintaining the support.
> >
> > Personally, my vote is a -1 here. In the PR, another core-dev, Ronald
> > had commented that support for explicitly removed a few years ago.
> >
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/VSQD4TVVI42I5NMDZ4OE43ZAPJV5M66G/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VQYFKTLDLIO5AEH7GAMZEOQMONDWFN5Y/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] importing does not dispatch to __builtins__.__getitem__ to lookup __import__

[Patrick Reader]

Hi there,

I've noticed that, when a frame's __builtins__ is a subclass of dict with an 
overridden __getitem__ method, this overriden method is not used by the 
IMPORT_NAME instruction to lookup __import__ in the dictionary; it uses the 
lookup function of normal dictionaries (via _PyDict_GetItemIdWithError). This 
is contrary to the behaviour of the similar LOAD_BUILD_CLASS, as well as the 
typical name lookup semantics of LOAD_GLOBAL/LOAD_NAME, which all use 
PyDict_CheckExact for a "fast path" before defaulting to PyObject_GetItem, 
which is unexpected.

Perhaps more seriously, if __builtins__ is not a dict at all, then it gets 
erroneously passed to some internal dict functions resulting in a mysterious 
SystemError ("Objects/dictobject.c:1440: bad argument to internal function") 
which, to me, indicates fragile behaviour that isn't supposed to happen.

I'm not sure if this intended, so I didn't want to open an issue yet. It also 
seems a highly specific use case and changing it would probably cause a bit of 
a slow-down in module importing so is perhaps not worth fixing. I just wanted 
to ask here in case this issue had been documented anywhere before, and to 
check if it might actually be supposed to happen before opening a bug report.

I cannot find evidence that this behaviour has changed at all in recent history 
and it seems to be the same on the main branch as in 3.9.6.

A short demo of these things is attached.

Links to relevant CPython code in v3.9.6:

IMPORT_NAME: https://github.com/python/cpython/blob/v3.9.6/Python/ceval.c#L5179

BUILD_CLASS: https://github.com/python/cpython/blob/v3.9.6/Python/ceval.c#L2316

LOAD_NAME: https://github.com/python/cpython/blob/v3.9.6/Python/ceval.c#L2488

LOAD_GLOBAL: https://github.com/python/cpython/blob/v3.9.6/Python/ceval.c#L2546

Thanks,

Patrick Reader

class MyDict(dict):
# keep a reference around to avoid infinite recursion
print = print
dict = dict
def __getitem__(self, key):
self.print("getting:", key)
# Can't use super here because we'd have to keep a reference around instead of looking it up
# in __builtins__ (to prevent infinite recursion), but then there's no __class__ cell which
# breaks the lookup mechanism. Instead, just refer to dict by name
return self.dict.__getitem__(self, key)

__builtins__ = MyDict(vars(__builtins__))

int# prints "getting: int"
__import__ # prints "getting: __import__"
class X: pass  # prints "getting: __build_class__"
import math# does not print "getting: __import__" because it uses dictobject internal lookup



# try these individually in the Python shell, because they all error on their own

__builtins__ = "not a dictionary"

int# TypeError: string indices must be integers (because it's trying to do effectively `"not a dictionary"["int"]`)
__import__ # same error
class X: pass  # same error (trying to load __build_class__)
import math# SystemError: Objects/dictobject.c:1440: bad argument to internal function
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZQMF6XC76J4APJPB3X6PGATG6CV5NN44/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python3 OSF/1 support?

[Chris Johns]

As someone who has ported Python 3 to another "less commonly used" 
system (RISC OS) I just do semi-regular updates rather than trying to 
keep it it totally up to date with "main".


I have some vague recollection of there being talk about a "second tier" 
of systems where there is at least a maintainer and build bot, but I'm 
not sure what, it anything, came from that.


Cheers

Chris

On 14/07/2021 15:44, Senthil Kumaran wrote:

On Wed, Jul 14, 2021 at 04:30:33AM +, Jay K wrote:

Hi. I have an Alpha/OSF machine up and running.
It is little slow, but it works ok.

I would like to run Python3 on it.

I have it "compiling and working". It was easy enough so far.
   https://github.com/python/cpython/pull/27063

Admitted, I haven't figured out what is happening with the posixsubprocess
module.

Yes? No? Maybe?

I can possibly provide ongoing "support" (I don't expect it will be much) and a
buildbot (if really required),
if I can get it working a bit more, if this is approved, etc. I haven't looked

It is difficult to maintain support for less commonly used systems. If
you maintain it personally (like using a cron) and look at the results
over time, you could see the difficulty in maintaining the support.

Personally, my vote is a -1 here. In the PR, another core-dev, Ronald
had commented that support for explicitly removed a few years ago.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VSQD4TVVI42I5NMDZ4OE43ZAPJV5M66G/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python3 OSF/1 support?

[Guido van Rossum]

I'd like to add that probably the most economic solution for the OP is to
just stay on a fixed version of Python and not bother trying to catch up
with newer Python versions.

On Wed, Jul 14, 2021 at 7:47 AM Senthil Kumaran  wrote:

> On Wed, Jul 14, 2021 at 04:30:33AM +, Jay K wrote:
> > Hi. I have an Alpha/OSF machine up and running.
> > It is little slow, but it works ok.
> >
> > I would like to run Python3 on it.
> >
> > I have it "compiling and working". It was easy enough so far.
> >   https://github.com/python/cpython/pull/27063
> >
> > Admitted, I haven't figured out what is happening with the
> posixsubprocess
> > module.
> >
> > Yes? No? Maybe?
> >
> > I can possibly provide ongoing "support" (I don't expect it will be
> much) and a
> > buildbot (if really required),
> > if I can get it working a bit more, if this is approved, etc. I haven't
> looked
>
> It is difficult to maintain support for less commonly used systems. If
> you maintain it personally (like using a cron) and look at the results
> over time, you could see the difficulty in maintaining the support.
>
> Personally, my vote is a -1 here. In the PR, another core-dev, Ronald
> had commented that support for explicitly removed a few years ago.
>
> --
> Senthil
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/QQOYHF77AFRZZ3ZPYZDNIPBGC2AEE7HN/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ROKW2XNTLXNWCQREF5BOBOOTIOTNRKE2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python3 OSF/1 support?

[Senthil Kumaran]

On Wed, Jul 14, 2021 at 04:30:33AM +, Jay K wrote:
> Hi. I have an Alpha/OSF machine up and running.
> It is little slow, but it works ok.
> 
> I would like to run Python3 on it.
> 
> I have it "compiling and working". It was easy enough so far.
>   https://github.com/python/cpython/pull/27063
> 
> Admitted, I haven't figured out what is happening with the posixsubprocess
> module.
> 
> Yes? No? Maybe?
> 
> I can possibly provide ongoing "support" (I don't expect it will be much) and 
> a
> buildbot (if really required),
> if I can get it working a bit more, if this is approved, etc. I haven't looked

It is difficult to maintain support for less commonly used systems. If
you maintain it personally (like using a cron) and look at the results
over time, you could see the difficulty in maintaining the support.

Personally, my vote is a -1 here. In the PR, another core-dev, Ronald
had commented that support for explicitly removed a few years ago.

-- 
Senthil
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QQOYHF77AFRZZ3ZPYZDNIPBGC2AEE7HN/
Code of Conduct: http://python.org/psf/codeofconduct/