[Python-Dev] Re: Understanding "is not safe" in typeobject.c

[Guido van Rossum]

That code is quite old. This comment tries to explain it:
```
/* Check that the use doesn't do something silly and unsafe like
   object.__new__(dict). To do this, we check that the
most derived base that's not a heap type is this type. */
```
I think you may have to special-case this and arrange for B.__new__() to be
called, like it or not.

(If you want us to change the code, please file a bpo bug report. I know
that's no fun, but it's the way to get the right people involved.)

On Mon, Feb 1, 2021 at 3:27 AM Phil Thompson via Python-Dev <
python-dev@python.org> wrote:

> Hi,
>
> I'm trying to understand the purpose of the check in tp_new_wrapper() of
> typeobject.c that results in the "is not safe" exception.
>
> I have the following class hierarchy...
>
> B -> A -> object
>
> ...where B and A are implemented in C. Class A has an implementation of
> tp_new which does a few context-specific checks before calling
> PyBaseObject_Type.tp_new() directly to actually create the object. This
> works fine.
>
> However I want to allow class B to be used with a Python mixin. A's
> tp_new() then has to do something similar to super().__new__(). I have
> tried to implement this by locating the type object after A in B's MRO,
> getting it's '__new__' attribute and calling it (using PyObject_Call())
> with B passed as the only argument. However I then get the "is not safe"
> exception, specifically...
>
> TypeError: object.__new__(B) is not safe, use B.__new__()
>
> I take the same approach for __init__() and that works fine.
>
> If I comment out the check in tp_new_wrapper() then everything works
> fine.
>
> So, am I doing something unsafe? If so, what?
>
> Or, is the check at fault in not allowing the case of a C extension type
> with its own tp_new?
>
> Thanks,
> Phil
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/HRGDEMURCJ5DSNEPMQPQR3R7VVDFA4ZX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/X5EDFSASK7RKYISS7MVMHHYWMRRUSNAM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Understanding "is not safe" in typeobject.c

[Greg Ewing]

On 2/02/21 12:13 am, Phil Thompson via Python-Dev wrote:

TypeError: object.__new__(B) is not safe, use B.__new__()


It's not safe because object.__new__ doesn't know about any
C-level initialisation that A or B need.

At the C level, there is always a *single* inheritance hierarchy.
The right thing is for B's tp_new to directly call A's tp_new,
which calls object's tp_new.

Don't worry about Python-level multiple inheritance; the
interpreter won't let you create an inheritance structure
that would mess this up.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IW3OX6Y324VSF4WLQHGA7EFHJQ6XEBH4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Understanding "is not safe" in typeobject.c

[Phil Thompson via Python-Dev]

On 01/02/2021 23:50, Greg Ewing wrote:

On 2/02/21 12:13 am, Phil Thompson via Python-Dev wrote:

TypeError: object.__new__(B) is not safe, use B.__new__()


It's not safe because object.__new__ doesn't know about any
C-level initialisation that A or B need.


But A.__new__ is calling object.__new__ and so can take care of its own 
needs after the latter returns.



At the C level, there is always a *single* inheritance hierarchy.


Why?


The right thing is for B's tp_new to directly call A's tp_new,
which calls object's tp_new.


I want my C-implemented class's __new__ to support cooperative 
multi-inheritance so my A class cannot assume that object.__new__ is the 
next in the MRO.


I did try to call the next-in-MRO's tp_new directly (rather that calling 
it's __new__ attribute) but that gave me recursion errors.



Don't worry about Python-level multiple inheritance; the
interpreter won't let you create an inheritance structure
that would mess this up.


Phil
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GZ2RF7TJ6MXDODPWCJB3PDC2Z3VDSQIQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Understanding "is not safe" in typeobject.c

[Phil Thompson via Python-Dev]

On 01/02/2021 19:06, Guido van Rossum wrote:

That code is quite old. This comment tries to explain it:
```
/* Check that the use doesn't do something silly and unsafe like
   object.__new__(dict). To do this, we check that the
most derived base that's not a heap type is this type. */
```


I understand what it is checking, but I don't understand why it is 
"silly and unsafe".


I think you may have to special-case this and arrange for B.__new__() 
to be

called, like it or not.


But it's already been called. The check fails when trying to 
subsequently call object.__new__().


(If you want us to change the code, please file a bpo bug report. I 
know

that's no fun, but it's the way to get the right people involved.)


Happy to do that but I first wanted to check if I was doing something 
"silly" - I'm still not sure.


Phil


On Mon, Feb 1, 2021 at 3:27 AM Phil Thompson via Python-Dev <
python-dev@python.org> wrote:


Hi,

I'm trying to understand the purpose of the check in tp_new_wrapper() 
of

typeobject.c that results in the "is not safe" exception.

I have the following class hierarchy...

B -> A -> object

...where B and A are implemented in C. Class A has an implementation 
of

tp_new which does a few context-specific checks before calling
PyBaseObject_Type.tp_new() directly to actually create the object. 
This

works fine.

However I want to allow class B to be used with a Python mixin. A's
tp_new() then has to do something similar to super().__new__(). I have
tried to implement this by locating the type object after A in B's 
MRO,
getting it's '__new__' attribute and calling it (using 
PyObject_Call())
with B passed as the only argument. However I then get the "is not 
safe"

exception, specifically...

TypeError: object.__new__(B) is not safe, use B.__new__()

I take the same approach for __init__() and that works fine.

If I comment out the check in tp_new_wrapper() then everything works
fine.

So, am I doing something unsafe? If so, what?

Or, is the check at fault in not allowing the case of a C extension 
type

with its own tp_new?

Thanks,
Phil
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/HRGDEMURCJ5DSNEPMQPQR3R7VVDFA4ZX/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZNJK6BJLXCMOOZNEDGNZZKT2YG4XUV57/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Understanding "is not safe" in typeobject.c

[Greg Ewing]

On 3/02/21 12:07 am, Phil Thompson wrote:

On 01/02/2021 23:50, Greg Ewing wrote:

At the C level, there is always a *single* inheritance hierarchy.


Why?


Because a C struct can only extend one other C struct.

I want my C-implemented class's __new__ to support cooperative 
multi-inheritance


I don't think this is possible. Here is what the C API docs have to
say about the matter:

---

Note

If you are creating a co-operative tp_new (one that calls a base type’s 
tp_new or __new__()), you must not try to determine what method to call 
using method resolution order at runtime. Always statically determine 
what type you are going to call, and call its tp_new directly, or via 
type->tp_base->tp_new. If you do not do this, Python subclasses of your 
type that also inherit from other Python-defined classes may not work 
correctly. (Specifically, you may not be able to create instances of 
such subclasses without getting a TypeError.)


---

(Source: https://docs.python.org/3.5/extending/newtypes.html)

This doesn't mean that your type can't be used in multiple inheritance,
just that __new__ methods in particular can't be cooperative.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KVUEGIRAXWPVD6BZLHPKUI5X7UBH3G2M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Understanding "is not safe" in typeobject.c

[Phil Thompson via Python-Dev]

On 02/02/2021 14:18, Greg Ewing wrote:

On 3/02/21 12:07 am, Phil Thompson wrote:

On 01/02/2021 23:50, Greg Ewing wrote:

At the C level, there is always a *single* inheritance hierarchy.


Why?


Because a C struct can only extend one other C struct.


Yes - I misunderstood what you meant by "at the C level".

I want my C-implemented class's __new__ to support cooperative 
multi-inheritance


I don't think this is possible. Here is what the C API docs have to
say about the matter:

---

Note

If you are creating a co-operative tp_new (one that calls a base
type’s tp_new or __new__()), you must not try to determine what method
to call using method resolution order at runtime. Always statically
determine what type you are going to call, and call its tp_new
directly, or via type->tp_base->tp_new. If you do not do this, Python
subclasses of your type that also inherit from other Python-defined
classes may not work correctly. (Specifically, you may not be able to
create instances of such subclasses without getting a TypeError.)

---

(Source: https://docs.python.org/3.5/extending/newtypes.html)

This doesn't mean that your type can't be used in multiple inheritance,
just that __new__ methods in particular can't be cooperative.


Thanks - that's fairly definitive, although I don't really understand 
why __new__ has this particular requirement.


Phil
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FWSIZUAGD4QRZQ2ZDKLE7MP4P76EIMKL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Understanding "is not safe" in typeobject.c

[Martin Teichmann]

Hi Phil, Hi List,

unfortunately you do not give enough code to reproduce what you are doing,
but just guessing roughly:

you say that you have a hierarchy like B -> A -> object, with B and A
implemented in C, and then want to use B with a mixin. Programmers with a
non-python background then often write

class MyClass(B, Mixin):
  "whatever"

this leads to an MRO of MyClass -> B -> Mixin -> A -> object. This is
horror if B and A are written in C, because suddenly B needs to do
something with Python code if it wants to have to do something with its
superclass Mixin, like creating a new object. I am just guessing that this
is what your code tries to do. And this is what the comment considers silly.

With

class MyClass(Mixin, B):
 "whatever"

there is no problem at all. We get an MRO of MyClass -> Mixin -> B -> A ->
object. There is no need for B to do anything special, being written in C
it already knows by itself how to construct A as well, no need to fiddle
with Python at all. In general, it is usually not necessary to deal with
super() in C code at all.

The statement that there is only single inheritance on the C level becomes
obvious once you look at the MRO: that one is always linear, it is always
effectively a single inheritance. This is also why you have to call super()
only once even if you have multiple superclasses: super() just follows the
effective single inheritance of the MRO.

Hope that helps.

Cheers

Martin
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WHTRIPKQMVCQPOYBFWRW6HI6KOBUYJU3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Understanding "is not safe" in typeobject.c

[Greg Ewing]

On 3/02/21 4:52 am, Phil Thompson wrote:

Thanks - that's fairly definitive, although I don't really understand 
why __new__ has this particular requirement.


The job of tp_new is to initialise the C struct. To do this,
it first has to initialise the fields of the struct it
inherits from, then initialise any fields of its own that
it adds, in that order.

Initialising the inherited fields must be done by calling
the tp_new for the struct that it inherits from. You don't
want to call the tp_new of some other class that might have
got inserted into the MRO, because you have no idea what
kind of C struct it expects to get.

Cooperative calling is a nice idea, but it requires rather
special conditions to make it work. All the methods must
have exactly the same signature, and it mustn't matter what
order they're called in. Those conditions don't apply to
__new__, especially at the C level where everything is much
more strict type-wise.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5AGXKE35LDQA6JP6EKXW7FUFR7YCC3RR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Understanding "is not safe" in typeobject.c

[Greg Ewing]

On 3/02/21 11:05 am, Martin Teichmann wrote:


     class MyClass(B, Mixin):
           "whatever"

this leads to an MRO of MyClass -> B -> Mixin -> A -> object.


If you do the tp_new stuff correctly at the C level, you can still
create such a class. The only limitation is that if Mixin has a __new__
method written in Python, it won't get called. So if Mixin needs to
do any initialisation, it will have to be in __init__, *and* all the
__init__ methods in the chain will need to be designed for
cooperative calling.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YOMFSCDKYYFNV4IRGFAMLQU7WQYNTO6C/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Understanding "is not safe" in typeobject.c

[Martin Teichmann]

Hi Greg, Hi List,

Greg wrote:

>  class MyClass(B, Mixin):
> >"whatever"
> >
> > this leads to an MRO of MyClass -> B -> Mixin -> A -> object.
>
> If you do the tp_new stuff correctly at the C level, you can still
> create such a class. The only limitation is that if Mixin has a __new__
> method written in Python, it won't get called.


Yes, it is indeed possible to write a class like this. But that would be,
as mentioned in the comment, silly. Now there is nothing bad with being
silly once in a while, especially while programming that cannot be avoided
at times, I am just warning to use such a construct in a wider context.

Because the limitation is actually much more general: if the Mixin has any
method written in Python, it won't get called, unless you do a lot of heavy
lifting. Now why do you write something in C at all? Two reasons: once for
speed, which we can effectively exclude here because the super() construct
is slow, or because you want to interact with a library written in another
language. Unless your library knows about cooperative inheritance, which is
very unlikely, the code from class B is not aware that it is not supposed
to call the method inherited from A, but the one from the Mixin instead.

So by writing

   class MyClass(B, Mixin):
"whatever"

I have major problems to understand what you actually want to achieve, what
you could not achieve by writing

class MyClass(Mixin, B):
"whatever"

Since one cannot achieve anything with the former that could not be
achieved by the latter, just with much less hassle, I would call the former
code silly.

That said, I might be wrong and there is indeed something one can achieve
with the former but not with the latter, I would be very pleased to hear
about this, so please come forward! (For sure, all is based on B and A
being written in C, not Python)

Cheers

Martin
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VQPMUJKMZ44MLUPULFERPLMM5IXEKFYE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Understanding "is not safe" in typeobject.c

[Phil Thompson via Python-Dev]

On 02/02/2021 23:08, Greg Ewing wrote:

On 3/02/21 4:52 am, Phil Thompson wrote:

Thanks - that's fairly definitive, although I don't really understand 
why __new__ has this particular requirement.


The job of tp_new is to initialise the C struct. To do this,
it first has to initialise the fields of the struct it
inherits from, then initialise any fields of its own that
it adds, in that order.


Understood.


Initialising the inherited fields must be done by calling
the tp_new for the struct that it inherits from. You don't
want to call the tp_new of some other class that might have
got inserted into the MRO, because you have no idea what
kind of C struct it expects to get.


I had assumed that some other magic in typeobject.c (eg. conflicting 
meta-classes) would have raised an exception before getting to this 
stage if there was a conflict.



Cooperative calling is a nice idea, but it requires rather
special conditions to make it work. All the methods must
have exactly the same signature, and it mustn't matter what
order they're called in. Those conditions don't apply to
__new__, especially at the C level where everything is much
more strict type-wise.


Thanks for the explanation.

Phil
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S5KRTD7M73SMBDADMMP5XM5CPT3BLGLD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Understanding "is not safe" in typeobject.c

[Greg Ewing]

On 3/02/21 10:35 pm, Phil Thompson wrote:

On 02/02/2021 23:08, Greg Ewing wrote:

you have no idea what
kind of C struct it expects to get.


I had assumed that some other magic in typeobject.c (eg. conflicting 
meta-classes) would have raised an exception before getting to this 
stage if there was a conflict.


"No idea" is probably an exaggeration -- there is checking
to ensure that all the C structs involved form a sequence
of extensions. But they still need to be initialised in the
right order, because __new__ methods for later ones assume
that the struct they're extending has already been
initialised.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/N54L3266FSMKEQPFPH3YS3XZ6GVJMGAU/
Code of Conduct: http://python.org/psf/codeofconduct/