[issue19070] In place operators of weakref.proxy() not returning self.

[Irit Katriel]

Change by Irit Katriel :


--
versions: +Python 3.10, Python 3.8, Python 3.9 -Python 2.6, Python 2.7, Python 
3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19070] In place operators of weakref.proxy() not returning self.

[shishkander]

shishkander added the comment:

This bug happens not just with operators, but with any call 
self.call_something()! For example, running this:

from __future__ import print_function
import weakref

class Test(object):
def method(self):
return type(self)

def test(obj):
print(type(obj), =, obj.method())

o = Test()
test(o)
test(weakref.proxy(o))


(also attached for convenience as t.py) 
produces in both python2.7 and python3.2:

$ python t.py 
class '__main__.Test' = class '__main__.Test'
type 'weakproxy' = class '__main__.Test'

--
nosy: +shishkander
Added file: http://bugs.python.org/file31983/t.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19070
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19070] In place operators of weakref.proxy() not returning self.

[Graham Dumpleton]

Graham Dumpleton added the comment:

@shishkander I can't see how what you are talking about has got anything to do 
with the issue with in place operators. The results from your test script are 
expected and normal.

What result are you expecting?

The one thing you cannot override in Python is what type() returns for an 
object. Thus is it completely normal for the weakref.proxy object to have a 
different type that what it wraps.

This is one of the reasons why in Python why should rarely ever do direct 
comparison of type objects. Instead you should use isinstance().

 import weakref
 class Test(object):
... pass
...
 test = Test()
 proxy = weakref.proxy(test)
 type(test)
class '__main__.Test'
 type(proxy)
type 'weakproxy'
 isinstance(test, Test)
True
 isinstance(proxy, Test)
True
 proxy.__class__
class '__main__.Test'

The isinstance() check will work because weakref.proxy will proxy __class__() 
method such that it returns the type of the wrapped object rather than of the 
proxy.

Now if your problem is with methods of wrapped objects which return self not 
having that self object some how automatically wrapped in another proxy, there 
isn't anything the proxy can be do about that. That is a situation where you as 
a user need to be careful about what you are doing. A way one can handle that 
is through derivation off a proxy object and override specific methods where 
you then in turn need to wrap the result, but I can see that easily becoming 
fragile when weakrefs are involved. Also, the weakref proxy in Python doesn't 
expose a class for doing that anyway.

One important thing to note is that where self is returned is returned by a 
normal method, it is still on you to have assigned the result to a variable so 
as to have started any possible problems. In the case of in place operators 
that is done under the covers by Python and you have no control over it. This 
is why the current behaviour as originally described is arguably broken as is 
breaks the expectations of what would logically happen for an in place operator 
when used via a proxy, something you have no control over.

So can you go back and explain what your specific problem is that you believe 
is the same issue as this bug report is, because so far I can't see any 
similarity based on your example code.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19070
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19070] In place operators of weakref.proxy() not returning self.

[shishkander]

shishkander added the comment:

@grahamd
In short, instead of this:
$ python t.py 
class '__main__.Test' = class '__main__.Test'
type 'weakproxy' = class '__main__.Test'

I expected to get this (note the second line):
class '__main__.Test' = class '__main__.Test'
type 'weakproxy' = type 'weakproxy'

I pass the proxy to the test() function, and I expect the argument inside the 
function to be a proxy. And that works - the type of obj in test() is 
weakproxy. 
However, the test function calls obj.method(), and I expected the method to get 
a proxy of obj as well, but *it does not* (the type of self is now Test)!

The reason I though this is the same bug is that your method doesn't work on a 
proxy any more!

def __iadd__(self, value):
  # self is not longer proxy! it's actually actual class instance now.
  ...

Now, if I call the method like this:
Test.method(weakref.proxy(o)), then obviously the self argument of the method 
is a proxy.

So, the crux of the matter is in how obj.method is handled inside Python 
interpreter. I think the current behavior is wrong, and if it is fixed, I'm 
certain your problem is fixed as well.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19070
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19070] In place operators of weakref.proxy() not returning self.

[Graham Dumpleton]

Graham Dumpleton added the comment:

The proxy is intended as a wrapper around an object, it is not intended to 
merge in some way with the wrapped object. The wrapped object shouldn't really 
ever be aware that it was being accessed via a proxy. Thus the expectation that 
the 'self' attribute of the methods of the wrapper object would actually be the 
proxy object is a strange one.

Can you explain why you need it to behave the way you are expecting? Also 
specifically indicate what requirement you have for needing the reference to 
the wrapped object to be a weakref?

Almost sounds a bit like you may be trying to use weakref.proxy in a way not 
intended.

It is technically possible to write an object proxy which would work how you 
are expecting, but the Python standard library doesn't provide an object proxy 
implementation to base such a thing on.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19070
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19070] In place operators of weakref.proxy() not returning self.

[shishkander]

shishkander added the comment:

@grahamd
Well, I read the PEP for weakproxy and python doc on weakref library, and I got 
an impression that weakproxy is really like a weak pointer in C++: i can pass 
it around as it was a strong pointer, and get an exception if actual object has 
been deleted. Well, but I see now that it was wrong.

My use case

If you are familiar with kazoo library, then what I am making is similar to 
KazooClient object. See here, for example:
https://kazoo.readthedocs.org/en/latest/basic_usage.html#connection-handling
The typical usage for my object is like this:

zk = KazooClient(hosts='127.0.0.1:2181')
zk.start()
# user *MUST* call zk.stop() otherwise there is a leak

Now I think it's not user-friendly to require the call to zk.stop(). I prefer 
the call to stop() to be called automatically from destructor of zk object 
whenever it goes out of scope. However, the background thread which is created 
in zk.start() also holds reference to zk object, and so __del__ is never 
called, the thread is never stopped, and the leak occurs.

Of course, I can do this to solve the problem above:

class Wrapper(object):
  def __init__(self, *args, **kwargs):
self.__obj = KazooClient(*args, **kwargs)
self.__obj.start()
  def __getattr__(self, attr):
return getattr(self.__obj, attr)
  def __del__(self):
self.__obj.stop()


But even this doesn't quite solve the problem of callbacks. To have callbacks 
from KazooClient, they have to be stored in Wrapper object instead, and 
KazooClient has to be given weak references to some special Wrapper method, 
which will call the real callback. When KazooClient executes callback, it 
operates on a weakproxy, but as soon as it calls wrapper.method() the weakproxy 
is transformed to actual instance, and that complicates things.

It still works with current weakref implementation, but it would be simpler and 
more flexible if the weakproxy was working as I expected, as then I don't have 
to use unnatural calls: Class.method(proxy, ...) instead of just 
proxy.method(...).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19070
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19070] In place operators of weakref.proxy() not returning self.

[Graham Dumpleton]

Graham Dumpleton added the comment:

The __del__() method is generally something to be avoided. As this is a design 
issue with how you are doing things, I would suggest you move the discussion to:

https://groups.google.com/forum/#!forum/comp.lang.python

You will no doubt get many suggestions there.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19070
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19070] In place operators of weakref.proxy() not returning self.

[Graham Dumpleton]

New submission from Graham Dumpleton:

When a weakref.proxy() is used to wrap a class instance which implements in 
place operators, when one applies the in place operator to the proxy, one could 
argue the variable holding the proxy should still be a reference to the proxy 
after the in place operator has been done. Instead the variable is replaced 
with the class instance the proxy was wrapping.

So for the code:

from __future__ import print_function

import weakref

class Class(object):
 def __init__(self, value):
 self.value = value
 def __iadd__(self, value):
 self.value += value
 return self

c = Class(1)

p = weakref.proxy(c)

print('p.value', p.value)
print('type(p)', type(p))

p += 1

print('p.value', p.value)
print('type(p)', type(p))

one gets:

$ python3.3 weakproxytest.py
p.value 1
type(p) class 'weakproxy'
p.value 2
type(p) class '__main__.Class'

One might expect type(p) at the end to still be class 'weakproxy'.

In the weakref.proxy() C code, all the operators are set up with preprocessor 
macros.

#define WRAP_BINARY(method, generic) \
static PyObject * \
method(PyObject *x, PyObject *y) { \
UNWRAP(x); \
UNWRAP(y); \
return generic(x, y); \
}

#define WRAP_TERNARY(method, generic) \
static PyObject * \
method(PyObject *proxy, PyObject *v, PyObject *w) { \
UNWRAP(proxy); \
UNWRAP(v); \
if (w != NULL) \
UNWRAP(w); \
return generic(proxy, v, w); \
}

These are fine for:

WRAP_BINARY(proxy_add, PyNumber_Add)
WRAP_BINARY(proxy_sub, PyNumber_Subtract)
WRAP_BINARY(proxy_mul, PyNumber_Multiply)
WRAP_BINARY(proxy_div, PyNumber_Divide)
WRAP_BINARY(proxy_floor_div, PyNumber_FloorDivide)
WRAP_BINARY(proxy_true_div, PyNumber_TrueDivide)
WRAP_BINARY(proxy_mod, PyNumber_Remainder)
WRAP_BINARY(proxy_divmod, PyNumber_Divmod)
WRAP_TERNARY(proxy_pow, PyNumber_Power)
WRAP_BINARY(proxy_lshift, PyNumber_Lshift)
WRAP_BINARY(proxy_rshift, PyNumber_Rshift)
WRAP_BINARY(proxy_and, PyNumber_And)
WRAP_BINARY(proxy_xor, PyNumber_Xor)
WRAP_BINARY(proxy_or, PyNumber_Or)

Because a result is being returned and the original is not modified.

Use of those macros gives the unexpected result for:

WRAP_BINARY(proxy_iadd, PyNumber_InPlaceAdd)
WRAP_BINARY(proxy_isub, PyNumber_InPlaceSubtract)
WRAP_BINARY(proxy_imul, PyNumber_InPlaceMultiply)
WRAP_BINARY(proxy_idiv, PyNumber_InPlaceDivide)
WRAP_BINARY(proxy_ifloor_div, PyNumber_InPlaceFloorDivide)
WRAP_BINARY(proxy_itrue_div, PyNumber_InPlaceTrueDivide)
WRAP_BINARY(proxy_imod, PyNumber_InPlaceRemainder)
WRAP_TERNARY(proxy_ipow, PyNumber_InPlacePower)
WRAP_BINARY(proxy_ilshift, PyNumber_InPlaceLshift)
WRAP_BINARY(proxy_irshift, PyNumber_InPlaceRshift)
WRAP_BINARY(proxy_iand, PyNumber_InPlaceAnd)
WRAP_BINARY(proxy_ixor, PyNumber_InPlaceXor)
WRAP_BINARY(proxy_ior, PyNumber_InPlaceOr)

This is because the macro returns the result from the API call, such as 
PyNumber_InPlaceAdd() where as it should notionally be returning 'proxy' so 
that the variable holding the weakref proxy instance is set to the proxy object 
again and not the result of the inner API call.

In changing this though there is a complication which one would have to deal 
with.

If the result of the inner API call such as PyNumber_InPlaceAdd() is the same 
as the original object wrapped by the weakref proxy, then all is fine.

What though should be done if it is different as notionally this means that the 
reference to the wrapped object would need to be changed to the new value.

The issue is that if one had to replace the reference to the wrapped object 
with a different one due to the in place operator, then notionally the whole 
existence of that weakref is invalidated as you would no longer be tracking the 
same object the weakref proxy was created for.

This odd situation is perhaps why the code was originally written the way it 
was, although that then sees the weakref proxy being replaced which could cause 
different problems with the callback not later being called since the weakref 
proxy can be destroyed before the object it wrapped. As there is nothing in the 
documentation of the code which calls out such a decision, not sure if it was 
deliberate or simply an oversight.

Overall I am not completely sure what the answer should be, so I am logging it 
as interesting behaviour. Maybe this odd case needs to be called out in the 
documentation in some way at least. That or in place operators simply shouldn't 
be allowed on a weakref proxy because of the issues it can cause either way.

--
components: Library (Lib)
messages: 198257
nosy: grahamd
priority: normal
severity: normal
status: open
title: In place operators of weakref.proxy() not returning self.
type: behavior
versions: Python 2.6, Python 2.7, Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19070
___

[issue19070] In place operators of weakref.proxy() not returning self.

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
nosy: +ncoghlan, tim.peters

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19070
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19070] In place operators of weakref.proxy() not returning self.

[Nick Coghlan]

Nick Coghlan added the comment:

Ouch :P

Perhaps the least-incompatible fix would be to switch to returning the proxy if 
the object returned is the proxied object (which would do the right thing for 
proxies to mutable objects), while preserving the current behaviour when the 
in-place operation returns a new object?

That behaviour is still somewhat broken, but there's no obviously non-broken 
behaviour in that case, and this broadly preserves compatibility with the 
status quo.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19070
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com