Re: Design thought for callbacks
Hi all, I apologize for taking so long to reply, but neither my work schedule
nor the weather have been kind in the past week. That said, I've been thinking
long and hard about what everyone has said, and have decided that it would be
useful to write a wrap-up email that attempts to encapsulate everything
everyone has said, as a record of sorts if nothing else. As such, this email
is fairly long and involved.
===
Analysis of the problem
===
My original question was 'what is the least surprising/most pythonic way to
write a callback API?' Through reading what everyone has said, I realized that
I wasn't being specific enough, simply because callback APIs can be quite
different. At the very least, the following questions need to be answered:
1) When a callback is registered, does it replace the prior callback?
2) If more than one callback can be registered, is there an ordering to them?
3) Can a callback be registered more than once?
4) When and how are callbacks deregistered?
5) Who is responsible for maintaining a strong reference to the callback?
As far as I know, there isn't a standard method to indicate to the caller that
one callback replaces another one except via well-written documentation. My
personal feeling is that callbacks that replace other callbacks should be
properties of the library. By implementing a setter, getter, and deleter for
each callback, the library makes it obvious that there is one and only one
callback active at a time. The only difficulty is making sure the user knows
that the library retains the callback, but this is a documentation problem.
I realized that ordering could be a problem when I read through the
documentation to asyncio.call_soon(). It promises that callbacks will be
called in the order in which they were registered. However, there are cases
where the order doesn't matter. Registration in both of these cases is fairly
simple; the former appends the callback to a list, while the latter adds it to
a set. The list or set can be a property of the library, and registration is
simply a matter of either inserting or adding. But this brings up point 3; if
a callback can be registered at most once and ordering matters, then we need
something that is both a sequence and a set. Subclassing either (or both)
collections.abc.MutableSequence or collections.abc.MutableSet will lead to
confusion due to unexpected violations of PEP 3119
(https://www.python.org/dev/peps/pep-3119/). Once again, the only option
appears to be careful documentation.
Registration is only half the problem. The other half is determining when a
callback should be unregistered. Some callbacks are one-shots and are
automatically unregistered as soon as they are called. Others will be called
each time an event occurs until they are explicitly unregistered from the
library. Which happens is another design choice that needs to be carefully
documented.
Finally, we come to the part that started my original question; who retains the
callback. I had originally asked everyone if it would be surprising to store
callbacks as weak references. The idea was that unless someone else maintained
a strong reference to the callback, it would be garbage collected, which would
save users from 'surprising' results such as the following:
"""
#! /usr/bin/env python
class Callback_object(object):
def __init__(self, msg):
self._msg = msg
def callback(self, stuff):
print("From {0!s}: {1!s}".format(self._msg, stuff))
class Fake_library(object):
def __init__(self):
self._callbacks = list()
def register_callback(self, callback):
self._callbacks.append(callback)
def execute_callbacks(self):
for thing in self._callbacks:
thing('Surprise!')
if __name__ == "__main__":
cbo = Callback_object("Evil Zombie")
lib = Fake_library()
lib.register_callback(cbo.callback)
# Way later, after the user forgot all about the callback above
cbo = Callback_object("Your Significant Other")
lib.register_callback(cbo.callback)
# And finally getting around to running all those callbacks.
lib.execute_callbacks()
"""
However, as others pointed out using a weak reference could actually increase
confusion rather than decrease it. The problem is that if there is a reference
cycle elsewhere in the code, it is possible that the zombie object is still
alive when it is supposed to be dead. This will likely be difficult to debug.
In addition, different types of callables have different requirements in order
to correctly store weak references to them. Both Ian Kelly and Fabio Zadrozny
provided solutions to this, with Fabio providing a link to his code at
http://pydev.blogspot.com.br/2015/02/design-for-client-side-applications-in.html.
Solution to my problem in particular
After con
Re: Design thought for callbacks
On Mon, Mar 2, 2015 at 4:04 AM, Cem Karan wrote: > On Feb 26, 2015, at 2:54 PM, Ian Kelly wrote: >> On Feb 26, 2015 4:00 AM, "Cem Karan" wrote: >> > >> > >> > On Feb 26, 2015, at 12:36 AM, Gregory Ewing >> > wrote: >> > >> > > Cem Karan wrote: >> > >> I think I see what you're talking about now. Does WeakMethod >> > >> (https://docs.python.org/3/library/weakref.html#weakref.WeakMethod) >> > >> solve >> > >> this problem? >> > > >> > > Yes, that looks like it would work. >> > >> > >> > Cool! >> >> Sometimes I wonder whether anybody reads my posts. I suggested a solution >> involving WeakMethod four days ago that additionally extends the concept to >> non-method callbacks (requiring a small amount of extra effort from the >> client in those cases, but I think that is unavoidable. There is no way that >> the framework can determine the appropriate lifetime for a closure-based >> callback.) > > I apologize about taking so long to reply to everyone's posts, but I've been > busy at home. > > Ian, it took me a while to do some research to understand WHY what you were > suggesting was important; you're right about storing the object as well as > the method/function separately, but I think that WeakMethod might solve that > completely, correct? Are there any cases where WeakMethod wouldn't work? WeakMethod only works for bound method objects. If you pass it a non-method function, you'll get a TypeError: >>> from weakref import WeakMethod >>> WeakMethod(lambda: None) Traceback (most recent call last): File "", line 1, in File "/usr/lib/python3.4/weakref.py", line 49, in __new__ .format(type(meth))) from None TypeError: argument should be a bound method, not This check uses duck typing, so you could perhaps write a method-like class with __self__ and __func__ attributes and pass that to the WeakMethod constructor in the case of non-methods. There's a bigger issue with this however, which is that WeakMethod works by keeping weak references to *both* the object and the function, meaning that as soon as the function has no other references, the WeakMethod expires even if the object is still alive. This isn't a problem for methods because it's the transience of the method object, not the underlying function, that WeakMethod seeks to work around. But it doesn't by itself do anything to solve the problem of closures or lambdas that may themselves be transient. Revisiting the implementation I suggested previously, I want to make a correction. This would be better solved with a WeakValueDictionary: class Listenable: def __init__(self): self._callbacks = weakref.WeakValueDictionary() def listen(self, callback, owner=None): if owner is None: if isinstance(callback, types.MethodType): owner = weakref.WeakMethod(callback) else: owner = callback # TODO: Should anything happen if the callback is already in the dict? self._callbacks[callback] = owner def do_callbacks(self, message): for callback in self._callbacks.keys(): callback(message) This approach has two benefits over the previous one: it simplifies the callback management a bit, and it avoids making the assumption that the owner is hashable (it assumes instead that the callback is hashable, but I think that's reasonable). -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 26, 2015, at 2:54 PM, Ian Kelly wrote: > On Feb 26, 2015 4:00 AM, "Cem Karan" wrote: > > > > > > On Feb 26, 2015, at 12:36 AM, Gregory Ewing > > wrote: > > > > > Cem Karan wrote: > > >> I think I see what you're talking about now. Does WeakMethod > > >> (https://docs.python.org/3/library/weakref.html#weakref.WeakMethod) solve > > >> this problem? > > > > > > Yes, that looks like it would work. > > > > > > Cool! > > Sometimes I wonder whether anybody reads my posts. I suggested a solution > involving WeakMethod four days ago that additionally extends the concept to > non-method callbacks (requiring a small amount of extra effort from the > client in those cases, but I think that is unavoidable. There is no way that > the framework can determine the appropriate lifetime for a closure-based > callback.) I apologize about taking so long to reply to everyone's posts, but I've been busy at home. Ian, it took me a while to do some research to understand WHY what you were suggesting was important; you're right about storing the object as well as the method/function separately, but I think that WeakMethod might solve that completely, correct? Are there any cases where WeakMethod wouldn't work? Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 26, 2015, at 7:04 PM, Fabio Zadrozny wrote: > > On Wed, Feb 25, 2015 at 9:46 AM, Cem Karan wrote: > > On Feb 24, 2015, at 8:23 AM, Fabio Zadrozny wrote: > > > Hi Cem, > > > > I didn't read the whole long thread, but I thought I'd point you to what > > I'm using in PyVmMonitor (http://www.pyvmmonitor.com/) -- which may already > > cover your use-case. > > > > Take a look at the callback.py at > > https://github.com/fabioz/pyvmmonitor-core/blob/master/pyvmmonitor_core/callback.py > > > > And its related test (where you can see how to use it): > > https://github.com/fabioz/pyvmmonitor-core/blob/master/_pyvmmonitor_core_tests/test_callback.py > > (note that it falls back to a strong reference on simple functions -- > > i.e.: usually top-level methods or methods created inside a scope -- but > > otherwise uses weak references). > > That looks like a better version of what I was thinking about originally. > However, various people on the list have convinced me to stick with strong > references everywhere. I'm working out a possible API right now, once I have > some code that I can use to illustrate what I'm thinking to everyone, I'll > post it to the list. > > Thank you for showing me your code though, it is clever! > > Thanks, > Cem Karan > > Hi Cem, > > Well, I decided to elaborate a bit on the use-case I have and how I use it > (on a higher level): > http://pydev.blogspot.com.br/2015/02/design-for-client-side-applications-in.html > > So, you can see if it may be worth for you or not (I agree that sometimes you > should keep strong references, but for my use-cases, weak references usually > work better -- with the only exception being closures, which is handled > different anyways but with the gotcha of having to manually unregister it). As I mentioned in an earlier post, I've been quite busy at home, and expect to be for a few days to come, so I apologize both for being so late posting, and for not posting my own API plans. Your blog post has given me quite a bit to think about, thank you! Do you mind if I work up an API similar to yours? I'm planning on using a different license (not LGPL), which is why I ask. Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 26, 2015, at 3:00 PM, Ethan Furman wrote: > On 02/26/2015 11:54 AM, Ian Kelly wrote: > >> Sometimes I wonder whether anybody reads my posts. > > It's entirely possible the OP wasn't ready to understand your solution four > days ago, but two days later the OP was. Thank you Ethan, that was precisely my problem. Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Wed, Feb 25, 2015 at 9:46 AM, Cem Karan wrote: > > On Feb 24, 2015, at 8:23 AM, Fabio Zadrozny wrote: > > > Hi Cem, > > > > I didn't read the whole long thread, but I thought I'd point you to what > I'm using in PyVmMonitor (http://www.pyvmmonitor.com/) -- which may > already cover your use-case. > > > > Take a look at the callback.py at > https://github.com/fabioz/pyvmmonitor-core/blob/master/pyvmmonitor_core/callback.py > > > > And its related test (where you can see how to use it): > https://github.com/fabioz/pyvmmonitor-core/blob/master/_pyvmmonitor_core_tests/test_callback.py > (note that it falls back to a strong reference on simple functions -- i.e.: > usually top-level methods or methods created inside a scope -- but > otherwise uses weak references). > > That looks like a better version of what I was thinking about originally. > However, various people on the list have convinced me to stick with strong > references everywhere. I'm working out a possible API right now, once I > have some code that I can use to illustrate what I'm thinking to everyone, > I'll post it to the list. > > Thank you for showing me your code though, it is clever! > > Thanks, > Cem Karan Hi Cem, Well, I decided to elaborate a bit on the use-case I have and how I use it (on a higher level): http://pydev.blogspot.com.br/2015/02/design-for-client-side-applications-in.html So, you can see if it may be worth for you or not (I agree that sometimes you should keep strong references, but for my use-cases, weak references usually work better -- with the only exception being closures, which is handled different anyways but with the gotcha of having to manually unregister it). Best Regards, Fabio -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On 02/26/2015 11:54 AM, Ian Kelly wrote: > Sometimes I wonder whether anybody reads my posts. It's entirely possible the OP wasn't ready to understand your solution four days ago, but two days later the OP was. -- ~Ethan~ signature.asc Description: OpenPGP digital signature -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 26, 2015 4:00 AM, "Cem Karan" wrote: > > > On Feb 26, 2015, at 12:36 AM, Gregory Ewing wrote: > > > Cem Karan wrote: > >> I think I see what you're talking about now. Does WeakMethod > >> (https://docs.python.org/3/library/weakref.html#weakref.WeakMethod) solve > >> this problem? > > > > Yes, that looks like it would work. > > > Cool! Sometimes I wonder whether anybody reads my posts. I suggested a solution involving WeakMethod four days ago that additionally extends the concept to non-method callbacks (requiring a small amount of extra effort from the client in those cases, but I think that is unavoidable. There is no way that the framework can determine the appropriate lifetime for a closure-based callback.) -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 26, 2015, at 12:36 AM, Gregory Ewing wrote: > Cem Karan wrote: >> I think I see what you're talking about now. Does WeakMethod >> (https://docs.python.org/3/library/weakref.html#weakref.WeakMethod) solve >> this problem? > > Yes, that looks like it would work. Cool! Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Cem Karan wrote: I think I see what you're talking about now. Does WeakMethod (https://docs.python.org/3/library/weakref.html#weakref.WeakMethod) solve this problem? Yes, that looks like it would work. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 24, 2015, at 4:19 PM, Gregory Ewing wrote: > random...@fastmail.us wrote: >> On Tue, Feb 24, 2015, at 00:20, Gregory Ewing wrote: >>> This is why I suggested registering a listener object >>> plus a method name instead of a callback. It avoids that >>> reference cycle, because there is no long-lived callback >>> object keeping a reference to the listener. >> How does that help? Everywhere you would have had a reference to the >> "callback object", you now have a reference to the listener object. > > The point is that the library can keep a weak reference > to the listener object, whereas it can't reliably keep > a weak reference to a bound method. I think I see what you're talking about now. Does WeakMethod (https://docs.python.org/3/library/weakref.html#weakref.WeakMethod) solve this problem? Note that I can force my users to use the latest stable version of python at all times, so WeakMethod IS available to me. Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 24, 2015, at 8:23 AM, Fabio Zadrozny wrote: > Hi Cem, > > I didn't read the whole long thread, but I thought I'd point you to what I'm > using in PyVmMonitor (http://www.pyvmmonitor.com/) -- which may already cover > your use-case. > > Take a look at the callback.py at > https://github.com/fabioz/pyvmmonitor-core/blob/master/pyvmmonitor_core/callback.py > > And its related test (where you can see how to use it): > https://github.com/fabioz/pyvmmonitor-core/blob/master/_pyvmmonitor_core_tests/test_callback.py > (note that it falls back to a strong reference on simple functions -- i.e.: > usually top-level methods or methods created inside a scope -- but otherwise > uses weak references). That looks like a better version of what I was thinking about originally. However, various people on the list have convinced me to stick with strong references everywhere. I'm working out a possible API right now, once I have some code that I can use to illustrate what I'm thinking to everyone, I'll post it to the list. Thank you for showing me your code though, it is clever! Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
random...@fastmail.us wrote: On Tue, Feb 24, 2015, at 00:20, Gregory Ewing wrote: This is why I suggested registering a listener object plus a method name instead of a callback. It avoids that reference cycle, because there is no long-lived callback object keeping a reference to the listener. How does that help? Everywhere you would have had a reference to the "callback object", you now have a reference to the listener object. The point is that the library can keep a weak reference to the listener object, whereas it can't reliably keep a weak reference to a bound method. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
I'm combining two messages into one, On Feb 24, 2015, at 12:29 AM, random...@fastmail.us wrote: > On Tue, Feb 24, 2015, at 00:20, Gregory Ewing wrote: >> Cem Karan wrote: >>> I tend to structure my code as a tree or DAG of objects. The owner refers >>> to >>> the owned object, but the owned object has no reference to its owner. With >>> callbacks, you get cycles, where the owned owns the owner. >> >> This is why I suggested registering a listener object >> plus a method name instead of a callback. It avoids that >> reference cycle, because there is no long-lived callback >> object keeping a reference to the listener. > > How does that help? Everywhere you would have had a reference to the > "callback object", you now have a reference to the listener object. > You're just shuffling deck chairs around: if B shouldn't reference A > because A owns B, then removing C from the B->C->A reference chain does > nothing to fix this. On Feb 24, 2015, at 12:45 AM, Gregory Ewing wrote: > Cem Karan wrote: >> On Feb 22, 2015, at 5:15 AM, Gregory Ewing >> wrote: >>> Perhaps instead of registering a callback function, you should be >>> registering the listener object together with a method name. >> I see what you're saying, but I don't think it gains us too much. If I store >> an object and an unbound method of the object, or if I store the bound method >> directly, I suspect it will yield approximately the same results. > > It would be weird and unpythonic to have to register both > an object and an unbound method, and if you use a bound > method you can't keep a weak reference to it. Greg, random832 said what I was thinking earlier, that you've only increased the diameter of your cycle without actually fixing it. Can you give a code example where your method breaks the cycle entirely? Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 23, 2015, at 7:29 AM, "Frank Millman" wrote: > > "Cem Karan" wrote in message > news:a3c11a70-5846-4915-bb26-b23793b65...@gmail.com... >> >> >> Good questions! That was why I was asking about 'gotchas' with WeakSets >> originally. Honestly, the only way to know for sure would be to write two >> APIs for doing similar things, and then see how people react to them. The >> problem is, how do you set up such a study so it is statistically valid? >> > > Just in case you missed Steven's comment on my 'gotcha', and my reply, it is > worth repeating that what I reported as a gotcha was not what it seemed. > > If you set up the callback as a weakref, and the listening object goes out > of scope, it will wait to be garbage collected. However, as far as I can > tell, the weakref is removed at the same time as the object is gc'd, so > there is no 'window' where the weakref exists but the object it is > referencing does not exist. > > My problem was that I had performed a cleanup operation on the listening > object before letting it go out of scope, and it was no longer in a valid > state to deal with the callback, resulting in an error. If you do not have > that situation, your original idea may well work. Thank you Frank, I did read Steve's comment to your reply earlier, but what you said in your original reply made sense to me. I don't have control over user code. That means that if someone wants to write code such that they perform some kind of cleanup and are no longer able to handle the callback, they are free to do so. While I can't prevent this from happening, I can make it as obvious as possible in my code that before you perform any cleanup, you also need to unregister from the library. That is my main goal in developing pythonic/obvious methods of registering callbacks. Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Cem Karan wrote: I tend to structure my code as a tree or DAG of objects. The owner refers to the owned object, but the owned object has no reference to its owner. With callbacks, you get cycles, where the owned owns the owner. This is why I suggested registering a listener object plus a method name instead of a callback. It avoids that reference cycle, because there is no long-lived callback object keeping a reference to the listener. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Cem Karan wrote: > > On Feb 21, 2015, at 12:27 PM, Steven D'Aprano > wrote: >> The simplest possible identity-based scheme would be something like this: >> >> >> # don't hate me for using a global variable >> CALLBACKS = [] >> >> def register(func): >>if func not in CALLBACKS: >> CALLBACKS.append(func) >> >> def unregister(func): >>try: >>CALLBACKS.remove(func) >>except ValueError: >>pass Oops! That's not identity-based, that's *equality* based. Both the `in` operator and the list `remove` method implicitly perform equality checks, not identity checks. Which means that they will work with methods, since method equality compares against the underlying function, which is the same: py> class Spam(object): ... def method(self): ... pass ... py> s = Spam() py> a = s.method py> b = s.method py> a is b False py> a == b True py> a.__func__ is b.__func__ True So, when I say this: >> That's probably a bit too simple, since it won't behave as expected with >> bound methods. The problem is that bound methods are generated on the >> fly, so this won't work: I was mistaken. > Are you sure about that? I just tested out the following code, and it > appears to work correctly: You are correct. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
"Cem Karan" wrote in message news:a3c11a70-5846-4915-bb26-b23793b65...@gmail.com... > > > Good questions! That was why I was asking about 'gotchas' with WeakSets > originally. Honestly, the only way to know for sure would be to write two > APIs for doing similar things, and then see how people react to them. The > problem is, how do you set up such a study so it is statistically valid? > Just in case you missed Steven's comment on my 'gotcha', and my reply, it is worth repeating that what I reported as a gotcha was not what it seemed. If you set up the callback as a weakref, and the listening object goes out of scope, it will wait to be garbage collected. However, as far as I can tell, the weakref is removed at the same time as the object is gc'd, so there is no 'window' where the weakref exists but the object it is referencing does not exist. My problem was that I had performed a cleanup operation on the listening object before letting it go out of scope, and it was no longer in a valid state to deal with the callback, resulting in an error. If you do not have that situation, your original idea may well work. Frank -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 22, 2015, at 5:29 PM, Laura Creighton wrote: > In a message of Sun, 22 Feb 2015 17:09:01 -0500, Cem Karan writes: > >> Documentation is a given; it MUST be there. That said, documenting >> something, but still making it surprising, is a bad idea. For >> example, several people have been strongly against using a WeakSet to >> hold callbacks because they expect a library to hold onto callbacks. >> If I chose not to do that, and used a WeakSet, then even if I >> documented it, it would still end up surprising people (and from the >> sound of it, more people would be surprised than not). > >> Thanks, Cem Karan > > No matter what you do, alas, will surprise the hell out of people > because callbacks do not behave as people expect. Among people who > have used callbacks, what you are polling is 'what are people > familiar with', and it seems for the people around here, now, > WeakSets are not what they are familiar with. And that's fine. I know that regardless of what I do, some people are going to be surprised. I'm trying to develop APIs that reduce that surprise as far as possible. That means I can spend more time coding and less time answering questions... :) > But that is not so surprising. How many people use WeakSets for > _anything_? I've never used them, aside from 'ooh! cool shiny > new language feature! Let's kick it around the park!' That people > aren't familiar with WeakSets doesn't mean all that much. Actually, I use them when building caches of stuff, and I use weak references when I have trees of stuff so the child nodes know of, but don't hold onto, their parents. But I agree with you, there aren't a huge number of use-cases. > The question I have is does this architecture make things harder, > easier or about the same to debug? To write tests for? to do Test > Driven Design with? Good questions! That was why I was asking about 'gotchas' with WeakSets originally. Honestly, the only way to know for sure would be to write two APIs for doing similar things, and then see how people react to them. The problem is, how do you set up such a study so it is statistically valid? Cem -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Sun, Feb 22, 2015 at 7:22 AM, Cem Karan wrote: > > On Feb 22, 2015, at 5:15 AM, Gregory Ewing > wrote: > >> Frank Millman wrote: >>> "In order to inform users that certain bits of state have changed, I >>> require them to register a callback with my code." >>> This sounds to me like a pub/sub scenario. When a 'listener' object comes >>> into existence it is passed a reference to a 'controller' object that holds >>> state. It wants to be informed when the state changes, so it registers a >>> callback function with the controller. >> >> Perhaps instead of registering a callback function, you >> should be registering the listener object together with >> a method name. >> >> You can then keep a weak reference to the listener object, >> since if it is no longer referenced elsewhere, it presumably >> no longer needs to be notified of anything. > > I see what you're saying, but I don't think it gains us too much. If I store > an object and an unbound method of the object, or if I store the bound method > directly, I suspect it will yield approximately the same results. Well, it ties the weak ref to the lifetime of the object owning the callback rather than to the lifetime of the potentially unreferenced callback itself. I'm not fond of the scheme though because it forces the callback to be a method, and I'd prefer not to make that assumption. Also, I just noticed that Python 3.4 adds a weakref.WeakMethod class that solves the problem for the bound method case. That still leaves the closure and lambda cases, but here's a thought: add an optional argument to the callback registration method that specifies what object to tie the weak ref to. Something like: class Listenable: def __init__(self): self._callbacks = weakref.WeakKeyDictionary() def listen(self, callback, owner=None): if owner is None: if isinstance(callback, types.MethodType): owner = weakref.WeakMethod(callback) else: owner = callback self._callbacks.setdefault(owner, []).append(callback) def do_callbacks(self, message): for callbacks in self._callbacks.values(): for callback in callbacks: callback(message) -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Mon, Feb 23, 2015 at 9:29 AM, Laura Creighton wrote: > But that is not so surprising. How many people use WeakSets for > _anything_? I've never used them, aside from 'ooh! cool shiny > new language feature! Let's kick it around the park!' That people > aren't familiar with WeakSets doesn't mean all that much. I haven't used weak *sets*, but I've used weak *mappings* on occasion. It's certainly not a common thing, but they have their uses. I have a MUD which must guarantee that there be no more than one instance of any given room (identified by a string that looks like a Unix path), but which will, if it can, flush rooms out of memory when nothing refers to them. So it has a mapping from the path strings to the instances, but with weak refs for the instances; if anything else is referring to that instance (eg a player character in the room), it'll hang around, and any time anyone else needs that room, they'll get the same instance back from the mapping; but any time the garbage collector notices that a room can be disposed of, it will be. Definitely not common though. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
In a message of Sun, 22 Feb 2015 17:09:01 -0500, Cem Karan writes: >Documentation is a given; it MUST be there. That said, documenting >something, but still making it surprising, is a bad idea. For >example, several people have been strongly against using a WeakSet to >hold callbacks because they expect a library to hold onto callbacks. >If I chose not to do that, and used a WeakSet, then even if I >documented it, it would still end up surprising people (and from the >sound of it, more people would be surprised than not). >Thanks, Cem Karan No matter what you do, alas, will surprise the hell out of people because callbacks do not behave as people expect. Among people who have used callbacks, what you are polling is 'what are people familiar with', and it seems for the people around here, now, WeakSets are not what they are familiar with. But that is not so surprising. How many people use WeakSets for _anything_? I've never used them, aside from 'ooh! cool shiny new language feature! Let's kick it around the park!' That people aren't familiar with WeakSets doesn't mean all that much. The question I have is does this architecture make things harder, easier or about the same to debug? To write tests for? to do Test Driven Design with? Laura -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 22, 2015, at 4:34 PM, Marko Rauhamaa wrote: > Cem Karan : > >> My goal is to make things as pythonic (whatever that means in this >> case) and obvious as possible. Ideally, a novice can more or less >> guess what will happen with my API without really having to read the >> documentation on it. > > If you try to shield your user from the complexities of asynchronous > programming, you will only cause confusion. You will definitely need to > document all nooks and crannies of the semantics of the callback API and > your user will have to pay attention to every detail of your spec. > > Your user, whether novice or an expert, will thank you for your > unambiguous specification even if it is complicated. Documentation is a given; it MUST be there. That said, documenting something, but still making it surprising, is a bad idea. For example, several people have been strongly against using a WeakSet to hold callbacks because they expect a library to hold onto callbacks. If I chose not to do that, and used a WeakSet, then even if I documented it, it would still end up surprising people (and from the sound of it, more people would be surprised than not). Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Cem Karan : > My goal is to make things as pythonic (whatever that means in this > case) and obvious as possible. Ideally, a novice can more or less > guess what will happen with my API without really having to read the > documentation on it. If you try to shield your user from the complexities of asynchronous programming, you will only cause confusion. You will definitely need to document all nooks and crannies of the semantics of the callback API and your user will have to pay attention to every detail of your spec. Your user, whether novice or an expert, will thank you for your unambiguous specification even if it is complicated. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 22, 2015, at 4:02 PM, Ethan Furman wrote: > On 02/22/2015 05:13 AM, Cem Karan wrote: > >> Output: >> From Evil Zombie: Surprise! >> From Your Significant Other: Surprise! >> >> In this case, the user made an error (just as Marko said in his earlier >> message), >> and forgot about the callback he registered with the library. The callback >> isn't >> really rising from the dead; as you say, either its been garbage collected, >> or it >> hasn't been. However, you may not be ready for a callback to be called at >> that >> moment in time, which means you're surprised by unexpected behavior. > > But the unexpected behavior is not a problem with Python, nor with your > library -- it's a bug in the fellow-programmer's > code, and you can't (or at least shouldn't) try to prevent those kinds of > bugs from manifesting -- they'll just get > bitten somewhere else by the same bug. I agree with you, but until a relatively new programmer has gotten used to what callbacks are and what they imply, I want to make things easy. For example, if the API subclasses collections.abc.MutableSet, and the documentation states that you can only add callbacks to this particular type of set, then a new programmer will naturally decide that either a) they need to dispose of the set, and if that isn't possible, then b) they need to delete their callback from the set. It won't occur to them that their live object will just magically 'go away'; its a member of a set! My goal is to make things as pythonic (whatever that means in this case) and obvious as possible. Ideally, a novice can more or less guess what will happen with my API without really having to read the documentation on it. Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On 02/22/2015 05:13 AM, Cem Karan wrote: > Output: > From Evil Zombie: Surprise! > From Your Significant Other: Surprise! > > In this case, the user made an error (just as Marko said in his earlier > message), > and forgot about the callback he registered with the library. The callback > isn't > really rising from the dead; as you say, either its been garbage collected, > or it > hasn't been. However, you may not be ready for a callback to be called at > that > moment in time, which means you're surprised by unexpected behavior. But the unexpected behavior is not a problem with Python, nor with your library -- it's a bug in the fellow-programmer's code, and you can't (or at least shouldn't) try to prevent those kinds of bugs from manifesting -- they'll just get bitten somewhere else by the same bug. -- ~Ethan~ signature.asc Description: OpenPGP digital signature -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 22, 2015, at 5:15 AM, Gregory Ewing wrote: > Frank Millman wrote: >> "In order to inform users that certain bits of state have changed, I require >> them to register a callback with my code." >> This sounds to me like a pub/sub scenario. When a 'listener' object comes >> into existence it is passed a reference to a 'controller' object that holds >> state. It wants to be informed when the state changes, so it registers a >> callback function with the controller. > > Perhaps instead of registering a callback function, you > should be registering the listener object together with > a method name. > > You can then keep a weak reference to the listener object, > since if it is no longer referenced elsewhere, it presumably > no longer needs to be notified of anything. I see what you're saying, but I don't think it gains us too much. If I store an object and an unbound method of the object, or if I store the bound method directly, I suspect it will yield approximately the same results. Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 22, 2015, at 7:46 AM, Marko Rauhamaa wrote: > Cem Karan : > >> On Feb 21, 2015, at 12:08 PM, Marko Rauhamaa wrote: >>> Maybe the logic of the receiving object isn't prepared for the callback >>> anymore after an intervening event. >>> >>> The problem then, of course, is in the logic and not in the callbacks. >> >> This was PRECISELY the situation I was thinking about. My hope was to >> make the callback mechanism slightly less surprising by allowing the >> user to track them, releasing them when they aren't needed without >> having to figure out where the callbacks were registered. However, it >> appears I'm making things more surprising rather than less. > > When dealing with callbacks, my advice is to create your objects as > explicit finite state machines. Don't try to encode the object state > implicitly or indirectly. Rather, give each and every state a symbolic > name and log the state transitions for troubleshooting. > > Your callbacks should then consider what to do in each state. There are > different ways to express this in Python, but it always boils down to a > state/transition matrix. > > Callbacks sometimes cannot be canceled after they have been committed to > and have been shipped to the event pipeline. Then, the receiving object > must brace itself for the impending spurious callback. Nononono, I'm NOT encoding anything implicitly! As Frank mentioned earlier, this is more of a pub/sub problem. E.g., 'USB dongle has gotten plugged in', or 'key has been pressed'. The user code needs to decide what to do next, the library code provides a nice, clean interface to some potentially weird hardware. Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 22, 2015, at 7:52 AM, Laura Creighton wrote: > In a message of Sun, 22 Feb 2015 07:16:14 -0500, Cem Karan writes: > >> This was PRECISELY the situation I was thinking about. My hope was >> to make the callback mechanism slightly less surprising by allowing >> the user to track them, releasing them when they aren't needed >> without having to figure out where the callbacks were registered. >> However, it appears I'm making things more surprising rather than >> less. > > You may be able to accomplish your goal by using a Queue with a > producer/consumer model. > see: > http://stackoverflow.com/questions/9968592/turn-functions-with-a-callback-into-python-generators > > especially the bottom of that. > > I haven't run the code, but it looks mostly reasonable, except that > you do not want to rely on the Queue maxsize being 1 here, and > indeed, I almost always want a bigger Queue in any case. Use > Queue.task_done if blocking the producer features in your design. > > The problem that you are up against is that callbacks are inherantly > confusing, even to programmers who are learning about them for the > first time. They don't fit people's internal model of 'how code works'. > There isn't a whole lot one can do about that except to > try to make the magic do as little as possible, so that more of the > code works 'the way people expect'. I think what you're suggesting is that library users register a Queue instead of a callback, correct? The problem is that I'll then have a strong reference to the Queue, which means I'll be pumping events into it after the user code has gone away. I was hoping to solve the problem of forgotten registrations in the library. Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Steven D'Aprano : > I don't know about Java's Hotspot, but I do know that CPython's ref counting > garbage collector has at least one advantage over the GC used by Jython and > IronPython: unlike them, open files are closed as soon as they are no > longer in use. You can't depend on that kind of behavior. Dangling resources may or may not be cleaned up, ever. > Oh, a bit of trivia: Apple is abandoning their garbage collector and going > back to a reference counter: > > https://developer.apple.com/news/?id=02202015a > > Word on Reddit is that Apple is concerned about performance and battery > life. That truly is a bit OT here. > It's like explicitly closing a file, either with file.close() or a context > manager. Both methods are explicit. Closing files and other resources are not directly related to GC. Here's the thing: GC relieves your from dynamic memory management. You are still on your own when it comes to other resources. > We're not trying to scare beginners, we're a group of moderately > experienced coders discussing "best practice" (or at least "reasonable > practice") when using callbacks. Who mentioned beginners? I'm abiding by the same best practices I'm advocating. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Mon, Feb 23, 2015 at 12:45 AM, Steven D'Aprano wrote: >> No no no. It's the other way around. _Something_ has to be doing those >> callbacks, and it's that _something_ that should be keeping them >> alive. The fact that it's a registered callback should itself *be* a >> reference (and not a weak reference), and should keep it alive. > > That's much more reasonable than what you said earlier: > > it seems wrong to have to stash a thing in a bucket in order > to keep its callbacks alive. I expect the callbacks themselves to > keep it alive. > > > So yes. If I bind a callback to a button, say, or a listener, then the > button (or listener) keeps the callback alive, *not* the callback keeping > the button or listener alive. I meant the same thing, but my terminology was poor. Yes, that's correct; it's not any sort of magic about it being a callback, but more that the one you register it with becomes the owner of something. Hence, no weak references. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Marko Rauhamaa wrote:
> Chris Angelico :
>
>> On Sun, Feb 22, 2015 at 7:34 PM, Marko Rauhamaa wrote:
>>> Refloops are not to be worried about, let alone removed.
>>
>> Why?
>
> Because the whole point of GC-languages is that you should stop worrying
> about memory. Trying to mastermind and micromanage GC in the application
> is, pardon my French, an antipattern.
While it would be nice to be able to stop worrying about memory, try to
calculate 1000**1000**1000 and see how that works for you.
Garbage collection enables us to *mostly* automate the allocation and
deallocation of memory. If doesn't mean we can forget about it. GC is an
abstraction that frees us from most of the grunt work of allocating memory,
but it doesn't mean that there is never any need to think about memory. GC
is a leaky abstraction. Depending on the implementation, it may cause
distracting and annoying pauses in your application and/or resource leaks.
Even if there are no pauses, GC still carries a performance penalty. Good
programmers need to be aware of the limitations of their tools, and be
prepared to code accordingly.
When writing programs for educational purposes, we should try to code in the
simplest and most elegant way with no thought given to annoying practical
matters. At least at first. But when writing programs for actual use, we
should write for the implication we have, not the one we wish we had.
>> They force the use of the much slower cycle-detecting GC, rather than
>> the quick and efficient CPython refcounter.
>
> Java's Hotspot doesn't bother with refcounters but is much faster than
> Python. CPython's refcounters are a historical accident that a Python
> application developer shouldn't even be aware of.
I don't know about Java's Hotspot, but I do know that CPython's ref counting
garbage collector has at least one advantage over the GC used by Jython and
IronPython: unlike them, open files are closed as soon as they are no
longer in use. Code like this may run out of operating system file handles
in Jython:
i = 0
while True:
f = open('/tmp/x%d' % i)
i += 1
while CPython will just keep going. I suppose it will *eventually* run out
of some resource, but probably not file handles.
Oh, a bit of trivia: Apple is abandoning their garbage collector and going
back to a reference counter:
https://developer.apple.com/news/?id=02202015a
Word on Reddit is that Apple is concerned about performance and battery
life.
P.S. A reminder that reference counting *is* a form of garbage collection.
>> I don't know how other Pythons work, but mark-and-sweep has its own
>> costs, and I don't know of any system that's both prompt and able to
>> detect refloops.
>
> It's exceedingly difficult (and pointless) to detect cycles in your
> object structures. Python is going to have to do a GC occasionally
> anyway. Yes, your worst-case response times are going to suffer, but
> that's the cost of doing business.
In *general*, you're right. Who wants to spend all their time worrying about
cycles when the GC can do it for you? But if cycles are rare, and in known
parts of your code where it is simple to break them when you're done,
there's no disadvantage to doing so. Leave the GC for the hard cases.
It's like explicitly closing a file, either with file.close() or a context
manager. When using CPython, it doesn't really matter whether you close the
file or not, since the ref counter will normally close it automatically as
soon as the file goes out of scope. But it is cheap and easy to do so, so
why not do it? Then, when it otherwise would matter, say you are running
under Jython, it doesn't because you've closed the file.
>> Helping it along means your program doesn't waste memory. Why such a
>> blanket statement?
>
> Because worrying Python programmers with evil spirits (reference loops)
> leads to awkward coding practices and takes away one of the main
> advantages of Python as a high-level programming language.
I think you exaggerate a tad. We're not trying to scare beginners, we're a
group of moderately experienced coders discussing "best practice" (or at
least "reasonable practice") when using callbacks.
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Cem Karan : > You were saying that you move your objects into a zombie state. I > assumed that you meant you marked them in some manner (e.g., setting > 'is_zombie' to True), Yes, but even better: self.set_state(ZOMBIE) > so that anything that has a strong reference to the object knows the > object is not supposed to be used anymore. The other way round: the zombie object knows to ignore callbacks sent its way. It's not the responsibility of the sender to mind the receiver's internal state. I nowadays tend to implement states as inner classes. Here's how I've implemented the zombie state of one class: class Delivery...: def __init__(...): ... class ZOMBIE(STATE): def handle_connected(self): pass def handle_eof(self): pass def handle_response(self, code, response): pass def handle_io_error(self, errcode): pass def zombifie(self): assert False def transaction_timeout(self): assert False Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 22, 2015, at 7:24 AM, Chris Angelico wrote: > On Sun, Feb 22, 2015 at 11:07 PM, Cem Karan wrote: >>> Correct. The GUI engine ultimately owns everything. Of course, this is >>> a very simple case (imagine a little notification popup; you don't >>> care about it, you don't need to know when it's been closed, the only >>> event on it is "hit Close to destroy the window"), and most usage >>> would have other complications, but it's not uncommon for me to build >>> a GUI program that leaves everything owned by the GUI engine. >>> Everything is done through callbacks. Destroy a window, clean up its >>> callbacks. The main window will have an "on-deletion" callback that >>> terminates the program, perhaps. It's pretty straight-forward. >> >> How do you handle returning information? E.g., the user types in a number >> and expects that to update the internal state of your code somewhere. > > Not sure what you mean by "returning". If the user types in a number > in a GUI widget, that would trigger some kind of on-change event, and > either the new text would be a parameter to the callback function, or > the callback could query the widget. In the latter case, I'd probably > have the callback as a closure, and thus able to reference the object. We're thinking of the same thing. I try to structure what little GUI code I write using the MVP pattern (http://en.wikipedia.org/wiki/Model-view-presenter), so I have these hub and spoke patterns. But you're right, if you have a partially evaluated callback that has the presenter as one of the parameters, that would do it for a GUI. I was thinking more of a DAG of objects, but now that I think about it, callbacks wouldn't make sense in that case. Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Chris Angelico wrote: > On Sun, Feb 22, 2015 at 9:32 PM, Steven D'Aprano > wrote: >> Why? Do you expect that the Python garbage collector special cases >> callbacks to keep them alive even when there are no references to them? >> How would it distinguish a callback from some other function? > > No no no. It's the other way around. _Something_ has to be doing those > callbacks, and it's that _something_ that should be keeping them > alive. The fact that it's a registered callback should itself *be* a > reference (and not a weak reference), and should keep it alive. That's much more reasonable than what you said earlier: it seems wrong to have to stash a thing in a bucket in order to keep its callbacks alive. I expect the callbacks themselves to keep it alive. So yes. If I bind a callback to a button, say, or a listener, then the button (or listener) keeps the callback alive, *not* the callback keeping the button or listener alive. But if there are no references to the button, or the listener, then it will be garbage-collected, which will free the references to the callback and allow it to be garbage-collected as well (if there are no further references to it). -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 22, 2015, at 7:12 AM, Marko Rauhamaa wrote: > Cem Karan : > >> On Feb 21, 2015, at 11:03 AM, Marko Rauhamaa wrote: >>> I use callbacks all the time but haven't had any problems with strong >>> references. >>> >>> I am careful to move my objects to a zombie state after they're done so >>> they can absorb any potential loose callbacks that are lingering in the >>> system. >> >> So, if I were designing a library for you, you would be willing to have >> a 'zombie' attribute on your callback, correct? This would allow the >> library to query its callbacks to ensure that only 'live' callbacks are >> called. How would you handle closures? > > Sorry, don't understand the question. You were saying that you move your objects into a zombie state. I assumed that you meant you marked them in some manner (e.g., setting 'is_zombie' to True), so that anything that has a strong reference to the object knows the object is not supposed to be used anymore. That way, regardless of where or how many times you've registered your object for callbacks, the library can do something like the following (banged out in my mail application, may have typos): """ _CALLBACKS = [] def execute_callbacks(): global _CALLBACKS _CALLBACKS = [x for x in _CALLBACKS if not x.is_zombie] for x in _CALLBACKS: x() """ That will lazily unregister callbacks that are in the zombie state, which will eventually lead to their collection by the garbage collector. It won't work for anything that you don't have a reference for (lambdas, etc.), but it should work in a lot of cases. Is this what you meant? Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 21, 2015, at 3:57 PM, Grant Edwards wrote: > On 2015-02-21, Cem Karan wrote: >> >> On Feb 21, 2015, at 12:42 AM, Chris Angelico wrote: >> >>> On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan wrote: In order to inform users that certain bits of state have changed, I require them to register a callback with my code. The problem is that when I store these callbacks, it naturally creates a strong reference to the objects, which means that if they are deleted without unregistering themselves first, my code will keep the callbacks alive. Since this could lead to really weird and nasty situations, I would like to store all the callbacks in a WeakSet (https://docs.python.org/3/library/weakref.html#weakref.WeakSet). That way, my code isn't the reason why the objects are kept alive, and if they are no longer alive, they are automatically removed from the WeakSet, preventing me from accidentally calling them when they are dead. My question is simple; is this a good design? If not, why not? Are there any potential 'gotchas' I should be worried about? >>> >>> No, it's not. I would advise using strong references - if the callback >>> is a closure, for instance, you need to hang onto it, because there >>> are unlikely to be any other references to it. If I register a >>> callback with you, I expect it to be called; I expect, in fact, that >>> that *will* keep my object alive. >> >> OK, so it would violate the principle of least surprise for you. > > And me as well. I would expect to be able to pass a closure as a > callback and not have to keep a reference to it. Perhaps that just a > leftover from working with other languages (javascript, scheme, etc.). > It doesn't matter if it's a string, a float, a callback, a graphic or > whatever: if I pass your function/library an object, I expect _you_ to > keep track of it until you're done with it. > >> Interesting. Is this a general pattern in python? That is, >> callbacks are owned by what they are registered with? > > I'm not sure what you mean by "owned" or why it matters that it's a > callback: it's an object that was passed to you: you need to hold onto > a reference to it until you're done with it, and the polite thing to > do is to delete references to it when you're done with it. I tend to structure my code as a tree or DAG of objects. The owner refers to the owned object, but the owned object has no reference to its owner. With callbacks, you get cycles, where the owned owns the owner. As a result, if you forget where your object has been registered, it may be kept alive when you aren't expecting it. My hope was that with WeakSets I could continue to preserve the DAG or tree while still having the benefits of callbacks. However, it looks like that is too surprising to most people. Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 21, 2015, at 12:27 PM, Steven D'Aprano
wrote:
> Cem Karan wrote:
>
>>
>> On Feb 21, 2015, at 8:15 AM, Chris Angelico wrote:
>>
>>> On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan wrote:
OK, so it would violate the principle of least surprise for you.
Interesting. Is this a general pattern in python? That is, callbacks
are owned by what they are registered with?
In the end, I want to make a library that offers as few surprises to the
user as possible, and no matter how I think about callbacks, they are
surprising to me. If callbacks are strongly-held, then calling 'del
foo' on a callable object may not make it go away, which can lead to
weird and nasty situations.
>
> How?
>
> The whole point of callbacks is that you hand over responsibility to another
> piece of code, and then forget about your callback. The library will call
> it, when and if necessary, and when the library no longer needs your
> callback, it is free to throw it away. (If I wish the callback to survive
> beyond the lifetime of your library's use of it, I have to keep a reference
> to the function.)
Marko mentioned it earlier; if you think you've gotten rid of all references to
some chunk of code, and it is still alive afterwards, that can be surprising.
Weakly-held callbacks mean that I (as the
programmer), know that objects will go away after the next garbage
collection (see Frank's earlier message), so I don't get 'dead'
callbacks coming back from the grave to haunt me.
>
> I'm afraid this makes no sense to me. Can you explain, or better still
> demonstrate, a scenario where "dead callbacks rise from the grave", so to
> speak?
"""
#! /usr/bin/env python
class Callback_object(object):
def __init__(self, msg):
self._msg = msg
def callback(self, stuff):
print("From {0!s}: {1!s}".format(self._msg, stuff))
class Fake_library(object):
def __init__(self):
self._callbacks = list()
def register_callback(self, callback):
self._callbacks.append(callback)
def execute_callbacks(self):
for thing in self._callbacks:
thing('Surprise!')
if __name__ == "__main__":
foo = Callback_object("Evil Zombie")
lib = Fake_library()
lib.register_callback(foo.callback)
# Way later, after the user forgot all about the callback above
foo = Callback_object("Your Significant Other")
lib.register_callback(foo.callback)
# And finally getting around to running all those callbacks.
lib.execute_callbacks()
"""
Output:
>From Evil Zombie: Surprise!
>From Your Significant Other: Surprise!
In this case, the user made an error (just as Marko said in his earlier
message), and forgot about the callback he registered with the library. The
callback isn't really rising from the dead; as you say, either its been garbage
collected, or it hasn't been. However, you may not be ready for a callback to
be called at that moment in time, which means you're surprised by unexpected
behavior.
So, what's the consensus on the list, strongly-held callbacks, or
weakly-held ones?
>>>
>>> I don't know about Python specifically, but it's certainly a general
>>> pattern in other languages. They most definitely are owned, and it's
>>> the only model that makes sense when you use closures (which won't
>>> have any other references anywhere).
>>
>> I agree about closures; its the only way they could work.
>
> *scratches head* There's nothing special about closures. You can assign them
> to a name like any other object.
>
> def make_closure():
>x = 23
>def closure():
>return x + 1
>return closure
>
> func = make_closure()
>
> Now you can register func as a callback, and de-register it when your done:
>
> register(func)
> unregister(func)
>
>
> Of course, if you thrown away your reference to func, you have no (easy) way
> of de-registering it. That's no different to any other object which is
> registered by identity. (Registering functions by name is a bad idea, since
> multiple functions can have the same name.)
>
> As an alternative, your callback registration function might return a ticket
> for the function:
>
> ticket = register(func)
> del func
> unregister(ticket)
>
> but that strikes me as over-kill. And of course, the simplest ticket is to
> return the function itself :-)
Agreed on all points; closures are just ordinary objects. The only difference
(in my opinion) is that they are 'fire and forget'; if you are registering or
tracking them then you've kind of defeated the purpose. THAT is what I meant
about how you handle closures.
>
>> When I was
>> originally thinking about the library, I was trying to include all types
>> of callbacks, including closures and callable objects. The callable
>> objects may pass themselves, or one of their methods to the library, or
>> may do something really weird.
>
> I don't think they can do anything too weir
Re: Design thought for callbacks
In a message of Sun, 22 Feb 2015 07:16:14 -0500, Cem Karan writes: >This was PRECISELY the situation I was thinking about. My hope was >to make the callback mechanism slightly less surprising by allowing >the user to track them, releasing them when they aren't needed >without having to figure out where the callbacks were registered. >However, it appears I'm making things more surprising rather than >less. You may be able to accomplish your goal by using a Queue with a producer/consumer model. see: http://stackoverflow.com/questions/9968592/turn-functions-with-a-callback-into-python-generators especially the bottom of that. I haven't run the code, but it looks mostly reasonable, except that you do not want to rely on the Queue maxsize being 1 here, and indeed, I almost always want a bigger Queue in any case. Use Queue.task_done if blocking the producer features in your design. The problem that you are up against is that callbacks are inherantly confusing, even to programmers who are learning about them for the first time. They don't fit people's internal model of 'how code works'. There isn't a whole lot one can do about that except to try to make the magic do as little as possible, so that more of the code works 'the way people expect'. Laura -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Cem Karan : > On Feb 21, 2015, at 12:08 PM, Marko Rauhamaa wrote: >> Maybe the logic of the receiving object isn't prepared for the callback >> anymore after an intervening event. >> >> The problem then, of course, is in the logic and not in the callbacks. > > This was PRECISELY the situation I was thinking about. My hope was to > make the callback mechanism slightly less surprising by allowing the > user to track them, releasing them when they aren't needed without > having to figure out where the callbacks were registered. However, it > appears I'm making things more surprising rather than less. When dealing with callbacks, my advice is to create your objects as explicit finite state machines. Don't try to encode the object state implicitly or indirectly. Rather, give each and every state a symbolic name and log the state transitions for troubleshooting. Your callbacks should then consider what to do in each state. There are different ways to express this in Python, but it always boils down to a state/transition matrix. Callbacks sometimes cannot be canceled after they have been committed to and have been shipped to the event pipeline. Then, the receiving object must brace itself for the impending spurious callback. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Sun, Feb 22, 2015 at 11:07 PM, Cem Karan wrote: >> Correct. The GUI engine ultimately owns everything. Of course, this is >> a very simple case (imagine a little notification popup; you don't >> care about it, you don't need to know when it's been closed, the only >> event on it is "hit Close to destroy the window"), and most usage >> would have other complications, but it's not uncommon for me to build >> a GUI program that leaves everything owned by the GUI engine. >> Everything is done through callbacks. Destroy a window, clean up its >> callbacks. The main window will have an "on-deletion" callback that >> terminates the program, perhaps. It's pretty straight-forward. > > How do you handle returning information? E.g., the user types in a number > and expects that to update the internal state of your code somewhere. Not sure what you mean by "returning". If the user types in a number in a GUI widget, that would trigger some kind of on-change event, and either the new text would be a parameter to the callback function, or the callback could query the widget. In the latter case, I'd probably have the callback as a closure, and thus able to reference the object. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 21, 2015, at 12:08 PM, Marko Rauhamaa wrote: > Steven D'Aprano : > >> Other than that, I cannot see how calling a function which has *not* >> yet been garbage collected can fail, just because the only reference >> still existing is a weak reference. > > Maybe the logic of the receiving object isn't prepared for the callback > anymore after an intervening event. > > The problem then, of course, is in the logic and not in the callbacks. This was PRECISELY the situation I was thinking about. My hope was to make the callback mechanism slightly less surprising by allowing the user to track them, releasing them when they aren't needed without having to figure out where the callbacks were registered. However, it appears I'm making things more surprising rather than less. Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Cem Karan : > On Feb 21, 2015, at 11:03 AM, Marko Rauhamaa wrote: >> I use callbacks all the time but haven't had any problems with strong >> references. >> >> I am careful to move my objects to a zombie state after they're done so >> they can absorb any potential loose callbacks that are lingering in the >> system. > > So, if I were designing a library for you, you would be willing to have > a 'zombie' attribute on your callback, correct? This would allow the > library to query its callbacks to ensure that only 'live' callbacks are > called. How would you handle closures? Sorry, don't understand the question. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 21, 2015, at 11:03 AM, Marko Rauhamaa wrote: > Chris Angelico : > >> On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan wrote: > >>> In order to inform users that certain bits of state have changed, I >>> require them to register a callback with my code. The problem is that >>> when I store these callbacks, it naturally creates a strong reference >>> to the objects, which means that if they are deleted without >>> unregistering themselves first, my code will keep the callbacks >>> alive. Since this could lead to really weird and nasty situations, >>> [...] >> >> No, it's not. I would advise using strong references - if the callback >> is a closure, for instance, you need to hang onto it, because there >> are unlikely to be any other references to it. If I register a >> callback with you, I expect it to be called; I expect, in fact, that >> that *will* keep my object alive. > > I use callbacks all the time but haven't had any problems with strong > references. > > I am careful to move my objects to a zombie state after they're done so > they can absorb any potential loose callbacks that are lingering in the > system. So, if I were designing a library for you, you would be willing to have a 'zombie' attribute on your callback, correct? This would allow the library to query its callbacks to ensure that only 'live' callbacks are called. How would you handle closures? Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 21, 2015, at 10:55 AM, Chris Angelico wrote: > On Sun, Feb 22, 2015 at 2:45 AM, Cem Karan wrote: >> OK, so if I'm reading your code correctly, you're breaking the cycle in your >> object graph by making the GUI the owner of the callback, correct? No other >> chunk of code has a reference to the callback, correct? > > Correct. The GUI engine ultimately owns everything. Of course, this is > a very simple case (imagine a little notification popup; you don't > care about it, you don't need to know when it's been closed, the only > event on it is "hit Close to destroy the window"), and most usage > would have other complications, but it's not uncommon for me to build > a GUI program that leaves everything owned by the GUI engine. > Everything is done through callbacks. Destroy a window, clean up its > callbacks. The main window will have an "on-deletion" callback that > terminates the program, perhaps. It's pretty straight-forward. How do you handle returning information? E.g., the user types in a number and expects that to update the internal state of your code somewhere. Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
somebody, I got confused with the indent level wrote: >> They force the use of the much slower cycle-detecting GC, rather than >> the quick and efficient CPython refcounter. Somebody has misunderstood something here. When it comes to efficient garbage collectors, refcounting is a turtle. The CPython one is no exception. Ref counting, however, is fairly easy to write. But when the PyPy project first replaced its refcounting gc with its very first and therefore not very efficient at all nursery gc ... that was the very first time when a bunch of python programs ran faster on pypy than on CPython. This was before pypy had a JIT. And today the pypy channel is full of people who want to link their C extension into some Python code running on PyPy, and who find that their C extension slows things down. There are lots of reasons for this, but one of the most common problems is 'this C extension is faking refcounting. All of this is wasted effort for PyPy and usually makes the thing unJITable as well.' Many of these people rewrite their C extension as pure Python and find that then, with PyPy, they get the speed improvements they were looking for. So: two points. One reason you might not want to rely on ref counting, because you expect your code to run under PyPy one day. and If you are interested in manipulating garbage collection -- especially if this is for your own pleasure and enjoyment, a worthy goal in my books -- you could do a lot worse than write your own gc in RPython for PyPy. The gc code is not mixed in with all of the other VM stuff, so a gc is small, and you don't have to worry about clobbering anything else while you are working. So it is great for experimenting, which was the whole point. Hacking gcs is fun! :) Laura -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Sun, Feb 22, 2015 at 9:32 PM, Steven D'Aprano wrote: > Why? Do you expect that the Python garbage collector special cases callbacks > to keep them alive even when there are no references to them? How would it > distinguish a callback from some other function? No no no. It's the other way around. _Something_ has to be doing those callbacks, and it's that _something_ that should be keeping them alive. The fact that it's a registered callback should itself *be* a reference (and not a weak reference), and should keep it alive. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Chris Angelico wrote: > On Sun, Feb 22, 2015 at 3:38 PM, Steven D'Aprano > wrote: >> But you are using it. You might not be using it by name, but you are >> using it via the callback function. What did you expect, that Python >> should read your mind and somehow intuit that you still care about this >> socket listener, but not some other socket listener that you are done >> with? >> >> You don't have to bind the listener to a name. Any reference will do. You >> can dump it in a bucket: >> >> bucket_of_stuff = [] >> bucket_of_stuff.append(some_function(a, b, c)) >> bucket_of_stuff.append(make_web_server()) >> bucket_of_stuff.append(socket(23, on_accept=client_connected)) > > Sure, and whether it's a name or a list-element reference doesn't > matter: it seems wrong to have to stash a thing in a bucket in order > to keep its callbacks alive. I expect the callbacks _themselves_ to > keep it alive. Why? Do you expect that the Python garbage collector special cases callbacks to keep them alive even when there are no references to them? How would it distinguish a callback from some other function? If I stuff a function in a list: [len] would you expect the presence of the function to keep the list alive when there are no references to the list? Apart from "But I really, really, REALLY want a magical pony that feeds itself and never poops and just disappears when I don't want it around!" wishful-thinking, which I *totally* get, I don't see how you think this is even possible. Maybe I'm missing something, but it seems to me that what you're wishing for is impossible. Perhaps if we had a special "callback" type which was treated as a special case by the garbage collector. But that opens up a big can of worms: how do you close/delete objects which are kept alive by the presence of callbacks if you don't have a reference to either the object or the callback? -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Sun, Feb 22, 2015 at 8:14 PM, Marko Rauhamaa wrote: >> Helping it along means your program doesn't waste memory. Why such a >> blanket statement? > > Because worrying Python programmers with evil spirits (reference loops) > leads to awkward coding practices and takes away one of the main > advantages of Python as a high-level programming language. Right, and I suppose that, by extension, we should assume that the Python interpreter can optimize this? def fib(x): if x<2: return x return fib(x-2)+fib(x-1) Just because a computer can, in theory, recognize that this is a pure function, doesn't mean that we can and should depend on that. If you want this to be optimized, you either fix your algorithm or explicitly memoize the function - you don't assume that Python can do it for you. Even when you write in a high level language, you need to understand how computers work. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Frank Millman wrote: "In order to inform users that certain bits of state have changed, I require them to register a callback with my code." This sounds to me like a pub/sub scenario. When a 'listener' object comes into existence it is passed a reference to a 'controller' object that holds state. It wants to be informed when the state changes, so it registers a callback function with the controller. Perhaps instead of registering a callback function, you should be registering the listener object together with a method name. You can then keep a weak reference to the listener object, since if it is no longer referenced elsewhere, it presumably no longer needs to be notified of anything. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Chris Angelico : > On Sun, Feb 22, 2015 at 7:34 PM, Marko Rauhamaa wrote: >> Refloops are not to be worried about, let alone removed. > > Why? Because the whole point of GC-languages is that you should stop worrying about memory. Trying to mastermind and micromanage GC in the application is, pardon my French, an antipattern. > They force the use of the much slower cycle-detecting GC, rather than > the quick and efficient CPython refcounter. Java's Hotspot doesn't bother with refcounters but is much faster than Python. CPython's refcounters are a historical accident that a Python application developer shouldn't even be aware of. > I don't know how other Pythons work, but mark-and-sweep has its own > costs, and I don't know of any system that's both prompt and able to > detect refloops. It's exceedingly difficult (and pointless) to detect cycles in your object structures. Python is going to have to do a GC occasionally anyway. Yes, your worst-case response times are going to suffer, but that's the cost of doing business. > Helping it along means your program doesn't waste memory. Why such a > blanket statement? Because worrying Python programmers with evil spirits (reference loops) leads to awkward coding practices and takes away one of the main advantages of Python as a high-level programming language. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Sun, Feb 22, 2015 at 7:34 PM, Marko Rauhamaa wrote: > Chris Angelico : > >> Or (a very common case for me) a callback saying "remote end is gone" >> (eg on a socket) might wipe out the callbacks, thus removing their >> refloops. > > Refloops are not to be worried about, let alone removed. Why? They force the use of the much slower cycle-detecting GC, rather than the quick and efficient CPython refcounter. I don't know how other Pythons work, but mark-and-sweep has its own costs, and I don't know of any system that's both prompt and able to detect refloops. Helping it along means your program doesn't waste memory. Why such a blanket statement? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Chris Angelico : > Or (a very common case for me) a callback saying "remote end is gone" > (eg on a socket) might wipe out the callbacks, thus removing their > refloops. Refloops are not to be worried about, let alone removed. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Sun, Feb 22, 2015 at 6:52 PM, Marko Rauhamaa wrote: > What I mean, though, is that you shouldn't think you need to create > object destructors where you routinely set all members to None. Sure, not *routinely*. It'd be a special case where it's not specifically a destructor, and its job is to break a reference cycle. For instance, you might have a close() method that clears out a bunch of references, which will then allow everything to get cleaned up promptly. Or (a very common case for me) a callback saying "remote end is gone" (eg on a socket) might wipe out the callbacks, thus removing their refloops. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Steven D'Aprano : > Marko Rauhamaa wrote: >> Grant Edwards : >>> the polite thing to do is to delete references to it when you're done >>> with it. >> >> I disagree with that recommendation. You should do the natural thing and >> not care who holds references to who. > > I don't understand this. What is "the natural thing" if not to delete > references to an object when you are done with it? Normally you just let > things go out of scope, but if that won't happen, you have to take active > steps, such as calling del or setting the reference to None. Obviously, if you are hoarding objects in a collection, you're going to land in trouble. What I mean, though, is that you shouldn't think you need to create object destructors where you routinely set all members to None. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Sat, Feb 21, 2015 at 1:57 PM, Grant Edwards wrote: > On 2015-02-21, Cem Karan wrote: >> >> On Feb 21, 2015, at 12:42 AM, Chris Angelico wrote: >> >>> On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan wrote: In order to inform users that certain bits of state have changed, I require them to register a callback with my code. The problem is that when I store these callbacks, it naturally creates a strong reference to the objects, which means that if they are deleted without unregistering themselves first, my code will keep the callbacks alive. Since this could lead to really weird and nasty situations, I would like to store all the callbacks in a WeakSet (https://docs.python.org/3/library/weakref.html#weakref.WeakSet). That way, my code isn't the reason why the objects are kept alive, and if they are no longer alive, they are automatically removed from the WeakSet, preventing me from accidentally calling them when they are dead. My question is simple; is this a good design? If not, why not? Are there any potential 'gotchas' I should be worried about? >>> >>> No, it's not. I would advise using strong references - if the callback >>> is a closure, for instance, you need to hang onto it, because there >>> are unlikely to be any other references to it. If I register a >>> callback with you, I expect it to be called; I expect, in fact, that >>> that *will* keep my object alive. >> >> OK, so it would violate the principle of least surprise for you. > > And me as well. I would expect to be able to pass a closure as a > callback and not have to keep a reference to it. Perhaps that just a > leftover from working with other languages (javascript, scheme, etc.). > It doesn't matter if it's a string, a float, a callback, a graphic or > whatever: if I pass your function/library an object, I expect _you_ to > keep track of it until you're done with it. > >> Interesting. Is this a general pattern in python? That is, >> callbacks are owned by what they are registered with? > > I'm not sure what you mean by "owned" or why it matters that it's a > callback: it's an object that was passed to you: you need to hold onto > a reference to it until you're done with it, and the polite thing to > do is to delete references to it when you're done with it. > >> So, what's the consensus on the list, strongly-held callbacks, or >> weakly-held ones? Count me in the weak-ref crowd. It may be a nuisance to keep a reference around on the object registering the callback, but it's preferable to the alternative of messing around with disposables in order to ensure that the callback gets cleaned up and doesn't create a memory leak. I would also rather have my code fail by losing a callback reference, which should be relatively easy to spot and diagnose, than to have said memory leak go unnoticed. -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
"Steven D'Aprano" wrote in message news:54e8af1b$0$12976$c3e8da3$54964...@news.astraweb.com... > Frank Millman wrote: > >> I tried something similar a while ago, and I did find a gotcha. >> >> The problem lies in this phrase - "if they are no longer alive, they are >> automatically removed from the WeakSet, preventing me from accidentally >> calling them when they are dead." >> >> I found that the reference was not removed immediately, but was waiting >> to >> be garbage collected. During that window, I could call the callback, >> which >> resulted in an error. > > I don't understand how this could possibly work. (Or fail to work, as the > case may be.) > > If the callback has been garbage collected, then you cannot call it, > because > you don't have any references to it and so cannot refer to it in any way. > > If the callback has *not* been garbage collected, then you can safely call > it. You have a reference to the callback, therefore it exists. (If Python > ever garbage collects an object that still has references to it, that > would > be a critical bug, and you would likely get some sort of seg fault). > > The only thing I can think of is you have a situation where your callback > refers to another object, B, via a weak reference. Once all the regular > strong references to the callback and B are gone, theoretically you could > have a race condition where the callback is waiting to be garbage > collected > but B has already been garbage collected. If, in that window, you call the > callback, *and* if the callback fails to correctly check that the weak > reference to B still exists, then you could get a Python exception. > > The solution is simple: anytime you have a weak reference, you must always > check it before you use it. > > Other than that, I cannot see how calling a function which has *not* yet > been garbage collected can fail, just because the only reference still > existing is a weak reference. > You are right. I tried to reproduce the problem and I can't. Before describing what I think was happening, I want to clarify something. Most of this thread uses the word 'callback' in the sense of an 'asynchronous' scenario - the caller wants something to happen some time in the future, and then forget about it, but it is important that it does actually happen eventually. That is not what I was doing, and it is not what I thought the OP was asking for. "In order to inform users that certain bits of state have changed, I require them to register a callback with my code." This sounds to me like a pub/sub scenario. When a 'listener' object comes into existence it is passed a reference to a 'controller' object that holds state. It wants to be informed when the state changes, so it registers a callback function with the controller. When the controller detects a change in state, it calls all the callback functions, thereby notifying each listener. When the listener goes out of scope, it is important that it deregisters with the controller. Now back to my scenario. You are right that so long as the controller maintains a reference to the callback function, the listener cannot be garbage collected, and therefore the callback will always succeed. As far as I can remember, I had a situation where the listener used the information to pass the information to a gui on a client. When the listener was no longer required, a close() fiunction was called which cleaned up and closed connections. When the callback was called after the listener was closed, whatever it was trying to do failed (I forget the details). Hope this makes sense. Frank -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Sun, Feb 22, 2015 at 3:38 PM, Steven D'Aprano wrote: > But you are using it. You might not be using it by name, but you are using > it via the callback function. What did you expect, that Python should read > your mind and somehow intuit that you still care about this socket > listener, but not some other socket listener that you are done with? > > You don't have to bind the listener to a name. Any reference will do. You > can dump it in a bucket: > > bucket_of_stuff = [] > bucket_of_stuff.append(some_function(a, b, c)) > bucket_of_stuff.append(make_web_server()) > bucket_of_stuff.append(socket(23, on_accept=client_connected)) Sure, and whether it's a name or a list-element reference doesn't matter: it seems wrong to have to stash a thing in a bucket in order to keep its callbacks alive. I expect the callbacks _themselves_ to keep it alive. But I can understand the opposite POV. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Chris Angelico wrote:
> On Sun, Feb 22, 2015 at 1:04 PM, Steven D'Aprano
> wrote:
>> Marko Rauhamaa wrote:
>>
>>> Grant Edwards :
>>>
the polite thing to do is to delete references to it when you're done
with it.
>>>
>>> I disagree with that recommendation. You should do the natural thing and
>>> not care who holds references to who.
>>
>> I don't understand this. What is "the natural thing" if not to delete
>> references to an object when you are done with it? Normally you just let
>> things go out of scope, but if that won't happen, you have to take active
>> steps, such as calling del or setting the reference to None.
>
> I think the disagreement here is over the interpretation of "done with
> it". If you drop all references to a connected socket object, Python
> can rightly assume that you're done with the file and want to close
> it; but what if you drop all references to a listening socket that's
> been configured to call a function whenever someone connects?
>
> def client_connected(sock):
> sock.send("Hello!\r\n")
> # whatever
>
> listener = socket(23, on_accept=client_connected)
>
> What should happen if that main socket isn't bound to a name? In my
> opinion, the fact that it's configured for callback mode should mean
> that it's kept alive. But it's also understandable to want to treat it
> as "done", that it can be disposed of. It seems weird to me that you
> should have to have a name somewhere that you'll never use, though.
But you are using it. You might not be using it by name, but you are using
it via the callback function. What did you expect, that Python should read
your mind and somehow intuit that you still care about this socket
listener, but not some other socket listener that you are done with?
If you have a servant that follows you around everywhere, throwing objects
away when you stop using them, then naturally if you stop using something
it will be thrown away. What did you expect?
You don't have to bind the listener to a name. Any reference will do. You
can dump it in a bucket:
bucket_of_stuff = []
bucket_of_stuff.append(some_function(a, b, c))
bucket_of_stuff.append(make_web_server())
bucket_of_stuff.append(socket(23, on_accept=client_connected))
So long as your bucket is still alive, the garbage collector won't collect
it or its contents.
Hypothetically we could have a system in place where you instruct the
garbage collector to not collect an object, then drop all references to it:
gc.never_collect(socket(23, on_accept=client_connected))
but that's a potential memory leak, because now you have no way of telling
the GC to collect it again once you've finished listening. If you never
finish listening, or at least not until your application shuts down, it
doesn't count as a leak. But in general, if listeners might come and go,
but you have no way for them to be garbage collected once they are done,
then that's a leak.
What's the easiest way for the GC to flag an object as "never collect this"?
It can keep a reference to it. Keeping a list of things you want to be kept
alive is simple, easy and obvious:
# hypothetically inside gc.py
_ALIVE = []
def never_collect(obj):
_ALIVE.append(obj)
but that's so trivial that it's not worth putting it in the gc module.
Besides, that means any other code could reach into the gc and remove your
listener from the "keep alive list" without your knowledge or permission.
It's simpler and safer to just keep it alive yourself:
alive = []
alive.append(socket(...))
but of course that's just my bucket of stuff under a different name.
Using the idiom "keep objects alive by keeping a reference to them" (e.g.
bind them to a name, or stick them in a list which you keep) makes things
much simpler. You can trivially flag objects as "collect" or "don't
collect" as needed, without having to import the gc module or memorise some
obscure API or worry about implementation details ("if I flag an object
as 'never collect' *twice*, do I have to unflag it twice to undo it?"). You
just use the regular interface to the garbage collector: the existence of a
reference, any reference, keeps an object alive.
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Sun, Feb 22, 2015 at 1:04 PM, Steven D'Aprano
wrote:
> Marko Rauhamaa wrote:
>
>> Grant Edwards :
>>
>>> the polite thing to do is to delete references to it when you're done
>>> with it.
>>
>> I disagree with that recommendation. You should do the natural thing and
>> not care who holds references to who.
>
> I don't understand this. What is "the natural thing" if not to delete
> references to an object when you are done with it? Normally you just let
> things go out of scope, but if that won't happen, you have to take active
> steps, such as calling del or setting the reference to None.
I think the disagreement here is over the interpretation of "done with
it". If you drop all references to a connected socket object, Python
can rightly assume that you're done with the file and want to close
it; but what if you drop all references to a listening socket that's
been configured to call a function whenever someone connects?
def client_connected(sock):
sock.send("Hello!\r\n")
# whatever
listener = socket(23, on_accept=client_connected)
What should happen if that main socket isn't bound to a name? In my
opinion, the fact that it's configured for callback mode should mean
that it's kept alive. But it's also understandable to want to treat it
as "done", that it can be disposed of. It seems weird to me that you
should have to have a name somewhere that you'll never use, though.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Marko Rauhamaa wrote: > Grant Edwards : > >> the polite thing to do is to delete references to it when you're done >> with it. > > I disagree with that recommendation. You should do the natural thing and > not care who holds references to who. I don't understand this. What is "the natural thing" if not to delete references to an object when you are done with it? Normally you just let things go out of scope, but if that won't happen, you have to take active steps, such as calling del or setting the reference to None. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Grant Edwards : > the polite thing to do is to delete references to it when you're done > with it. I disagree with that recommendation. You should do the natural thing and not care who holds references to who. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On 2015-02-21, Cem Karan wrote: > > On Feb 21, 2015, at 12:42 AM, Chris Angelico wrote: > >> On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan wrote: >>> In order to inform users that certain bits of state have changed, I require >>> them to register a callback with my code. The problem is that when I store >>> these callbacks, it naturally creates a strong reference to the objects, >>> which means that if they are deleted without unregistering themselves >>> first, my code will keep the callbacks alive. Since this could lead to >>> really weird and nasty situations, I would like to store all the callbacks >>> in a WeakSet >>> (https://docs.python.org/3/library/weakref.html#weakref.WeakSet). That >>> way, my code isn't the reason why the objects are kept alive, and if they >>> are no longer alive, they are automatically removed from the WeakSet, >>> preventing me from accidentally calling them when they are dead. My >>> question is simple; is this a good design? If not, why not? Are there any >>> potential 'gotchas' I should be worried about? >>> >> >> No, it's not. I would advise using strong references - if the callback >> is a closure, for instance, you need to hang onto it, because there >> are unlikely to be any other references to it. If I register a >> callback with you, I expect it to be called; I expect, in fact, that >> that *will* keep my object alive. > > OK, so it would violate the principle of least surprise for you. And me as well. I would expect to be able to pass a closure as a callback and not have to keep a reference to it. Perhaps that just a leftover from working with other languages (javascript, scheme, etc.). It doesn't matter if it's a string, a float, a callback, a graphic or whatever: if I pass your function/library an object, I expect _you_ to keep track of it until you're done with it. > Interesting. Is this a general pattern in python? That is, > callbacks are owned by what they are registered with? I'm not sure what you mean by "owned" or why it matters that it's a callback: it's an object that was passed to you: you need to hold onto a reference to it until you're done with it, and the polite thing to do is to delete references to it when you're done with it. > So, what's the consensus on the list, strongly-held callbacks, or > weakly-held ones? -- Grant -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Cem Karan wrote: > > On Feb 21, 2015, at 8:15 AM, Chris Angelico wrote: > >> On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan wrote: >>> OK, so it would violate the principle of least surprise for you. >>> Interesting. Is this a general pattern in python? That is, callbacks >>> are owned by what they are registered with? >>> >>> In the end, I want to make a library that offers as few surprises to the >>> user as possible, and no matter how I think about callbacks, they are >>> surprising to me. If callbacks are strongly-held, then calling 'del >>> foo' on a callable object may not make it go away, which can lead to >>> weird and nasty situations. How? The whole point of callbacks is that you hand over responsibility to another piece of code, and then forget about your callback. The library will call it, when and if necessary, and when the library no longer needs your callback, it is free to throw it away. (If I wish the callback to survive beyond the lifetime of your library's use of it, I have to keep a reference to the function.) >>> Weakly-held callbacks mean that I (as the >>> programmer), know that objects will go away after the next garbage >>> collection (see Frank's earlier message), so I don't get 'dead' >>> callbacks coming back from the grave to haunt me. I'm afraid this makes no sense to me. Can you explain, or better still demonstrate, a scenario where "dead callbacks rise from the grave", so to speak? >>> So, what's the consensus on the list, strongly-held callbacks, or >>> weakly-held ones? >> >> I don't know about Python specifically, but it's certainly a general >> pattern in other languages. They most definitely are owned, and it's >> the only model that makes sense when you use closures (which won't >> have any other references anywhere). > > I agree about closures; its the only way they could work. *scratches head* There's nothing special about closures. You can assign them to a name like any other object. def make_closure(): x = 23 def closure(): return x + 1 return closure func = make_closure() Now you can register func as a callback, and de-register it when your done: register(func) unregister(func) Of course, if you thrown away your reference to func, you have no (easy) way of de-registering it. That's no different to any other object which is registered by identity. (Registering functions by name is a bad idea, since multiple functions can have the same name.) As an alternative, your callback registration function might return a ticket for the function: ticket = register(func) del func unregister(ticket) but that strikes me as over-kill. And of course, the simplest ticket is to return the function itself :-) > When I was > originally thinking about the library, I was trying to include all types > of callbacks, including closures and callable objects. The callable > objects may pass themselves, or one of their methods to the library, or > may do something really weird. I don't think they can do anything too weird. They have to pass a callable object. Your library just calls that object. You shouldn't need to care whether it is a function, a method, a type, a callable instance, or something else. You just call it, and when you're done calling it forever, you just throw it away. > Although I just realized that closures may cause another problem. In my > code, I expect that many different callbacks can be registered for the > same event. Unregistering means you request to be unregistered for the > event. How do you do that with a closure? Aren't they anonymous? Not unless you create them using lambda. Using the make_closure function above: py> func = make_closure() py> func.__name__ 'closure' Of course, if you call make_closure twice, both functions will have the same internal name. You can set the function __name__ and __qualname__ to fix that. This is how the functools.wraps decorator works. But that's a red herring. Don't register functions by name! Not all callable objects have names, and those that do, you may have multiple *distinct* callbacks with the same name. There are two reasonable approaches: unregister by identity, or by returning a ticket which uniquely identifies the callback. The user is responsible for keeping track of their own ticket. If I lose it, I can't unregister my callback any more. So sad, sucks to be me. The simplest possible identity-based scheme would be something like this: # don't hate me for using a global variable CALLBACKS = [] def register(func): if func not in CALLBACKS: CALLBACKS.append(func) def unregister(func): try: CALLBACKS.remove(func) except ValueError: pass That's probably a bit too simple, since it won't behave as expected with bound methods. The problem is that bound methods are generated on the fly, so this won't work: register(instance.spam) # later unregister(instance.spam) # a different instance! I would have to do this: boun
Re: Design thought for callbacks
Steven D'Aprano : > Other than that, I cannot see how calling a function which has *not* > yet been garbage collected can fail, just because the only reference > still existing is a weak reference. Maybe the logic of the receiving object isn't prepared for the callback anymore after an intervening event. The problem then, of course, is in the logic and not in the callbacks. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Frank Millman wrote: > I tried something similar a while ago, and I did find a gotcha. > > The problem lies in this phrase - "if they are no longer alive, they are > automatically removed from the WeakSet, preventing me from accidentally > calling them when they are dead." > > I found that the reference was not removed immediately, but was waiting to > be garbage collected. During that window, I could call the callback, which > resulted in an error. I don't understand how this could possibly work. (Or fail to work, as the case may be.) If the callback has been garbage collected, then you cannot call it, because you don't have any references to it and so cannot refer to it in any way. If the callback has *not* been garbage collected, then you can safely call it. You have a reference to the callback, therefore it exists. (If Python ever garbage collects an object that still has references to it, that would be a critical bug, and you would likely get some sort of seg fault). The only thing I can think of is you have a situation where your callback refers to another object, B, via a weak reference. Once all the regular strong references to the callback and B are gone, theoretically you could have a race condition where the callback is waiting to be garbage collected but B has already been garbage collected. If, in that window, you call the callback, *and* if the callback fails to correctly check that the weak reference to B still exists, then you could get a Python exception. The solution is simple: anytime you have a weak reference, you must always check it before you use it. Other than that, I cannot see how calling a function which has *not* yet been garbage collected can fail, just because the only reference still existing is a weak reference. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
Chris Angelico : > On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan wrote: >> In order to inform users that certain bits of state have changed, I >> require them to register a callback with my code. The problem is that >> when I store these callbacks, it naturally creates a strong reference >> to the objects, which means that if they are deleted without >> unregistering themselves first, my code will keep the callbacks >> alive. Since this could lead to really weird and nasty situations, >> [...] > > No, it's not. I would advise using strong references - if the callback > is a closure, for instance, you need to hang onto it, because there > are unlikely to be any other references to it. If I register a > callback with you, I expect it to be called; I expect, in fact, that > that *will* keep my object alive. I use callbacks all the time but haven't had any problems with strong references. I am careful to move my objects to a zombie state after they're done so they can absorb any potential loose callbacks that are lingering in the system. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Sun, Feb 22, 2015 at 2:45 AM, Cem Karan wrote: > OK, so if I'm reading your code correctly, you're breaking the cycle in your > object graph by making the GUI the owner of the callback, correct? No other > chunk of code has a reference to the callback, correct? Correct. The GUI engine ultimately owns everything. Of course, this is a very simple case (imagine a little notification popup; you don't care about it, you don't need to know when it's been closed, the only event on it is "hit Close to destroy the window"), and most usage would have other complications, but it's not uncommon for me to build a GUI program that leaves everything owned by the GUI engine. Everything is done through callbacks. Destroy a window, clean up its callbacks. The main window will have an "on-deletion" callback that terminates the program, perhaps. It's pretty straight-forward. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 21, 2015, at 9:36 AM, Chris Angelico wrote:
> On Sun, Feb 22, 2015 at 1:07 AM, Cem Karan wrote:
>> I agree about closures; its the only way they could work. When I was
>> originally thinking about the library, I was trying to include all types of
>> callbacks, including closures and callable objects. The callable objects
>> may pass themselves, or one of their methods to the library, or may do
>> something really weird.
>>
>> Although I just realized that closures may cause another problem. In my
>> code, I expect that many different callbacks can be registered for the same
>> event. Unregistering means you request to be unregistered for the event.
>> How do you do that with a closure? Aren't they anonymous?
>>
>
> They're objects, same as any other, so the caller can hang onto a
> reference and then say "now remove this one". Simple example:
>
> callbacks = []
> def register_callback(f): callbacks.append(f)
> def unregister_callback(f): callbacks.remove(f)
> def do_callbacks():
>for f in callbacks:
>f()
>
> def make_callback(i):
>def inner():
>print("Callback! %d"%i)
>register_callback(inner)
>return inner
>
> make_callback(5)
> remove_me = make_callback(6)
> make_callback(7)
> unregister_callback(remove_me)
> do_callbacks()
Yeah, that's pretty much what I thought you'd have to do, which kind of defeats
the purpose of closures (fire-and-forget things). BUT it does answer my
question, so no complaints about it!
So, either you keep a reference to your own closure, which means that the
library doesn't really need to, or the library keeps hold of it for you, in
which case you don't have a reasonable way of removing it.
> The other option is for your callback registration to return some kind
> of identifier, which can later be used to unregister the callback.
> This is a good way of avoiding reference cycles (the ID could be a
> simple integer - maybe the length of the list prior to the new
> callback being appended, and then the unregistration process is simply
> "callbacks[id] = None", and you skip the Nones when iterating), and
> even allows you to register the exact same function more than once,
> for what that's worth.
That would work. In the cases where someone might register & unregister many
callbacks, you might use UUIDs as keys instead (avoids the ABA problem).
> When I do GUI programming, this is usually how things work. For
> instance, I use GTK2 (though usually with Pike rather than Python),
> and I can connect a signal to a callback function. Any given signal
> could have multiple callbacks attached to it, so it's similar to your
> case. I frequently depend on the GTK engine retaining a reference to
> my function (and thus to any data it requires), as I tend not to hang
> onto any inner objects that don't need retention. Once the parent
> object is destroyed, all its callbacks get dereferenced. Consider this
> simplified form:
>
> def popup_window():
>w = Window()
># Add layout, info, whatever it takes
>btn = Button("Close")
>w.add(btn) # actually it'd be added to a layout
>btn.signal_connect("clicked", lambda *args: w.destroy())
>
> The GUI back end will hang onto a reference to the window, because
> it's currently on screen; to the button, because it's attached to the
> window; and to my function, because it's connected to a button signal.
> Then when you click the button, the window gets destroyed, which
> destroys the button, which unregisters all its callbacks. At that
> point, there are no refs to the function, so it can get disposed of.
> That button function was the last external reference to the window,
> and now that it's not on screen, its Python object can also be
> disposed of, as can the button inside. So it'll all clean up fairly
> nicely; as long as the callback gets explicitly deregistered, that's
> the end of everything.
OK, so if I'm reading your code correctly, you're breaking the cycle in your
object graph by making the GUI the owner of the callback, correct? No other
chunk of code has a reference to the callback, correct?
Thanks,
Cem Karan
--
https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Sun, Feb 22, 2015 at 1:07 AM, Cem Karan wrote:
> I agree about closures; its the only way they could work. When I was
> originally thinking about the library, I was trying to include all types of
> callbacks, including closures and callable objects. The callable objects may
> pass themselves, or one of their methods to the library, or may do something
> really weird.
>
> Although I just realized that closures may cause another problem. In my
> code, I expect that many different callbacks can be registered for the same
> event. Unregistering means you request to be unregistered for the event. How
> do you do that with a closure? Aren't they anonymous?
>
They're objects, same as any other, so the caller can hang onto a
reference and then say "now remove this one". Simple example:
callbacks = []
def register_callback(f): callbacks.append(f)
def unregister_callback(f): callbacks.remove(f)
def do_callbacks():
for f in callbacks:
f()
def make_callback(i):
def inner():
print("Callback! %d"%i)
register_callback(inner)
return inner
make_callback(5)
remove_me = make_callback(6)
make_callback(7)
unregister_callback(remove_me)
do_callbacks()
The other option is for your callback registration to return some kind
of identifier, which can later be used to unregister the callback.
This is a good way of avoiding reference cycles (the ID could be a
simple integer - maybe the length of the list prior to the new
callback being appended, and then the unregistration process is simply
"callbacks[id] = None", and you skip the Nones when iterating), and
even allows you to register the exact same function more than once,
for what that's worth.
When I do GUI programming, this is usually how things work. For
instance, I use GTK2 (though usually with Pike rather than Python),
and I can connect a signal to a callback function. Any given signal
could have multiple callbacks attached to it, so it's similar to your
case. I frequently depend on the GTK engine retaining a reference to
my function (and thus to any data it requires), as I tend not to hang
onto any inner objects that don't need retention. Once the parent
object is destroyed, all its callbacks get dereferenced. Consider this
simplified form:
def popup_window():
w = Window()
# Add layout, info, whatever it takes
btn = Button("Close")
w.add(btn) # actually it'd be added to a layout
btn.signal_connect("clicked", lambda *args: w.destroy())
The GUI back end will hang onto a reference to the window, because
it's currently on screen; to the button, because it's attached to the
window; and to my function, because it's connected to a button signal.
Then when you click the button, the window gets destroyed, which
destroys the button, which unregisters all its callbacks. At that
point, there are no refs to the function, so it can get disposed of.
That button function was the last external reference to the window,
and now that it's not on screen, its Python object can also be
disposed of, as can the button inside. So it'll all clean up fairly
nicely; as long as the callback gets explicitly deregistered, that's
the end of everything.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Fri, Feb 20, 2015 at 9:42 PM, Chris Angelico wrote: > No, it's not. I would advise using strong references - if the callback > is a closure, for instance, you need to hang onto it, because there > are unlikely to be any other references to it. If I register a > callback with you, I expect it to be called; I expect, in fact, that > that *will* keep my object alive. For that matter, if the callback is a method, you need to hang onto it, because method wrappers are generated on demand, so the method would be removed from the valid callbacks instantly. Weak references for callbacks are broken. -- Devin -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 21, 2015, at 8:37 AM, Mark Lawrence wrote: > On 21/02/2015 05:41, Frank Millman wrote: >> >> "Cem Karan" wrote in message >> news:33677ae8-b2fa-49f9-9304-c8d937842...@gmail.com... >>> Hi all, I'm working on a project that will involve the use of callbacks, >>> and I want to bounce an idea I had off of everyone to make sure I'm not >>> developing a bad idea. Note that this is for python 3.4 code; I don't >>> need to worry about any version of python earlier than that. >>> >>> In order to inform users that certain bits of state have changed, I >>> require them to register a callback with my code. The problem is that >>> when I store these callbacks, it naturally creates a strong reference to >>> the objects, which means that if they are deleted without unregistering >>> themselves first, my code will keep the callbacks alive. Since this could >>> lead to really weird and nasty situations, I would like to store all the >>> callbacks in a WeakSet >>> (https://docs.python.org/3/library/weakref.html#weakref.WeakSet). That >>> way, my code isn't the reason why the objects are kept alive, and if they >>> are no longer alive, they are automatically removed from the WeakSet, >>> preventing me from accidentally calling them when they are dead. My >>> question is simple; is this a good design? If not, why not? >>> Are there any potential 'gotchas' I should be worried about? >>> >> >> I tried something similar a while ago, and I did find a gotcha. >> >> The problem lies in this phrase - "if they are no longer alive, they are >> automatically removed from the WeakSet, preventing me from accidentally >> calling them when they are dead." >> >> I found that the reference was not removed immediately, but was waiting to >> be garbage collected. During that window, I could call the callback, which >> resulted in an error. >> >> There may have been a simple workaround. Perhaps someone else can comment. >> >> Frank Millman >> > > https://docs.python.org/3/library/gc.html has a collect function. That seems > like a simple workaround, but whether or not it classifies as a good solution > I'll leave to others, I'm not qualified to say. Unfortunately, depending on how many objects you have in your object graph, it can slow your code down a fair amount. I think Frank is right about how a WeakSet might be a bad idea in this case. You really need to know if an object is alive or dead, and not some indeterminate state. Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 21, 2015, at 8:15 AM, Chris Angelico wrote: > On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan wrote: >> OK, so it would violate the principle of least surprise for you. >> Interesting. Is this a general pattern in python? That is, callbacks are >> owned by what they are registered with? >> >> In the end, I want to make a library that offers as few surprises to the >> user as possible, and no matter how I think about callbacks, they are >> surprising to me. If callbacks are strongly-held, then calling 'del foo' on >> a callable object may not make it go away, which can lead to weird and nasty >> situations. Weakly-held callbacks mean that I (as the programmer), know >> that objects will go away after the next garbage collection (see Frank's >> earlier message), so I don't get 'dead' callbacks coming back from the grave >> to haunt me. >> >> So, what's the consensus on the list, strongly-held callbacks, or >> weakly-held ones? > > I don't know about Python specifically, but it's certainly a general > pattern in other languages. They most definitely are owned, and it's > the only model that makes sense when you use closures (which won't > have any other references anywhere). I agree about closures; its the only way they could work. When I was originally thinking about the library, I was trying to include all types of callbacks, including closures and callable objects. The callable objects may pass themselves, or one of their methods to the library, or may do something really weird. Although I just realized that closures may cause another problem. In my code, I expect that many different callbacks can be registered for the same event. Unregistering means you request to be unregistered for the event. How do you do that with a closure? Aren't they anonymous? > If you're expecting 'del foo' to destroy the object, then you have a > bigger problem than callbacks, because that's simply not how Python > works. You can't _ever_ assume that deleting something from your local > namespace will destroy the object, because there can always be more > references. So maybe you need a more clear way of saying "I'm done > with this, get rid of it". Agreed about 'del', and I don't assume that the object goes away at the point. The problem is debugging and determining WHY your object is still around. I know a combination of logging and gc.get_referrers() will probably help you figure out why something is still around, but I'm trying to avoid that headache. I guess the real problem is how this creates cycles in the call graph. User code effectively owns the library code, which via callbacks owns the user code. I have no idea what the best point the cycle is to break it, and not surprise someone down the road. The only idea I have is to redesign the library a little, and make anything that accepts a callback actually be a subclass of collections.abc.Container, or even collections.abc.MutableSet. That makes it very obvious that the object owns the callback, and that you will need to remove your object to unregister it. The only problem is how to handle closures; since they are anonymous, how do you decide which one to remove? Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On 21/02/2015 05:41, Frank Millman wrote: "Cem Karan" wrote in message news:33677ae8-b2fa-49f9-9304-c8d937842...@gmail.com... Hi all, I'm working on a project that will involve the use of callbacks, and I want to bounce an idea I had off of everyone to make sure I'm not developing a bad idea. Note that this is for python 3.4 code; I don't need to worry about any version of python earlier than that. In order to inform users that certain bits of state have changed, I require them to register a callback with my code. The problem is that when I store these callbacks, it naturally creates a strong reference to the objects, which means that if they are deleted without unregistering themselves first, my code will keep the callbacks alive. Since this could lead to really weird and nasty situations, I would like to store all the callbacks in a WeakSet (https://docs.python.org/3/library/weakref.html#weakref.WeakSet). That way, my code isn't the reason why the objects are kept alive, and if they are no longer alive, they are automatically removed from the WeakSet, preventing me from accidentally calling them when they are dead. My question is simple; is this a good design? If not, why not? Are there any potential 'gotchas' I should be worried about? I tried something similar a while ago, and I did find a gotcha. The problem lies in this phrase - "if they are no longer alive, they are automatically removed from the WeakSet, preventing me from accidentally calling them when they are dead." I found that the reference was not removed immediately, but was waiting to be garbage collected. During that window, I could call the callback, which resulted in an error. There may have been a simple workaround. Perhaps someone else can comment. Frank Millman https://docs.python.org/3/library/gc.html has a collect function. That seems like a simple workaround, but whether or not it classifies as a good solution I'll leave to others, I'm not qualified to say. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 21, 2015, at 12:41 AM, Frank Millman wrote: > > "Cem Karan" wrote in message > news:33677ae8-b2fa-49f9-9304-c8d937842...@gmail.com... >> Hi all, I'm working on a project that will involve the use of callbacks, >> and I want to bounce an idea I had off of everyone to make sure I'm not >> developing a bad idea. Note that this is for python 3.4 code; I don't >> need to worry about any version of python earlier than that. >> >> In order to inform users that certain bits of state have changed, I >> require them to register a callback with my code. The problem is that >> when I store these callbacks, it naturally creates a strong reference to >> the objects, which means that if they are deleted without unregistering >> themselves first, my code will keep the callbacks alive. Since this could >> lead to really weird and nasty situations, I would like to store all the >> callbacks in a WeakSet >> (https://docs.python.org/3/library/weakref.html#weakref.WeakSet). That >> way, my code isn't the reason why the objects are kept alive, and if they >> are no longer alive, they are automatically removed from the WeakSet, >> preventing me from accidentally calling them when they are dead. My >> question is simple; is this a good design? If not, why not? >> Are there any potential 'gotchas' I should be worried about? >> > > I tried something similar a while ago, and I did find a gotcha. > > The problem lies in this phrase - "if they are no longer alive, they are > automatically removed from the WeakSet, preventing me from accidentally > calling them when they are dead." > > I found that the reference was not removed immediately, but was waiting to > be garbage collected. During that window, I could call the callback, which > resulted in an error. > > There may have been a simple workaround. Perhaps someone else can comment. THAT would be one heck of a gotcha! Must have been fun debugging that one! Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan wrote: > OK, so it would violate the principle of least surprise for you. > Interesting. Is this a general pattern in python? That is, callbacks are > owned by what they are registered with? > > In the end, I want to make a library that offers as few surprises to the user > as possible, and no matter how I think about callbacks, they are surprising > to me. If callbacks are strongly-held, then calling 'del foo' on a callable > object may not make it go away, which can lead to weird and nasty situations. > Weakly-held callbacks mean that I (as the programmer), know that objects > will go away after the next garbage collection (see Frank's earlier message), > so I don't get 'dead' callbacks coming back from the grave to haunt me. > > So, what's the consensus on the list, strongly-held callbacks, or weakly-held > ones? I don't know about Python specifically, but it's certainly a general pattern in other languages. They most definitely are owned, and it's the only model that makes sense when you use closures (which won't have any other references anywhere). If you're expecting 'del foo' to destroy the object, then you have a bigger problem than callbacks, because that's simply not how Python works. You can't _ever_ assume that deleting something from your local namespace will destroy the object, because there can always be more references. So maybe you need a more clear way of saying "I'm done with this, get rid of it". ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Feb 21, 2015, at 12:42 AM, Chris Angelico wrote: > On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan wrote: >> In order to inform users that certain bits of state have changed, I require >> them to register a callback with my code. The problem is that when I store >> these callbacks, it naturally creates a strong reference to the objects, >> which means that if they are deleted without unregistering themselves first, >> my code will keep the callbacks alive. Since this could lead to really >> weird and nasty situations, I would like to store all the callbacks in a >> WeakSet (https://docs.python.org/3/library/weakref.html#weakref.WeakSet). >> That way, my code isn't the reason why the objects are kept alive, and if >> they are no longer alive, they are automatically removed from the WeakSet, >> preventing me from accidentally calling them when they are dead. My >> question is simple; is this a good design? If not, why not? Are there any >> potential 'gotchas' I should be worried about? >> > > No, it's not. I would advise using strong references - if the callback > is a closure, for instance, you need to hang onto it, because there > are unlikely to be any other references to it. If I register a > callback with you, I expect it to be called; I expect, in fact, that > that *will* keep my object alive. OK, so it would violate the principle of least surprise for you. Interesting. Is this a general pattern in python? That is, callbacks are owned by what they are registered with? In the end, I want to make a library that offers as few surprises to the user as possible, and no matter how I think about callbacks, they are surprising to me. If callbacks are strongly-held, then calling 'del foo' on a callable object may not make it go away, which can lead to weird and nasty situations. Weakly-held callbacks mean that I (as the programmer), know that objects will go away after the next garbage collection (see Frank's earlier message), so I don't get 'dead' callbacks coming back from the grave to haunt me. So, what's the consensus on the list, strongly-held callbacks, or weakly-held ones? Thanks, Cem Karan -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan wrote: > In order to inform users that certain bits of state have changed, I require > them to register a callback with my code. The problem is that when I store > these callbacks, it naturally creates a strong reference to the objects, > which means that if they are deleted without unregistering themselves first, > my code will keep the callbacks alive. Since this could lead to really weird > and nasty situations, I would like to store all the callbacks in a WeakSet > (https://docs.python.org/3/library/weakref.html#weakref.WeakSet). That way, > my code isn't the reason why the objects are kept alive, and if they are no > longer alive, they are automatically removed from the WeakSet, preventing me > from accidentally calling them when they are dead. My question is simple; is > this a good design? If not, why not? Are there any potential 'gotchas' I > should be worried about? > No, it's not. I would advise using strong references - if the callback is a closure, for instance, you need to hang onto it, because there are unlikely to be any other references to it. If I register a callback with you, I expect it to be called; I expect, in fact, that that *will* keep my object alive. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Design thought for callbacks
"Cem Karan" wrote in message news:33677ae8-b2fa-49f9-9304-c8d937842...@gmail.com... > Hi all, I'm working on a project that will involve the use of callbacks, > and I want to bounce an idea I had off of everyone to make sure I'm not > developing a bad idea. Note that this is for python 3.4 code; I don't > need to worry about any version of python earlier than that. > > In order to inform users that certain bits of state have changed, I > require them to register a callback with my code. The problem is that > when I store these callbacks, it naturally creates a strong reference to > the objects, which means that if they are deleted without unregistering > themselves first, my code will keep the callbacks alive. Since this could > lead to really weird and nasty situations, I would like to store all the > callbacks in a WeakSet > (https://docs.python.org/3/library/weakref.html#weakref.WeakSet). That > way, my code isn't the reason why the objects are kept alive, and if they > are no longer alive, they are automatically removed from the WeakSet, > preventing me from accidentally calling them when they are dead. My > question is simple; is this a good design? If not, why not? > Are there any potential 'gotchas' I should be worried about? > I tried something similar a while ago, and I did find a gotcha. The problem lies in this phrase - "if they are no longer alive, they are automatically removed from the WeakSet, preventing me from accidentally calling them when they are dead." I found that the reference was not removed immediately, but was waiting to be garbage collected. During that window, I could call the callback, which resulted in an error. There may have been a simple workaround. Perhaps someone else can comment. Frank Millman -- https://mail.python.org/mailman/listinfo/python-list
