Re: [Python-Dev] Coroutines and PEP 380
After much consideration, and playing with PEP380, I've changed my stance on this. Full blown coroutines are the proper way forward. greenlet doesn't cut it because the Python interpreter isn't aware of the context switches. Profiling, debugging and tracebacks are completely broken by this. Stackless would need to be merged, and that's clearly not going to happen. I built a basic scheduler and had a go at enhancing the stdlib using PEP380, here are some examples making use of this style: https://bitbucket.org/anacrolix/green380/src/8f7fdc20a8ce/examples After realising it was a dead-end, I read up on Mark's ideas, there's some really good stuff in there: http://www.dcs.gla.ac.uk/~marks/ http://hotpy.blogspot.com/ If someone can explain what's stopping real coroutines being into Python (3.3), that would be great. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coroutines and PEP 380
Glyph wrote: Yes, but you /can/ look at a 'yield' and conclude that you /might/ need a lock, and that you have to think about it. My concern is that you will end up with vastly more 'yield from's than places that require locks, so most of them are just noise. If you bite your nails over whether a lock is needed every time you see one, they will cause you a lot more anxiety than they alleviate. Sometimes there's no alternative, but wherever I can, I avoid thinking, especially hard thinking. This maxim has served me very well throughout my programming career ;-). There are already well-known techniques for dealing with concurrency that minimise the amount of hard thinking required. You devise some well-behaved abstractions, such as queues, and put all your hard thinking into implementing them. Then you build the rest of your code around those abstractions. That way you don't have to rely on crutches such as explicitly marking everything that might cause a task switch, because it doesn't matter. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coroutines and PEP 380
My concern is that you will end up with vastly more 'yield from's than places that require locks, so most of them are just noise. If you bite your nails over whether a lock is needed every time you see one, they will cause you a lot more anxiety than they alleviate. Not necessarily. The yield from's follow the blocking control flow, which is surprisingly less common than you might think. Parts of your code naturally arise as not requiring blocking behaviour in the same manner as in Haskell where parts of your code are identified as requiring the IO monad. Sometimes there's no alternative, but wherever I can, I avoid thinking, especially hard thinking. This maxim has served me very well throughout my programming career ;-). I'd replace hard thinking with future confusion here. There are already well-known techniques for dealing with concurrency that minimise the amount of hard thinking required. You devise some well-behaved abstractions, such as queues, and put all your hard thinking into implementing them. Then you build the rest of your code around those abstractions. That way you don't have to rely on crutches such as explicitly marking everything that might cause a task switch, because it doesn't matter. It's my firm belief that this isn't sufficient. If this were true, then the Python internals could be improved by replacing the GIL with a series of channels/queues or what have you. State is complex, and without guarantees of immutability, it's just not practical to try to wrap every state object in some protocol to be passed back and forth on queues. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coroutines and PEP 380
Glyph wrote: [Guido] mentions the point that coroutines that can implicitly switch out from under you have the same non-deterministic property as threads: you don't know where you're going to need a lock or lock-like construct to update any variables, so you need to think about concurrency more deeply than if you could explicitly always see a 'yield'. I'm not convinced that being able to see 'yield's will help all that much. In any system that makes substantial use of generator-based coroutines, you're going to see 'yield from's all over the place, from the lowest to the highest levels. But that doesn't mean you need a correspondingly large number of locks. You can't look at a 'yield' and conclude that you need a lock there or tell what needs to be locked. There's no substitute for deep thought where any kind of theading is involved, IMO. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coroutines and PEP 380
On Fri, Jan 20, 2012 at 8:41 AM, Greg greg.ew...@canterbury.ac.nz wrote: Glyph wrote: [Guido] mentions the point that coroutines that can implicitly switch out from under you have the same non-deterministic property as threads: you don't know where you're going to need a lock or lock-like construct to update any variables, so you need to think about concurrency more deeply than if you could explicitly always see a 'yield'. I'm not convinced that being able to see 'yield's will help all that much. In any system that makes substantial use of generator-based coroutines, you're going to see 'yield from's all over the place, from the lowest to the highest levels. But that doesn't mean you need a correspondingly large number of locks. You can't look at a 'yield' and conclude that you need a lock there or tell what needs to be locked. There's no substitute for deep thought where any kind of theading is involved, IMO. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com I wasn't aware that Guido had brought this up, and I believe what he says to be true. Preemptive coroutines, are just a hack around the GIL, and reduce OS overheads. It's the explicit nature of the enhanced generators that is their greatest value. FWIW, I wrote a Python 3 compatible equivalent to gevent (also greenlet based, and also very similar to Brett's et al coroutine proposal), which didn't really solve the concurrency problems I hoped. There were no guarantees whether functions would switch out, so all the locking and threading issues simply reemerged, albeit with also needing to have all calls non-blocking, losing compatibility with any routine that didn't make use of nonblocking calls and/or expose it's yield in the correct way, but reducing GIL contention. Overall not worth it. In short, implicit coroutines are just a GIL work around, that break compatibility for little gain. Thanks Glyph for those links. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coroutines and PEP 380
On Jan 19, 2012, at 4:41 PM, Greg wrote: Glyph wrote: [Guido] mentions the point that coroutines that can implicitly switch out from under you have the same non-deterministic property as threads: you don't know where you're going to need a lock or lock-like construct to update any variables, so you need to think about concurrency more deeply than if you could explicitly always see a 'yield'. I'm not convinced that being able to see 'yield's will help all that much. Well, apparently we disagree, and I work on such a system all day, every day :-). It was nice to see that Matt Joiner also agreed for very similar reasons, and at least I know I'm not crazy. In any system that makes substantial use of generator-based coroutines, you're going to see 'yield from's all over the place, from the lowest to the highest levels. But that doesn't mean you need a correspondingly large number of locks. You can't look at a 'yield' and conclude that you need a lock there or tell what needs to be locked. Yes, but you can look at a 'yield' and conclude that you might need a lock, and that you have to think about it. Further exploration of my own feelings on the subject grew a bit beyond a good length for a reply here, so if you're interested in my thoughts you can have a look at my blog: http://glyph.twistedmatrix.com/2012/01/concurrency-spectrum-from-callbacks-to.html. There's no substitute for deep thought where any kind of theading is involved, IMO. Sometimes there's no alternative, but wherever I can, I avoid thinking, especially hard thinking. This maxim has served me very well throughout my programming career ;-). -glyph ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coroutines and PEP 380
Matt Joiner wrote: Just to clarify, this differs in functionality from enhanced generators by allowing you to yield from an arbitrary call depth rather than having to yield from through a chain of calling generators? Furthermore there's no syntactical change except to the bottommost frame doing a co_yield? Does this capture the major differences? Yes. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coroutines and PEP 380
Glyph wrote: On Jan 17, 2012, at 5:03 PM, Mark Shannon wrote: Lets start controversially: I don't like PEP 380, I think it's a kludge. Too late; it's already accepted. There's not much point in making controversial statements about it now. Why is it too late? Presenting this as a fait accompli does not make it any better. The PEP mailing list is closed to most people, so what forum for debate is there? I think that CPython should have proper coroutines, rather than add more bits and pieces to generators in an attempt to make them more like coroutines. By proper coroutines, you mean implicit coroutines (cooperative threads) rather than explicit coroutines (cooperative generators). Nothing implicit about it. Python has been going in the explicit direction on this question for a long time. (And, in my opinion, this is the right direction to go, but that's not really relevant here.) You can use asymmetric coroutines with a scheduler to provide cooperative threads if you want, but coroutines not have to be used as threads. The key advantages of my coroutine implmentation over PEP 380 are: 1. No syntax change. 2. Code can be used in coroutines without modification. 3. No stack unwinding is required at a yield point. I think this discussion would be more suitable for python-ideas though, since you have a long row to hoe here. There's already a PEP - http://www.python.org/dev/peps/pep-0219/ - apparently deferred and not rejected, which you may want to revisit. There are several libraries which can give you cooperative threading already; I assume you're already aware of greenlet and stackless, but I didn't see what advantages your proposed implementation provides over those. I would guess that one of the first things you should address on python-ideas is why adopting your implementation would be a better idea than just bundling one of those with the standard library :). Already been discussed: http://mail.python.org/pipermail/python-ideas/2011-October/012571.html All of the objections to coroutines (as I propose) also apply to PEP 380. The advantage of my implementation over greenlets is portability. I suspect stackless is actually fairly similar to what I have done, I haven't checked in detail. Cheers, Mark. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coroutines and PEP 380
On Jan 18, 2012, at 4:23 AM, Mark Shannon wrote: Glyph wrote: On Jan 17, 2012, at 5:03 PM, Mark Shannon wrote: Lets start controversially: I don't like PEP 380, I think it's a kludge. Too late; it's already accepted. There's not much point in making controversial statements about it now. Why is it too late? Because discussion happens before the PEP is accepted. See the description of the workflow in http://www.python.org/dev/peps/pep-0001/. The time to object to PEP 380 was when those threads were going on. Presenting this as a fait accompli does not make it any better. But it is[1] a fait accompli, whether you like it or not; I'm first and foremost informing you of the truth, not trying to make you feel better (or worse). Secondly, I am trying to forestall a long and ultimately pointless conversation :). The PEP mailing list is closed to most people, The PEP mailing list is just where you submit your PEPs, and where the PEP editors do their work. I'm not on it, but to my understanding of the process, there's not really any debate there. so what forum for debate is there? python-ideas, and then this mailing list, in that order. Regarding PEP 380 specifically, there's been quite a bit. See for example http://thread.gmane.org/gmane.comp.python.devel/102161/focus=102164. Keep in mind that the purpose of debate in this context is to inform Guido's opinion. There's no voting involved, although he will occasionally delegate decisions about particular PEPs to people knowledgeable in a relevant area. I think this discussion would be more suitable for python-ideas though [...] Already been discussed: http://mail.python.org/pipermail/python-ideas/2011-October/012571.html If you're following the PEP process, then the next step would be for you (having built some support) to author a new PEP, or to resurrect the deferred Stackless PEP with some new rationale - personally I'd recommend the latter. My brief skimming of the linked thread doesn't indicate you have a lot of strong support though, just some people who would be somewhat interested. So I still think it bears more discussion there, especially on the motivation / justification side of things. All of the objections to coroutines (as I propose) also apply to PEP 380. You might want to see the video of Guido's Fireside Chat last year http://pycon.tv/#/video/100. Skip to a little before 15:00. He mentions the point that coroutines that can implicitly switch out from under you have the same non-deterministic property as threads: you don't know where you're going to need a lock or lock-like construct to update any variables, so you need to think about concurrency more deeply than if you could explicitly always see a 'yield'. I have more than one painful event in my past (as he refers to it) indicating that microthreads have the same problem as real threads :). (And yes, they're microthreads, even if you don't have an elaborate scheduling construct. If you can switch to another stack by making a function call, then you are effectively context switching, and it can become arbitrarily complex. Any coroutine in a system may introduce an arbitrarily complex microthread scheduler just by calling a function that yields to it.) -glyph ([1]: Well actually it isn't, note the dashed line from Accepted to Rejected in the workflow diagram. But you have to have a really darn good reason, and championing the rejection of a pep that Guido has explicitly accepted and has liked from pretty much the beginning is going to be very, very hard.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coroutines and PEP 380
PEP380 and Mark's coroutines could coexist, so I really don't it's too late matters. Furthermore, PEP380 has utility in its own right without considering its use for explicit coroutines. I would like to see these coroutines considered, but as someone else mentioned, coroutines via PEP380 enhanced generators have some interesting characteristics, from my experimentations they feel monadic. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Coroutines and PEP 380
Hi all. Lets start controversially: I don't like PEP 380, I think it's a kludge. I think that CPython should have proper coroutines, rather than add more bits and pieces to generators in an attempt to make them more like coroutines. I have mentioned this before, but this time I have done something about it :) I have a working, portable, (asymmetric) coroutine implementation here: https://bitbucket.org/markshannon/hotpy_coroutines Its all standard C, no messing with the C stack, just using standard techniques to convert recursion to iteration (in the VM not at the Python level) and a revised internal calling convention to make CPython stackless: https://bitbucket.org/markshannon/hotpy_stackless Then I've added a Coroutine class and fiddled with the implementation of YIELD_VALUE to support it. I think the stackless implementation is pretty solid, but the coroutine stuff needs some refinement. I've not tested it well (it passes the test suite, but I've added no new tests). It is (surprisingly) a bit faster than tip (on my machine). There are limitations: all calls must be Python-to-Python calls, which rules out most __xxx__ methods. It might be worth special casing __iter__, but I've not done that yet. To try it out: import coroutine To send a value to a coroutine: co.send(val) where co is a Coroutine() To yield a value: coroutine.co_yield(val) send() is a method, co_yield is a function. Here's a little program to demonstrate: import coroutine class Node: def __init__(self, l, item, r): self.l = l self.item = item self.r = r def make_tree(n): if n == 0: return Node(None, n, None) else: return Node(make_tree(n-1), n, make_tree(n-1)) def walk_tree(t, f): if t is not None: walk_tree(t.l, f) f(t) walk_tree(t.r, f) def yielder(t): coroutine.co_yield(t.item) def tree_yielder(t): walk_tree(t, yielder) co = coroutine.Coroutine(tree_yielder, (make_tree(2),)) while True: print(co.send(None)) Which will output: 0 1 0 2 0 1 0 None Traceback (most recent call last): File co_demo.py, line 30, in module print(co.send(None)) TypeError: can't send to a halted coroutine Cheers, Mark. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coroutines and PEP 380
Just to clarify, this differs in functionality from enhanced generators by allowing you to yield from an arbitrary call depth rather than having to yield from through a chain of calling generators? Furthermore there's no syntactical change except to the bottommost frame doing a co_yield? Does this capture the major differences? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coroutines and PEP 380
Mark Shannon wrote: I think that CPython should have proper coroutines, rather than add more bits and pieces to generators in an attempt to make them more like coroutines. I have mentioned this before, but this time I have done something about it :) I have a working, portable, (asymmetric) coroutine implementation here: https://bitbucket.org/markshannon/hotpy_coroutines As a user, this sounds cool! ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coroutines and PEP 380
On Jan 17, 2012, at 5:03 PM, Mark Shannon wrote: Lets start controversially: I don't like PEP 380, I think it's a kludge. Too late; it's already accepted. There's not much point in making controversial statements about it now. I think that CPython should have proper coroutines, rather than add more bits and pieces to generators in an attempt to make them more like coroutines. By proper coroutines, you mean implicit coroutines (cooperative threads) rather than explicit coroutines (cooperative generators). Python has been going in the explicit direction on this question for a long time. (And, in my opinion, this is the right direction to go, but that's not really relevant here.) I think this discussion would be more suitable for python-ideas though, since you have a long row to hoe here. There's already a PEP - http://www.python.org/dev/peps/pep-0219/ - apparently deferred and not rejected, which you may want to revisit. There are several libraries which can give you cooperative threading already; I assume you're already aware of greenlet and stackless, but I didn't see what advantages your proposed implementation provides over those. I would guess that one of the first things you should address on python-ideas is why adopting your implementation would be a better idea than just bundling one of those with the standard library :). -glyph ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
