Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On Wed, Feb 14, 2007, [EMAIL PROTECTED] wrote: > > As I said, I don't have time to write the PEPs myself, but I might fix > some specific bugs if there were a clear set of issues preventing this > from moving forward. Better integration with the standard library > would definitely be a big win for both Twisted and Python. Here's where I'm coming from: My first experience with Twisted was excellent: I needed a web server in fifteen minutes to do my PyCon presentation, and it Just Worked. My second experience with Twisted? Well, I didn't really have one. My first experience was Twisted 1.1, and when I tried installing 2.0 on my Mac (about 1.5 years ago), it just didn't work. Combined with the difficulty of using the documentation and the fact that I was in a hurry, I rejected the Twisted solution. (My company needed an FTP server that did a callback every time a file got uploaded -- something that I expect would be very simple for Twisted.) -- Aahz ([EMAIL PROTECTED]) <*> http://www.pythoncraft.com/ "I disrespectfully agree." --SJM ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 03:32 pm, [EMAIL PROTECTED] wrote: >[EMAIL PROTECTED] wrote: >>When you boil it down, Twisted's event loop ... >But that is exactly the problem I have with Twisted. For HTTP ... >From that word on out, you have completely lost the plot. I am talking about Twisted's _event loop_. I specifically mentioned that the higher levels (and specifically "web") are separate, and can be discussed separately. If there were interest in porting the simpler XXXHTTPServer APIs in the stdlib to be event-driven and work with a Twisted-style event loop, it would be a relatively simple hack. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 02:22 pm, [EMAIL PROTECTED] wrote: >On Wed, Feb 14, 2007, [EMAIL PROTECTED] wrote: >> >> As I said, I don't have time to write the PEPs myself, but I might fix >> some specific bugs if there were a clear set of issues preventing this >> from moving forward. Better integration with the standard library >> would definitely be a big win for both Twisted and Python. > >Here's where I'm coming from: > >My first experience with Twisted was excellent: I needed a web server in >fifteen minutes to do my PyCon presentation, and it Just Worked. > >My second experience with Twisted? Well, I didn't really have one. Thanks for the feedback, Aahz... I'm a little confused, though. Is this just a personal anecdote, or were you trying to make a larger point about Twisted's suitability for the stdlib? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
[EMAIL PROTECTED] schrieb: > On 02:20 am, [EMAIL PROTECTED] wrote: > >> If Twisted is designed so that it absolutely *has* to use its own >> special event mechanism, and everything else needs to be modified >> to suit its requirements, then it's part of the problem, not part >> of the solution. > > I've often heard this complaint, usually of the form "that's > twisted-specific". The thing is, Twisted isn't specific. Its event > mechanism isn't "special". In fact it's hard to imagine how it might > be made less "special" than it currently is. > > Whatever your event dispatch mechanism, *some* code has to be > responsible for calling the OS API of select(), > WaitForMultipleEvents(), g_main_loop_run(), or whatever. Twisted > actually imposes very few requirements for code to participate in > this, and was designed from day one specifically to be a generalized > mainloop mechanism which would not limit you to one underlying > multiplexing implementation, event-dispatch mechanism, or operating > system if you used its API. When I last looked at twisted (that is several years ago), there were several reactors - win32reactor, wxreactor, maybe even more. And they didn't even work too well. The problems I remember were that the win32reactor was limited to a only handful of handles, the wxreactor didn't process events when a wx modal dialog boy was displayed, and so on. Has this changed? Thanks, Thomas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 08:52 pm, [EMAIL PROTECTED] wrote: >When I last looked at twisted (that is several years ago), there were >several reactors - win32reactor, wxreactor, maybe even more. Yes. That's intentional. Each of those systems offers its own event loop, and each reactor implements the basic operations in terms of those loops. They all have the same API. Application code does 'from twisted.internet import reactor; reactor.listenTCP', 'reactor.callLater', etc. Only the very top-most level decides which reactor the application will use. >And they >didn't even work too well. The problems I remember were that the win32reactor >was limited to a only handful of handles, the wxreactor didn't process >events when a wx modal dialog boy was displayed, and so on. Has this changed? win32eventreactor is limited to 64 handles because WaitForMultipleObjects is limited to 64 handles. wxreactor's event loop is ghastly and actually did pause when a modal dialog box is displayed (that has since been fixed). Process support on win32 now works in the default select reactor as well as the gtk reactor, so win32reactor is mostly a curiosity at this point (useful mainly if you want to implement your own GDI-based GUI, as PyUI did at one point), and its limitations are not as serious for Twisted as a whole. In other words, Twisted exposes the platform limitations in its platform-specific event loop implementations, and only works around them where it's possible to do so without masking platform functionality. For servers, the epoll, poll, and select reactors work just fine. The select reactor does have a maximum of FD_SETSIZE simultaneous sockets as well, but it is very easy to switch reactors if you need something more scalable. For clients, the best GUI toolkit for Twisted applications at this point is GTK, but WX, QT and Cocoa can all be made to work with a minimum of hassle. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
[EMAIL PROTECTED] wrote: > On 08:52 pm, [EMAIL PROTECTED] wrote: > > >When I last looked at twisted (that is several years ago), there were > >several reactors - win32reactor, wxreactor, maybe even more. > > Only the very top-most level decides which reactor the application will use. This is a worry, because it implies that there has to *be* a top level that knows what kind of reactor the whole application will use, and all parts of the application need to know that they will be getting their reactor from that top level. That may not be the case. For example, you want to incorporate some piece of event-driven code written by someone else into your gtk application. But it wasn't written with gtk in mind, so it doesn't know to use a gtkreactor, or how to get a reactor that it can use from your application. This is not my idea of what another poster called a "one-stop shop" -- a common API that different pieces of code can call independently without having to know about each other. To my mind, there shouldn't be a "reactor" object exposed to the application at all. There should just be functions for setting up callbacks. The choice of implementation should be made somewhere deep inside the library, based on what platform is being used. So at this point I'm skeptical that the Twisted API for these things should be adopted as-is. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Greg Ewing wrote: > [EMAIL PROTECTED] wrote: >> On 08:52 pm, [EMAIL PROTECTED] wrote: >> >> >When I last looked at twisted (that is several years ago), there were >> >several reactors - win32reactor, wxreactor, maybe even more. >> >> Only the very top-most level decides which reactor the application will use. > > This is a worry, because it implies that there has to > *be* a top level that knows what kind of reactor the > whole application will use, and all parts of the > application need to know that they will be getting > their reactor from that top level. > > That may not be the case. For example, you want to > incorporate some piece of event-driven code written > by someone else into your gtk application. But it > wasn't written with gtk in mind, so it doesn't know > to use a gtkreactor, or how to get a reactor that it > can use from your application. > If the borrowed code takes a reactor parameter then presumably the top-level code can pass the appropriate reactor type in. If the borrowed code uses a fixed reactor then it's difficult to see how changes to the Twisted API could help. "Incorporating some piece of event-driven code written by someone else" implies specific assumptions about event types and delivery, surely. That's why it's difficult to port code between GUI toolkits, for example, and even more so to write code that runs on several toolkits without change. > This is not my idea of what another poster called a > "one-stop shop" -- a common API that different pieces > of code can call independently without having to know > about each other. > > To my mind, there shouldn't be a "reactor" object > exposed to the application at all. There should just > be functions for setting up callbacks. The choice of > implementation should be made somewhere deep inside > the library, based on what platform is being used. > You seem to be arguing for libraries that contain platform dependencies to handle multiple platforms. Glyph seems to prefer the ability for the library caller to pass in handlers for platform-dependent features. > So at this point I'm skeptical that the Twisted > API for these things should be adopted as-is. > Since Glyph has already stated his opinion that Twisted isn't yet ready for adoption as-is this doesn't add to the discussion. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
At 01:31 PM 2/15/2007 +1300, Greg Ewing wrote: >To my mind, there shouldn't be a "reactor" object >exposed to the application at all. There should just >be functions for setting up callbacks. The choice of >implementation should be made somewhere deep inside >the library, based on what platform is being used. *shudder*. I, on the other hand, prefer to assume that there is no one "top level" and certainly no requirement for a single event loop or reactor. peak.events, for example, lets you have multiple event loops running in the same or different threads. One of these can be Twisted's reactor, if you like, but the framework doesn't impose this singleton-ness on you. (IIUC, Twisted uses a singleton for at least two fairly good reasons: ease of programming, and the fact that certain platforms demand it for performance reasons. I just don't happen to agree that this limitation should be applied across *all* platforms. And there are good solutions for context-specific pseudo-singletons to address the API issue, although they might not be as high-performance as Twisted applications might prefer. Everything's a trade-off.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 2/14/07, Greg Ewing <[EMAIL PROTECTED]> wrote: [EMAIL PROTECTED] wrote: > On 08:52 pm, [EMAIL PROTECTED] wrote: > > >When I last looked at twisted (that is several years ago), there were > >several reactors - win32reactor, wxreactor, maybe even more. > > Only the very top-most level decides which reactor the application will use. This is a worry, because it implies that there has to *be* a top level that knows what kind of reactor the whole application will use, and all parts of the application need to know that they will be getting their reactor from that top level. That may not be the case. For example, you want to incorporate some piece of event-driven code written by someone else into your gtk application. But it wasn't written with gtk in mind, so it doesn't know to use a gtkreactor, or how to get a reactor that it can use from your application. This is not my idea of what another poster called a "one-stop shop" -- a common API that different pieces of code can call independently without having to know about each other. To my mind, there shouldn't be a "reactor" object exposed to the application at all. There should just be functions for setting up callbacks. The choice of implementation should be made somewhere deep inside the library, based on what platform is being used. Eh, your own example seems to argue the opposite. If the choice for reactor was made somewhere deep inside the library, how does it know to use the GTK reactor? Only the code using GTK knows that it's going to need the GTK reactor. The generic code just uses whatever reactor was selected, automatically. The reason the decision should be made at the topmost level is that this is the most visible and adaptive location. A library or framework has to handle a boundless number of variations of environment, use and co-operation with other libraries and frameworks, whereas the topmost script knows exactly what it wants to do. A library that uses PyGTK can say 'I need the GTK reactor', but it would be wrong. All it needs is a reactor that interacts nicely with GTK. If there are multiple reactors that integrate nicely with GTK, there is no reason for the library to insist on one particular implementation. And it could be in the way, if a second library needs a different kind of integration and it knows of a reactor that supports both that *and* GTK integration. And if the whole system breaks, for some reason, the first place the programmer will look is in his own code, not deep inside another library. For what it's worth, I think a single event-handling mainloop API in the standard library is a *great* idea. Using part of Twisted for it (suitably adapted) is an even greater idea: a whole lot of work and brainpower went into Twisted's core, and it wasn't wasted. Duplicating the effort would be a big mistake and a terrible waste. A from-scratch implementation of an event-handling mechanism will not solve any of the "issues" most people have with Twisted. It will have many more bugs than Twisted (all the ones Twisted had and fixed, sometimes very obscure and hard-to-find bugs), it will not be significantly less complex (Twisted is as complex as it is because it had to be, to do what it is supposed to do) and it will not require any less of a brain-warp to understand (because that's just what event-driven programming takes.) If I wasn't short on free time (spending most of it on Py3k and, soon, py3k-forward-compatibility) I would do all the necessary Twisted-integration work myself :-) -- Thomas Wouters <[EMAIL PROTECTED]> Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 12:31 am, [EMAIL PROTECTED] wrote: >[EMAIL PROTECTED] wrote: >> On 08:52 pm, [EMAIL PROTECTED] wrote: >> >> >When I last looked at twisted (that is several years ago), there were >> >several reactors - win32reactor, wxreactor, maybe even more. >> >> Only the very top-most level decides which reactor the application will use. > >This is a worry, because it implies that there has to >*be* a top level that knows what kind of reactor the >whole application will use, and all parts of the >application need to know that they will be getting >their reactor from that top level. The default reactor is the most portable one, 'select', and if no other reactor is installed, that's the one that will be used. >That may not be the case. For example, you want to >incorporate some piece of event-driven code written >by someone else into your gtk application. But it >wasn't written with gtk in mind, so it doesn't know >to use a gtkreactor, or how to get a reactor that it >can use from your application. "from twisted.internet import reactor" is the way you get at the reactor, regardless of which one is currently installed. There can only be one reactor active at any given time, because at the very bottom of the event-handling machinery _some_ platform multiplexing API must be called, and that is mostly what the reactor represents. The GTK reactor does not have its own API. It simply allows you to use GTK APIs as well, by back-ending to the glib mainloop. That is, in fact, the whole point of having a "reactor" API in the first place. >This is not my idea of what another poster called a >"one-stop shop" -- a common API that different pieces >of code can call independently without having to know >about each other. >To my mind, there shouldn't be a "reactor" object >exposed to the application at all. There should just >be functions for setting up callbacks. That's what the Twisted "reactor" object *is*, exactly. Functions (well, methods) for setting up callbacks. >The choice of >implementation should be made somewhere deep inside >the library, based on what platform is being used. The "deep inside the library" decision is actually a policy decision made by a server's administrator, or dependent upon the GUI library being used if you need to interact with a GUI event loop. Perhaps the default selection could be better, but installing a reactor is literally one line of code, or a single command-line option to the "twistd" daemon. See: http://twistedmatrix.com/projects/core/documentation/howto/choosing-reactor.html It is completely transparent to the application, _unless_ the application wants to make use of platform-specific features. See the following for more information: http://www.cs.wustl.edu/~schmidt/PDF/reactor-siemens.pdf although technically Twisted's "reactor" is more like the slightly higher level POSA "proactor" pattern; asyncore is more like a true "reactor" in the sense discussed in that paper. Twisted exposes various APIs for "setting up callbacks" exactly as you describe: http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.IReactorTCP.html http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.IReactorTime.html >So at this point I'm skeptical that the Twisted >API for these things should be adopted as-is Given that your supporting arguments were almost exactly the opposite of the truth in every case, I think this conclusion should be re-examined :). If you're interested in how a normal Twisted application looks, see this tutorial: http://twistedmatrix.com/projects/core/documentation/howto/servers.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Steve Holden wrote: > If the borrowed code takes a reactor parameter then presumably the > top-level code can pass the appropriate reactor type in. Since there should only be one reactor at a time in any given application, it shouldn't have to be passed in -- it could be held in a global variable deep inside the library. Only the code which creates the reactor initially needs to know about that variable, or even that there is such a thing as a reactor. > "Incorporating some piece of event-driven code written by someone else" > implies specific assumptions about event types and delivery, surely. It requires agreement on how to specify the event types and what to do in response, but that's all it should require. The way I envisage it, setting up an event callback should be like opening a file -- there's only one way to do it, and you don't have to worry about what the rest of the application is doing. You don't have to get passed an object that knows how to open files -- it's a fundamental service provided by the system. You just use it. > That's why it's difficult to port code between GUI toolkits, for > example, and even more so to write code that runs on several toolkits > without change. Just in case it's not clear, the events I'm talking about are things like file and socket I/O, not GUI events. Trying to use two different GUIs at once is not something I'm addressing. Rather, you should be able to write code that does e.g. some async socket I/O, and embed it in a GUI app using e.g. gtk, without having to modify it to take account of the fact that it's working in a gtk environment, or having to parameterise it to allow for such things. > You seem to be arguing for libraries that contain platform dependencies > to handle multiple platforms. I'm arguing that as much of the platform dependency as possible should be in the asyncore library (or whatever replaces it). The main application code *might* have to give it a hint such as "this app uses gtk", but no more than that. And ideally, I'd prefer it not to even have to do that -- pygtk should do whatever is necessary to hook itself into asyncore if at all possible, not the other way around. > Since Glyph has already stated his opinion that Twisted isn't yet ready > for adoption as-is this doesn't add to the discussion. Okay, but one of the suggestions made seemed to be "why not just use the Twisted API". I'm putting forward a possible reason. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Thomas Wouters wrote: > If the choice for > reactor was made somewhere deep inside the library, how does it know to > use the GTK reactor? In my ideal world, there wouldn't actually be a gtk reactor -- there would only be a Linux reactor, a MacOSX reactor, a Windows reactor, etc. Things like pygtk would be adapted to hook into the platform's standard reactor. Less ideal would be for pygtk to intall a gtk reactor when it gets imported. The danger with this approach is that two libraries could fight over which kind of reactor to use. You suggest that the top level could choose some other reactor that is compatible with both libraries. That seems like a rather hit-and-miss approach -- such a reactor might exist, or it might not. If not, you're out of luck. And the chance of finding a suitable reactor gets smaller as the number of libraries increases. > The reason the decision should be made at the topmost level is that this > is the most visible and adaptive location. This is where my vision is fundamentally different: you shouldn't have to *make* a decision in the first place. All event-driven libraries should be made to use the same substrate on any given platform. Then they can coexist without the need for any top-level choices. I know that will be hard to do, but it's the only way out of this mess that I can see. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 2/14/07, Greg Ewing <[EMAIL PROTECTED]> wrote: I know that will be hard to do, but it's the only way out of this mess that I can see. That depends on what you consider messy about it. *I* don't like the idea of something in the Python installation deciding which reactor to use. It's my application, and I'm damn well going to tell it what to do. If that means it doesn't work as I expected, it's my own fault :-) In any case, your idea requires a lot of changes in external, non-Python code -- PyGTK simply exposes the GTK mainloop, which couldn't care less about Python's idea of a perfect event reactor model. While those issues are being settled, we'll have to cope with selecting the right reactor manually. It's not all that different from what you want, in any case. The PerfectReactor can be added later, all current reactors aliased to it, and no one would have to change a single line of code. -- Thomas Wouters <[EMAIL PROTECTED]> Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Phillip J. Eby wrote: > peak.events, for example, lets you have multiple event loops > running in the same or different threads. Different threads is okay if you're willing to use threads, but you might not. The reason you're using an event loop may well be precisely so that you *don't* have to use threads. And... how do you run multiple event loops simultaneously in the *same* thread? That sounds self-contradictory to me. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 2/14/07, Greg Ewing <[EMAIL PROTECTED]> wrote: Phillip J. Eby wrote: > peak.events, for example, lets you have multiple event loops > running in the same or different threads. Different threads is okay if you're willing to use threads, but you might not. The reason you're using an event loop may well be precisely so that you *don't* have to use threads. And... how do you run multiple event loops simultaneously in the *same* thread? That sounds self-contradictory to me. If all (or all-but-one) of them have a 'run one iteration' method, you can call that from the 'main' mainloop. Or you can design all mainloops as coroutines and have them call each other. (I haven't looked at Phillip's approach at all, but something tells me coroutines are involved :-) -- Thomas Wouters <[EMAIL PROTECTED]> Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On Thu, 15 Feb 2007 15:47:39 +1300, Greg Ewing <[EMAIL PROTECTED]> wrote: >Steve Holden wrote: > >> If the borrowed code takes a reactor parameter then presumably the >> top-level code can pass the appropriate reactor type in. > >Since there should only be one reactor at a time in >any given application, it shouldn't have to be passed >in -- it could be held in a global variable deep >inside the library. Only the code which creates the >reactor initially needs to know about that variable, >or even that there is such a thing as a reactor. Whether or not the premise here is accurate may be out of scope for this thread. Or it may not be. I dunno. However, I do want to point out that it is not necessarily correct that there should be only one reactor at a time in a given application. PJE has already explained that peak.events can have multiple reactors. Twisted is tied to one, but this may not always be the case. Whether there is a default reactor for applications that don't care about the ability to have more than one at a time is yet another question which may be worth examining. These are the kinds of things which should be spelled out in a PEP, including the rationale for any particular policy decisions (which should be kept to an absolute minimum) are made. > >> "Incorporating some piece of event-driven code written by someone else" >> implies specific assumptions about event types and delivery, surely. > >It requires agreement on how to specify the event types >and what to do in response, but that's all it should >require. > >The way I envisage it, setting up an event callback >should be like opening a file -- there's only one way >to do it, and you don't have to worry about what the >rest of the application is doing. You don't have to >get passed an object that knows how to open files -- >it's a fundamental service provided by the system. >You just use it. If we suppose that files and sockets are supported in roughly the same way, and we suppose that sockets are supported in the way that Twisted supports them, then there is no difficulty supporting files in this way. :) > >> That's why it's difficult to port code between GUI toolkits, for >> example, and even more so to write code that runs on several toolkits >> without change. > >Just in case it's not clear, the events I'm talking >about are things like file and socket I/O, not GUI >events. Trying to use two different GUIs at once is >not something I'm addressing. Alright, good. Getting two different GUI libraries to play together is a pretty hairy task indeed, and well worth keeping separate from this one. :) > >Rather, you should be able to write code that does >e.g. some async socket I/O, and embed it in a GUI >app using e.g. gtk, without having to modify it to >take account of the fact that it's working in a >gtk environment, or having to parameterise it to >allow for such things. Excellent. To be clear, this is how the Twisted model works, with respect to integration with GUI toolkits. I would not enjoy working with a system in which this was not the case. > >> You seem to be arguing for libraries that contain platform dependencies >> to handle multiple platforms. > >I'm arguing that as much of the platform dependency >as possible should be in the asyncore library (or >whatever replaces it). Certainly. Library code doesn't care if the event loop is driven by select or poll or epoll or /dev/poll or kqueue or aio or iocp or win32 events or realtime signals or kaio or whatever gnarly thing is hidden in gtk or whatever gnarly thing is hidden inside qt or whatever gnarly thing is hidden inside COM or whatever gnarly thing is hidden inside wxWidgets. It cares about what features are available. It requests them somehow, and uses them. If they are unavailable, then it can decide whether the lack is catastrophic and give up or if it can be worked around somehow. The way a Twisted application does this is based on interfaces. Assuming interfaces continue to not be present in the stdlib, a stdlib event loop would have to find some other API for presenting this information, but it is not a very hard problem to solve. >The main application code >*might* have to give it a hint such as "this app >uses gtk", but no more than that. And ideally, I'd >prefer it not to even have to do that -- pygtk >should do whatever is necessary to hook itself >into asyncore if at all possible, not the other >way around. There is some advantage to declaring things up front, lest you get into the situation where you are partway through using code which will suddenly begin to demand Gtk at the same time as you are partway through using code which will suddenly begin to demand Qt, at which point you are in trouble. But this is another minor point. > >> Since Glyph has already stated his opinion that Twisted isn't yet ready >> for adoption as-is this doesn't add to the discussion. > >Okay, but one of the suggestions made seemed to be >"why not just use the Tw
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On Thu, 15 Feb 2007 16:18:40 +1300, Greg Ewing <[EMAIL PROTECTED]> wrote: > [snip] > >This is where my vision is fundamentally different: >you shouldn't have to *make* a decision in the first >place. All event-driven libraries should be made to >use the same substrate on any given platform. Then >they can coexist without the need for any top-level >choices. > >I know that will be hard to do, but it's the only >way out of this mess that I can see. > Thomas already pointed this out, but I'm repeating it anyway. This vision represents an impossible reality at present. You will not get Gtk or Qt or wxWidgets to use Python's event notification API. If you are really very interested in solving this problem, go to the developers of each platform those toolkits run on and sell them on a unified event notification API. Once they have adopted, implemented, and deployed it, you can go to the Gtk, Qt, and wxWidgets teams and tell them to port all of their code to that new API. Then, you can have a unified model in Python. Until then, the practical compromise with almost zero negative consequences (sometimes, one extra piece of configuration will be required - compare this to how the logging module works ;) is to optionally allow explicit reactor selection. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
At 04:25 PM 2/15/2007 +1300, Greg Ewing wrote: >Phillip J. Eby wrote: >>peak.events, for example, lets you have multiple event loops running in >>the same or different threads. > >Different threads is okay if you're willing to use threads, >but you might not. The reason you're using an event loop >may well be precisely so that you *don't* have to use >threads. > >And... how do you run multiple event loops simultaneously >in the *same* thread? When one is nested inside the other. This isn't a common need, but it's occasionally useful if you need to switch back and forth between blocking and non-blocking code. For example, suppose that you have some code that wants to offer a synchronous interface to an asynchronous library... and the synchronous code is being called from a FastCGI "accept" event loop. The inner code can't use the outer event loop, because the outer loop isn't going to proceed until the inner code is finished. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Thomas Wouters wrote: > If all (or all-but-one) of them have a 'run one iteration' method, you > can call that from the 'main' mainloop. But without some way of blocking until an event arrives for *either* loop, you have to resort to some kind of busy polling, which is not elegant. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Thomas Wouters wrote: > *I* don't like the idea of something in the Python installation > deciding which reactor to use. I wouldn't mind if some way were provided of changing the reactor if you want. I'd just like to see a long term goal of making it unnecessary as far as possible. > In any case, your idea requires a lot of changes in external, non-Python > code -- PyGTK simply exposes the GTK mainloop, which couldn't care less > about Python's idea of a perfect event reactor model. On unix at least, I don't think it should be necessary to change gtk, only pygtk. If it can find out the file descriptor of the connection to the X server, it can plug that into the reactor, and then call gtk_main_iteration_do() whenever something comes in on it. A similar strategy ought to work for any X11-based toolkit that exposes a function to perform one iteration of its main loop. Mileage on other platforms may vary. > The PerfectReactor can be added later, all current reactors > aliased to it, and no one would have to change a single line > of code. Sure. The other side to all this is the client side, i.e. the code that installs event callbacks. At the moment there's no clear direction to take, so everyone makes their own choice -- some use asyncore, some use Twisted, some use the gtk event loop, some roll their own, etc. If it were made known that asyncore or some other thing in the stdlib was intended to become the standard, then it would give people some guidance as to how to write future event-driven code. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Phillip J. Eby wrote: > At 04:25 PM 2/15/2007 +1300, Greg Ewing wrote: >> Phillip J. Eby wrote: >>> peak.events, for example, lets you have multiple event loops running in >>> the same or different threads. >> Different threads is okay if you're willing to use threads, >> but you might not. The reason you're using an event loop >> may well be precisely so that you *don't* have to use >> threads. >> >> And... how do you run multiple event loops simultaneously >> in the *same* thread? > > When one is nested inside the other. This isn't a common need, but it's > occasionally useful if you need to switch back and forth between blocking > and non-blocking code. For example, suppose that you have some code that > wants to offer a synchronous interface to an asynchronous library... and > the synchronous code is being called from a FastCGI "accept" event > loop. The inner code can't use the outer event loop, because the outer > loop isn't going to proceed until the inner code is finished. I actually have some code that works a little bit like this. I have a SelectLoop class that the application knows about and explicitly attaches transport objects to, and queues events on the loop. I have two ways to perform some actions in a "blocking" way. 1) I can create a new SelectLoop and move some transports temporarily from the main loop to the temporary loop to ignore all other events for a while. 2) I can re-enter the SelectLoop run() method and keep reacting to all events. Once the right event occurs - I exit from the nested run() and continue on. This does require a lot more mental effort though. Note that my code is nowhere near as ambitious as Twisted, but I watned to have the above flexibility. The code really only has to work on AIX and Linux, and in a limited way, Windows. - Dave ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 2/14/07, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > When one is nested inside the other. This isn't a common need, but it's > occasionally useful if you need to switch back and forth between blocking > and non-blocking code. For example, suppose that you have some code that > wants to offer a synchronous interface to an asynchronous library... and > the synchronous code is being called from a FastCGI "accept" event > loop. The inner code can't use the outer event loop, because the outer > loop isn't going to proceed until the inner code is finished. This would also let you wrap sys.stdout.write() in a nested event loop so as to allow print statements to still work while you use have it set to non-blocking mode, but I could see it argued that using print statements at all is wrong at that point. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Thomas Wouters schrieb: > If all (or all-but-one) of them have a 'run one iteration' method, you > can call that from the 'main' mainloop. That doesn't really work (and neither do variations involving coroutines. Either the 'run one iteration' method blocks until one even arrives, in which case it may block for a long time while other loops have events pending (which then won't get processed). Or each 'run one iteration' method has a polling mode (e.g. with a time-out), in which case you get busy wait (you can fiddle with the time-out value to trade business vs. responsiveness). The 'run one iteration' approach doesn't support writing applications that are single-threaded, responsive, and idle when there is nothing to do. I can't believe that coroutines help in any way here. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Greg Ewing schrieb: > On unix at least, I don't think it should be necessary > to change gtk, only pygtk. If it can find out the file > descriptor of the connection to the X server, it can > plug that into the reactor, and then call > gtk_main_iteration_do() whenever something comes in > on it. That is insufficient. The gtk main loop has more input sources than just the connection to X: - timers can be registered, which are called when the time comes - idle handlers can be registered which are called when there are no other events - child handlers are invoked when a child process terminates - additional file descriptors can be registered (probably used for sockets primarily) - a generalzed 'event source' can be hooked into it, with C functions for prepare, check, dispatch, and finalize See http://www.gtk.org/api/2.6/glib/glib-The-Main-Event-Loop.html Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
I was the one on the Stackless list who last September or so
proposed the idea of monkeypatching and I'm including that
idea in my presentation for PyCon. See my early rough draft
at http://www.stackless.com/pipermail/stackless/2007-February/002212.html
which contains many details about using Stackless, though
none on the Stackless implementation. (A lot on how to tie things together.)
So people know, I am an applications programmer and not a
systems programmer. Things like OS-specific event mechanisms
annoy and frustrate me. If I could do away with hardware and
still write useful programs I would.
I have tried 3 times to learn Twisted. The first time I found
and reported various problems and successes. See emails at
http://www.twistedmatrix.com/pipermail/twisted-python/2003-June/thread.html
The second time was to investigate a way to report upload
progress: http://twistedmatrix.com/trac/ticket/288
and the third was to compare Allegra and Twisted
http://www.dalkescientific.com/writings/diary/archive/2006/08/28/levels_of_abstraction.html
In all three cases I've found it hard to use Twisted because
the code didn't do as I expected it to do and when something
went wrong I got results which were hard to interpret. I
believe others have similar problems and is one reason Twisted
is considered to be "a big, complicated, inseparable hairy mess."
I find the Stackless code also hard to understand. Eg,
I don't know where the watchdog code is for the "run()"
command. It uses several layers of macros and I haven't
been able get it straight in my head. However, so far
I've not run into strange errors in Stackless that I
have in Twisted.
I find the normal Python code relatively easy to understand.
Stackless only provides threadlets. It does no I/O.
Richard Tew developed a "stacklesssocket" module which emulates
the API for the stdlib "socket" module. I tweaked it a
bit and showed that by doing the monkeypatch
import stacklesssocket
import sys
sys.modules["socket"] = stacklesssocket
then code like "urllib.urlopen" became Stackless compatible.
Eg, in my PyCon talk draft I show something like
import slib
# must monkeypatch before any other module imports "socket"
slib.use_monkeypatch()
import urllib2
import time
import hashlib
def fetch_and_reverse(host):
t1 = time.time()
s = urllib2.urlopen("http://"+host+"/";).read()[::-1]
dt = time.time() - t1
digest = hashlib.md5(s).hexdigest()
print "hash of %r/ = %s in %.2f s" % (host, digest, dt)
slib.main_tasklet(fetch_and_reverse)("www.python.org")
slib.main_tasklet(fetch_and_reverse)("docs.python.org")
slib.main_tasklet(fetch_and_reverse)("planet.python.org")
slib.run_all()
where the three fetches occur in parallel.
The choice of asyncore is, I think, done because 1) it
prevents needing an external dependency, 2) asyncore is
smaller and easier to understand than Twisted, and
3) it was for demo/proof of concept purposes. While
tempting to improve that module I know that Twisted
has already gone though all the platform-specific crap
and I don't want to go through it again myself. I don't
want to write a reactor to deal with GTK, and one for
OS X, and one for ...
Another reason I think Twisted is considered "tangled-up
Deep Magic, only for Wizards Of The Highest Order" is because
it's infused with event-based processing. I've done a lot
of SAX processing and I can say that few people think that
way or want to go through the process of learning how.
Compare, for example, the following
f = urllib2.urlopen("http://example.com/";)
for i, line in enumerate(f):
print ("%06d" % i), repr(line)
with the normal equivalent in Twisted or other
async-based system.
Yet by using the Stackless socket monkeypatch, this
same code works in an async framework. And the underlying
libraries have a much larger developer base than Twisted.
Want NNTP? "import nntplib" Want POP3? "import poplib"
Plenty of documentation about them too.
On the Stackless mailing list I have proposed someone work
on a talk for EuroPython titled "Stackless and Twisted".
Andrew Francis has been looking into how to do that.
All the earlier quotes were lifted from glyph. Here's another:
> When you boil it down, Twisted's event loop is just a
> notification for "a connection was made", "some data was
> received on a connection", "a connection was closed", and
> a few APIs to listen or initiate different kinds of
> connections, start timed calls, and communicate with threads.
> All of the platform details of how data is delivered to the
> connections are abstracted away.. How do you propose we
> would make a less "specific" event mechanism?
What would I need to do to extract this Twisted core so
I could replace asyncore? I know at minimum I need
"twisted.internet" and "twisted.python" (the latter for
logging) and "twisted.persisted" for "styles.Ephemeral".
But I say this hesitantly recalling the frustrations
I had in dealing with a co
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 2/14/07, Greg Ewing <[EMAIL PROTECTED]> wrote:
> Thomas Wouters wrote:
>
> > *I* don't like the idea of something in the Python installation
> > deciding which reactor to use.
>
> I wouldn't mind if some way were provided of changing
> the reactor if you want. I'd just like to see a long
> term goal of making it unnecessary as far as possible.
>
> > In any case, your idea requires a lot of changes in external, non-Python
> > code -- PyGTK simply exposes the GTK mainloop, which couldn't care less
> > about Python's idea of a perfect event reactor model.
>
> On unix at least, I don't think it should be necessary
> to change gtk, only pygtk. If it can find out the file
> descriptor of the connection to the X server, it can
> plug that into the reactor, and then call
> gtk_main_iteration_do() whenever something comes in
> on it.
>
> A similar strategy ought to work for any X11-based
> toolkit that exposes a function to perform one
> iteration of its main loop.
>
> Mileage on other platforms may vary.
>
> > The PerfectReactor can be added later, all current reactors
> > aliased to it, and no one would have to change a single line
> > of code.
>
> Sure.
>
> The other side to all this is the client side, i.e. the
> code that installs event callbacks. At the moment there's
> no clear direction to take, so everyone makes their own
> choice -- some use asyncore, some use Twisted, some use
> the gtk event loop, some roll their own, etc.
There is no single PerfectReactor. There are several use cases where
you need to wait on >1 different event systems, which guarantees at
least two OS threads (and two event loops). In general it's nice to
have a single Python event loop ("the reactor") to act on said threads
(e.g. something just sitting on a mutex waiting for messages) but
waiting for IO to occur should *probably* happen on one or more
ancillary threads -- one per event system (e.g. select, GTK,
WaitForMultipleEvents, etc.)
-bob
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On Thu, 15 Feb 2007 02:36:22 -0700, Andrew Dalke <[EMAIL PROTECTED]> wrote: >I was the one on the Stackless list who last September or so >proposed the idea of monkeypatching and I'm including that >idea in my presentation for PyCon. See my early rough draft >at http://www.stackless.com/pipermail/stackless/2007-February/002212.html >which contains many details about using Stackless, though >none on the Stackless implementation. (A lot on how to tie things together.) > >So people know, I am an applications programmer and not a >systems programmer. Things like OS-specific event mechanisms >annoy and frustrate me. If I could do away with hardware and >still write useful programs I would. What a wonderful world it would be. :) > > [snip] > >In all three cases I've found it hard to use Twisted because >the code didn't do as I expected it to do and when something >went wrong I got results which were hard to interpret. I >believe others have similar problems and is one reason Twisted >is considered to be "a big, complicated, inseparable hairy mess." > >I find the Stackless code also hard to understand. Eg, >I don't know where the watchdog code is for the "run()" >command. It uses several layers of macros and I haven't >been able get it straight in my head. However, so far >I've not run into strange errors in Stackless that I >have in Twisted. > As you point out below, however, Twisted and stackless achieve different goals. >I find the normal Python code relatively easy to understand. > >Stackless only provides threadlets. It does no I/O. >Richard Tew developed a "stacklesssocket" module which emulates >the API for the stdlib "socket" module. I tweaked it a >bit and showed that by doing the monkeypatch > > import stacklesssocket > import sys > sys.modules["socket"] = stacklesssocket > >then code like "urllib.urlopen" became Stackless compatible. >Eg, in my PyCon talk draft I show something like > It may be of interest to you to learn that a Twisted developer implement this model several years ago. It has not been further developed for a handful of reasons, at the core of which is the fact that it is very similar to pre-emptive threading in terms of application-level complexity. You gave several examples of the use of existing code which expects a blocking socket interface and which "just works" when the socket module is changed in this way. However, this is a slight simplification. Code written without expecting a context switch (exactly what happens when a socket operation is performed in this model) is not necessarily correct when context switches are suddenly introduced. Consider this extremely trivial example: x = 0 def foo(conn): global x a = x + 1 b = ord(conn.recv(1)) x = a + b return x Clearly, foo is not threadsafe. Global mutable state is a terrible, terrible thing. The point to note is that by introducing a context switch at the conn.recv(1) call, the same effect is achieved as by any other context switch: it becomes possible for foo to return an inconsistent result or otherwise corrupt its own state if another piece of code violates its assumptions and changes x while it is waiting for the recv call to complete. Is urllib2 threadsafe? I have heard complaints that it is not. I have looked at the code, and at least in its support for caching, it appears not to be. Perhaps it can be made threadsafe, but in requiring that, the advantage of having a whole suite of modules which will "just work" with a transparently context switching socket module are mostly lost. > > [snip - urllib2/tasklet example] > >The choice of asyncore is, I think, done because 1) it >prevents needing an external dependency, But if some new event loop is introduced into the standard library, then using it also will not require an external dependency. ;) >2) asyncore is >smaller and easier to understand than Twisted, While I hear this a lot, applications written with Twisted _are_ shorter and contain less irrelevant noise in the form of boilerplate than the equivalent asyncore programs. This may not mean that Twisted programs are easier to understand, but it is at least an objectively measurable metric. >and >3) it was for demo/proof of concept purposes. >While >tempting to improve that module I know that Twisted >has already gone though all the platform-specific crap >and I don't want to go through it again myself. I don't >want to write a reactor to deal with GTK, and one for >OS X, and one for ... > Now if we can only figure out a way for everyone to benefit from this without tying too many brains up in knots. :) > >Another reason I think Twisted is considered "tangled-up >Deep Magic, only for Wizards Of The Highest Order" is because >it's infused with event-based processing. I've done a lot >of SAX processing and I can say that few people think that >way or want to go through the process of learning how. > >Compare, for example, the following > > f = urllib2.
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On Thu, Feb 15, 2007 at 09:19:30AM -0500, Jean-Paul Calderone wrote:
> >That feels like 6 layers too many, given that
> > _logrun(selectable, _drdw, selectable, method, dict)
> > return context.call({ILogContext: newCtx}, func, *args, **kw)
> > return self.currentContext().callWithContext(ctx, func, *args, **kw)
> > return func(*args, **kw)
> > getattr(selectable, method())
> > klass(number, string)
> >
> >are all generic calls.
>
> I know function calls are expensive in Python, and method calls even more
> so... but I still don't understand this issue. Twisted's call stack is too
> deep? It is fair to say it is deep, I guess, but I don't see how that is a
> problem. If it is, I don't see how it is specific to this discussion.
It's hard to debug the resulting problem. Which level of the *12*
levels in the stack trace is responsible for a bug? Which of the *6*
generic calls is calling the wrong thing because a handler was set up
incorrectly or the wrong object provided? The code is so 'meta' that
it becomes effectively undebuggable.
--amk
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On Thu, 15 Feb 2007 10:46:05 -0500, "A.M. Kuchling" <[EMAIL PROTECTED]> wrote:
>On Thu, Feb 15, 2007 at 09:19:30AM -0500, Jean-Paul Calderone wrote:
>> >That feels like 6 layers too many, given that
>> > _logrun(selectable, _drdw, selectable, method, dict)
>> > return context.call({ILogContext: newCtx}, func, *args, **kw)
>> > return self.currentContext().callWithContext(ctx, func, *args, **kw)
>> > return func(*args, **kw)
>> > getattr(selectable, method())
>> > klass(number, string)
>> >
>> >are all generic calls.
>>
>> I know function calls are expensive in Python, and method calls even more
>> so... but I still don't understand this issue. Twisted's call stack is too
>> deep? It is fair to say it is deep, I guess, but I don't see how that is a
>> problem. If it is, I don't see how it is specific to this discussion.
>
>It's hard to debug the resulting problem. Which level of the *12*
>levels in the stack trace is responsible for a bug? Which of the *6*
>generic calls is calling the wrong thing because a handler was set up
>incorrectly or the wrong object provided? The code is so 'meta' that
>it becomes effectively undebuggable.
I've debugged plenty of Twisted applications. So it's not undebuggable. :)
Application code tends to reside at the bottom of the call stack, so Python's
traceback order puts it right where you're looking, which makes it easy to
find. For any bug which causes something to be set up incorrectly and only
later manifests as a traceback, I would posit that whether there is 1 frame or
12, you aren't going to get anything useful out of the traceback. Standard
practice here is just to make exception text informative, I think, but this is
another general problem with Python programs and event loops, not one specific
to either Twisted itself or the particular APIs Twisted exposes.
As a personal anecdote, I've never once had to chase a bug through any of the
6 "generic calls" singled out. I can't think of a case where I've helped any
one else who had to do this, either. That part of Twisted is very old, it is
_very_ close to bug-free, and application code doesn't have very much control
over it at all. Perhaps in order to avoid scaring people, there should be a
way to elide frames from a traceback (I don't much like this myself, I worry
about it going wrong and chopping out too much information, but I have heard
other people ask for it)?
Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Bob Ippolito wrote:
> There is no single PerfectReactor. There are several use cases where
> you need to wait on >1 different event systems, which guarantees at
> least two OS threads (and two event loops). In general it's nice to
> have a single Python event loop ("the reactor") to act on said threads
> (e.g. something just sitting on a mutex waiting for messages) but
> waiting for IO to occur should *probably* happen on one or more
> ancillary threads -- one per event system (e.g. select, GTK,
> WaitForMultipleEvents, etc.)
Why couldn't PerfectReactor be a reactor for other reactors? A sort of
concentrator for these multiple event systems and multiple threads.
You ask to listen to sockets, so it instantiates a singleton
PerfectReactor which instantiates a select() reactor and listens to it
directly in a single-threaded manner. If you then ask to listen to
Win32 messages, the PerfectReactor instantiates a GetMessage() reactor.
Then, realizing it has two "event systems", it spawns a thread for each
child reactor with a listener that serializes the incoming events into
the PerfectReactor's queue. Bingo, your application doesn't need to be
written thread-safe, PerfectReactor is platform-agnostic, and you don't
have to know in advance all the event types you might ever listen to.
Sorry if this is a dumb question (or if I'm mangling the terminology),
/larry/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Larry Hastings schrieb:
> Bob Ippolito wrote:
>> There is no single PerfectReactor. There are several use cases where
>> you need to wait on >1 different event systems, which guarantees at
>> least two OS threads (and two event loops). In general it's nice to
>> have a single Python event loop ("the reactor") to act on said threads
>> (e.g. something just sitting on a mutex waiting for messages) but
>> waiting for IO to occur should *probably* happen on one or more
>> ancillary threads -- one per event system (e.g. select, GTK,
>> WaitForMultipleEvents, etc.)
> Why couldn't PerfectReactor be a reactor for other reactors? A sort of
> concentrator for these multiple event systems and multiple threads.
Because you can't write a single function that blocks (idle) until
an event in one of the systems occurs. None of the system is truly
asynchronous. All APIs, eventually, rely on synchronous, blocking,
operating system calls. In Unix, the call to make is typically
select or poll, and you know it is safe to make because you have
all even sources as file descriptors and you know nothing else
needs to be done before one of the events occurs.
Now, for these generalized event loops, it may not be possible anymore
to combine all event sources into a single blocking call. If you have
timeouts, you have to find out what the minimum timeout is (to make
it the timeout for the blocking call, assuming that supports a timeout).
If you have idle tasks, you have to invoke all idle tasks of all
reactors (which you may not be able to get because there is no API
to fetch them).
Even more difficult: if these use different OS event mechanisms
(on systems that have multiple such systems). E.g. try combinging
a Win32 GetMessage/PostMessage loop with a WaitForMultipleObjects
loop with a WinSock select call; then add the Win32 RPC libraries
to that (i.e. try to determine the named pipe handles you have to
pass to WaitForMultipleObjects).
> You ask to listen to sockets, so it instantiates a singleton
> PerfectReactor which instantiates a select() reactor and listens to it
> directly in a single-threaded manner. If you then ask to listen to
> Win32 messages, the PerfectReactor instantiates a GetMessage() reactor.
> Then, realizing it has two "event systems", it spawns a thread for each
> child reactor with a listener that serializes the incoming events into
> the PerfectReactor's queue.
Ah, threads :-( It turns out that you need to invoke GetMessage in the
context of the thread in which the window was created. In a different
thread, you won't get any messages. (This is indeed one of the biggest
limitations of USER32, which was only overcome in Vista when the
"native" GUI programming model is based on DirectX - the problem is
then of course solved only for applications that use the DirectX UI
API).
> Bingo, your application doesn't need to be
> written thread-safe, PerfectReactor is platform-agnostic, and you don't
> have to know in advance all the event types you might ever listen to.
>
> Sorry if this is a dumb question (or if I'm mangling the terminology),
Integrating with threads might be a solution in some cases, and
a problem in others. You can't assume it is a universal solution.
Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Martin v. Löwis wrote: > Now, for these generalized event loops, it may not be possible anymore > to combine all event sources into a single blocking call. Right, that's why my proposal assumed that each disparate event source would need its own thread. > Ah, threads :-( It turns out that you need to invoke GetMessage in the > context of the thread in which the window was created. In a different > thread, you won't get any messages. Oof! I'm embarrassed to have forgotten that. But that's not a fatal problem. It means that on Windows the PerfectReactor must service the blocking GetMessage loop, and these other threads notify the PerfectReactor of new events by sending a message. (Either that, or, it could poll GetMessage and its incoming event queue without ever blocking. But that is obviously suboptimal.) I think I've done this sort of thing before, in fact. Of course, in the absence of any windows, the Windows PerfectReactor could fall back to a mutex. Abstract this inside PerfectReactor and its event sources wouldn't notice the difference. > Integrating with threads might be a solution in some cases, and a > problem in others. You can't assume it is a universal solution. Universal? Yeah, I doubt it too. But perhaps it would be "good enough for nearly all cases". In the cases where it wasn't, it could throw an "I can't listen to that type of event right now" exception, forcing you to fall back to preconfiguring your central reactor by hand. Anyway, like many folks I'm hoping this whole conversation results in establishing basic standard duck-typed conventions--what other languages might call "interfaces"--for event senders and receivers. In which case this non-universal threaded approach would simply be one of several to choose from. I'd be interested to hear about other situations where threading would cause a problem. My suspicion is that Windows is the hard one, and as I've shown that one is solvable. Thanks for the thoughtful reply, /larry/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
>> Ah, threads :-( It turns out that you need to invoke GetMessage in the >> context of the thread in which the window was created. In a different >> thread, you won't get any messages. > > I'd be interested to hear about other situations where threading would > cause a problem. My suspicion is that Windows is the hard one, and as > I've shown that one is solvable. > > I've tried something similar on Linux, with gtk an wx. You can run the gtk main loop in its own thread, but because gtk is not thread safe, you have to grab a mutex everytime you run gtk code outside the thread the mainloop is running in. So you have to surround your calls to the gtk api with calls to gtk.threads_enter and gtk.threads_leave. Except for callbacks of course, because they are executed in the main thread... Doable, but not fun. The same goes for wx. Then all hell breaks loose when you try to use both gtk and wx at the same time. That's because on Linux, the wx main loop calls the gtk mainloop behind the scenes. As far as I know, that problem can not be solved from python. So yes that strategy can work, but it's no silver bullet. Cheers, Baptiste ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 2/15/07, Baptiste Carvello <[EMAIL PROTECTED]> wrote: > >> Ah, threads :-( It turns out that you need to invoke GetMessage in the > >> context of the thread in which the window was created. In a different > >> thread, you won't get any messages. > > > > I'd be interested to hear about other situations where threading would > > cause a problem. My suspicion is that Windows is the hard one, and as > > I've shown that one is solvable. > > > > > I've tried something similar on Linux, with gtk an wx. > > You can run the gtk main loop in its own thread, but because gtk is not thread > safe, you have to grab a mutex everytime you run gtk code outside the thread > the > mainloop is running in. So you have to surround your calls to the gtk api with > calls to gtk.threads_enter and gtk.threads_leave. Except for callbacks of > course, because they are executed in the main thread... Doable, but not fun. > > The same goes for wx. Then all hell breaks loose when you try to use both gtk > and wx at the same time. That's because on Linux, the wx main loop calls the > gtk > mainloop behind the scenes. As far as I know, that problem can not be solved > from python. > > So yes that strategy can work, but it's no silver bullet. And it's worse on Windows and Mac OS X where some GUI API calls *must* happen on a particular thread or they simply don't work. -bob ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Jean-Paul Calderone <[EMAIL PROTECTED]> wrote: > On Thu, 15 Feb 2007 02:36:22 -0700, Andrew Dalke <[EMAIL PROTECTED]> wrote: [snip] > >2) asyncore is > >smaller and easier to understand than Twisted, > > While I hear this a lot, applications written with Twisted _are_ shorter and > contain less irrelevant noise in the form of boilerplate than the equivalent > asyncore programs. This may not mean that Twisted programs are easier to > understand, but it is at least an objectively measurable metric. In my experience, the boilerplate is generally incoming and outgoing buffers. If both had better (optional default) implementations, and perhaps a way of saying "use the default implementations of handle_close, etc.", then much of the boilerplate would vanish. People would likely implement a found_terminator method and be happy. > >and > >3) it was for demo/proof of concept purposes. > >While > >tempting to improve that module I know that Twisted > >has already gone though all the platform-specific crap > >and I don't want to go through it again myself. I don't > >want to write a reactor to deal with GTK, and one for > >OS X, and one for ... > > Now if we can only figure out a way for everyone to benefit from this without > tying too many brains up in knots. :) Whenever I need to deal with these kinds of things (in wxPython specifically), I usually set up a wxTimer to signal asyncore.poll(timeout=0), but I'm lazy, and rarely need significant throughput in my GUI applications. [snip] > >Yet by using the Stackless socket monkeypatch, this > >same code works in an async framework. And the underlying > >libraries have a much larger developer base than Twisted. > >Want NNTP? "import nntplib" Want POP3? "import poplib" > >Plenty of documentation about them too. > > This is going to come out pretty harshly, for which I can only apologize in > advance, but it bears mention. The quality of protocol implementations in the > standard library is bad. As in "not good". Twisted's NNTP support is better > (even if I do say so myself - despite only having been working on by myself, > when I knew almost nothing about Twisted, and having essentially never been > touched since). Twisted's POP3 support is fantastically awesome. Next to > imaplib, twisted.mail.imap4 is a sparkling diamond. And each of these > implements the server end of the protocol as well: you won't find that in the > standard library for almost any protocol. Protocol support is hit and miss. NNTP in Python could be better, but that's not an asyncore issue (being that nntplib isn't implemented using asyncore), that's an "NNTP in Python could be done better" issue. Is it worth someone's time to patch it, or should they just use Twisted? Well, if we start abandoning stdlib modules, "because they can always use Twisted", then we may as well just ship Twisted with Python. - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
> On Thu, 15 Feb 2007 10:46:05 -0500, "A.M. Kuchling" <[EMAIL PROTECTED]> wrote:
> >It's hard to debug the resulting problem. Which level of the *12*
> >levels in the stack trace is responsible for a bug? Which of the *6*
> >generic calls is calling the wrong thing because a handler was set up
> >incorrectly or the wrong object provided? The code is so 'meta' that
> >it becomes effectively undebuggable.
On 2/15/07, Jean-Paul Calderone <[EMAIL PROTECTED]> wrote,
> I've debugged plenty of Twisted applications. So it's not undebuggable. :)
Hence the word "effectively". Or are you offering to be on-call
within 5 minutes for anyone wanting to debug code? Not very
scalable that.
The code I was talking about took me an hour to track down
and I could only find the location be inserting a "print traceback"
call to figure out where I was.
> Application code tends to reside at the bottom of the call stack, so Python's
> traceback order puts it right where you're looking, which makes it easy to
> find.
As I also documented, Twisted tosses a lot of the call stack. Here
is the complete and full error message I got:
Error: [Failure instance: Traceback (failure with no frames):
twisted.internet.error.ConnectionRefusedError: Connection was refused
by other side: 22: Invalid argument.
]
I wrote the essay at
http://www.dalkescientific.com/writings/diary/archive/2006/08/28/levels_of_abstraction.html
to, among others, show just how hard it is to figure things
out in Twisted.
> For any bug which causes something to be set up incorrectly and only
> later manifests as a traceback, I would posit that whether there is 1 frame or
> 12, you aren't going to get anything useful out of the traceback.
I posit that tracebacks are useful.
Consider:
def blah():
make_network_request("A")
make_network_request("B")
where "A" and "B" are encoded as part of a HTTP POST payload
to the same URI.
If there's an error in the network connection - eg, the implementation
for "B" on the server dies so the connection closes w/o a response -
then knowning that the call for "B" failed and not "A" is helpful
during debugging.
The low level error message cannot report that. Yes, I could put
my own try blocks around everything and contextualize all of the
error messages so they are semantically correct for the given
level of code. But that I would be a lot of code, hard to test, and
not cost effective.
> Standard practice here is just to make exception text informative,
> I think,
If you want to think of it as "exception text" then consider that
the stack trace is "just" more text for the message.
>but this is another general problem with Python programs
> and event loops, not one specific to either Twisted itself or the
> particular APIs Twisted exposes.
The thread is "Twisted Isn't Specific", as a branch of a discussion
on microthreads in the standard library. As someone experimenting
with Stackless and how it can be used on top of an async library
I feel competent enough to comment on the latter topic.
As someone who has seen the reverse Bozo bit set by Twisted
people on everyone who makes the slightest comment towards
using any other async library, and has documented evidence as
to just why one might do so, I also feel competent enough to
comment on the branch topic.
My belief is that there are too many levels of generiticity in
Twisted. This makes is hard for an outsider to come in and
use the system. By "use" I include 1) understanding how the
parts go together, 2) diagnose problems and 3) adding new
features that Twisted doesn't support.
Life is trade offs. A Twisted trade off is generiticity at the
cost of understandability. Remember, this is all my belief,
backed by examples where I did try to understand. My
experience with other networking packages have been much
easier, including with asyncore and Allegra. They are not
as general purpose, but it's hard for me to believe the extra
layers in Twisted are needed to get that extra whatever
functionality.
My other belief is that async programming is hard for most
people, who would rather do "normal" programming instead
of "inside-out" programming. Because of this 2nd belief I
am interested in something like Stackless on top of an
async library.
> As a personal anecdote, I've never once had to chase a bug through any of the
> 6 "generic calls" singled out. I can't think of a case where I've helped any
> one else who had to do this, either. That part of Twisted is very old, it is
> _very_ close to bug-free, and application code doesn't have very much control
> over it at all. Perhaps in order to avoid scaring people, there should be a
> way to elide frames from a traceback (I don't much like this myself, I worry
> about it going wrong and chopping out too much information, but I have heard
> other people ask for it)?
Even though I said some of this earlier I'll elaborate for clarification.
The specific bug I was tracking down had *no* traceback. There w
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Martin v. Löwis wrote: > That is insufficient. The gtk main loop has more input > sources than just the connection to X: > - timers > - idle handlers > - child handlers > - additional file descriptors > - a generalzed 'event source' When gtk is not the central event mechanism, there's no need to use the gtk event loop for these things -- you can just use the central event mechanism directly. The pygtk APIs for setting these up can redirect them to the appropriate place, to accommodate existing code that uses the gtk event loop for them. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On Thu, 15 Feb 2007 13:55:31 -0800, Josiah Carlson <[EMAIL PROTECTED]> wrote: > >Jean-Paul Calderone <[EMAIL PROTECTED]> wrote: > [snip] >> >> Now if we can only figure out a way for everyone to benefit from this without >> tying too many brains up in knots. :) > >Whenever I need to deal with these kinds of things (in wxPython >specifically), I usually set up a wxTimer to signal >asyncore.poll(timeout=0), but I'm lazy, and rarely need significant >throughput in my GUI applications. And I guess you also don't mind that on OS X this is often noticably broken? :) > [snip] > >Protocol support is hit and miss. NNTP in Python could be better, but >that's not an asyncore issue (being that nntplib isn't implemented using >asyncore), that's an "NNTP in Python could be done better" issue. Is it >worth someone's time to patch it, or should they just use Twisted? Well, >if we start abandoning stdlib modules, "because they can always use >Twisted", then we may as well just ship Twisted with Python. > We could always replace the stdlib modules with thin compatibility layers based on the Twisted protocol implementations. It's trivial to turn an asynchronous API into a synchronous one. I think you are correct in marking this an unrelated issue, though. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Jean-Paul Calderone <[EMAIL PROTECTED]> wrote: > > On Thu, 15 Feb 2007 13:55:31 -0800, Josiah Carlson <[EMAIL PROTECTED]> wrote: > > > >Jean-Paul Calderone <[EMAIL PROTECTED]> wrote: > > [snip] > >> > >> Now if we can only figure out a way for everyone to benefit from this > >> without > >> tying too many brains up in knots. :) > > > >Whenever I need to deal with these kinds of things (in wxPython > >specifically), I usually set up a wxTimer to signal > >asyncore.poll(timeout=0), but I'm lazy, and rarely need significant > >throughput in my GUI applications. > > And I guess you also don't mind that on OS X this is often noticably broken? > :) I don't own a Mac, and so far, of the perhaps dozen or so Mac users of the software that does this, I've heard no reports of it being broken. >From what I understand, wxTimers work on all supported platforms (which includes OSX), and if asyncore.poll() is broken on Macs, then someone should file a bug report. If it's asyncore's fault, assign it to me, otherwise someone with Mac experience needs to dig into it. > > [snip] > >Protocol support is hit and miss. NNTP in Python could be better, but > >that's not an asyncore issue (being that nntplib isn't implemented using > >asyncore), that's an "NNTP in Python could be done better" issue. Is it > >worth someone's time to patch it, or should they just use Twisted? Well, > >if we start abandoning stdlib modules, "because they can always use > >Twisted", then we may as well just ship Twisted with Python. > > We could always replace the stdlib modules with thin compatibility layers > based on the Twisted protocol implementations. It's trivial to turn an > asynchronous API into a synchronous one. I think you are correct in marking > this an unrelated issue, though. If the twisted folks (or anyone else) want to implement a "shim" that pretends to be nntplib, it's their business whether calling twisted.internet.monkeypatch.nntplib() does what the name suggests. :) That is to say, I don't believe anyone would be terribly distraught if there was an easy way to use Twisted without drinking the kool-aid. Then again, I do believe that it makes sense to patch the standard library whenever possible - if Twisted has better parsing of nntp, smtp, pop3, imap4, etc. responses, then perhaps we should get the original authors to sign a PSF contributor agreement, and we could translate whatever is better. - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
Larry Hastings schrieb: > Oof! I'm embarrassed to have forgotten that. But that's not a fatal > problem. It means that on Windows the PerfectReactor must service the > blocking GetMessage loop, and these other threads notify the > PerfectReactor of new events by sending a message. ... > I'd be interested to hear about other situations where threading would > cause a problem. My suspicion is that Windows is the hard one, and as > I've shown that one is solvable. As others have reported: if you have a gtk reactor and a wx reactor to support, in addition to a "regular" Win32 reactor (e.g. PythonWin), your approach still won't work. They all do GetMessage behind the scenes (sp?), yet you would run them in multiple threads, so they get their own messages. Plus they are not thread-safe. Speaking of which: Tcl has a "multiple interpreter" model where each interpreter is single-threaded. You can build Tcl either single- or multi-threaded; for the multi-threaded Tcl, you still can't share data across threads directly, but instead, you create an interpreter per thread. Then, all operations on that interpreter must occur in this thread, including the reactor calls. So if the PerfectReactor is to support Tcl/Tk, it needs to run the Tcl even loop in the Tcl thread, which is normally the main thread (in Tkinter, it's the thread where you originally created the tkapp object). OTOH, PerfectReactor may also need to run some Win32 event loop in the same thread. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
"Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > > Larry Hastings schrieb: > > Oof! I'm embarrassed to have forgotten that. But that's not a fatal > > problem. It means that on Windows the PerfectReactor must service the > > blocking GetMessage loop, and these other threads notify the > > PerfectReactor of new events by sending a message. > ... > > I'd be interested to hear about other situations where threading would > > cause a problem. My suspicion is that Windows is the hard one, and as > > I've shown that one is solvable. > > As others have reported: if you have a gtk reactor and a wx reactor > to support, in addition to a "regular" Win32 reactor (e.g. PythonWin), > your approach still won't work. They all do GetMessage behind the > scenes (sp?), yet you would run them in multiple threads, so they > get their own messages. Plus they are not thread-safe. At least on the wx side of things, there exist ways of scheduling calls to be executed inside its event dispatching loop (wx.CallAfter, wx.FutureCall, wx.PostEvent), which allows for worker threads to hand data back to the main thread (the creator of the wx.App instance, like Tk). I would be surprised if the other toolkits didn't have something similar, couldn't be manipulated into using Queues to communicate in some way, or couldn't be manipulated into directly sending a message on win32. - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
> As others have reported: if you have a gtk reactor and a wx reactor to > support, in addition to a "regular" Win32 reactor (e.g. PythonWin), > your approach still won't work. They all do GetMessage behind the > scenes (sp?), yet you would run them in multiple threads, so they get > their own messages. Plus they are not thread-safe. I'm not sure what your point is here. Let me see if I can clarify things, at least from my side. I suggest it is possible to implement a PerfectReactor. I define this PerfectReactor as a singleton top-level reactor which runs in the main thread of a Python application. All incoming events would be serialized to this single reactor, which would dispatch them to their appropriate listeners. When asked to monitor disparate event systems, the PerfectReactor would act as a concentrator of multiple other reactors, with various ugly things going on hidden in the implementation. The goal would be that modular libraries of event listeners could be written in an agnostic manner, simply requesting to listen to whatever events they need to, and the main application wouldn't have to know in advance what reactors to configure or how to configure them. You seem to assert it is not possible to implement such a PerfectReactor, and list counter-examples. So far so good. If you're saying "it is impossible for any implementation to simultaneously support more than one of the Win32, gtk, and wx reactors": fine, this hypothetical PerfectReactor can't do the impossible, but that's not an argument against a PerfectReactor. If it *is* possible to support more than one of these simultaneously through precise configuration, I assert that a PerfectReactor could support that configuration too, and the goal would be to have it automatically configure itself. You point out the gtk reactor and the wx reactor must be run in the main thread, because they run their own message pumps. What I had in mind was that, when you asked to start listening to Win32 events, the Win32 reactor would tell the PerfectReactor "I must be run in the main thread", and it would (re-) configure that to be the main thread's listener; the notification mechanism for other threads saying "there's a new event in your queue" would switch to using Win32 messages. If the gtk and wx reactors had the same restriction, then whichever was first introduced to the PerfectReactor first would win, and introducing subsequent reactors would throw an exception. (Again, can't do the impossible.) However, if gtk and wx expose enough of the plumbing of their message pumps, this central Win32 message pump could identify messages intended for the gtk and wx reactor message pumps and inject them. The gtk and wx reactors wouldn't call GetMessage, but they could call the rest of their message loops. Therefore they could all be run in the main thread. (And yes, "behind the scenes" is correct.) > if the PerfectReactor is to support Tcl/Tk, it needs to run the Tcl > even loop in the Tcl thread, which is normally the main thread (in > Tkinter, it's the thread where you originally created the tkapp > object). OTOH, PerfectReactor may also need to run some Win32 event > loop in the same thread. Again, I'm not sure of your point. If you're saying "it is impossible to run multithreaded Tcl and any message pump (Win32, gtk, wx) in the same application", then fine, PerfectReactor cannot do the impossible. If it is only possible to do with clever configuration, then I maintain a PerfectReactor could self-configure to support this configuration at runtime. Cheers, /larry/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
[EMAIL PROTECTED] wrote: > When you boil it down, Twisted's event loop is just a notification for > "a connection was made", "some data was received on a connection", "a > connection was closed", and a few APIs to listen or initiate different > kinds of connections, start timed calls, and communicate with threads. > All of the platform details of how data is delivered to the > connections are abstracted away. How do you propose we would make a > less "specific" event mechanism? But that is exactly the problem I have with Twisted. For HTTP it creates its own set of notifications instead of structuring the code similar to SocketServer (UDP and TCP), BaseHTTPServer, SimpleHTTPServer etc which are well understood in the python community and e.g. used by medusa and asyncore. Having to completely restructure one's own code is a high price. Giving control away into a big framework that calls my own code for not so easy to understand reasons (for a twisted noob) does not give me a warm feeling. It's o.k. for complex applications like web servers but for small networking applications I'd like to have a chance to understand what's going on. Asyncore is so simple that it's easy to follow when I let it do the select() for me. That said, I conclude that the protocol implementations are superb but unfortunately to tightly coupled to the Twisted philosophy, sitting in the middle, trying to orchestrate instead of being easy to integrate. Joachim ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
On 16 Feb, 06:30 pm, [EMAIL PROTECTED] wrote: >I suggest it is possible to implement a PerfectReactor. I don't think this constitutes a sufficient existence proof. Perhaps you could write a prototype? There are a bunch of existing reactors you could either import or copy/paste from to bootstrap it, so you wouldn't necessarily need to do many of the platform-specific hacks it has already implemented for those loops. Personally I am highly skeptical that this is possible, and I didn't find any of your arguments convincing, but I would be ecstatic if it worked; it isn't *fun* maintaining 12 or so subtly incompatible implementations of the same interface :). I also think this discussion would be more appropriate to continue on the Twisted list, as we have veered pretty far afield of stdlib enhancements and are now talking about multiplexing method implementations, but I'll follow it wherever it continues. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
