On 9/5/07, Marcin Kasperski <[EMAIL PROTECTED]> wrote: > Has anybody tried merging pylons techniques with twisted? > In general it should be possible as twisted contains WSGI > container... > > Twisted has its own offerings (nevow), but > I am not truly happy with them...
It's certainly worth assessing what would be required to make Pylons work with Twisted, or to make a Pylons application that works both under Twisted and the traditional WSGI servers. Nevow and the other Twisted frameworks are very different from what people looking for a Rails-like framework with a "normal" template language would be willing to use, plus Twisted's Deferreds and callbacks are enough to make many other programmers run screaming from the room. Also, the Deferreds require a compatible event handler (reactor), which would make the application incompatible with a non-Twisted server. But given that the other megaframeworks have not yet been ported to Twisted, we could be the first! Twisted Web 1 had a WSGI server, but it was buggy and wasn't ported to Twisted Web 2. That would be the first hurdle. Searching for "wsgi" at twistedmatrix.com brings several results suggesting that WSGI support is still in pre-alpha state. http://twistedmatrix.com/trac/ticket/2545 http://twistedmatrix.com/trac/ticket/2753 > Concerns? Well, apart from necessity to put some work, I feel that in > pylons world multitasking=multithreading. I'm sure that was never meant to be a requirement. Pylons apps should work under multiprocess servers, and if they don't (e.g., session locking), this should be fixed. Generally, making a program thread safe is a superset of making it multiprocess safe, so I assume this has already been done. But asynchronous servers raise a third set of issues which we perhaps haven't addressed... > What need not to be a case > in twisted context. Many twisted tasks may run within the same thread, > sharing the same threadlocal in spite of being separate simultaneous > tasks... The WSGI support they're working on seems to run the application in a separate thread, in which case Pylons could block or spawn child threads like it normally does. It would be Twisted WSGI's responsibility to defer for the return value and to handle the start_response callback somehow. The first question that comes to my mind is, would the Pylons app be a long-running application or would it be re-instantiated for every request? The latter would incur overhead. Does Twisted WSGI know how to instantiate an application class and then call the instance, or is it expecting an actual function to call? If you want to run a Pylons application directly under Twisted -- not in a separate thread -- several other issues come up. You may have to write a custom interface, patch Pylons and Beaker, etc. Coming back to the issue of running several tasks asynchronously in the same thread, this comes down to proper scoping. Each Pylons request is a "scope" containing the request/response/c/session/cache, etc. Requests belong to application instances, each with a config dict and 'g' variable. (This is called "request scope" vs "application scope".) We already have the possibility of two Pylons applications -- or two instances of the same Pylons application -- running simultaneously in the same process via paste.composite or whatever it's called. Pylons is apparently fine with this because this scenario has been known for a while. The only issue that has been that it *may* require a custom app_scope function for SQLAlchemy: a function that returns some kind of application ID. (There was debate about whether this was necessary to prevent SQLAlchemy sessions from being shared between applications. Given that every request resets the session to its initial state, and the request monopolizes the thread during its entire lifetime, it may not matter if thread-local sessions are used serially by different applications. See the bottom of "SQLAlchemy 0.4 for people in a hurry" in the Pylons wiki for details.) The issue for Pylons in an asynchronous environment is, do the StackedObjectProxies return the appropriate request/response/c/g/session/cache for the current request. I don't know the answer to that. Given that the request determines the application, perhaps these all could be keyed on id(request), but it may be a headache to pass the ID around within Pylons and to the StackedObjectProxies. The other issue is where Pylons or typical applications do I/O blocking. My first guess is at the socket I/O, which Twisted would handle, and in SQLAlchemy database queries. Twisted apps use a special DBAPI wrapper (http://twistedmatrix.com/projects/core/documentation/howto/enterprise.html). Given its callback-style queries and separate connection pool, I don't see how you'd integrate it with SQLAlchemy. Perhaps you could use parts of it; e.g., manually create a ResultProxy from a DBAPI cursor,or write a function that takes a query and returns a ResultProxy. If all file operations are considered "blocking" too, that would affect Beaker sessions, Beaker caching, and Mako's template caching. So it definitely would be much easier to get Twisted WSGI working so that it can run a non-Twisted application in a thread, and make sure it can call a long-running app (returned by myapp.config.middleware.make_app) repeatedly, with each call in its own thread. This would bypass Paste's ThreadPool if I understand correctly (which is tied to PasteHTTPServer), so Twisted would be doing all the thread management unless your application or libraries spawned its own threads. The alternative, making Pylons and its dependencies into a non-blocking Twisted application, would be much more work, and it may not be possible without making incompatible changes that would break Pylons' normal usage. -- Mike Orr <[EMAIL PROTECTED]> --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~----------~----~----~----~------~----~------~--~---
