Re: Beta testers needed for a high performance Python application server
Graham Dumpleton wrote: Yes that is a viable option, as still are existing fastcgi solutions for Apache, lighttpd and nginx. Fast cgi is a good technology, but it's not well documented or well supported. For some reason, the Apache people don't like it. It used to be part of the Apache distribution, but that ended years ago. It's more reliable than using things like mod_python, where you have application code running in the web server's address space. That creates both security problems and robustness problems. If an fcgi process crashes, it is automatically replaced by a fresh copy of the program at the next request. Other activity in progress is not affected. Also, fcgi processes are reloaded after some number of requests, so minor memory leaks won't choke the system over time. The main problem is that the documentation is terrible, and the monitoring and administrative tools are absent. Little problems include the fact that if you put an .fcgi file in the CGI directory, it's run silently, and inefficiently, in CGI mode. This is because Apache has CGI built into it at a level below the level at which it recognizes other files. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Re: Beta testers needed for a high performance Python application server
John Nagle [EMAIL PROTECTED] writes: Fast cgi is a good technology, but it's not well documented or well supported. For some reason, the Apache people don't like it. It used to be part of the Apache distribution, but that ended years ago. It seems to be coming back into favor. See: http://cryp.to/publications/fastcgi/ I've thought for a long time that the right way to do it is with the SCM_RIGHTS ancillary message on unix domain sockets, that lets you pass file descriptors around between processes. One of these days... -- http://mail.python.org/mailman/listinfo/python-list
Re: Beta testers needed for a high performance Python application server
Minor Gordon a écrit : (snip otherwise intersting stuff) Background: I'm in this to help write a story for Python and web applications. Everyone likes to go on about Ruby on Rails, and as far as I can tell there's nothing that approaches Rails in Python. You may have missed Django and Pylons then. Beta testers: should be intermediate to advanced Python programmers with demanding applications, particularly web applications with databases and AJAX. The C++ is portable to Win32, Linux, and OS X, with no mandatory libraries beyond python-dev. Please contact me if you're interested: firstname.lastname at cl.cam.ac.uk. I can only second Graham here: setup a project page with source code downloads, install instructions and (at least minimal) software documentation, and if possible a tracker. -- http://mail.python.org/mailman/listinfo/python-list
Re: Beta testers needed for a high performance Python application server
Why not just put it on the net somewhere and tell us where it is? People aren't generally going to want to help or even look at it if you treat it like a proprietary application. So, put the documentation and code up somewhere for all to see. http://www.yieldserver.com:8081/ -- http://mail.python.org/mailman/listinfo/python-list
Re: Beta testers needed for a high performance Python application server
Paul Rubin wrote: John Nagle [EMAIL PROTECTED] writes: Fast cgi is a good technology, but it's not well documented or well supported. For some reason, the Apache people don't like it. It used to be part of the Apache distribution, but that ended years ago. It seems to be coming back into favor. See: http://cryp.to/publications/fastcgi/ That paper is from 2002. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Re: Beta testers needed for a high performance Python application server
Il Tue, 25 Mar 2008 20:31:39 +, Minor Gordon ha scritto: Hello all, I'm looking for beta testers for a high performance, event-driven Python application server I've developed. About the server: the front end and other speed-critical parts of the server are written in portable, multithreaded C++. The back end is an embedded CPython interpreter. The server is much faster than anything in pure Python, and it can compete with C servers (including e.g. lighttpd for file workloads) or outdo them (e.g. anything behind Apache) until CPython consumes a single processor. Have you tried my WSGI implementation for Nginx? http://hg.mperillo.ath.cx/nginx/mod_wsgi/ Its not a general purpose solution, but it can be of interest. On the Python side it supports WSGI (the server can handle the static and dynamic requests of MoinMoin with a handful of lines), the DB API with blocking calls offloaded to a connection in a separate thread (MySQL, SQLite supported), Google's ctemplate, gzipping responses, file caching, reading and writing to URIs as a client, AJAX integration, debugging as a Python extension, and a lot of other features. The core Python API is event-driven, using continuations like Twisted but much cleaner (continuations are any callables, there are no special objects anywhere). The Python back end also supports Stackless Python so all of the continuation machinery can be hidden behind tasklet switching. I have recently added support for asynchronous application. There are two examples: an application that execute a query to PostgreSQL and an application that execute an HTTP request with pycurl: http://hg.mperillo.ath.cx/nginx/mod_wsgi/file/tip/examples/nginx-postgres- async.py http://hg.mperillo.ath.cx/nginx/mod_wsgi/file/tip/examples/nginx-curl.py Note that ngx.poll extension is still experimental. [...] Manlio Perillo -- http://mail.python.org/mailman/listinfo/python-list
Re: Beta testers needed for a high performance Python application server
Il Wed, 26 Mar 2008 00:22:38 -0700, John Nagle ha scritto: Graham Dumpleton wrote: Yes that is a viable option, as still are existing fastcgi solutions for Apache, lighttpd and nginx. Fast cgi is a good technology, Well, not really so good: http://twistedmatrix.com/pipermail/twisted-web/2006-April/002598.html but it's not well documented or well supported. For some reason, the Apache people don't like it. It used to be part of the Apache distribution, but that ended years ago. It's more reliable than using things like mod_python, where you have application code running in the web server's address space. That creates both security problems and robustness problems. If an fcgi process crashes, it is automatically replaced by a fresh copy of the program at the next request. Other activity in progress is not affected. Also, fcgi processes are reloaded after some number of requests, so minor memory leaks won't choke the system over time. The problem is simple: why put an extra server layer between an HTTP client and an HTTP server? Moreover, you should not use mod_python as an example. The WSGI module for Apache has a lot of feature for reducing these problems; and as an alternative you can use the WSGI implementation for Nginx. [...] Manlio Perillo -- http://mail.python.org/mailman/listinfo/python-list
Re: Beta testers needed for a high performance Python application server
On Mar 26, 7:31 am, Minor Gordon [EMAIL PROTECTED] wrote: Hello all, I'm looking for beta testers for a high performance, event-driven Python application server I've developed. About the server: the front end and other speed-critical parts of the server are written in portable, multithreaded C++. The back end is an embedded CPython interpreter. The server is much faster than anything in pure Python, and it can compete with C servers (including e.g. lighttpd for file workloads) or outdo them (e.g. anything behind Apache) until CPython consumes a single processor. On the Python side it supports WSGI (the server can handle the static and dynamic requests of MoinMoin with a handful of lines), the DB API with blocking calls offloaded to a connection in a separate thread (MySQL, SQLite supported), Google's ctemplate, gzipping responses, file caching, reading and writing to URIs as a client, AJAX integration, debugging as a Python extension, and a lot of other features. The core Python API is event-driven, using continuations like Twisted but much cleaner (continuations are any callables, there are no special objects anywhere). The Python back end also supports Stackless Python so all of the continuation machinery can be hidden behind tasklet switching. Background: I'm in this to help write a story for Python and web applications. Everyone likes to go on about Ruby on Rails, and as far as I can tell there's nothing that approaches Rails in Python. I want to code quickly in Python like I can with Rails, but without sacrificing single node performance on many cores. Beta testers: should be intermediate to advanced Python programmers with demanding applications, particularly web applications with databases and AJAX. The C++ is portable to Win32, Linux, and OS X, with no mandatory libraries beyond python-dev. Please contact me if you're interested: firstname.lastname at cl.cam.ac.uk. Why not just put it on the net somewhere and tell us where it is? People aren't generally going to want to help or even look at it if you treat it like a proprietary application. So, put the documentation and code up somewhere for all to see. BTW, multiprocess web servers such as Apache can quite happily make use of multiple cores. Even within a single Apache multithread process it can still use multiple cores quite happily because all the underlying network code and static file handling code is in C and not subject to the GIL. So, as much as people like to bash up on the GIL, within Apache it is not necessarily as big a deal as people make out. Also, an event driven model, or even a large dependence on multithreading, can actually be worse for Python web applications where using it means one can handle a much larger number of concurrent requests. This is because you greatly increase the risk of code having large requirements for transient memory being hit at the same time. The result can be large unpredictable blowouts in memory requirements for the process, with that memory then being held by the process and not necessarily able to be released back to operating system. Thus, for large Python web applications, use of a multiprocess web server, where each worker process is single threaded, is in various way still best as it provides the most predictable memory profile. Finally, the speed of the underlying web server (except for CGI) generally has minimal bearing on the performance of a large Python web application. This is because that isn't where the bottlenecks are. The real bottlenecks are generally in the application code itself and in any access to a back end database. Thus, pursuing absolute speed is a bit of a fools errand when you consider that any performance gain you may have over a competing solution may only end up resulting in somewhat less than 1% difference when one looks at overall request time. Graham -- http://mail.python.org/mailman/listinfo/python-list
Re: Beta testers needed for a high performance Python application server
I'm looking for beta testers for a high performance, event-driven Python application server I've developed. About the server: the front end and other speed-critical parts of the server are written in portable, multithreaded C++. ... Why not just put it on the net somewhere and tell us where it is? People aren't generally going to want to help or even look at it if you treat it like a proprietary application. So, put the documentation and code up somewhere for all to see. BTW, multiprocess web servers such as Apache can quite happily make use of multiple cores. Even within a single Apache multithread process it can still use multiple cores quite happily because all the underlying network code and static file handling code is in C and not subject to the GIL. So, as much as people like to bash up on the GIL, within Apache it is not necessarily as big a deal as people make out. BTW nginx now has a mod_wsgi too, if someone is looking for an Apache replacement. -- damjan -- http://mail.python.org/mailman/listinfo/python-list
Re: Beta testers needed for a high performance Python application server
On Mar 26, 11:00 am, Damjan [EMAIL PROTECTED] wrote: I'm looking for beta testers for a high performance, event-driven Python application server I've developed. About the server: the front end and other speed-critical parts of the server are written in portable, multithreaded C++. ... Why not just put it on the net somewhere and tell us where it is? People aren't generally going to want to help or even look at it if you treat it like a proprietary application. So, put the documentation and code up somewhere for all to see. BTW, multiprocess web servers such as Apache can quite happily make use of multiple cores. Even within a single Apache multithread process it can still use multiple cores quite happily because all the underlying network code and static file handling code is in C and not subject to the GIL. So, as much as people like to bash up on the GIL, within Apache it is not necessarily as big a deal as people make out. BTW nginx now has a mod_wsgi too, if someone is looking for an Apache replacement. Yes that is a viable option, as still are existing fastcgi solutions for Apache, lighttpd and nginx. Because the bottlenecks are generally going to be in upper application layers it really comes down to personal preferences as to which you want to use, which you find easier to configure and manage, plus which you trust as being the most secure and stable. For truly high demand sites you should also be looking at spreading load across multiple hosts, not just because of performance but also because of redundancy. You would also be serving media off a different web server to your Python web application and configuring each web server to the specific task it is doing. So, there is a lot more to in than raw speed of underlying technology and so one should always treat with caution something that is being sold as some better way of doing things. This applies to Apache mod_wsgi, nginx mod_wsgi or anything else for that matter. All have benefits, but they also have shortcomings in different areas which may not always make them suitable for all applications, or at least they may need to be configured in specific ways to make them perform best for specific applications. Anyway, because it isn't that simple is why I'd like to see some actual documentation for this new contender posted in a public place, along with code being browsable so one can evaluate it without having to sign up for some closed beta program. Graham -- http://mail.python.org/mailman/listinfo/python-list
