Re: [Web-SIG] WSGI and start_response
On Thu, Apr 15, 2010 at 01:35, Graham Dumpleton graham.dumple...@gmail.com wrote: If that isn't done, we will be here in another year still arguing about whether some aspect of the specification should be changed or removed based on some individuals perceived need. I agree, WSGI 1.1 should be more like HTML5 in that it tries to more meticulously describe/unify existing implementations. Such a significant change as removing the requirement for write() should also not be done within a minor version of the WSGI specification because anything that works with WSGI 1.0 should still work with WSGI 1.1 and vice versa. On that basis it can't really be entertained until WSGI 2.0 where incompatible changes would be allowed. I think it's a good idea to consider for 2.0, certainly. Cheers, Dirkjan ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
Dirkjan Ochtman ha scritto: [...] Such a significant change as removing the requirement for write() should also not be done within a minor version of the WSGI specification because anything that works with WSGI 1.0 should still work with WSGI 1.1 and vice versa. On that basis it can't really be entertained until WSGI 2.0 where incompatible changes would be allowed. I think it's a good idea to consider for 2.0, certainly. Ehm, the purpose of WSGI 2.0 is precisely to remove start_response and write callable with it... Manlio ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
On Thu, Apr 15, 2010 at 11:09, Manlio Perillo manlio_peri...@libero.it wrote: Ehm, the purpose of WSGI 2.0 is precisely to remove start_response and write callable with it... Right, there you go! Cheers, Dirkjan ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
Dirkjan Ochtman ha scritto: On Tue, Apr 13, 2010 at 14:46, Graham Dumpleton graham.dumple...@gmail.com wrote: The last attempt was to have WSGI 1.1 as clarifications and Python 3.X. And when I say 'last attempt', yes there have been people who have stepped up to try and get this to happen in the past. I think you would be the 3rd time, excluding me in general having tried to push it in the past and also given up. You really should perhaps look back through the archive of WEB-SIG posts on Google Groups to understand the history and how this always seems to just go around in circles. :-) I've been on Web-SIG for quite a while now, exactly to keep track of these issues. Since there doesn't seem to be much traction, I figured it would be time to just get a new PEP together. To limit the amount of work, I'd go in the direction of having a single WSGI 2.0 PEP incorporating your suggestions (maybe minus the number 3), everything required for Python 3 (as outlined by your wiki page). If you volunteer for this task, I have some suggestions: * target WSGI 1.1, not WSGI 2.0 * take the original WSGI 1.0 spec text * start to integrate all changes documented by Graham * I would really like to have changes integrates as a series of diff, using del and ins HTML elements. Unfortunately docutils seems to not have support for this, but should not be hard to implement. I can help. * You should keep a separate, unofficial document, with the rationale of the changes. You can just copy the content of Graham blog post, and reformatting it, if this is ok for Graham * For each of the main changea, start a thread on this mailing list asking for votation. If, after 1 week, there is no vote against it, consider it approved If we are really going to approve WSGI 1.1, I have a request: remove the ``write`` callable. Rationale: * it was added in WSGI 1.0 only for compatibility * new code does not use it * this will force applications under development that still use the ``write`` callable to be fixed. See work on Mercurial * it is very easy for current implementations to support both WSGI 1.0 and WSGI 1.1 * legacy application will continue to work * removing of the ``write`` callable will make middlewares more easy to write Thanks Manlio ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
On 15 April 2010 02:53, Manlio Perillo manlio_peri...@libero.it wrote: Dirkjan Ochtman ha scritto: On Tue, Apr 13, 2010 at 14:46, Graham Dumpleton graham.dumple...@gmail.com wrote: The last attempt was to have WSGI 1.1 as clarifications and Python 3.X. And when I say 'last attempt', yes there have been people who have stepped up to try and get this to happen in the past. I think you would be the 3rd time, excluding me in general having tried to push it in the past and also given up. You really should perhaps look back through the archive of WEB-SIG posts on Google Groups to understand the history and how this always seems to just go around in circles. :-) I've been on Web-SIG for quite a while now, exactly to keep track of these issues. Since there doesn't seem to be much traction, I figured it would be time to just get a new PEP together. To limit the amount of work, I'd go in the direction of having a single WSGI 2.0 PEP incorporating your suggestions (maybe minus the number 3), everything required for Python 3 (as outlined by your wiki page). If you volunteer for this task, I have some suggestions: * target WSGI 1.1, not WSGI 2.0 * take the original WSGI 1.0 spec text * start to integrate all changes documented by Graham * I would really like to have changes integrates as a series of diff, using del and ins HTML elements. Unfortunately docutils seems to not have support for this, but should not be hard to implement. I can help. * You should keep a separate, unofficial document, with the rationale of the changes. You can just copy the content of Graham blog post, and reformatting it, if this is ok for Graham * For each of the main changea, start a thread on this mailing list asking for votation. If, after 1 week, there is no vote against it, consider it approved If we are really going to approve WSGI 1.1, I have a request: remove the ``write`` callable. Rationale: * it was added in WSGI 1.0 only for compatibility * new code does not use it * this will force applications under development that still use the ``write`` callable to be fixed. See work on Mercurial * it is very easy for current implementations to support both WSGI 1.0 and WSGI 1.1 * legacy application will continue to work * removing of the ``write`` callable will make middlewares more easy to write This is in part why an attempt to come up with a new WSGI 1.X specification, even one that covers just the obvious and justified changes because of actual problems, keeps failing. That is, parties with vested interests or a desire for their little pet change to be made because it helps them, keep poking their heads up and disrupting the process. Given how long this has taken, all that should happen at this point is a codification of what wsgiref implements for Python 3.X along with readline() argument change and obvious other clarifications where actual use shows the original WSGI specification was wrong or where it wasn't practical. If that isn't done, we will be here in another year still arguing about whether some aspect of the specification should be changed or removed based on some individuals perceived need. Such a significant change as removing the requirement for write() should also not be done within a minor version of the WSGI specification because anything that works with WSGI 1.0 should still work with WSGI 1.1 and vice versa. On that basis it can't really be entertained until WSGI 2.0 where incompatible changes would be allowed. Graham ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
P.J. Eby ha scritto:
At 10:18 PM 4/8/2010 +0200, Manlio Perillo wrote:
Suppose I have an HTML template file, and I want to use a sub request.
...
${subrequest('/header/'}
...
The problem with this code is that, since Mako will buffer all generated
content, the result response body will contain incorrect data.
It will first contain the response body generated by the sub request,
then the content generated from the Mako template (XXX I have not
checked this, but I think it is how it works).
Okay, I'm confused even more now. It seems to me like what you've just
described is something that's fundamentally broken, even if you're not
using WSGI at all.
If you are referring to Mako being turned in a generator, yes, this
implementation is rather obscure.
I wrote it as a proof of concept.
Before this, I wrote a more polite implementation:
http://paste.pocoo.org/show/201324/
So, when executing a sub request, it is necessary to flush (that is,
send to Nginx, in my case) the content generated from the template
before the sub request is done.
This seems to only makes sense if you're saying that the subrequest *has
to* send its output directly to the client, rather than to the parent
request.
Yes, this is how subrequests work in Nginx. And I assume the same is
true for Apache.
If the subrequest sends its output to the parent request (as a
sane implementation would), then there is no problem.
You are forgetting that Nginx is not an application server.
Why should the subrequest output returned to the parent?
This would only make it less efficient.
Likewise, if the
subrequest is sent to a buffer that's then inserted into the parent
invocation.
Anything else seems utterly insane to me, unless you're basically taking
a bunch of legacy CGI code using 'print' statements and hacking it into
something else. (Which is still insane, just differently. ;-) )
We are talking about subrequest implementation in a efficient web server
written in C, like Nginx and Apache.
Ah, you are right sorry.
But this is not required for the Mako example (I was focusing on that
example).
As far as I can tell, that example is horribly wrong. ;-)
I agree ;-)
But when using the greenlet middleware, and when using the function for
flushing Mako buffer, some data will be yielded *before* the application
returns and status and headers are passed to Nginx.
And that's probably because sharing a single output channel between the
parent and child requests is a bad idea. ;-)
No, this is not specific to subrequests.
As an example, here you can find an up to date greenlet adapters:
http://bitbucket.org/mperillo/txwsgi/src/tip/txwsgi/greenlet.py
The ``write_adapter`` **needs** to yield some data before WSGI
application return, because this is how the write callable workd.
The exposed ``gsuspend`` function, instead, will cause an empty string
to be yielded to the server, before the WSGI application returns.
(Specifically, it's an increase in temporal coupling, I believe. I
know it's some kind of coupling between functions that's considered bad,
I just don't remember if that's the correct name for it.)
Nginx code contains some coupling; I assume this is done because it was
designed with efficiency in mind.
[...]
It's true that dropping start_response() means you can't yield empty
strings prior to determining your headers, yes.
- yielding is for server push or
sending blocks of large files, not tiny strings.
Again, consider the use of sub requests.
yielding a not large block is the only choice you have.
No, it isn't. You can buffer your output and yield empty strings until
you're ready to flush.
As I wrote, this will not work if you want to use subrequest support
from Nginx.
Unless, of course, you implement sub request support in pure Python (or
using SSI - Server Side Include).
I don't see why it has to be pure, actually. It just that the
subrequest needs to send data to the invoker rather than sending it
straight to the client.
You may say this, but it is not how subrequests are implemented in Nginx
;-).
That's the bit that's crazy in your example -- it's not a scenario that
WSGI 2 should support, and I'd consider the fact that WSGI 1 lets you do
it to be a bug, not a feature. ;-)
Are you referring to the bad Mako example, or to the
``greenlet_adapter`` idea?
That being said, I can see that removing start_response() closes a
loophole that allows async apps to *potentially* exist under WSGI 1 (as
long as you were able to tolerate the resulting crappy API).
However, to fix that crappy API requires greenlets or threads, at which
point you might as well just use WSGI 2. In the Nginx case, you can
either do WSGI 1 in C and then use an adapter to provide WSGI 2, or you
can expose your C API to Python and write a small greenlets-using Python
wrapper to support suspending.
But this is already implemented using the
Re: [Web-SIG] WSGI and start_response
On 13 April 2010 20:41, Manlio Perillo manlio_peri...@libero.it wrote: So, when executing a sub request, it is necessary to flush (that is, send to Nginx, in my case) the content generated from the template before the sub request is done. This seems to only makes sense if you're saying that the subrequest *has to* send its output directly to the client, rather than to the parent request. Yes, this is how subrequests work in Nginx. And I assume the same is true for Apache. No that is not true for Apache. Apache content handlers write output into what is called a bucket brigade. For a normal sub request this may be the bucket brigade of the parent request and so be processed by the output filters of the parent request. You can however code the mechanics of the sub request to override that and do something else with the data pushed into that bucket brigade. Although it can be done it gets a bit complicated to have the data written back into the bucket brigade pulled back into the context of a parent request. This is because the data is written from the context of the sub request where as at same time the parent request is going to want to pull it. Thus need to use threading and have to fire off the sub request in its own thread with a queue of some sort being used to communicate between the two. So messy, but technically it should be possible with custom Python code specific to Apache to fire off a subrequest and the result of the sub request be an iterable which yields data which itself could be yielded from the context of the parent application such that the content could then be processed and modified by a WSGI middleware wrapper. Graham ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
On Tue, Apr 13, 2010 at 13:13, Graham Dumpleton graham.dumple...@gmail.com wrote: There is no such thing as a WSGI 2.0 PEP and there is no proper concensus either on what it should look like. Thus if you see anything claiming to implement WSGI 2.0, then it isn't and you should only view it as an experimental proposal. You are warned. :-) Do you (or someone else) have a status on where WSGI 2 is? IIRC WSGI 1 isn't really usable with Python 3.x, so it seems about time we get something going again (AIUI this is blocking Werkzeug from being ported to 3.x, for example). Cheers, Dirkjan ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
Dirkjan Ochtman ha scritto: On Tue, Apr 13, 2010 at 13:13, Graham Dumpleton graham.dumple...@gmail.com wrote: There is no such thing as a WSGI 2.0 PEP and there is no proper concensus either on what it should look like. Thus if you see anything claiming to implement WSGI 2.0, then it isn't and you should only view it as an experimental proposal. You are warned. :-) Do you (or someone else) have a status on where WSGI 2 is? IIRC WSGI 1 isn't really usable with Python 3.x, so it seems about time we get something going again (AIUI this is blocking Werkzeug from being ported to 3.x, for example). WSGI 2.0 ideas are here: http://wsgi.org/wsgi/WSGI_2.0 But it does not have support for Python 3.x. Some corrections to WSGI 1.0 are here: http://wsgi.org/wsgi/Amendments_1.0 You may add support to Python 3.x in existing WSGI 1.0 implementation, but your implementation will end up to something that is no more WSGI 1.0. Manlio ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
On 13 April 2010 21:20, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Tue, Apr 13, 2010 at 13:13, Graham Dumpleton graham.dumple...@gmail.com wrote: There is no such thing as a WSGI 2.0 PEP and there is no proper concensus either on what it should look like. Thus if you see anything claiming to implement WSGI 2.0, then it isn't and you should only view it as an experimental proposal. You are warned. :-) Do you (or someone else) have a status on where WSGI 2 is? IIRC WSGI 1 isn't really usable with Python 3.x, so it seems about time we get something going again (AIUI this is blocking Werkzeug from being ported to 3.x, for example). WSGI 2.0 isn't about Python 3.X, it is about removing start_response(). Python 3.X support can be catered for by clarifications in the WSGI 1.0 specification and to a degree how Python 3.X is implemented is dictated by existing practice in the form of what wsgiref implemented in Python 3.1. The Apache/mod_wsgi implementation has had Python 3.X support for over a year using the same interpretation. I believe latest CherryPy WSGI server code is also providing Python 3.X support. Apache/mod_wsgi tried to push the issue of a new definition/specification to cater for Python 3.X by actually identifying itself as WSGI 1.1. The attempts at ratifying that didn't happen however, but then no one has turned around either to complain about Apache/mod_wsgi identifying itself as WSGI 1.1 and so it has been left that way and not reverted to WSGI 1.0. So, in effect existing practice has determined how WSGI on Python 3.X should be implemented and given how long this has been going on, nothing is likely to change that now. You can however see a summary of how it is being interpreted at: http://code.google.com/p/modwsgi/wiki/SupportForPython3X Graham ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
On Tue, Apr 13, 2010 at 13:39, Graham Dumpleton graham.dumple...@gmail.com wrote: WSGI 2.0 isn't about Python 3.X, it is about removing start_response(). Okay, so it is orthogonal, right? Python 3.X support can be catered for by clarifications in the WSGI 1.0 specification and to a degree how Python 3.X is implemented is dictated by existing practice in the form of what wsgiref implemented in Python 3.1. The Apache/mod_wsgi implementation has had Python 3.X support for over a year using the same interpretation. I believe latest CherryPy WSGI server code is also providing Python 3.X support. Apache/mod_wsgi tried to push the issue of a new definition/specification to cater for Python 3.X by actually identifying itself as WSGI 1.1. The attempts at ratifying that didn't happen however, but then no one has turned around either to complain about Apache/mod_wsgi identifying itself as WSGI 1.1 and so it has been left that way and not reverted to WSGI 1.0. So, in effect existing practice has determined how WSGI on Python 3.X should be implemented and given how long this has been going on, nothing is likely to change that now. You can however see a summary of how it is being interpreted at: http://code.google.com/p/modwsgi/wiki/SupportForPython3X So that page has 8 points required for 3.x, which apparently wsgiref and CherryPy also adhere to? And you have 5 simplifications that, as far as we know, have only been been implemented in mod_wsgi? Cheers, Dirkjan ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
On 13 April 2010 21:46, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Tue, Apr 13, 2010 at 13:39, Graham Dumpleton graham.dumple...@gmail.com wrote: WSGI 2.0 isn't about Python 3.X, it is about removing start_response(). Okay, so it is orthogonal, right? Python 3.X support can be catered for by clarifications in the WSGI 1.0 specification and to a degree how Python 3.X is implemented is dictated by existing practice in the form of what wsgiref implemented in Python 3.1. The Apache/mod_wsgi implementation has had Python 3.X support for over a year using the same interpretation. I believe latest CherryPy WSGI server code is also providing Python 3.X support. Apache/mod_wsgi tried to push the issue of a new definition/specification to cater for Python 3.X by actually identifying itself as WSGI 1.1. The attempts at ratifying that didn't happen however, but then no one has turned around either to complain about Apache/mod_wsgi identifying itself as WSGI 1.1 and so it has been left that way and not reverted to WSGI 1.0. So, in effect existing practice has determined how WSGI on Python 3.X should be implemented and given how long this has been going on, nothing is likely to change that now. You can however see a summary of how it is being interpreted at: http://code.google.com/p/modwsgi/wiki/SupportForPython3X So that page has 8 points required for 3.x, which apparently wsgiref and CherryPy also adhere to? And you have 5 simplifications that, as far as we know, have only been been implemented in mod_wsgi? They are not simplications. They are clarifications or just describing existing practice. They are not necessarily mod_wsgi specific. (1) Implemented by everyone already otherwise cgi.FieldStorage doesn't work. (2) Implemented by the more reputable WSGI servers such as CherryPy and Paste WSGI servers. Not implemented by wsgiref. An implementation that doesn't implement it and which doesn't discard request content yet supports HTTP 1.1 request pipelining is arguably broken. A correctly implemented WSGI middleware already has to cope with an empty string as end sentinel anyway to detect premature end of input. (3) Only implemented by Apache/mod_wsgi that I know of for sure. Although it makes it work like a proper file like object as specification suggests wsgi.input is, would raise potential issues with WSGI middleware which depend on it then not being able to be used with WSGI 1.0 and so intent was that it not be made a requirement. (4) This is a statement about WSGI middleware, not the WSGI server. A WSGI middleware that doesn't do it is arguably broken and can generate incorrect data. Thus is a clarification of obligations only. (5) A WSGI server shouldn't do this now as doing so is technically a voilation of HTTP. Thus is a clarification of obligations only. You can find more detail on these at: http://blog.dscpl.com.au/2009/10/details-on-wsgi-10-amendmentsclarificat.html Graham ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
On Tue, Apr 13, 2010 at 14:01, Graham Dumpleton graham.dumple...@gmail.com wrote: They are not simplications. They are clarifications or just describing existing practice. They are not necessarily mod_wsgi specific. Sorry, I didn't mean to imply they were mod_wsgi specific, and they definitely look sane/like an improvement to me. Okay, so, would it be valuable to have both 1.1 and 2.0? I.e. if we start writing a new spec now, should we aim for just a 2.0 that copes with everything at once, or first work on an 1.1 that has clarifications/improvements and is otherwise relatively compatible with 1.0 (but also with python 3.x)? Who's up for some PEP-writing? Cheers, Dirkjan ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
On 13 April 2010 22:12, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Tue, Apr 13, 2010 at 14:01, Graham Dumpleton graham.dumple...@gmail.com wrote: They are not simplications. They are clarifications or just describing existing practice. They are not necessarily mod_wsgi specific. Sorry, I didn't mean to imply they were mod_wsgi specific, and they definitely look sane/like an improvement to me. Okay, so, would it be valuable to have both 1.1 and 2.0? I.e. if we start writing a new spec now, should we aim for just a 2.0 that copes with everything at once, or first work on an 1.1 that has clarifications/improvements and is otherwise relatively compatible with 1.0 (but also with python 3.x)? Who's up for some PEP-writing? The last attempt was to have WSGI 1.1 as clarifications and Python 3.X. And when I say 'last attempt', yes there have been people who have stepped up to try and get this to happen in the past. I think you would be the 3rd time, excluding me in general having tried to push it in the past and also given up. You really should perhaps look back through the archive of WEB-SIG posts on Google Groups to understand the history and how this always seems to just go around in circles. :-) Graham ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
On Tue, Apr 13, 2010 at 14:46, Graham Dumpleton graham.dumple...@gmail.com wrote: The last attempt was to have WSGI 1.1 as clarifications and Python 3.X. And when I say 'last attempt', yes there have been people who have stepped up to try and get this to happen in the past. I think you would be the 3rd time, excluding me in general having tried to push it in the past and also given up. You really should perhaps look back through the archive of WEB-SIG posts on Google Groups to understand the history and how this always seems to just go around in circles. :-) I've been on Web-SIG for quite a while now, exactly to keep track of these issues. Since there doesn't seem to be much traction, I figured it would be time to just get a new PEP together. To limit the amount of work, I'd go in the direction of having a single WSGI 2.0 PEP incorporating your suggestions (maybe minus the number 3), everything required for Python 3 (as outlined by your wiki page). The idea is that, as soon as there is a draft PEP, we can circulate it on Web-SIG for a little bit before bringing it to python-dev. No single person should really be able to block all this. I'll try to write something up that incorporates all of your notes. Cheers, Dirkjan ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
On 13 April 2010 23:55, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Tue, Apr 13, 2010 at 14:46, Graham Dumpleton graham.dumple...@gmail.com wrote: The last attempt was to have WSGI 1.1 as clarifications and Python 3.X. And when I say 'last attempt', yes there have been people who have stepped up to try and get this to happen in the past. I think you would be the 3rd time, excluding me in general having tried to push it in the past and also given up. You really should perhaps look back through the archive of WEB-SIG posts on Google Groups to understand the history and how this always seems to just go around in circles. :-) I've been on Web-SIG for quite a while now, exactly to keep track of these issues. Since there doesn't seem to be much traction, I figured it would be time to just get a new PEP together. To limit the amount of work, I'd go in the direction of having a single WSGI 2.0 PEP Limiting your work by going to WSGI 2.0 and dropping start_response() potentially causes more work for anyone implementing WSGI. This is because if no official statement about Python 3.X is made about WSGI 1.0 (with start_response()), and the only option is WSGI 2.0 (without start_response()), people are likely going to be less inclined to move to Python 3.X for WSGI because to do so means potentially more drastic changes to their code. People talk about WSGI 2.0 - WSGI 1.0 adapters, but if WSGI 1.0 on Python 3.X is formalised, you can do that. incorporating your suggestions (maybe minus the number 3), everything required for Python 3 (as outlined by your wiki page). The whole point of not doing (3) in WSGI 1.1 was that it was seen that it could only be done, if it were to be done, at a point where the API was broken anyway, ie., when start_response() was dropped in WSGI 2.0. So, it shouldn't be ruled out in WSGI 2.0 as a possible change. If you do, then it likely would never be able to be incorporated. So WSGI 2.0 is the only chance to clean that up. The idea is that, as soon as there is a draft PEP, we can circulate it on Web-SIG for a little bit before bringing it to python-dev. No single person should really be able to block all this. I'll try to write something up that incorporates all of your notes. Graham ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
On 9 April 2010 07:53, P.J. Eby p...@telecommunity.com wrote: Also, note that with Nginx (as with Apache, if I'm not wrong), even if application yields small strings, the server can still do some buffering in order to increase performance. In which case, it's in violation of the WSGI spec. The spec requires eparately-yielded strings to be flushed to OS-level buffering. True, and Apache/mod_wsgi does best effort on that. Output filters at Apache level can be a problem with that though as a flush bucket in Apache bucket chains is a request only and an output filter can decide not to flush through all data. For example, mod_deflate may buffer partial data in order to get enough for next block of compressed data. This is the exception rather than the norm, and if no such output filters exists, then separately yield strings should be flushed right through to the socket. So, one can try and satisfy that requirement in WSGI, but in practice it cannot always be achieved because you may have absolutely no control over the underlying web server. Graham ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
At 04:08 PM 4/8/2010 +0200, Manlio Perillo wrote: Hi. Some time ago I objected the decision to remove start_response function from next version WSGI, using as rationale the fact that without start_callable, asynchronous extension are impossible to support. Now I have found that removing start_response will also make impossible to support coroutines (or, at least, some coroutines usage). Here is an example (this is the same example I posted few days ago): http://paste.pocoo.org/show/199202/ Forgetting about the write callable, the problem is that the application starts to yield data when tmpl.render_unicode function is called. Please note that this has *nothing* to do with asynchronus applications. The code should work with *all* WSGI implementations. In the pasted example, the Mako render_unicode function is turned into a generator, with a simple function that allows to flush the current buffer. Can someone else confirm that this code is impossible to support in WSGI 2.0? I don't understand why it's a problem. See my previous post here: http://mail.python.org/pipermail/web-sig/2009-September/003986.html for a sketch of a WSGI 1-to-2 converter. It takes a WSGI 1 application callable as the input, and returns a WSGI 2 function. ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
Aaron Watters ha scritto: someone remind me: where is the canonical WSGI 2 spec? http://wsgi.org/wsgi/WSGI_2.0 I assume there is a way to wrap WSGI 1 applications without breaking them? Or is this the regex--re fiasco all over again? start_response can be implemented by a function that will store the status code and response headers. There should be a sample WSGI 2.0 implementation for CGI, and a sample WSGI 1.0 - 2.0 adapter. This adapter should be able to support the coroutine example, http://paste.pocoo.org/show/199202/ but I would like to test. write callable, as far as I know, can not be implemented. [...] Regards Manlio ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
At 04:59 PM 4/8/2010 +0200, Manlio Perillo wrote: Aaron Watters ha scritto: someone remind me: where is the canonical WSGI 2 spec? http://wsgi.org/wsgi/WSGI_2.0 I assume there is a way to wrap WSGI 1 applications without breaking them? Or is this the regex--re fiasco all over again? start_response can be implemented by a function that will store the status code and response headers. There should be a sample WSGI 2.0 implementation for CGI, and a sample WSGI 1.0 - 2.0 adapter. This adapter should be able to support the coroutine example, http://paste.pocoo.org/show/199202/ but I would like to test. write callable, as far as I know, can not be implemented. Implementing it requires greenlets or threads, but it's implementable. See: http://mail.python.org/pipermail/web-sig/2009-September/003986.html (Btw, I've noticed that this early sketch of mine doesn't support the case where an application is a generator, because start_response won't have been called when the application returns. This can be fixed, but it requires the addition of a wrapper class and a few other annoying details. It also doesn't support exc_info properly, so it's still a ways from being a correct WSGI 1 server implementation. Getting rid of all these little variations, though, is the goal of having a WSGI 2 - it's difficult to write *any* middleware to be completely WSGI 1 compliant.) ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
P.J. Eby ha scritto: At 05:40 PM 4/8/2010 +0200, Manlio Perillo wrote: With WSGI 2.0 we will end up with: - WSGI 1.0, a full featured protocol, but with hard to implement middlewares - WSGI 2.0, a simple protocol, with more easy to implement middlewares but without support for some advanced applications Let me see if I understand what you're saying. You want to support suspending an application, without using greenlets or threads. What I'm trying to do is: * as in the example I posted, turn Mako render function in a generator. The reason is that I would lite to to implement support for Nginx subrequests. During a subrequest, the generated response body is sent directly to the client, so it is necessary to be able to flush the Mako buffer * implement the simple suspend/resume extension, as described here: http://comments.gmane.org/gmane.comp.python.twisted.web/632 Note that my ngx_http_wsgi_module already support asynchronous web server, since when the application returns a generator and sending a yielded buffer to the client would block, execution of WSGI application is suspended, and resumed when the socket is ready to send data. The suspend/resume extension allows an application to explicitly suspend/resume execution, so it is a nice complement for an asynchronous server. I would like to propose this extension for wsgiorg namespace. Not that, however, greenlets are still required, since it will make the code much more usable. Under WSGI 1, you can do this by yielding empty strings before calling start_response. No, in this case this is not what I need to do. I need to call start_response, since the greenlet middleware will yield data to the caller before the application returns. Under WSGI 2, you can only do this by directly suspending execution, e.g. via greenlet or eventlets or some similar API provided by the server. Is this your objection? In WSGI 2 what I want to do is not really possible. The reason is that I don't use greenlets in the C module (I'm not even sure greenlets can be used in my ngx_http_wsgi module) Execution is suspended using the normal suspend extension. The problem is with the greenlet middleware that will force a different code flow. As far as I know, nobody has actually implemented an async app facility for WSGI 1, although it sounds like perhaps you're trying to design or implement such a thing now. Right. My previous attempt was a failure, since the extensions have severe usability problem. It is the same problem you have with Twisted deferred. In this case every function that call a function that use the async extension must be a generator. In my new attempt I plan to: 1) Implement the simple suspend/resume extension 2) Implement a Python extension module that wraps the Nginx events system. 3) Implement a pure Python WSGI middleware that, using greenlets, will enable normal applications to take advantage of Nginx async features. This middleware will have the same purpose as the Hub available in gevent If so, then there's nothing stopping you from implementing a WSGI 1 server and providing a WSGI 2 adapter, since as you point out, WSGI 2 is easier to implement on top of WSGI 1 than the other way around. Yes, this is what I would like to do. Do you think it will possible to implement all the requirements of WSGI 2 (including Python 3.x support) in a simple adapter on top of WSGI 1.0 ? And what about applications that need to use the WSGI 1.0 API but require to run with Python 3.x? Thanks Manlio ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
At 08:06 PM 4/8/2010 +0200, Manlio Perillo wrote: What I'm trying to do is: * as in the example I posted, turn Mako render function in a generator. The reason is that I would lite to to implement support for Nginx subrequests. By subrequest, do you mean that one request is invoking another, like one WSGI application calling multiple other WSGI applications to render one page containing contents from more than one? During a subrequest, the generated response body is sent directly to the client, so it is necessary to be able to flush the Mako buffer I don't quite understand this, since I don't know what Mako is, or, if it's a template engine, what flushing its buffer would have to do with WSGI buffering. Under WSGI 1, you can do this by yielding empty strings before calling start_response. No, in this case this is not what I need to do. Well, if that's not when you're needing to suspend the application, then I don't see what you're losing in WSGI 2. I need to call start_response, since the greenlet middleware will yield data to the caller before the application returns. I still don't understand you. In WSGI 1, the only way to suspend execution (without using greenlets) prior to determining the headers is to yield empty strings. I'm beginning to wonder if maybe what you're saying is that you want to be able to write an application function in the form of a generator? If so, be aware that any WSGI 1 app written as: def app(environ, start_response): start_response(status, headers) yield foo yield bar can be written as a WSGI 2 app thus: def app(environ, start_response): def respond(): yield foo yield bar return status, headers, respond() This is also a good time for people to learn that generators are usually a *very bad* way to write WSGI apps - yielding is for server push or sending blocks of large files, not tiny strings. In general, if you're yielding more than one block, you're almost certainly doing WSGI wrong. The typical HTML, XML, or JSON output that's 99% of a webapp's requests should be transmitted as a single string, rather than as a series of snippets. IOW, the absence of generator support in WSGI 2 is a feature, not a bug. In my new attempt I plan to: 1) Implement the simple suspend/resume extension 2) Implement a Python extension module that wraps the Nginx events system. 3) Implement a pure Python WSGI middleware that, using greenlets, will enable normal applications to take advantage of Nginx async features. I think maybe I'm understanding a little better now -- you want to implement the WSGI gateway entirely in C, without using any Python, and without using the greenlet API directly. I think I've been unable to understand because I'm thinking in terms of a server implemented in Python, or at least that has the WSGI part implemented in Python. Do you think it will possible to implement all the requirements of WSGI 2 (including Python 3.x support) in a simple adapter on top of WSGI 1.0 ? My practical experience with Python 3 is essentially nonexistent, but being able to implement WSGI 2 in terms of WSGI 1 is a *design requirement* for WSGI 2; it's likely that much early use and development of WSGI 2 will be done through such an adapter. And what about applications that need to use the WSGI 1.0 API but require to run with Python 3.x? That's a tougher nut to crack; again, my practical experience with Python 3 is essentially nonexistent. ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] WSGI and start_response
P.J. Eby ha scritto:
At 08:06 PM 4/8/2010 +0200, Manlio Perillo wrote:
What I'm trying to do is:
* as in the example I posted, turn Mako render function in a generator.
The reason is that I would lite to to implement support for Nginx
subrequests.
By subrequest, do you mean that one request is invoking another, like
one WSGI application calling multiple other WSGI applications to render
one page containing contents from more than one?
Yes.
During a subrequest, the generated response body is sent directly to
the client, so it is necessary to be able to flush the Mako buffer
I don't quite understand this, since I don't know what Mako is, or, if
it's a template engine, what flushing its buffer would have to do with
WSGI buffering.
Ah, sorry.
Mako is a template engine.
Suppose I have an HTML template file, and I want to use a sub request.
html
head.../head
body
div${subrequest('/header/'}/div
...
/body
/html
The problem with this code is that, since Mako will buffer all generated
content, the result response body will contain incorrect data.
It will first contain the response body generated by the sub request,
then the content generated from the Mako template (XXX I have not
checked this, but I think it is how it works).
So, when executing a sub request, it is necessary to flush (that is,
send to Nginx, in my case) the content generated from the template
before the sub request is done.
Since Mako does not return a generator (I asked the author, and it was
too hard to implement), I use a greenlet in order to turn the Mako
render function in a generator.
Under
WSGI 1, you can do this by yielding empty strings before calling
start_response.
No, in this case this is not what I need to do.
Well, if that's not when you're needing to suspend the application, then
I don't see what you're losing in WSGI 2.
I need to call start_response, since the greenlet middleware will yield
data to the caller before the application returns.
I still don't understand you. In WSGI 1, the only way to suspend
execution (without using greenlets) prior to determining the headers is
to yield empty strings.
Ah, you are right sorry.
But this is not required for the Mako example (I was focusing on that
example).
I'm beginning to wonder if maybe what you're saying is that you want to
be able to write an application function in the form of a generator?
The greenlet middleware return a generator, in order to work.
If
so, be aware that any WSGI 1 app written as:
def app(environ, start_response):
start_response(status, headers)
yield foo
yield bar
can be written as a WSGI 2 app thus:
def app(environ, start_response):
def respond():
yield foo
yield bar
return status, headers, respond()
The problem, as I wrote, is that with the greenlet middleware, the
application needs not to return a generator.
def app(environ):
tmpl = ...
body = tmpl.render(...)
return status, headers, [body]
This is a very simple WSGI application.
But when using the greenlet middleware, and when using the function for
flushing Mako buffer, some data will be yielded *before* the application
returns and status and headers are passed to Nginx.
This is also a good time for people to learn that generators are usually
a *very bad* way to write WSGI apps
It's the only way to be able to suspend execution, when the WSGI
implementation is embedded in an async web server not written in Python.
The reason is that you can not use (XXX check me) greenlets in C code,
you should probably use something like http://code.google.com/p/coev/
Greenlets can be used in gevent, as an example, because scheduling is
under control of Python code.
This is not the case with Nginx.
- yielding is for server push or
sending blocks of large files, not tiny strings.
Again, consider the use of sub requests.
yielding a not large block is the only choice you have.
Unless, of course, you implement sub request support in pure Python (or
using SSI - Server Side Include).
Another use case is when you have a very large page, and you want to
return some data as soon as possible to avoid the user to abort request
if it takes some time.
Also, note that with Nginx (as with Apache, if I'm not wrong), even if
application yields small strings, the server can still do some buffering
in order to increase performance.
In ngx_http_wsgi_module buffering is optional (and disabled by default).
In the sub request example, it means that if both the main request
response body and sub request response body are small, Nginx can buffer
all the data in memory before sending it to the client (XXX I need to
check this).
In general, if you're
yielding more than one block, you're almost certainly doing WSGI wrong.
The typical HTML, XML, or JSON output that's 99% of a webapp's requests
should be transmitted as a
Re: [Web-SIG] wsgi write=start_response() and iterable return?
--- On Mon, 1/4/10, P.J. Eby p...@telecommunity.com wrote: From: P.J. Eby p...@telecommunity.com Subject: Re: [Web-SIG] wsgi write=start_response() and iterable return? To: Aaron Watters arw1...@yahoo.com, web-sig@python.org Date: Monday, January 4, 2010, 4:38 PM At 08:42 AM 1/4/2010 -0800, Aaron Watters wrote: From: Aaron Watters arw1...@yahoo.com If an application returns an iterable response and *also* calls the write()... what is supposed to happen? After carefully considering all the responses on this issue ;c) I came up with the following strategy for dealing with calls to write() in combination with an iterable response: see http://listtree.appspot.com/listtreeNotes/qFxCJOYB2xkf2vyQS5L$AA This wrapper implementation diverts calls to write() into the iterable response so the rest of the system can ignore the write() function(). I'd be very happy if some of you would take a quick look and see if this makes sense to you. Do note that an application which calls write() from an iterator body is *not* WSGI compliant, as described under: In practice, however, wsgiref.handlers treats write() and yield as interchangeable, and wsgiref.validate doesn't complain if an application calls write() from within an iteration.. :-( ... And I'm not sure that a complicated program might not do this even it was not intended if it uses both idioms. In fact since WHIFF is designed to combine external components you probably can confuse WHIFF (or other intrastructure tools) into mixing the modes even if the tool doesn't mix modes directly. I'd like to see the write() callable removed from future versions of WSGI, and wrappers like the one I referenced could provide backwards compatibility for old style apps. -- Aaron Watters === less is more ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] wsgi write=start_response() and iterable return?
From: Aaron Watters arw1...@yahoo.com If an application returns an iterable response and *also* calls the write()... what is supposed to happen? After carefully considering all the responses on this issue ;c) I came up with the following strategy for dealing with calls to write() in combination with an iterable response: see http://listtree.appspot.com/listtreeNotes/qFxCJOYB2xkf2vyQS5L$AA This wrapper implementation diverts calls to write() into the iterable response so the rest of the system can ignore the write() function(). I'd be very happy if some of you would take a quick look and see if this makes sense to you. Thanks in advance, -- Aaron Watters === less is more ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
Re: [Web-SIG] wsgi write=start_response() and iterable return?
At 08:42 AM 1/4/2010 -0800, Aaron Watters wrote: From: Aaron Watters arw1...@yahoo.com If an application returns an iterable response and *also* calls the write()... what is supposed to happen? After carefully considering all the responses on this issue ;c) I came up with the following strategy for dealing with calls to write() in combination with an iterable response: see http://listtree.appspot.com/listtreeNotes/qFxCJOYB2xkf2vyQS5L$AA This wrapper implementation diverts calls to write() into the iterable response so the rest of the system can ignore the write() function(). I'd be very happy if some of you would take a quick look and see if this makes sense to you. Do note that an application which calls write() from an iterator body is *not* WSGI compliant, as described under: http://www.python.org/dev/peps/pep-0333/#the-write-callable Applications MUST NOT invoke write() from within their return iterable, and therefore any strings yielded by the iterable are transmitted after all strings passed to write() have been sent to the client. In practice, however, wsgiref.handlers treats write() and yield as interchangeable, and wsgiref.validate doesn't complain if an application calls write() from within an iteration.. :-( ___ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
