The WSGI specification doesn't really say much about chunked transfer encoding for content sent within the body of a request. The only thing that appears to apply is the comment:
WSGI servers must handle any supported inbound "hop-by-hop" headers on their own, such as by decoding any inbound Transfer-Encoding, including chunked encoding if applicable. What does this really mean in practice though? As a means of getting feedback on what is the correct approach I'll go through how the CherryPy WSGI server handles it. The problem is that the CherryPy approach raises a few issues which makes me wander if it is doing it in the most appropriate way. In CherryPy, when it sees that the Transfer-Encoding is set to 'chunked' while parsing the HTTP headers, it will at that point, even before it has called start_response for the WSGI application, read in all content from the body of the request. CherryPy reads in the content like this for two reasons. The first is so that it can then determine the overall length of the content that was available and set the CONTENT_LENGTH value in the WSGI environ. The second reason is so that it can read in any additional HTTP header fields that may occur in the trailer after the last data chunk and also incorporate them into the WSGI environ. The first issue with what it does is that it has read in all the content. This denies a WSGI application the ability to stream content from the body of a request and process it a bit at a time. If the content is huge, that it buffers it can also mean the application process size will grow significantly. The second issue, although I am confused on whether the CherryPy WSGI server actually implements this correctly, is that if the client was expecting to see a 100 continue response, this will need to be sent back to the client before any content can be read. When chunked transfer encoding is not used, such a 100 continue response would in a good WSGI server only be sent when the WSGI application called read() on wsgi.input for the first time. Ie., the 100 continue indicates that the application which is consuming the data is actually ready to start processing it. What CherryPy WSGI server is doing is circumventing that and the client could think the final consumer application is ready before it actually is. Note that I am assuming here that 100 continue is still usable in conjunction with chunked transfer encoding. In CherryPy WSGI server it only actually sends the 100 continue after it attempts to try and read content in the presence of a chunked transfer encoding header. Not sure if this is actually a bug or not. CherryPy WSGI server also doesn't wait until first read() by WSGI application before sending back the 100 continue either and instead sends it as soon as the headers are parsed. This may be fine, but possibly not most optimal as it denies an application the ability to fail a request and avoid a client sending the actual content. Now, to my mind, the preferred approach would be that the content would not be read up front like this and instead CONTENT_LENGTH would simply be unset in the WSGI environ. >From prior discussions related to input filtering on the list, a WSGI application shouldn't really be paying much attention to CONTENT_LENGTH anyway and should just be using read() to get data until it returns an empty string. Thus, for chunked data, that it doesn't know the content length up front shouldn't matter as it should just call read() until there is no more. BTW, it may not be this simple for something like a proxy, but that is a discussion for another time. Doing this also means that the 100 continue only gets sent when the application is ready and there is no need to for the content to be buffered up. That it is the actual application which is consuming the data and not some intermediary means that an application could implement some mechanism whereby it reads some data, acts on that and starts sending some data in response. The client then might send more data based on that response which the application only then reads, send more data as response etc. Thus an end to end communication stream can be established where the actual overall content length of the request could never be established up front. The only problem with deferring any reading of data to when the application wants to actually read it, is that if the overall length of content in the request is bounded, there is no way to get access to the additional headers in the trailer of the request and have them available in the WSGI environ since processing of the WSGI environ has already occurred before any data was read. So, what gives. What should a WSGI server do for chunked transfer encoding on a request? I may not totally understand 100 continue and chunked transfer encoding and am happy to be correct in my understanding of them, but what CherryPy WSGI server does doesn't seem right to me at first look. Graham _______________________________________________ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com