Re: POST without Content-Length
On Sun, 8 Aug 2004, [ISO-8859-15] André Malo wrote: * Nick Kew [EMAIL PROTECTED] wrote: On Sun, 8 Aug 2004, [ISO-8859-15] André Malo wrote: A CGI script therefore should never trust Content-Length, but just read stdin until it meets an EOF. That is well-known to fail in CGI. A CGI must use Content-Length. Folks; please! CGI/1.0 is a crap interface. We all know that. Do not break its semantics or make it worse; instead define a nice CGI/2.0 (and there is a nice version number in the ENV')s, fix the input filter to not do stupid things, and make the world a better place by introducing a AddHandler cgi2-script .cgi or (shudder) Script2Alias. DW
Re: POST without Content-Length
Hi, On Sun, 08 Aug 2004 00:24:15 +0200, Nick Kew wrote: ... Therefore it should be safe to assume if no Content-Length and no chunked headers are present there MUST follow an optional body with the connection-close afterwards as 'persistent connection' MUST NOT be present. Nope. GET requests routinely have keep-alive, but don't have bodies. Just FYI squid-3.0 release notes: request_entities New squid.conf directive request_entities on/off.If set to on then Squid will allow GET/HEAD requests with request entities, even if such entites are undefined in the HTTP specification. (Henrik Nordstrom) Anyway I have no longer standards compliance complaints as httpd-2.1 looks OK: On Sun, 08 Aug 2004 00:07:24 +0200, Roy T. Fielding wrote: ... An HTTP request with no content-length and no tranfer-encoding has no body, period: The presence of a message-body in a request is signaled by the inclusion of a Content-Length or Transfer-Encoding header field in the request's message-headers. Oops, with the greatest regards to RFC author :-), Lace -- Jan Kratochvil; Captive: free r/w NTFS Filesystem; http://www.jankratochvil.net/
Re: POST without Content-Length
Hi, On Sun, 08 Aug 2004 00:07:24 +0200, Roy T. Fielding wrote: ... An HTTP request with no content-length and no tranfer-encoding has no body, period: OK. Now just a question whether to not to implement HTTP-request sanity check as is implemented in squid clientCheckContentLength(): PUT+POST: require a request entity: HTTP_LENGTH_REQUIRED = 411 GET+HEAD: forbid a request entity if !$request_entities: HTTP_LENGTH_REQUIRED = 411 configurable $request_entities: allow GET/HEAD requests with request entities, even if such entites are undefined in the HTTP specification. Unfortunately there is no HTTP specification which methods MUST/MUST NOT/etc. (RFC 2119) include HTTP request bodies. Lace -- Jan Kratochvil; Captive: free r/w NTFS Filesystem; http://www.jankratochvil.net/
Re: POST without Content-Length
--On Saturday, August 7, 2004 8:49 AM +0200 Jan Kratochvil [EMAIL PROTECTED] wrote: according to RFC2616 (HTTP/1.1) section 4.4 it is valid to do POST method without specifying Content-Length. Mobile phone SonyEricsson P900 sends such requests to its specified WAP proxy (behaving in HTTP way - I assume a bug): http://www.jankratochvil.net/priv/SonyEricssonP900.HTTPrequest.gz Unfortunately no httpd/apache or squid versions support such requests. * httpd assumes no body at all and tries to interpret the message body as another HTTP request. * squid complains with 411 (length required) Um, Apache HTTP Server 2.0 (and higher) certainly reads chunked input. Our httpd-test perl-framework explicitly checks for this behavior. -- justin
Re: POST without Content-Length
On Sat, 07 Aug 2004 09:06:18 +0200, Justin Erenkrantz wrote: ... Um, Apache HTTP Server 2.0 (and higher) certainly reads chunked input. Our httpd-test perl-framework explicitly checks for this behavior. -- justin Oops, OK, my mistake during summarizing the bugreport today. That request in the previous mail is OK as it is yet chunked encoded. Unfortunately httpd proxy will forward it dechunked although it will not fill-in any Content-Length. httpd is no longer to handle it on its input afterwards. The sniffed forwarded request is at: http://www.jankratochvil.net/priv/SonyEricssonP900-forwarded.HTTPrequest.gz (tested in Red Hat Fedora Core 2 httpd-2.0.50-2.1) Thanks, Lace -- Jan Kratochvil; Captive: free r/w NTFS Filesystem; http://www.jankratochvil.net/
Re: POST without Content-Length
--On Saturday, August 7, 2004 9:18 AM +0200 Jan Kratochvil [EMAIL PROTECTED] wrote: Unfortunately httpd proxy will forward it dechunked although it will not fill-in any Content-Length. httpd is no longer to handle it on its input afterwards. The sniffed forwarded request is at: http://www.jankratochvil.net/priv/SonyEricssonP900-forwarded.HTTPrequest .gz That's a slightly different story. 2.1 has the fix for this (proxy_http.c r1.166), but it never got back ported to 2.0. 2.0's STATUS says: * Rewrite how proxy sends its request to allow input bodies to morph the request bodies. Previously, if an input filter changed the request body, the original C-L would be sent which would be incorrect. Due to HTTP compliance, we must either send the body T-E: chunked or include a C-L for the request body. Connection: Close is not an option. [jerenkrantz2002/12/08 21:37:27] +1: stoddard, striker, jim -1: brianp (we need a more robust solution than what's in 2.1 right now) jerenkrantz (should be fixed, but I don't have time to do this) At this date (about 20 months later), I have no earthly idea what was wrong. But, I'd suggest trying httpd-2.0 HEAD (aka httpd-2.1) and see if that fixes it. Perhaps someone can remember why I agreed with Brian and what I never fixed... ;-) -- justin
Re: POST without Content-Length
On Sat, 7 Aug 2004, Justin Erenkrantz wrote: That's a slightly different story. 2.1 has the fix for this (proxy_http.c r1.166), but it never got back ported to 2.0. We have a lot of proxy updates in 2.1, which are presumably getting test-driven over time. How would one go about proposing a wholesale backport? 2.0's STATUS says: * Rewrite how proxy sends its request to allow input bodies to morph the request bodies. Previously, if an input filter changed the request body, the original C-L would be sent which would be incorrect. This is basically the same as an output filter changing the content-length. In the 2.0 architecture, the filter must take responsibility for not sending a bogus length. The only difference is that Connection: close is an option in output. Due to HTTP compliance, we must either send the body T-E: chunked or include a C-L for the request body. Connection: Close is not an option. [jerenkrantz2002/12/08 21:37:27] +1: stoddard, striker, jim -1: brianp (we need a more robust solution than what's in 2.1 right now) jerenkrantz (should be fixed, but I don't have time to do this) At this date (about 20 months later), I have no earthly idea what was wrong. But, I'd suggest trying httpd-2.0 HEAD (aka httpd-2.1) and see if that fixes it. Perhaps someone can remember why I agreed with Brian and what I never fixed... ;-) -- justin Hmmm, did your fix merely chunk content, or compute C-L, or was it smart enough to do the Right Thing according to whether the backend is HTTP/1.1[1], whether the content is short enough to fit in one heap bucket, or whatever other criteria might be applied? [1] Presumably we can only assume HTTP/1.1 backend in a controlled - reverse proxy - case, and where the admin has configured it? -- Nick Kew
Re: POST without Content-Length
--On Saturday, August 7, 2004 6:14 PM +0100 Nick Kew [EMAIL PROTECTED] wrote: We have a lot of proxy updates in 2.1, which are presumably getting test-driven over time. How would one go about proposing a wholesale backport? How about releasing 2.2 instead? ;-) Hmmm, did your fix merely chunk content, or compute C-L, or was it smart enough to do the Right Thing according to whether the backend is HTTP/1.1[1], whether the content is short enough to fit in one heap bucket, or whatever other criteria might be applied? [1] Presumably we can only assume HTTP/1.1 backend in a controlled - reverse proxy - case, and where the admin has configured it? Actually, now that I think about it, that was the issue. It needs to only send chunks conditionally based on some config setting. -- justin
RE: POST without Content-Length
Justin Erenkrantz wrote: We have a lot of proxy updates in 2.1, which are presumably getting test-driven over time. How would one go about proposing a wholesale backport? How about releasing 2.2 instead? ;-) How about backport :) JaeanFrederic, Henri and myself take a liberty and make a lots of updates to proxy. The general idea was to make a successor of jk2, but we were convinced to take a look at proxy framework. So we did, and make a connection pool for http (20% performance boost over current proxy implementation), added AJP13 module (proxy_ajp) with proxy_balancer on the way all working with 2.0.50. You may find the sources at jakarta-tomcat-connectors/ajp. The question to backport is IMO more political then technological. The users will switch to 2.1/2.2 no mater if the ASF artificially 'hides' the technology from 2.0. Of course the pragmatism will make a 2.0 alive for couple of years (as well as 1.3), but that doesn't mater. What maters is a stable user-aware solution. If that prime principle (although conservative) means that we do not have enough proven infrastructure to test the software, but just rely on user experience, then I'm in favor to make all the 'advanced technology' as 2.2 goal. OTOH hiding something just to 'boost' a new version is not fair. Regards, MT. smime.p7s Description: S/MIME cryptographic signature
Re: POST without Content-Length
* Mladen Turk [EMAIL PROTECTED] wrote:
The question to backport is IMO more political then technological. The users
will switch to 2.1/2.2 no mater if the ASF artificially 'hides' the
technology from 2.0. Of course the pragmatism will make a 2.0 alive for
couple of years (as well as 1.3), but that doesn't mater. What maters is a
stable user-aware solution. If that prime principle (although conservative)
means that we do not have enough proven infrastructure to test the software,
but just rely on user experience, then I'm in favor to make all the
'advanced technology' as 2.2 goal. OTOH hiding something just to 'boost' a
new version is not fair.
I think, you've got something wrong. As of the branch to 2.1, we've marked 2.0
stable and more or less freezed. That has nothing to do with being fair or
not. The development just happens in 2.1 and beyond. Big changes to 2.0 are
always dangerous with regard to stability. (therefore RTC policy).
Backporting all stuff from 2.1 to 2.0 doesn't do any good.
nd
--
sub the($){+shift} sub answer (){ord q
[* It is always 42! *] }
print the answer
# André Malo # http://pub.perlig.de/ #
Re: POST without Content-Length
Hi, On Sat, 07 Aug 2004 09:35:40 +0200, Justin Erenkrantz wrote: ... At this date (about 20 months later), I have no earthly idea what was wrong. But, I'd suggest trying httpd-2.0 HEAD (aka httpd-2.1) and see if that fixes it. Thanks for the great support - httpd-2.0 HEAD 2004-08-07 really fixes it. It even provides env variable proxy-sendchunks to select between compatible Content-Length (default) and performance-wise chunked. On Sat, 07 Aug 2004 19:14:37 +0200, Nick Kew wrote: On Sat, 7 Aug 2004, Justin Erenkrantz wrote: That's a slightly different story. 2.1 has the fix for this (proxy_http.c r1.166), but it never got back ported to 2.0. We have a lot of proxy updates in 2.1, which are presumably getting test-driven over time. How would one go about proposing a wholesale backport? FYI Fedora Core 2 httpd already backports httpd-2.1 version of proxy_http.c although it was not so new snapshot to include resolving of my issues. Current CVS snapshot I Bugzilled them as https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=129391 FYI backport of current mod_proxy is technically trivia - just copying raw mod_proxy.c mod_proxy.h proxy_http.c although it brings new domain-remapping functionality there. ... This is basically the same as an output filter changing the content-length. In the 2.0 architecture, the filter must take responsibility for not sending a bogus length. The only difference is that Connection: close is an option in output. Although the proxy is OK now there still remains one problem: I think HTTP server MUST accept the request: POST ... HTTP/1.0 or HTTP/1.1 [ no Content-Length ] [ no Transfer-Encoding ] Connection: close [ or even no Connection header at all] \r\n DATA according to RFC2616 section 4.4. Even httpd-2.1/CVS just assumes empty body. squid up to squid/2.5.STABLE5 at least responds by 411 Length Required. Regards, Lace -- Jan Kratochvil; Captive: free r/w NTFS Filesystem; http://www.jankratochvil.net/
RE: POST without Content-Length
André Malo wrote: OTOH hiding something just to 'boost' a new version is not fair. I think, you've got something wrong. Perhaps :) As of the branch to 2.1, we've marked 2.0 stable and more or less freezed. That has nothing to do with being fair or not. The development just happens in 2.1 and beyond. Big changes to 2.0 are always dangerous with regard to stability. (therefore RTC policy). Dangerous? It works or it doesnt. Backporting all stuff from 2.1 to 2.0 doesn't do any good. Of course, but if something works with 2.0, then it perhaps deserve a second look. It won't kill anyone :) Regards, MT. smime.p7s Description: S/MIME cryptographic signature
Re: POST without Content-Length
On Sat, 7 Aug 2004, Jan Kratochvil wrote: Hi, Thanks for the great support - httpd-2.0 HEAD 2004-08-07 really fixes it. It even provides env variable proxy-sendchunks to select between compatible Content-Length (default) and performance-wise chunked. Sounds pretty complete to me. Of course you'd need to stick to C-L unless you *know* the backend accepts chunks. It occurs to me that a similar situation arises with CGI and chunked input. The CGI spec guarantees a content-length header, so presumably(?) the code for dealing with that is already there somewhere, and will figure in the AP_CHUNKED_DECHUNK option to the old handler-read functions. We have a lot of proxy updates in 2.1, which are presumably getting test-driven over time. How would one go about proposing a wholesale backport? FYI Fedora Core 2 httpd already backports httpd-2.1 version of proxy_http.c although it was not so new snapshot to include resolving of my issues. Current CVS snapshot I Bugzilled them as https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=129391 FYI backport of current mod_proxy is technically trivia - just copying raw mod_proxy.c mod_proxy.h proxy_http.c It would also need proxy_util.c. Not sure about ftp or connect. although it brings new domain-remapping functionality there. Indeed. Although the proxy is OK now there still remains one problem: I think HTTP server MUST accept the request: POST ... HTTP/1.0 or HTTP/1.1 [ no Content-Length ] [ no Transfer-Encoding ] Connection: close [ or even no Connection header at all] \r\n DATA according to RFC2616 section 4.4. Even httpd-2.1/CVS just assumes empty body. squid up to squid/2.5.STABLE5 at least responds by 411 Length Required. Surely that only applies if the server can infer there's a request body. How does it do that with neither C-L nor T-E to indicate a body? If we could infer a body in such a case, then AIUI the following applies: If a request contains a message-body and a Content-Length is not given, the server SHOULD respond with 400 (bad request) if it cannot determine the length of the message, or with 411 (length required) if it wishes to insist on receiving a valid Content-Length. Maybe we should infer a body (and hence apply the above logic) in any POST or PUT request? If we do that, it begs the question of how to treat unknown HTTP/extension methods (cf DAV), and suggests perhaps RequireRequestBody should be made a configuration directive. -- Nick Kew
Re: POST without Content-Length
* Nick Kew [EMAIL PROTECTED] wrote: It occurs to me that a similar situation arises with CGI and chunked input. The CGI spec guarantees a content-length header, ah, no. | * CONTENT_LENGTH | | The length of the said content as given by the client. That's rather, *if* the client says something about the length, then CONTENT_LENGTH tells about it. One should not trust it anyway, since inflating compressed content with mod_deflate (for example), changes the length, but neither changes the Header nor the environment variable. Without joining the discussion :-) nd -- +[++-][-]++.[++-]+++.--.+. .[-]---.+++[-]+.+.+.+++[-].---.+++[-] +..+[]+.+[-]+.+++[-]+.--..++ [---].+[+-].++..--.+++[-]+.
Re: POST without Content-Length
On Sat, 07 Aug 2004 21:38:19 +0200, Nick Kew wrote: On Sat, 7 Aug 2004, Jan Kratochvil wrote: ... Current CVS snapshot I Bugzilled them as https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=129391 FYI backport of current mod_proxy is technically trivia - just copying raw mod_proxy.c mod_proxy.h proxy_http.c It would also need proxy_util.c. Not sure about ftp or connect. It is not about would - it WORKS4ME fine with just those 3 files in the Bugzilla post above. ... Maybe we should infer a body (and hence apply the above logic) in any POST or PUT request? If we do that, it begs the question of how to treat unknown HTTP/extension methods (cf DAV), and suggests perhaps RequireRequestBody should be made a configuration directive. What would happen in this case httpd would infer a body while no body would be found there? * In the case of a 'connection close' nothing, empty body would be found. * In the case of a 'persistent connection': * RFC2616 section 8.1.2.1: In order to remain persistent, all messages on the connection MUST have a self-defined message length (i.e., one not defined by closure of the connection), as described in section 4.4. Therefore 'persistent connection' is not allowed in this case. Therefore it should be safe to assume if no Content-Length and no chunked headers are present there MUST follow an optional body with the connection-close afterwards as 'persistent connection' MUST NOT be present. Regards, Lace -- Jan Kratochvil; Captive: free r/w NTFS Filesystem; http://www.jankratochvil.net/
Re: POST without Content-Length
What would happen in this case httpd would infer a body while no body would be found there? * In the case of a 'connection close' nothing, empty body would be found. * In the case of a 'persistent connection': * RFC2616 section 8.1.2.1: In order to remain persistent, all messages on the connection MUST have a self-defined message length (i.e., one not defined by closure of the connection), as described in section 4.4. Therefore 'persistent connection' is not allowed in this case. Therefore it should be safe to assume if no Content-Length and no chunked headers are present there MUST follow an optional body with the connection-close afterwards as 'persistent connection' MUST NOT be present. No, because looking for body when no body is present is an expensive operation. An HTTP request with no content-length and no tranfer-encoding has no body, period: The presence of a message-body in a request is signaled by the inclusion of a Content-Length or Transfer-Encoding header field in the request's message-headers. Roy
Re: POST without Content-Length
On Saturday, August 7, 2004, at 01:17 PM, André Malo wrote: * Nick Kew [EMAIL PROTECTED] wrote: It occurs to me that a similar situation arises with CGI and chunked input. The CGI spec guarantees a content-length header, ah, no. | * CONTENT_LENGTH | | The length of the said content as given by the client. That's rather, *if* the client says something about the length, then CONTENT_LENGTH tells about it. One should not trust it anyway, since inflating compressed content with mod_deflate (for example), changes the length, but neither changes the Header nor the environment variable. CGI would happen after mod_deflate. If mod_deflate changes the request body without also (un)setting content-length, then it is broken. However, I suspect you are thinking of a response body, not the request. Roy
Re: POST without Content-Length
Thanks for the great support - httpd-2.0 HEAD 2004-08-07 really fixes it. It even provides env variable proxy-sendchunks to select between compatible Content-Length (default) and performance-wise chunked. Sounds pretty complete to me. Of course you'd need to stick to C-L unless you *know* the backend accepts chunks. If the client sent chunks, then it is safe to assume that the proxy can send chunks as well. Generally speaking, user agents only send chunks to applications that they know will accept chunks. Roy
Re: POST without Content-Length
On Sat, 7 Aug 2004, Roy T.Fielding wrote: Thanks for the great support - httpd-2.0 HEAD 2004-08-07 really fixes it. It even provides env variable proxy-sendchunks to select between compatible Content-Length (default) and performance-wise chunked. Sounds pretty complete to me. Of course you'd need to stick to C-L unless you *know* the backend accepts chunks. If the client sent chunks, then it is safe to assume that the proxy can send chunks as well. Generally speaking, user agents only send chunks to applications that they know will accept chunks. The client could be sending chunks precisely because it's designed to work with a proxy that is known to accept them. That doesn't imply any knowledge of the backend(s) proxied, which might be anything up to and including the 'net in general. Also bear in mind that we were discussing (also) the case where the request came with C-L but an input filter invalidated it. -- Nick Kew
Re: POST without Content-Length
* Roy T. Fielding [EMAIL PROTECTED] wrote: On Saturday, August 7, 2004, at 01:17 PM, André Malo wrote: * Nick Kew [EMAIL PROTECTED] wrote: It occurs to me that a similar situation arises with CGI and chunked input. The CGI spec guarantees a content-length header, ah, no. | * CONTENT_LENGTH | | The length of the said content as given by the client. That's rather, *if* the client says something about the length, then CONTENT_LENGTH tells about it. One should not trust it anyway, since inflating compressed content with mod_deflate (for example), changes the length, but neither changes the Header nor the environment variable. CGI would happen after mod_deflate. If mod_deflate changes the request body without also (un)setting content-length, then it is broken. Huh? Input filters are pulled, so they run *after* the handler has been started. And - CONTENT_LENGTH (if any - It's unset for chunked as well) still reflects the Content-Length sent by the client. So the current behaviour is correct in all cases. A CGI script therefore should never trust Content-Length, but just read stdin until it meets an EOF. However, I suspect you are thinking of a response body, not the request. No. nd -- package Hacker::Perl::Another::Just;print [EMAIL PROTECTED] split/::/ =__PACKAGE__]}~; # André Malo # http://www.perlig.de #
Re: POST without Content-Length
On Sat, 7 Aug 2004, Jan Kratochvil wrote: What would happen in this case httpd would infer a body while no body would be found there? Just consider a bog-standard GET. Therefore it should be safe to assume if no Content-Length and no chunked headers are present there MUST follow an optional body with the connection-close afterwards as 'persistent connection' MUST NOT be present. Nope. GET requests routinely have keep-alive, but don't have bodies. -- Nick Kew
Re: POST without Content-Length
On Sun, 8 Aug 2004, [ISO-8859-15] André Malo wrote: A CGI script therefore should never trust Content-Length, but just read stdin until it meets an EOF. That is well-known to fail in CGI. A CGI must use Content-Length. -- Nick Kew
Re: POST without Content-Length
If the client sent chunks, then it is safe to assume that the proxy can send chunks as well. Generally speaking, user agents only send chunks to applications that they know will accept chunks. The client could be sending chunks precisely because it's designed to work with a proxy that is known to accept them. That doesn't imply any knowledge of the backend(s) proxied, which might be anything up to and including the 'net in general. Theoretically, yes. However, in practice, that is never the case. Either a user agent is using generic stuff like HTML forms, which will always result in a content-length if there is a body, or it is using custom software designed to work with custom server apps. There are no other real-world examples, and thus it is safe to use chunks if the client used chunks. Also bear in mind that we were discussing (also) the case where the request came with C-L but an input filter invalidated it. I was not discussing that case. The answer to that case is don't do that. Fix the input filter if it is doing something stupid. Roy
Re: POST without Content-Length
* Nick Kew [EMAIL PROTECTED] wrote: On Sun, 8 Aug 2004, [ISO-8859-15] André Malo wrote: A CGI script therefore should never trust Content-Length, but just read stdin until it meets an EOF. That is well-known to fail in CGI. A CGI must use Content-Length. Hmm. any pointers where this is specified? I didn't have any problems with this until now - but in trusting the C-L variable. (Besides the fact, that Content-Length doesn't need to be set). nd -- Wenn nur Ingenieure mit Diplom programmieren würden, hätten wir wahrscheinlich weniger schlechte Software. Wir hätten allerdings auch weniger gute Software. -- Felix von Leitner in dasr
Re: POST without Content-Length
CGI would happen after mod_deflate. If mod_deflate changes the request body without also (un)setting content-length, then it is broken. Huh? Input filters are pulled, so they run *after* the handler has been started. And - CONTENT_LENGTH (if any - It's unset for chunked as well) still reflects the Content-Length sent by the client. So the current behaviour is correct in all cases. No, it is broken in all cases. CGI scripts cannot handle chunked input and they cannot handle bodies without content-length -- that is how the interface was designed. You would have to define a CGI+ interface to get some other behavior. A CGI script therefore should never trust Content-Length, but just read stdin until it meets an EOF. We cannot redefine CGI. It is a legacy crap interface. Input filters either have to be disabled for CGI or replaced with a buffering system that takes HTTP/1.1 in and supplies CGI with the correct metadata and body. Roy
Re: POST without Content-Length
On Sat, 7 Aug 2004, Roy T.Fielding wrote: If the client sent chunks, then it is safe to assume that the proxy can send chunks as well. Generally speaking, user agents only send chunks to applications that they know will accept chunks. The client could be sending chunks precisely because it's designed to work with a proxy that is known to accept them. That doesn't imply any knowledge of the backend(s) proxied, which might be anything up to and including the 'net in general. Theoretically, yes. However, in practice, that is never the case. On the contrary! I myself have done a great deal of work on a proxy for mobile devices, for a household-name Client. The client software makes certain assumptions of the proxy that would not be valid on the Web at large. But the backend *is* the web at large. Also bear in mind that we were discussing (also) the case where the request came with C-L but an input filter invalidated it. I was not discussing that case. The answer to that case is don't do that. Fix the input filter if it is doing something stupid. That was one of the cases that started this thread. I don't have an example of this, but someone did. -- Nick Kew
Re: POST without Content-Length
A CGI script therefore should never trust Content-Length, but just read stdin until it meets an EOF. That is well-known to fail in CGI. A CGI must use Content-Length. Hmm. any pointers where this is specified? I didn't have any problems with this until now - but in trusting the C-L variable. CGI doesn't require standard input to be closed by the server -- Apache just happens to do that for the sake of old scripts that used fgets to read line-by-line. Other servers do things differently, which is why reading til EOF does not work across platforms. Roy
Re: POST without Content-Length
* Roy T. Fielding [EMAIL PROTECTED] wrote: A CGI script therefore should never trust Content-Length, but just read stdin until it meets an EOF. That is well-known to fail in CGI. A CGI must use Content-Length. Hmm. any pointers where this is specified? I didn't have any problems with this until now - but in trusting the C-L variable. CGI doesn't require standard input to be closed by the server -- Apache just happens to do that for the sake of old scripts that used fgets to read line-by-line. Other servers do things differently, which is why reading til EOF does not work across platforms. Oh well :-( I've glanced over the spec again. ...good that's only an outdated draft :^) sorry for the confusion. nd -- kann mir jemand sagen, was genau @-Domains sind? Ein Mythos. Ein Werbetrick. Verarsche. Nenn es wie du willst... -- Alexandra Buss und Björn Höhrmann in dciwam
Re: POST without Content-Length
On the contrary! I myself have done a great deal of work on a proxy for mobile devices, for a household-name Client. The client software makes certain assumptions of the proxy that would not be valid on the Web at large. But the backend *is* the web at large. But then the client is either using non-standard HTML forms or non-standard HTTP, neither of which is our concern. It doesn't make any sense to code a general proxy that assumes all chunked requests are meant to be length-delimited just because someone might write themselves a custom client that sends everything chunked. Those people can write their own proxies (or at least configure them to be sub-optimal). Roy
Re: POST without Content-Length
On Sat, Aug 07, 2004 at 03:36:33PM -0700, Roy T. Fielding wrote: CGI would happen after mod_deflate. If mod_deflate changes the request body without also (un)setting content-length, then it is broken. Huh? Input filters are pulled, so they run *after* the handler has been started. And - CONTENT_LENGTH (if any - It's unset for chunked as well) still reflects the Content-Length sent by the client. So the current behaviour is correct in all cases. No, it is broken in all cases. CGI scripts cannot handle chunked input and they cannot handle bodies without content-length -- that is how the interface was designed. You would have to define a CGI+ interface to get some other behavior. A CGI script therefore should never trust Content-Length, but just read stdin until it meets an EOF. We cannot redefine CGI. It is a legacy crap interface. Input filters either have to be disabled for CGI or replaced with a buffering system that takes HTTP/1.1 in and supplies CGI with the correct metadata and body. Actually, IMHO, RFC 2616 Section 4.4 clearly defines expected behavior: For compatibility with HTTP/1.0 applications, HTTP/1.1 requests containing a message-body MUST include a valid Content-Length header field unless the server is known to be HTTP/1.1 compliant. If a request contains a message-body and a Content-Length is not given, the server SHOULD respond with 400 (bad request) if it cannot determine the length of the message, or with 411 (length required) if it wishes to insist on receiving a valid Content-Length. Since the Apache server can not know if CGI requires C-L, I conclude that CGI scripts are broken if they require C-L and do not return 411 Length Required when the CGI/1.1 CONTENT_LENGTH environment variable is not present. It's too bad that CGI.pm and cgi-lib.pl are both broken in this respect. Fixing them would be simple and that would take care of the vast majority of legacy apps. For custom apps, supporting T-E chunked and C-L in CGI is trivial, and only requires that the calling app use a get_input() or similar abstraction instead of read()ing directly from stdin. In Apache 1.3, ap_setup_client_block(r, REQUEST_CHUNKED_PASS); just passes along chunks to target applications. Adding support to mod_cgi in Apache2 for passing chunks is also straightforward. Cheers, Glenn
Re: POST without Content-Length
Hi, On Sun, 08 Aug 2004 00:50:13 +0200, Roy T. Fielding wrote: On the contrary! I myself have done a great deal of work on a proxy for mobile devices, for a household-name Client. The client software makes certain assumptions of the proxy that would not be valid on the Web at large. But the backend *is* the web at large. But then the client is either using non-standard HTML forms or non-standard HTTP, neither of which is our concern. This whole thread started due to a commercial GSM mobile phone: User-Agent: SonyEricssonP900/R102 Profile/MIDP-2.0 Configuration/CLDC-1.0 Rev/MR4 , it sends HTTP/1.1 chunked requests to its HTTP proxy although you will access general web sites. The chunked body is apparently created on the fly, each chunk as a specific body element generated by a part of P900 code. Lace -- Jan Kratochvil; Captive: free r/w NTFS Filesystem; http://www.jankratochvil.net/
Re: POST without Content-Length
Since the Apache server can not know if CGI requires C-L, I conclude that CGI scripts are broken if they require C-L and do not return 411 Length Required when the CGI/1.1 CONTENT_LENGTH environment variable is not present. It's too bad that CGI.pm and cgi-lib.pl are both broken in this respect. Fixing them would be simple and that would take care of the vast majority of legacy apps. CGI was defined in 1993. HTTP/1.0 in 1993-95. HTTP/1.1 in 1995-97. I think it is far-fetched to believe that CGI scripts are broken because they don't understand a feature introduced three years after CGI was done. I certainly didn't expect CGI scripts to change when I was editing HTTP. I probably expected that someone would define a successor to CGI that was closer in alignment to HTTP, but that never happened (instead, servlets were defined as a copy of the already-obsolete CGI interface rather than something sensible like an HTTP proxy interface). *shrug* CGI is supposed to be a simple interface for web programming. It is not supposed to be a fast interface, a robust interface, or a long-term interface -- just a simple one that works on multiple independent web server implementations. Roy
Re: POST without Content-Length
On Saturday, August 7, 2004, at 05:21 PM, Jan Kratochvil wrote: This whole thread started due to a commercial GSM mobile phone: User-Agent: SonyEricssonP900/R102 Profile/MIDP-2.0 Configuration/CLDC-1.0 Rev/MR4 , it sends HTTP/1.1 chunked requests to its HTTP proxy although you will access general web sites. The chunked body is apparently created on the fly, each chunk as a specific body element generated by a part of P900 code. So stick a proxy in front of it that waits for the entire body on every request and converts it to a content-length. I am not saying that it isn't possible -- it is just stupid for a general-purpose proxy to do that (just as it is stupid to deploy a cell phone with such a lazy HTTP implementation). Roy
Re: POST without Content-Length
On Sat, Aug 07, 2004 at 05:24:19PM -0700, Roy T. Fielding wrote: Since the Apache server can not know if CGI requires C-L, I conclude that CGI scripts are broken if they require C-L and do not return 411 Length Required when the CGI/1.1 CONTENT_LENGTH environment variable is not present. It's too bad that CGI.pm and cgi-lib.pl are both broken in this respect. Fixing them would be simple and that would take care of the vast majority of legacy apps. Here's my argument, notwithstanding current practice and the extremely poor quality of many, many CGI programs out there: [snip] CGI is supposed to be a simple interface for web programming. You and your cohorts did a great job with it. It's been quite extensible and robust (which is not to say it is perfect :)). [snip] CGI was defined in 1993. HTTP/1.0 in 1993-95. HTTP/1.1 in 1995-97. OK, so any robust CGI written after 1995 that is intended to be gatewayed from an HTTP web server should grok all standard HTTP/1.0 request methods and should know which ones it supports, and which ones _require_ a message body (e.g. POST, PUT). And, naturally, the CGI should know it is speaking to an HTTP/1.x server on the other end of the gateway because of the CGI/1.1 SERVER_PROTOCOL environment variable. Now, I always read the CGI/1.1 spec as _requiring_ a CONTENT_LENGTH if there was any message body. (-- Note the qualification). http://hoohoo.ncsa.uiuc.edu/cgi/env.html The following environment variables are specific to the request being fulfilled by the gateway program: [...] # CONTENT_TYPE For queries which have attached information, such as HTTP POST and PUT, this is the content type of the data. # CONTENT_LENGTH The length of the said content as given by the client. It is implied that CONTENT_LENGTH should be set to 0 by the HTTP side of the gateway when a request method that requires a message body has an empty body. Therefore, on the other side of the gateway, any robust little CGI should check for CONTENT_LENGTH, because it wants to be able to detect premature end of content. (It doesn't even have to know that the HTTP server required a message body for that method) If a script expects content on stdin and does not find CONTENT_LENGTH, then even very old CGI scripts should abort with 400 Bad Request. Better, the script should return 411 Length Required if it requires CONTENT_LENGTH and one is not present. Of course, if the script does not require that content be present, and CONTENT_LENGTH is not present, scripts that do not support HTTP/1.1 semantics (i.e. Transfer-Encoding) will assume no content and will silently ignore the content that was submitted. However, since the content is not required, one must assume that no corruption will occur because of its absense; only some missing information results (RFC 2616 says that C-L should be ignored if T-E is present, and so an HTTP/1.1 server might pass CONTENT_LENGTH=0 to such scripts along with HTTP_TRANSFER_ENCODING=chunked, and still be proper. Scripts that require content, should then definitely return 400 Bad Request) In any case, CGI scripts are welcome to support HTTP/1.1 and T-E if they see appropriate SERVER_PROTOCOL and HTTP_TRANSFER_ENCODING environment variables. Cheers, Glenn
