Re: [PATCH] Send chunked responses if body size is unknown.

2010-08-23 Thread Alex Rousskov

On 08/22/2010 04:49 AM, Amos Jeffries wrote:

Alex Rousskov wrote:

Send chunked responses if body size is unknown.

Apply HTTP chunked transfer encoding to the response body if all of
the following conditions are met:

* client claims HTTP version 1.1 or later support
* response does not have a Content-Length header already
* response does not use multipart/byteranges encoding
* connection is persistent

If we decide to send chunked reply, chunked_reply flag is set. Chunked
encoding is done in ClientSocketContext::packChunk(). The last-chunk
is sent only when clientReplyContext complete flag is set.



This feature was requested to make Squid work with HTTP/1.1 clients
that can handle chunked responses but cannot handle connection
closures in the middle of a transaction sequence. The earlier version
of the patch (for Squid v3.1) was tested in production.

N.B. A bug in Squid may result in server-side code not treating
premature server-side connection termination as an error. That bug
results in Squid client-side sending a complete chunked response to
the client instead of omitting the last-chunk to indicate a truncated
response. Fixing that bug is outside this project scope (but we might
have a patch for it somewhere, I need to check).



+1.

Though it's worth noting that the logic as given also excludes chunking
in HTTP 2.0, 3.0, etc


Indeed, I missed that bug. I have another patch somewhere that adds 
proper comparison operators to the HttpVersion class. Will use that 
instead of the hand-made comparison.


Thank you,

Alex.


Re: [PATCH] Send chunked responses if body size is unknown.

2010-08-22 Thread Amos Jeffries

Alex Rousskov wrote:

Send chunked responses if body size is unknown.

Apply HTTP chunked transfer encoding to the response body if all of the 
following conditions are met:


* client claims HTTP version 1.1 or later support
* response does not have a Content-Length header already
* response does not use multipart/byteranges encoding
* connection is persistent

If we decide to send chunked reply, chunked_reply flag is set. Chunked 
encoding is done in ClientSocketContext::packChunk(). The last-chunk is 
sent only when clientReplyContext complete flag is set.




This feature was requested to make Squid work with HTTP/1.1 clients that 
can handle chunked responses but cannot handle connection closures in 
the middle of a transaction sequence. The earlier version of the patch 
(for Squid v3.1) was tested in production.


N.B. A bug in Squid may result in server-side code not treating 
premature server-side connection termination as an error. That bug 
results in Squid client-side sending a complete chunked response to the 
client instead of omitting the last-chunk to indicate a truncated 
response. Fixing that bug is outside this project scope (but we might 
have a patch for it somewhere, I need to check).




+1.

Though it's worth noting that the logic as given also excludes chunking 
in HTTP 2.0, 3.0, etc


Amos
--
Please be using
  Current Stable Squid 2.7.STABLE9 or 3.1.6
  Beta testers wanted for 3.2.0.1


[PATCH] Send chunked responses if body size is unknown.

2010-08-20 Thread Alex Rousskov

Send chunked responses if body size is unknown.

Apply HTTP chunked transfer encoding to the response body if all of the 
following conditions are met:


* client claims HTTP version 1.1 or later support
* response does not have a Content-Length header already
* response does not use multipart/byteranges encoding
* connection is persistent

If we decide to send chunked reply, chunked_reply flag is set. Chunked 
encoding is done in ClientSocketContext::packChunk(). The last-chunk is 
sent only when clientReplyContext complete flag is set.




This feature was requested to make Squid work with HTTP/1.1 clients that 
can handle chunked responses but cannot handle connection closures in 
the middle of a transaction sequence. The earlier version of the patch 
(for Squid v3.1) was tested in production.


N.B. A bug in Squid may result in server-side code not treating 
premature server-side connection termination as an error. That bug 
results in Squid client-side sending a complete chunked response to the 
client instead of omitting the last-chunk to indicate a truncated 
response. Fixing that bug is outside this project scope (but we might 
have a patch for it somewhere, I need to check).




[PATCH] Send chunked responses if body size is unknown.

2010-08-20 Thread Alex Rousskov

Send chunked responses if body size is unknown.

Apply HTTP chunked transfer encoding to the response body if all of the 
following conditions are met:


* client claims HTTP version 1.1 or later support
* response does not have a Content-Length header already
* response does not use multipart/byteranges encoding
* connection is persistent

If we decide to send chunked reply, chunked_reply flag is set. Chunked 
encoding is done in ClientSocketContext::packChunk(). The last-chunk is 
sent only when clientReplyContext complete flag is set.




This feature was requested to make Squid work with HTTP/1.1 clients that 
can handle chunked responses but cannot handle connection closures in 
the middle of a transaction sequence. The earlier version of the patch 
(for Squid v3.1) was tested in production.


N.B. A bug in Squid may result in server-side code not treating 
premature server-side connection termination as an error. That bug 
results in Squid client-side sending a complete chunked response to the 
client instead of omitting the last-chunk to indicate a truncated 
response. Fixing that bug is outside this project scope (but we might 
have a patch for it somewhere, I need to check).


Send chunked responses if body size is unknown.

Apply HTTP chunked transfer encoding to the response body if all of
the following conditions are met:

* client claims HTTP version 1.1 or later support
* response does not have a Content-Length header already
* response does not use multipart/byteranges encoding
* connection is persistent

If we decide to send chunked reply, chunked_reply flag is set. Chunked
encoding is done in ClientSocketContext::packChunk(). The last-chunk
is sent only when clientReplyContext complete flag is set.

=== modified file 'compat/types.h'
--- compat/types.h	2010-07-09 13:23:03 +
+++ compat/types.h	2010-08-19 18:25:20 +
@@ -99,37 +99,47 @@
 #ifndef PRId64
 #ifdef _SQUID_MSWIN_		/* Windows native port using MSVCRT */
 #define PRId64 I64d
 #elif SIZEOF_INT64_T  SIZEOF_LONG
 #define PRId64 lld
 #else
 #define PRId64 ld
 #endif
 #endif
 
 #ifndef PRIu64
 #ifdef _SQUID_MSWIN_		/* Windows native port using MSVCRT */
 #define PRIu64 I64u
 #elif SIZEOF_INT64_T  SIZEOF_LONG
 #define PRIu64 llu
 #else
 #define PRIu64 lu
 #endif
 #endif
 
+#ifndef PRIX64
+#ifdef _SQUID_MSWIN_		/* Windows native port using MSVCRT */
+#define PRIX64 I64X
+#elif SIZEOF_INT64_T  SIZEOF_LONG
+#define PRIX64 llX
+#else
+#define PRIX64 lX
+#endif
+#endif
+
 #ifndef HAVE_MODE_T
 typedef unsigned short mode_t;
 #endif
 
 #ifndef HAVE_FD_MASK
 typedef unsigned long fd_mask;
 #endif
 
 #ifndef HAVE_SOCKLEN_T
 typedef int socklen_t;
 #endif
 
 #ifndef HAVE_MTYP_T
 typedef long mtyp_t;
 #endif
 
 #endif /* SQUID_TYPES_H */

=== modified file 'src/client_side.cc'
--- src/client_side.cc	2010-08-18 23:43:22 +
+++ src/client_side.cc	2010-08-20 03:50:08 +
@@ -875,62 +875,81 @@ ClientSocketContext::noteSentBodyBytes(s
 assert (http-range_iter.debt() = -1);
 }
 
 bool
 ClientHttpRequest::multipartRangeRequest() const
 {
 return request-multipartRangeRequest();
 }
 
 bool
 ClientSocketContext::multipartRangeRequest() const
 {
 return http-multipartRangeRequest();
 }
 
 void
 ClientSocketContext::sendBody(HttpReply * rep, StoreIOBuffer bodyData)
 {
 assert(rep == NULL);
 
-if (!multipartRangeRequest()) {
+if (!multipartRangeRequest()  !http-request-flags.chunked_reply) {
 size_t length = lengthToSend(bodyData.range());
 noteSentBodyBytes (length);
 AsyncCall::Pointer call = commCbCall(33, 5, clientWriteBodyComplete,
  CommIoCbPtrFun(clientWriteBodyComplete, this));
 comm_write(fd(), bodyData.data, length, call );
 return;
 }
 
 MemBuf mb;
 mb.init();
-packRange(bodyData, mb);
+if (multipartRangeRequest())
+packRange(bodyData, mb);
+else
+packChunk(bodyData, mb);
 
 if (mb.contentSize()) {
 /* write */
 AsyncCall::Pointer call = commCbCall(33, 5, clientWriteComplete,
  CommIoCbPtrFun(clientWriteComplete, this));
 comm_write_mbuf(fd(), mb, call);
 }  else
 writeComplete(fd(), NULL, 0, COMM_OK);
 }
 
+/**
+ * Packs bodyData into mb using chunked encoding. Packs the last-chunk
+ * if bodyData is empty.
+ */
+void
+ClientSocketContext::packChunk(const StoreIOBuffer bodyData, MemBuf mb)
+{
+const uint64_t length =
+static_castuint64_t(lengthToSend(bodyData.range()));
+noteSentBodyBytes(length);
+
+mb.Printf(%PRIX64\r\n, length);
+mb.append(bodyData.data, length);
+mb.Printf(\r\n);
+}
+
 /** put terminating boundary for multiparts */
 static void
 clientPackTermBound(String boundary, MemBuf * mb)
 {
 mb-Printf(\r\n-- SQUIDSTRINGPH --\r\n, SQUIDSTRINGPRINT(boundary));
 debugs(33, 6, clientPackTermBound: buf offset:   mb-size);
 }
 
 /** appends a part HTTP header