Re: Long response header problem
Ok, Did some additional checks, It should be --- src/http.cc 2008-04-01 13:54:38.0 +0200 +++ src/http.cc 2008-04-21 16:42:19.0 +0200 @@ -75,7 +75,7 @@ surrogateNoStore = false; fd = fwd-server_fd; readBuf = new MemBuf; -readBuf-init(4096, SQUID_TCP_SO_RCVBUF); +readBuf-init( SQUID_TCP_SO_RCVBUF, SQUID_TCP_SO_RCVBUF); orig_request = HTTPMSGLOCK(fwd-request); if (fwd-servers) Which is btw. the coding used in ICAP where SQUID_TCP_SO_RCVBUF = 16384 Regards, Axel Hi there, I ran, or better a customer ran into a problem today which sounded like this bug. http://www.squid-cache.org/bugs/show_bug.cgi?id=2001 So I applied the attached patch to squid-3.0.STABLE4 and did a quick test. Still the same problem. By cache.log looks like this ... comm_read_try: FD 16, size 4094, retval 2896, errno 0 ... HttpMsg::parse: failed to find end of headers (eof: 0) ... http.cc(1050) needs more at 2896 http.cc(1206) may read up to 1199 bytes from FD 16 ... comm_select(): got FD 16 events=1 monitoring=19 F-read_handler=1 F-write_handler=0 comm_select(): Calling read handler on FD 16 comm_read_try: FD 16, size 1198, retval 1198, errno 0 ... HttpMsg::parse: failed to find end of headers (eof: 0) ... http.cc(1050) needs more at 4094 http.cc(1206) may read up to 1 bytes from FD 16 ... comm_select(): got FD 16 events=1 monitoring=19 F-read_handler=0 F-write_handler=0 comm_select(): no read handler for FD 16 and so on and so on. So I checked the coding in http.cc and changed it as follows. --- src/http.cc 2008-04-01 13:54:38.0 +0200 +++ src/http.cc 2008-04-21 16:42:19.0 +0200 @@ -75,7 +75,7 @@ surrogateNoStore = false; fd = fwd-server_fd; readBuf = new MemBuf; -readBuf-init(4096, SQUID_TCP_SO_RCVBUF); +readBuf-init(16384, SQUID_TCP_SO_RCVBUF); orig_request = HTTPMSGLOCK(fwd-request); if (fwd-servers) Now it works but I am not sure if a.) this is a good solution and b.) a stable one :-). Maybe someone with more knowledge can do a check. Regards, Axel Westerhold DTS Systeme GmbH
Long response header problem
Hi there, I ran, or better a customer ran into a problem today which sounded like this bug. http://www.squid-cache.org/bugs/show_bug.cgi?id=2001 So I applied the attached patch to squid-3.0.STABLE4 and did a quick test. Still the same problem. By cache.log looks like this ... comm_read_try: FD 16, size 4094, retval 2896, errno 0 ... HttpMsg::parse: failed to find end of headers (eof: 0) ... http.cc(1050) needs more at 2896 http.cc(1206) may read up to 1199 bytes from FD 16 ... comm_select(): got FD 16 events=1 monitoring=19 F-read_handler=1 F-write_handler=0 comm_select(): Calling read handler on FD 16 comm_read_try: FD 16, size 1198, retval 1198, errno 0 ... HttpMsg::parse: failed to find end of headers (eof: 0) ... http.cc(1050) needs more at 4094 http.cc(1206) may read up to 1 bytes from FD 16 ... comm_select(): got FD 16 events=1 monitoring=19 F-read_handler=0 F-write_handler=0 comm_select(): no read handler for FD 16 and so on and so on. So I checked the coding in http.cc and changed it as follows. --- src/http.cc 2008-04-01 13:54:38.0 +0200 +++ src/http.cc 2008-04-21 16:42:19.0 +0200 @@ -75,7 +75,7 @@ surrogateNoStore = false; fd = fwd-server_fd; readBuf = new MemBuf; -readBuf-init(4096, SQUID_TCP_SO_RCVBUF); +readBuf-init(16384, SQUID_TCP_SO_RCVBUF); orig_request = HTTPMSGLOCK(fwd-request); if (fwd-servers) Now it works but I am not sure if a.) this is a good solution and b.) a stable one :-). Maybe someone with more knowledge can do a check. Regards, Axel Westerhold DTS Systeme GmbH
Re: Long response header problem
And one more. It might be this patch which solved the issue
--- src/http.cc 2008-04-01 13:54:38.0 +0200
+++ src/http.cc 2008-04-21 19:11:47.0 +0200
@@ -1200,7 +1200,7 @@
void
HttpStateData::maybeReadVirginBody()
{
-int read_sz = replyBodySpace(readBuf-spaceSize());
+int read_sz = replyBodySpace(readBuf-potentialSpaceSize());
debugs(11,9, HERE (flags.do_next_read ? may : wont)
read up to read_sz bytes from FD fd);
spaceSize will only return the size left from initial size. This will result
in read_sz2 and return some lines down in http.cc.
PotentialSpaceSize will return max_capacity - terminatedSize which seems
more logical.
Regards,
Axel Westerhold
DTS Systeme GmbH
Ok,
Did some additional checks,
It should be
--- src/http.cc 2008-04-01 13:54:38.0 +0200
+++ src/http.cc 2008-04-21 16:42:19.0 +0200
@@ -75,7 +75,7 @@
surrogateNoStore = false;
fd = fwd-server_fd;
readBuf = new MemBuf;
-readBuf-init(4096, SQUID_TCP_SO_RCVBUF);
+readBuf-init( SQUID_TCP_SO_RCVBUF, SQUID_TCP_SO_RCVBUF);
orig_request = HTTPMSGLOCK(fwd-request);
if (fwd-servers)
Which is btw. the coding used in ICAP where SQUID_TCP_SO_RCVBUF = 16384
Regards,
Axel
Hi there,
I ran, or better a customer ran into a problem today which sounded like this
bug.
http://www.squid-cache.org/bugs/show_bug.cgi?id=2001
So I applied the attached patch to squid-3.0.STABLE4 and did a quick test.
Still the same problem. By cache.log looks like this
...
comm_read_try: FD 16, size 4094, retval 2896, errno 0
...
HttpMsg::parse: failed to find end of headers (eof: 0)
...
http.cc(1050) needs more at 2896
http.cc(1206) may read up to 1199 bytes from FD 16
...
comm_select(): got FD 16 events=1 monitoring=19 F-read_handler=1
F-write_handler=0
comm_select(): Calling read handler on FD 16
comm_read_try: FD 16, size 1198, retval 1198, errno 0
...
HttpMsg::parse: failed to find end of headers (eof: 0)
...
http.cc(1050) needs more at 4094
http.cc(1206) may read up to 1 bytes from FD 16
...
comm_select(): got FD 16 events=1 monitoring=19 F-read_handler=0
F-write_handler=0
comm_select(): no read handler for FD 16
and so on and so on. So I checked the coding in http.cc and changed it as
follows.
--- src/http.cc 2008-04-01 13:54:38.0 +0200
+++ src/http.cc 2008-04-21 16:42:19.0 +0200
@@ -75,7 +75,7 @@
surrogateNoStore = false;
fd = fwd-server_fd;
readBuf = new MemBuf;
-readBuf-init(4096, SQUID_TCP_SO_RCVBUF);
+readBuf-init(16384, SQUID_TCP_SO_RCVBUF);
orig_request = HTTPMSGLOCK(fwd-request);
if (fwd-servers)
Now it works but I am not sure if a.) this is a good solution and b.) a
stable
one :-).
Maybe someone with more knowledge can do a check.
Regards,
Axel Westerhold
DTS Systeme GmbH
Re: Long response header problem
Hmm.. can't seem to reproduce this.
The proposed change do not fix the problem, just hides it a bit.
See my last mail of three (:-) sorry Not my best day)
The 3.0.STABLE4 code already bumps the read size to 1KB minimum when
headers haven't been successfully parsed yet. See
HttpStateData::maybeReadVirginBody()
Do you have an example URL triggering the problem?
Yes and no. The Url is including a authentication dialog I can't give you
the username and password for. I'll check if I can come up with something
similar.
Are you using ICAP?
ICAP is off for this test.
Any other interesting details about your configuration?
Nothing special. Actually the bug showed up on STABLE1 and I tested with a
STABLE4 without modifications (failed) patched with the longresp patch
(failed).
Regards
Henrik
As said, see my third mail:
SNIP
--- src/http.cc 2008-04-01 13:54:38.0 +0200
+++ src/http.cc 2008-04-21 19:11:47.0 +0200
@@ -1200,7 +1200,7 @@
void
HttpStateData::maybeReadVirginBody()
{
-int read_sz = replyBodySpace(readBuf-spaceSize());
+int read_sz = replyBodySpace(readBuf-potentialSpaceSize());
debugs(11,9, HERE (flags.do_next_read ? may : wont)
read up to read_sz bytes from FD fd);
spaceSize will only return the size left from initial size. This will result
in read_sz2 and return some lines down in http.cc.
PotentialSpaceSize will return max_capacity - terminatedSize which seems
more logical.
---SNIP
Regards,
Axel
Re: Long response header problem
mån 2008-04-21 klockan 21:35 +0200 skrev Axel Westerhold:
--- src/http.cc 2008-04-01 13:54:38.0 +0200
+++ src/http.cc 2008-04-21 19:11:47.0 +0200
@@ -1200,7 +1200,7 @@
void
HttpStateData::maybeReadVirginBody()
{
-int read_sz = replyBodySpace(readBuf-spaceSize());
+int read_sz = replyBodySpace(readBuf-potentialSpaceSize());
debugs(11,9, HERE (flags.do_next_read ? may : wont)
read up to read_sz bytes from FD fd);
Ok, that's a quite different change. But still not right. See below.
spaceSize will only return the size left from initial size. This will result
in read_sz2 and return some lines down in http.cc.
PotentialSpaceSize will return max_capacity - terminatedSize which seems
more logical.
No it's not. We do not want this buffer to grow unless absoultely
needed. The upper limit on buffer size is just a safe guard to make sure
something realize when things run completely out of bound.
Regarding how it handles long headers, look a few lines down... it only
returns there if the header has been parsed. If the header has not yet
been parsed it allows the buffer to grow by reading at least 1024 octets
more..
if (read_sz 2) {
if (flags.headers_parsed)
return;
else
read_sz = 1024;
}
But there is one cosmetic problem here in that we log the expected read
size before adjustment, with the adjustment being silent in debug logs..
Regards
Henrik
Re: Long response header problem
Axel Westerhold wrote:
mån 2008-04-21 klockan 21:35 +0200 skrev Axel Westerhold:
--- src/http.cc 2008-04-01 13:54:38.0 +0200
+++ src/http.cc 2008-04-21 19:11:47.0 +0200
@@ -1200,7 +1200,7 @@
void
HttpStateData::maybeReadVirginBody()
{
-int read_sz = replyBodySpace(readBuf-spaceSize());
+int read_sz = replyBodySpace(readBuf-potentialSpaceSize());
debugs(11,9, HERE (flags.do_next_read ? may : wont)
read up to read_sz bytes from FD fd);
Ok, that's a quite different change. But still not right. See below.
spaceSize will only return the size left from initial size. This will result
in read_sz2 and return some lines down in http.cc.
PotentialSpaceSize will return max_capacity - terminatedSize which seems
more logical.
No it's not. We do not want this buffer to grow unless absoultely
needed. The upper limit on buffer size is just a safe guard to make sure
something realize when things run completely out of bound.
Regarding how it handles long headers, look a few lines down... it only
returns there if the header has been parsed. If the header has not yet
been parsed it allows the buffer to grow by reading at least 1024 octets
more..
if (read_sz 2) {
if (flags.headers_parsed)
return;
else
read_sz = 1024;
}
But there is one cosmetic problem here in that we log the expected read
size before adjustment, with the adjustment being silent in debug logs..
Regards
Henrik
Uhmmm,
See my mybeReadVirginBody() from Stable4
Any chance that you re using CVS ?
Do you mean you want CVS access?
We use Bazaar for Squid-3. Details:
http://wiki.squid-cache.org/Squid3VCS
Or rsync has the latest patched source:
rsync -avz rsync://squid-cache.org/source/squid-3.0
Amos
--
Please use Squid 2.6.STABLE19 or 3.0.STABLE4
