Re: [squid-users] headers say HIT, logs say MISS, payload is truncated...
On mån, 2008-10-27 at 15:56 -0700, Neil Harkins wrote: > I'd like to help and see this get fixed, but as I said earlier, > it happens on about 16% of our test requests, only when > there's 750~1050 reqs/second going through the box, > and pretty much disappears under 500 reqs/s (off-peak). Ouch.. > Is this except significant?: Hard to say, but probably not. It's just reading of the Vary/ETag index, finding that the request matches the object with key 968D51EAA0C2BCF5688EAB92E8F56EE4. Do your server support ETag on these objects? And do it properly report different ETag values for the different variants? Or are you using broken_vary_encoding to work around server brokenness? > Note that I've since changed our load balancer to rewrite "Accept-Encoding: " > to "Accept-Encoding: identity" in case squid didn't like a null header, > (although the example in the RFC implies that "Accept-Encoding: " is valid: > http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.3) > but the timeouts still happened, although I didn't grab debugging then. Accept-Encoding header without a value is not really a valid HTTP header (what you probably want is no Accept-Encoding header at all). But Squid should work fine in both cases as it's just two different Vary request fingerprints. The blank example is an error in the specifications descriptive text and has been corrected in the errata. If you look closely at the above reference you'll notice the BNF says 1#(...) which means at least one. The BNF is the authorative definition of the syntax of this header, the rest of the text just tries to explain it.. Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] headers say HIT, logs say MISS, payload is truncated...
On Mon, Oct 27, 2008 at 2:56 PM, Henrik Nordstrom <[EMAIL PROTECTED]> wrote: > On mån, 2008-10-27 at 12:23 -0700, Neil Harkins wrote: >> The timeout is because the Content-Length header is bigger than the >> payload it sent. >> Every http client/server will hang in that situation. This isn't >> simply a misreported >> HIT<->MISS in the log, this is absolutely a significant bug where >> collapsed forwarding is >> mixing up the metadata from the two branches of our Vary: >> Accept-Encoding (gzip and not), >> i.e. giving the headers and content as non-gzip, but the amount of >> payload it reads from >> the cache and sends is based on the gzip size. Disabling >> collapsed_forwarding fixed it. > > Please file a bug report on this. Preferably including "squid -k debug" > cache.log output and "tcpdump -s 0 -w traffic.pcap" traces. I'd like to help and see this get fixed, but as I said earlier, it happens on about 16% of our test requests, only when there's 750~1050 reqs/second going through the box, and pretty much disappears under 500 reqs/s (off-peak). I managed to catch only 1 instance of the timeout in a 17 Gig cache.log with debugging enabled. Is this except significant?: 2008/10/24 17:09:23| storeLocateVaryRead: accept-encoding="gzip" 0x3b28978 seen_offset=204 buf_offset=0 size=85 2008/10/24 17:09:23| storeLocateVaryRead: Key: 968D51EAA0C2BCF5688EAB92E8F56EE4 2008/10/24 17:09:23| storeLocateVaryRead: 0x3b28978 seen_offset=289 buf_offset=0 2008/10/24 17:09:23| storeClientCopy: 4F3F9F8F4461796C3C469E610B6E2D5C, seen 289, want 289, size 4096, cb 0x46169d, cbdata 0x3b28978 [snip] 2008/10/24 17:09:23| storeLocateVaryRead: accept-encoding="" 0x3b28978 seen_offset=204 buf_offset=0 size=178 2008/10/24 17:09:23| storeLocateVaryRead: Key: 968D51EAA0C2BCF5688EAB92E8F56EE4 2008/10/24 17:09:23| storeLocateVaryRead: MATCH! 968D51EAA0C2BCF5688EAB92E8F56EE4 (null) 2008/10/24 17:09:23| storeClientCopy: 4F3F9F8F4461796C3C469E610B6E2D5C, seen 382, want 382, size 4096, cb 0x46169d, cbdata 0x3b28978 Note that I've since changed our load balancer to rewrite "Accept-Encoding: " to "Accept-Encoding: identity" in case squid didn't like a null header, (although the example in the RFC implies that "Accept-Encoding: " is valid: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.3) but the timeouts still happened, although I didn't grab debugging then. -neil
Re: [squid-users] headers say HIT, logs say MISS, payload is truncated...
On mån, 2008-10-27 at 12:23 -0700, Neil Harkins wrote: > The timeout is because the Content-Length header is bigger than the > payload it sent. > Every http client/server will hang in that situation. This isn't > simply a misreported > HIT<->MISS in the log, this is absolutely a significant bug where > collapsed forwarding is > mixing up the metadata from the two branches of our Vary: > Accept-Encoding (gzip and not), > i.e. giving the headers and content as non-gzip, but the amount of > payload it reads from > the cache and sends is based on the gzip size. Disabling > collapsed_forwarding fixed it. Please file a bug report on this. Preferably including "squid -k debug" cache.log output and "tcpdump -s 0 -w traffic.pcap" traces. http://bugs.squid-cache.org/ Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] headers say HIT, logs say MISS, payload is truncated...
On Sat, Oct 25, 2008 at 8:54 AM, Henrik Nordstrom <[EMAIL PROTECTED]> wrote: > On fre, 2008-10-24 at 15:44 -0700, Neil Harkins wrote: >> We are using collapsed_forwarding here. I haven't tried disabling it yet. >> >> Unfortunately, since the problem appears to be load-related, I've been >> unable to reproduce for a tcpdump or running squid in debug thus far. > > The mismatch in HIT/MISS is most likely related to collapsed forwarding. > Collapsed requests gets somewhere inbetween an hit or miss, and may well > be reported a little inconsistent. > > Have no idea on the timeout issue unless there is a communication issue > between Squid and your web server. The timeout is because the Content-Length header is bigger than the payload it sent. Every http client/server will hang in that situation. This isn't simply a misreported HIT<->MISS in the log, this is absolutely a significant bug where collapsed forwarding is mixing up the metadata from the two branches of our Vary: Accept-Encoding (gzip and not), i.e. giving the headers and content as non-gzip, but the amount of payload it reads from the cache and sends is based on the gzip size. Disabling collapsed_forwarding fixed it. -neil
Re: [squid-users] headers say HIT, logs say MISS, payload is truncated...
On fre, 2008-10-24 at 15:44 -0700, Neil Harkins wrote: > We are using collapsed_forwarding here. I haven't tried disabling it yet. > > Unfortunately, since the problem appears to be load-related, I've been > unable to reproduce for a tcpdump or running squid in debug thus far. The mismatch in HIT/MISS is most likely related to collapsed forwarding. Collapsed requests gets somewhere inbetween an hit or miss, and may well be reported a little inconsistent. Have no idea on the timeout issue unless there is a communication issue between Squid and your web server. Regards Henrik signature.asc Description: This is a digitally signed message part
