Stefan Hartmann wrote:
Amos Jeffries wrote:
Hello,
i am running squid as reverse proxy in front of a web server farm. We
are trying to implement Content-Compression, and it gets broken from
"time to time".
The www-servers are windows IIS 5, and the compression is done using a
ISAPI Filter (no, not the original broken M$ filter from the server).
We are using Version 2.7.STABLE6 in our setup. The www-servers are all
sending a "Vary: Accept-Enconding" header, and the setup is working
perfectly in my test scenarios. We have no "broken_vary_encoding"
configured, and no ETag in the responses (we are only using Expire:
Headers).
We installed the ISAPI Filters last week without putting the "Vary:
Accept-Encoding" header on the www-servers in place, and blocked the
"Accept-Encoding:" and the "Vary:" headers at squid, waiting for
maintainance window to activate it. The site worked without any problem.
During the last maintainance window, we activated the "Accept-Encoding:"
and "Vary:" Headers (no longer blocking it in squid), and set up the
WWW-Servers to send "Vary: Accept-Encoding" headers, and it works -
sometimes with some browsers.
The failure we see are content-pages which are ending after some kB of
correct data. ie the homepage is about 150 kB uncompressed, compressed
around 30 kB (this is why we want compression), and the Serverfarm
delivered Content-Pages consisting of the first 18 to 25 kB
(uncompressed, different sizes possible) of the complete page, never
coming to an end. This never happened in our test setup.
The pages were (as intended) cached by squid, so we had the situation
that for example Internet Explorer was working, but Firefox got the
short page. And vice versa, sometimes Firefox worked, but IE failed. And
sometimes all browsers worked.
From tonight logs:
11:30 pm to 01:00 am: IE pages broken
01:00 am to 09:30 am: all working
09:30 am to 10:15 am: Firefox pages broken
10:15 am to 11:00 am: IE and Firefox pages broken
Ok, perhaps the ISAPI filter is faulty in some conditions we did not
test, with some browsers or bots or... so we uninstalled the ISAPI
filter from all WWW-Servers, but left the "Vary: Accept-Encoding" header
in place.
Result: The error did not stop! I had to block the "Accept-Encoding:"
and "Vary:" in squid to get the site working properly.
Next step was to remove the Vary: Header from the WWW-Servers and not
blocking the "Accept-Encoding:" and "Vary:" headers in squid: the site
is working properly.
So... are there any issues regarding squid and WWW-Servers sending
"Vary: Accept-Encoding" (without actually doing Content-Compression)?
When the error occurs, our logs are showing connections with "short"
pages (ie 18 kB vs. 150 kB normaly), which are obviously aborted after
900 seconds:
Mon Apr 27 15:38:29 2009 900301 111.111.111.111 TCP_MISS/200 18806 GET
http://real.server.de/ - DEFAULT_PARENT/real.server.de text/html
[
Accept-Encoding: gzip, deflate
User-Agent: Nutscrape/1.0 (CP/M; 8-bit)
Host: real.server.de
Cookie: WT_SET=id=213.253.......
Cache-Control: max-age=259200
]
[
HTTP/1.0 200 OK
Date: Mon, 27 Apr 2009 13:23:29 GMT
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
Realserver-info: BuildTime: 27.04.2009 15:23:29; TimeSpan:
00:00:02.6719434; CacheTime: 120; Server: WWW31
Publisher: Real-Server
Expires: Mon, 27 Apr 2009 13:25:29 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 168830
X-Cache: HIT from accel3
Connection: close
]
Please help!
Since the browser seems to be eratic, I assume that one particular client
request is causing some bad data to enter squid cache and being served for
all following clients for a period.
Look to the requests at the beginning of the time when things break. If
you can find the exact conditions or client it will be much easier to
track through the logs on later occurances.
It sounds a little bit like:
http://squidproxy.wordpress.com/2008/04/29/chunked-decoding/
except for a few factors that don't fit:
IIS 5 is not known for this issue,
2.7 has a decoding hack to fix it
and Vary: seemed to show relevance.
I'd try raising the debug_options levels for request processing a bit and
see what becomes visible.
Amos,
thanks for the reply. debugging is somewhat tricky, since the serverfarm
has to handle lots of traffic (around 200 Mio content pages per month)
and debugging the real servers would generate a (too) huge amount of
data. And in my test scenario i don`t get the error...
I will try to filter the "bad" requests. The idea is to stop the
Accept-Encoding headers if the are "crazy", ie (all seen live)
Accept-Encoding: FFFF, FFFFFFF
Accept-Encoding: mzip, meflate
Accept-Encoding: identity, deflate, gzip
Accept-Encoding: gzip;q=1.0, deflate;q=0.8, chunked;q=0.6,
identity;q=0.4, *;q=0
Accept-Encoding: gzip, deflate, x-gzip, identity; q=0.9
Accept-Encoding: gzip,deflate,bzip2
Accept-Encoding: nnnnndeflate
Accept-Encoding: x-gzip, gzip
Accept-Encoding: gzip,identity
Accept-Encoding: gzip, deflate, compress;q=0.9
Accept-Encoding: gzip,deflate,X.509
lol. Thanks.
and only let pass these two:
Accept-Encoding: gzip,deflate
Accept-Encoding: gzip, deflate
first one is Firefox, the other is IE. This will match in about 80-90%
of all requests, which would be ok.
so i tried
acl zipit req_header Accept-Encoding ^gzip,deflate$
acl zipit req_header Accept-Encoding ^gzip, deflate$
[...]
header_access Accept-Encoding allow zipit
but something seems to be wrong with the regex above, squid will let
pass not only "gzip,deflate" as i would expect but also
"gzip,deflate,xx" and "gzip,xx". "bla" will be blocked.
seems like squid will let pass the header if it starts with gzip,
disregarding the rest. am i wrong with my regex?
Squid splits on whitespace. The space in your second pattern makes that
into three patterns.
I'd use:
acl zipit req_header Accept-Encoding ^gzip,.deflate$
Amos
--
Please be using
Current Stable Squid 2.7.STABLE6 or 3.0.STABLE14
Current Beta Squid 3.1.0.7
