Re: [squid-users] Another HTTP 1.1 Question
tis 2007-03-13 klockan 11:57 +0800 skrev Adrian Chadd: Are ECN/Window Scaling options which can be fiddled via syscalls per socket FD? Probably not for ECN, but it looks like the window scaling can be influenced by the TCP_WINDOW_CLAMP setsockopt. Adding a TCP setsockopt for turning off ECN shouldn't be hard however.. If they are then we could sneak them into the retry forwarding logic and cache if/when they work. The window scaling problem is unfortunately not so easy to detect in Squid as it causes connections to hang after the request has been sent and acknowledged making it just look like the server takes ages to process the request, so it generally never reaches a retry condition.. The ECN problem is easier as it causes connections to fail to establish. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: [squid-users] Another HTTP 1.1 Question
On 3/14/07, Henrik Nordstrom [EMAIL PROTECTED] wrote: The window scaling problem is unfortunately not so easy to detect in Squid as it causes connections to hang after the request has been sent and acknowledged making it just look like the server takes ages to process the request, so it generally never reaches a retry condition.. I notice that the window may be scaled many times during the course of packet exchanges. I also noticed that the size of the window on the acknowledge packet immediately before the hang was different nearly every time. So, my question is: Is the symptom which exposes the window scaling problem simply the packet sequence 'request - response - hang (aka no subsequent packets)'? Chris
Re: [squid-users] Another HTTP 1.1 Question
ons 2007-03-14 klockan 11:33 -0400 skrev Chris Nighswonger: I notice that the window may be scaled many times during the course of packet exchanges. No, the window scaling is selected during the SYN handshake only. The window size may change, but not the scale. I also noticed that the size of the window on the acknowledge packet immediately before the hang was different nearly every time. So, my question is: Is the symptom which exposes the window scaling problem simply the packet sequence 'request - response - hang (aka no subsequent packets)'? Yes. Plus the additional test that everything works when window scaling is disabled. The firewall bug causes the origin server to think that your window size is way too small, far below the threshold for sending any response data.. If the symptoms is the same with windows scaling disabled then it's most likely just a overloaded server not capable to respond within a reasonable timeframe. The problem was (and still is) unnoticed by most as the major operating system seen on desktops is still optimized for LAN connections, not caring much for long distance high latency connections.. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: [squid-users] Another HTTP 1.1 Question
On 3/11/07, Henrik Nordstrom [EMAIL PROTECTED] wrote: sön 2007-03-11 klockan 16:38 +0800 skrev Adrian Chadd: If someone would like a fun weekend project - write something to sniff out these broken connections and insert temporary ip routes for it. Another idea would be a test tool to see why a site is broken.. Known issues: - ECN - Window Scaling - Forgetting Vary - Mixing up ETag (same ETag on multiple incompatible entities) - Various malformed responses * Double content length * Malformed headers * Repeated single-value headers If I knew more about the structure of these items I'd give it a whirl. As it is, I just have come up to the bottom level of understanding tcp window scaling. FWIW, I complained to the ncsecu and got a call from their IT dept today. It seems that using the words firewall, bank, and broken in the same sentence caused a stir. Apparently they did an OS upgrade on their Symantec (?) firewall recently. I'm not the only one complaining. We'll see if Symantec fixes it. Thanks for the help again, Henrik. I would have been lost on this one without it. Chris
Re: [squid-users] Another HTTP 1.1 Question
mån 2007-03-12 klockan 19:44 -0400 skrev Chris Nighswonger: - ECN - Window Scaling - Forgetting Vary - Mixing up ETag (same ETag on multiple incompatible entities) - Various malformed responses * Double content length * Malformed headers * Repeated single-value headers If I knew more about the structure of these items I'd give it a whirl. The first two is TCP/IP related. Detection is done by trying to communicate with the server both with and without the feature enabled. The rest is HTTP protocol: Forgetting Vary: Many servers forget to emit a Vary: accept-encoding header in gzip:ed responses. Tested by sending requests with and without Accept-Encoding: gzip and comparing the results. If Content-Encoding of the responses differ but Accept-Encoding isn't mentioned in vary then the server is broken. Badly broken if the Vary is missing on the gzip:ed variant. Note: always responding with gzip is ok, even without vary. Mixing up ETag: Effectively the same test as above, but comparing the ETag header of the responses. If Content-Encoding differs but ETag is the same the server is broken. Double content length: Server response has more than one content-length header. Critical if the value differs. Malformed headers: Server response contains various crap mixed with the response headers. Have seen for example various system error messages etc.. (i.e. Failed to open ). A well formed HTTP header follows the pattern ^[A-Za-z][-A-Za-z0-9]*:.* and anything in the headers not matching this is malformed. Technically a few more characters is allowed in the header name, but all known headers follow this pattern. Repeated single-value headers: The exact same test as for content-length, but for other less critical HTTP headers specified as single-valued not allowing more than one value. I.e. Content-Type, Content-Encoding, ETag, Date, etc. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: [squid-users] Another HTTP 1.1 Question
On Tue, Mar 13, 2007, Henrik Nordstrom wrote: The first two is TCP/IP related. Detection is done by trying to communicate with the server both with and without the feature enabled. Are ECN/Window Scaling options which can be fiddled via syscalls per socket FD? If they are then we could sneak them into the retry forwarding logic and cache if/when they work. Adrian
Re: [squid-users] Another HTTP 1.1 Question
On Fri, Mar 09, 2007, Chris Nighswonger wrote: On 3/9/07, Henrik Nordstrom [EMAIL PROTECTED] wrote: tor 2007-03-08 klockan 20:25 -0500 skrev Chris Nighswonger: ip route add $THEIR_IP/32 via $MY_GATEWAY window 65535 which only limits window scaling for that destination without interfering with your other connections ---end snip--- Any thoughts? Better or worse than the other? Obviously better. Nice! Works great happy users every network admin's dream This method may be something to post in the wiki on the page you mentioned regarding these issues. If someone would like a fun weekend project - write something to sniff out these broken connections and insert temporary ip routes for it. That's one more step towards a completely transparent transparent squid. Adrian
Re: [squid-users] Another HTTP 1.1 Question
sön 2007-03-11 klockan 16:38 +0800 skrev Adrian Chadd: If someone would like a fun weekend project - write something to sniff out these broken connections and insert temporary ip routes for it. Another idea would be a test tool to see why a site is broken.. Known issues: - ECN - Window Scaling - Forgetting Vary - Mixing up ETag (same ETag on multiple incompatible entities) - Various malformed responses * Double content length * Malformed headers * Repeated single-value headers Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: [squid-users] Another HTTP 1.1 Question
On 3/9/07, Henrik Nordstrom [EMAIL PROTECTED] wrote: tor 2007-03-08 klockan 20:25 -0500 skrev Chris Nighswonger: ip route add $THEIR_IP/32 via $MY_GATEWAY window 65535 which only limits window scaling for that destination without interfering with your other connections ---end snip--- Any thoughts? Better or worse than the other? Obviously better. Nice! Works great happy users every network admin's dream This method may be something to post in the wiki on the page you mentioned regarding these issues. For what it's worth I had no problem loading the start page using Firefox via Squid-2. What OS are you running squid-2 over? Linux Fedora Core 6, but I had both window scaling and ecn disabled since earlier tests with other broken sites.. ok. My squid is on FC6 as well. Thanks so much for the help Henrik. Chris
[squid-users] Another HTTP 1.1 Question
Hi all, I have a site (www.ncsecu.org) which has been working fine via Squid 2.6STABLE9. Several days ago it broke. Doing some investigation with wireshark, it looks like the site has switched to HTTP 1.1. I have checked with the list archives and understand that there is no real support for 1.1 in squid at present. I have tried the header_access workaround. I also found a post by Henrik suggesting this patch: http://www.henriknordstrom.net/code/squid-http11.patch which appears to be a broken link. Since this site is a banking site and many of my users bank there, I need to come up with a workaround. I have thought to setup a firewall rule combined with client configuration to bypass the proxy for this site. I don't like this solution because it is so specific and requires manual changes to configurations at the client level. Any thoughts/suggestions/other workarounds in place for HTTP 1.1/Squid? Thanks, Chris
Re: [squid-users] Another HTTP 1.1 Question
On Thu, Mar 08, 2007, Chris Nighswonger wrote: Hi all, I have a site (www.ncsecu.org) which has been working fine via Squid 2.6STABLE9. Several days ago it broke. Doing some investigation with wireshark, it looks like the site has switched to HTTP 1.1. I have checked with the list archives and understand that there is no real support for 1.1 in squid at present. I have tried the header_access workaround. I also found a post by Henrik suggesting this patch: Hi, You can try the Squid-2 snapshots which include the below patch. http://www.squid-cache.org/Versions/v2/HEAD/ http://www.henriknordstrom.net/code/squid-http11.patch which appears to be a broken link. Adrian
Re: [squid-users] Another HTTP 1.1 Question
On 3/8/07, Adrian Chadd [EMAIL PROTECTED] wrote: You can try the Squid-2 snapshots which include the below patch. http://www.squid-cache.org/Versions/v2/HEAD/ Here is what I have done: 1. My current install is via yum (rpm). 2. I have configured with the same options returned from a '#squid -v' and done a 'make' 3. I have backup my current squid.conf Here is the question: Do I do a 'make install,' then replace the new 'squid.conf' with my original, and start squid back up? (This is a production box and I really don't want to bust it.) Thanks, Chris
Re: [squid-users] Another HTTP 1.1 Question
Quoting Chris Nighswonger [EMAIL PROTECTED]: On 3/8/07, Adrian Chadd [EMAIL PROTECTED] wrote: You can try the Squid-2 snapshots which include the below patch. http://www.squid-cache.org/Versions/v2/HEAD/ Here is what I have done: 1. My current install is via yum (rpm). 2. I have configured with the same options returned from a '#squid -v' and done a 'make' 3. I have backup my current squid.conf Here is the question: Do I do a 'make install,' then replace the new 'squid.conf' with my original, and start squid back up? (This is a production box and I really don't want to bust it.) Thanks, Chris If it was me I would do a cp on my current squid directory, then when installing do a ./configure --someother directory. For example: if squid is installed in /var/squid you could install the new version in /usr/local/squid. Doing a ./configure --help will give you the exact options. Then after installing you can either edit the new squid.conf to suit, stop your old squid and start new squid with a ./squid -z to build the cache directorys and then do a ./squid. If everything goes south, then you can go back to your old version and figure out why the new one didnt work. ddh -- Dwayne Hottinger Network Administrator Harrisonburg City Public Schools
Re: [squid-users] Another HTTP 1.1 Question
tor 2007-03-08 klockan 08:38 -0500 skrev Chris Nighswonger: Hi all, I have a site (www.ncsecu.org) which has been working fine via Squid 2.6STABLE9. Several days ago it broke. Doing some investigation with wireshark, it looks like the site has switched to HTTP 1.1. Can the brokenness be tested without having an account at the site? It may be worth trying 2.6.STABLE10, it has significant workarounds for broken HTTP/1.1 servers. (i.e. the bulk of the http11 patch mentioned earlier). No need to go to Squid-2.HEAD for this. The related code is the same in 2.6.STABLE10. The patch is no longer in my code directory because a) That version of the patch was a bit broken. b) Development of HTTP/1.1 support has moved to devel.squid-cache.org Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: [squid-users] Another HTTP 1.1 Question
On Thu, Mar 08, 2007, [EMAIL PROTECTED] wrote: If it was me I would do a cp on my current squid directory, then when installing do a ./configure --someother directory. For example: if squid is installed in /var/squid you could install the new version in /usr/local/squid. Doing a ./configure --help will give you the exact options. Then after installing you can either edit the new squid.conf to suit, stop your old squid and start new squid with a ./squid -z to build the cache directorys and then do a ./squid. If everything goes south, then you can go back to your old version and figure out why the new one didnt work. I normally do this: ./configure --prefix=/usr/local/squid-VERSION make make install cd /usr/local rm squid (its a symlink!) ln -s squid-VERSION squid cp /path/to/normal/squid.conf /usr/local/squid-VERSION/etc/squid.conf That way I can have multiple squids installed to try but have my init script only start /usr/local/squid/sbin/squid . Adrian
Re: [squid-users] Another HTTP 1.1 Question
On 3/8/07, Henrik Nordstrom [EMAIL PROTECTED] wrote: tor 2007-03-08 klockan 08:38 -0500 skrev Chris Nighswonger: Hi all, I have a site (www.ncsecu.org) which has been working fine via Squid 2.6STABLE9. Several days ago it broke. Doing some investigation with wireshark, it looks like the site has switched to HTTP 1.1. Can the brokenness be tested without having an account at the site? It can. The default page will not load with squid in-line. No errors at all in access.log. The browser just hangs. This happens after squid forwards an HTTP 1.0 packet. The entire packet exchange dies at this point. With squid out of line, the same packe is HTTP 1.1 and the page loads right up. It may be worth trying 2.6.STABLE10, it has significant workarounds for broken HTTP/1.1 servers. (i.e. the bulk of the http11 patch mentioned earlier). No need to go to Squid-2.HEAD for this. The related code is the same in 2.6.STABLE10. I'll grab it then. Thanks. Chris
Re: [squid-users] Another HTTP 1.1 Question
tor 2007-03-08 klockan 18:46 -0500 skrev Chris Nighswonger: It can. The default page will not load with squid in-line. No errors at all in access.log. The browser just hangs. This happens after squid forwards an HTTP 1.0 packet. The entire packet exchange dies at this point. With squid out of line, the same packe is HTTP 1.1 and the page loads right up. That smells more like a tcp windows issue than an HTTP/1.1 issue.. If on Linux try the following workaround: echo 0 /proc/sys/net/ipv4/tcp_window_scaling this works around quite many broken firewalls not coping well with window scaling, but significantly scarifies performance over long distance connections (measured in RTT * bandwidth, not miles)... For what it's worth I had no problem loading the start page using Firefox via Squid-2. Testing.. Ah, yes. There is a broken firewall at this site crashing window scaling.. http://wiki.squid-cache.org/SquidFaq/SystemWeirdnesses?highlight=%28window%29%7C%28scaling%29#head-699d810035c099c8b4bff21e12bb365438a21027 Someone should contact the site operators explaining the problem to them.. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: [squid-users] Another HTTP 1.1 Question
On 3/8/07, Henrik Nordstrom [EMAIL PROTECTED] wrote: tor 2007-03-08 klockan 18:46 -0500 skrev Chris Nighswonger: It can. The default page will not load with squid in-line. No errors at all in access.log. The browser just hangs. This happens after squid forwards an HTTP 1.0 packet. The entire packet exchange dies at this point. With squid out of line, the same packe is HTTP 1.1 and the page loads right up. That smells more like a tcp windows issue than an HTTP/1.1 issue.. ok If on Linux try the following workaround: echo 0 /proc/sys/net/ipv4/tcp_window_scaling this works around quite many broken firewalls not coping well with window scaling, but significantly scarifies performance over long distance connections (measured in RTT * bandwidth, not miles)... I can try this tomorrow. For what it's worth I had no problem loading the start page using Firefox via Squid-2. Up untill a week or so ago, I had no problems with Firefox/Squid-2 either... :( Testing.. Ah, yes. There is a broken firewall at this site crashing window scaling.. http://wiki.squid-cache.org/SquidFaq/SystemWeirdnesses?highlight=%28window%29%7C%28scaling%29#head-699d810035c099c8b4bff21e12bb365438a21027 Someone should contact the site operators explaining the problem to them.. I read the info at the link above. Are you suggesting that the issue is a broken firewall on ncsecu.org? If that is the issue, I'll have a chat with them. Thanks again, Chris
Re: [squid-users] Another HTTP 1.1 Question
tor 2007-03-08 klockan 19:13 -0500 skrev Chris Nighswonger: I read the info at the link above. Are you suggesting that the issue is a broken firewall on ncsecu.org? If that is the issue, I'll have a chat with them. Yes. It is a broken firewall on ncsecu.org causing this problem. It fails to handle the tcp window scaling option. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: [squid-users] Another HTTP 1.1 Question
Quoting Adrian Chadd [EMAIL PROTECTED]: On Thu, Mar 08, 2007, [EMAIL PROTECTED] wrote: If it was me I would do a cp on my current squid directory, then when installing do a ./configure --someother directory. For example: if squid is installed in /var/squid you could install the new version in /usr/local/squid. Doing a ./configure --help will give you the exact options. Then after installing you can either edit the new squid.conf to suit, stop your old squid and start new squid with a ./squid -z to build the cache directorys and then do a ./squid. If everything goes south, then you can go back to your old version and figure out why the new one didnt work. I normally do this: ./configure --prefix=/usr/local/squid-VERSION make make install cd /usr/local rm squid (its a symlink!) ln -s squid-VERSION squid cp /path/to/normal/squid.conf /usr/local/squid-VERSION/etc/squid.conf That way I can have multiple squids installed to try but have my init script only start /usr/local/squid/sbin/squid . Adrian Cool. Yep. Works very well. Sort of the same thing only more elegant. -- Dwayne Hottinger Network Administrator Harrisonburg City Public Schools
Re: [squid-users] Another HTTP 1.1 Question
On 3/8/07, Henrik Nordstrom [EMAIL PROTECTED] wrote: That smells more like a tcp windows issue than an HTTP/1.1 issue.. If on Linux try the following workaround: echo 0 /proc/sys/net/ipv4/tcp_window_scaling this works around quite many broken firewalls not coping well with window scaling, but significantly scarifies performance over long distance connections (measured in RTT * bandwidth, not miles)... Here: http://lwn.net/Articles/92727/ I found this workaround: ---snip--- With kernel 2.6.17.13 or higher, you can also do: THEIR_IP=1.2.3.4 MY_GATEWAY=5.6.7.8 ip route add $THEIR_IP/32 via $MY_GATEWAY window 65535 which only limits window scaling for that destination without interfering with your other connections ---end snip--- Any thoughts? Better or worse than the other? For what it's worth I had no problem loading the start page using Firefox via Squid-2. What OS are you running squid-2 over? Thanks, Chris
Re: [squid-users] Another HTTP 1.1 Question
tor 2007-03-08 klockan 20:25 -0500 skrev Chris Nighswonger: ip route add $THEIR_IP/32 via $MY_GATEWAY window 65535 which only limits window scaling for that destination without interfering with your other connections ---end snip--- Any thoughts? Better or worse than the other? Obviously better. For what it's worth I had no problem loading the start page using Firefox via Squid-2. What OS are you running squid-2 over? Linux Fedora Core 6, but I had both window scaling and ecn disabled since earlier tests with other broken sites.. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
