RE: [squid-users] 304 response preventing site from loading
tor 2010-09-30 klockan 13:24 +1000 skrev Paul Freeman: However on further investigation, I don't think this is the case in this instance. For some reason, the squid GET request to www.mhhe.com (IP 12.26.55.139) takes a long time to be completed - approx. 2 minutes. Some data is returned quickly but then there is a period where on my squid server I see a TCP Previous Segment lost then squid server sending Dup ACKs to www.mhhe.com and www.mhhe.com sending TCP Retransmissions for the same segment. The Retransmission RTTs to ACK the one segment are at 0.2,4,8,16,32 and 60 seconds. After that segment has finally been received, the rest of the data is received OK. This smells like TCP window scaling issues in a firewall somewhere. Try as a test: echo 0 /proc/sys/net/ipv4/tcp_window_scaling note that this is somewhat intrusive and reduces performance of TCP in general, but is an easy way of testing for the problem. Regards Henrik
Re: [squid-users] 304 response preventing site from loading
- Henrik Nordström hen...@henriknordstrom.net wrote: tor 2010-09-30 klockan 13:24 +1000 skrev Paul Freeman: However on further investigation, I don't think this is the case in this instance. For some reason, the squid GET request to www.mhhe.com (IP 12.26.55.139) takes a long time to be completed - approx. 2 minutes. Some data is returned quickly but then there is a period where on my squid server I see a TCP Previous Segment lost then squid server sending Dup ACKs to www.mhhe.com and www.mhhe.com sending TCP Retransmissions for the same segment. The Retransmission RTTs to ACK the one segment are at 0.2,4,8,16,32 and 60 seconds. After that segment has finally been received, the rest of the data is received OK. This smells like TCP window scaling issues in a firewall somewhere. Try as a test: echo 0 /proc/sys/net/ipv4/tcp_window_scaling note that this is somewhat intrusive and reduces performance of TCP in general, but is an easy way of testing for the problem. Regards Henrik Henrik, Firstly, I am struck with the coincidence of the timing of your response! After not looking at this issue for 4 weeks, I tackled it again today, and within minutes of adopting a workaround, your message was sent! This is the second odd coincidence in the past few hours - the first was finding two machines with the same MAC (from the factory, it seems) on our network... :-( I have not had time to try another version of squid, and Paul's test of 3.x indicated it may not help, so I set about trying a workaround. I am using cisco ip policy route-maps to redirect http traffic to the transparent squid box. I excluded the 12.26.55.139, and instead nat'd that traffic. This works fine, and page loads are within 30 seconds. It is messy, however, since our network is currently a mix of machines using wpad auto proxy, manual proxy, and two different redirect methods to a transparent squid. (a Cisco 6500 MSFC, and an Aruba 6000 wireless controller both handle redirects for http traffic for their VLANs). So adding an exception like this requires changes in quite a few places. As an aside, we had to abandon WCCP2 with our Cisco 6500 MSFC1, as it appeared to be unstable under heavy loads, and would slow down during peak periods, and actually crashed once. I will look into the tcp window scaling idea and report back. Thanks! Shawn Wright I.T. Manager Shawnigan Lake School
Re: [squid-users] 304 response preventing site from loading
- Henrik Nordström hen...@henriknordstrom.net wrote: tor 2010-09-30 klockan 13:24 +1000 skrev Paul Freeman: However on further investigation, I don't think this is the case in this instance. For some reason, the squid GET request to www.mhhe.com (IP 12.26.55.139) takes a long time to be completed - approx. 2 minutes. Some data is returned quickly but then there is a period where on my squid server I see a TCP Previous Segment lost then squid server sending Dup ACKs to www.mhhe.com and www.mhhe.com sending TCP Retransmissions for the same segment. The Retransmission RTTs to ACK the one segment are at 0.2,4,8,16,32 and 60 seconds. After that segment has finally been received, the rest of the data is received OK. This smells like TCP window scaling issues in a firewall somewhere. Try as a test: echo 0 /proc/sys/net/ipv4/tcp_window_scaling note that this is somewhat intrusive and reduces performance of TCP in general, but is an easy way of testing for the problem. Regards Henrik I disabled tcp window scaling on both the squid box (linux 2.6.15), and our linux firewall (2.4.28), and saw no change in behaviour. What we're seeing certainly matches the description of tcp window scaling problems I've read about, so I was hopeful it might work. Thanks for the help.
Re: [squid-users] 304 response preventing site from loading
Paul, Firstly, thanks for your very thorough testing! Your results mirror our experience - in one instance I did see the page load complete after some 5 minutes or more when going through proxy. Our server OS is Ubuntu 6.06LTS 32bit, kernel 2.6.15-26. We did away with authentication in the summer after 10 years, as it caused too many problems with dumb apps and devices, and enabled transparent support with WCCP at that time, mainly for wireless devices. This is the first site we've encountered so far that does not work. Many sites like itunes now work as we allow SSL traffic to be SNATed by our firewall, bypassing squid. Shawn Wright I.T. Manager Shawnigan Lake School Please direct requests for support to helpd...@shawnigan.ca - Paul Freeman paul.free...@eml.com.au wrote: Shawn I have seen Amos' reply regarding a possible bug in the version of squid you are using and his suggestion to upgrade and try again. After seeing your question, I did some testing using different versions of squid I have access to - Squid 3.1.8 and Squid 2.6stable18. Both squid installations are using authentication (Kerberos/NTLM for 3.1.8 and ntlm/basic for 2.6stable18) and are running on Ubuntu - 3.1.8 on Ubuntu 10.04LTS and 2.6stable18 on Ubuntu 8.04LTS. Transparent interception is _not_ being used in either installation. I tested using Firefox v 3.6.3 and found that going direct (not using squid) works OK (approx 30 sec page load) but going via squid 3.1.8 or squid 2.6stable eventually works but is very slow (approx 4-5minutes to load the entire page contents). Basically, I have found these squid versions both work and load the page successfully but for me, the page is slow to load when using squid compared with going direct. I have investigated this further and the problem may be related to some aspect related to networking on my squid server OS (linux) rather than squid but I am not sure. For those who are interested, please read on ... (a bit long) :-) Regards Paul Freeman The discussion below refers to my investigation using squid 3.1.8. It is running on Ubuntu 10.04LTS and was compiled from a source package created by Amos Jeffries (thanks Amos). The client workstation is running Windows XP SP3. Doing some wireshark packet traces of the traffic leads me to think the slowness is in retrieving two urls: http://www.dushkin.com/web-cgi/olc/nytfeed.pl?DCID=984N=3 http://www.dushkin.com/web-cgi/olc/gencurrentnew.pl?DCID=984N=3 Both the GET requests for these urls get 302 re-direct responses as follows (same order as urls above): http://www.mhcls.com/cls/web-cgi/olc/nytfeed.pl?DCID=984N=3 http://www.mhcls.com/cls/web-cgi/olc/gencurrentnews.pl?DCID=984N=3 Requests to these re-direct urls also receive 302 re-direct responses as follows (same order as urls above): http://www.mhhe.com/cls/?DCID=984N=3 http://www.mhhe.com/cls/?DCID=984N=3 It is this last url (http://www.mhhe.com/cls/?DCID=984N=3) that seems to take a long to retrieve by squid. I originally thought the slowness may have to do with the HTTP/1.1 feature of Transfer-Encoding: chunked as I have come across this in some other work I have been doing recently. This header is included in the www.dushkin.com and www.mhcls.com 302 re-direct responses. I noticed in the header the word chunked is all lower case. This does not appear to be in violation of the HTTP/1.1 spec but some versions of squid use a case sensitive compare for Chunked (capital C) and thus do not match on chunked. IN some instances and squid versions, the Transfer-Encoding: chunked/Chunked header can cause squid to not be able to determine when all the data to fulfil the GET request has been supplied and so it waits. Eventually the web server replying to the GET request will timeout the connection (timeout various depending on the web server but can be of the order of a minute or more), sending a TCP RST. Search the squid-users mailing list for more info on this one. However on further investigation, I don't think this is the case in this instance. For some reason, the squid GET request to www.mhhe.com (IP 12.26.55.139) takes a long time to be completed - approx. 2 minutes. Some data is returned quickly but then there is a period where on my squid server I see a TCP Previous Segment lost then squid server sending Dup ACKs to www.mhhe.com and www.mhhe.com sending TCP Retransmissions for the same segment. The Retransmission RTTs to ACK the one segment are at 0.2,4,8,16,32 and 60 seconds. After that segment has finally been received, the rest of the data is received OK. The reply headers from the GET to www.mhhe.com are as follows: HTTP/1.0 200 OK Server: MHttpd/3.2 (UAI; sparc-solaris2.6; Meta-HTML/5.06) Date: Thu, 30 Sep 2010 00:06:25 GMT Expires: Wed 29 Sep 2010 00:06:25 GMT Last-modified: Thu Sep 2010 00:05:25 GMT Content-length: 13858
Re: [squid-users] 304 response preventing site from loading
On Wed, 29 Sep 2010 16:26:10 -0700 (PDT), Shawn Wright swri...@shawnigan.ca wrote: We have encountered a site which does appear to like going through our squid proxy, either regular or transparent mode. I have contacted the company, but was hoping someone here could quickly try this site and tell me if it works through your squid proxy. http://highered.mcgraw-hill.com/sites/0072421975/student_view0/index.html It is a school textbook site, and the menus will not load when going through proxy. We have a transparent proxy which uses WCCP2 from a Cisco Cat6500 to redirect the packets from the clients. I have also disabled wccp and tried a manual proxy config from the browser. The only thing that works is a SNAT from the client through our firewall, which isn't a solution for us. The squid log shows: (squid 2.6-20) 1285802182.732 403 10.3.0.144 TCP_REFRESH_HIT/304 186 GET http://highered.mcgraw-hill.com/sites/0072421975/student_view0/index.html - DIRECT/204.8.133.213 - 1285802182.906 160 10.3.0.144 TCP_REFRESH_HIT/304 185 GET http://highered.mcgraw-hill.com/olcweb/styles/v2/wheat/css.css - DIRECT/204.8.133.213 - snip a whole pile of apparently working requests Squid is asking for new data, the website server is reporting that Squid copy is correct (304) and so Squid sends it to the client (HIT). I can't tell from the log whether the client sent an IMS request to squid or if it was created by Squid and the wrong reply sent back. That is an old bug IIRC. Your Squid is quite old and this may be fixed in more recent versions. Please try an upgrade and see if the problem remains. Amos
RE: [squid-users] 304 response preventing site from loading
Shawn I have seen Amos' reply regarding a possible bug in the version of squid you are using and his suggestion to upgrade and try again. After seeing your question, I did some testing using different versions of squid I have access to - Squid 3.1.8 and Squid 2.6stable18. Both squid installations are using authentication (Kerberos/NTLM for 3.1.8 and ntlm/basic for 2.6stable18) and are running on Ubuntu - 3.1.8 on Ubuntu 10.04LTS and 2.6stable18 on Ubuntu 8.04LTS. Transparent interception is _not_ being used in either installation. I tested using Firefox v 3.6.3 and found that going direct (not using squid) works OK (approx 30 sec page load) but going via squid 3.1.8 or squid 2.6stable eventually works but is very slow (approx 4-5minutes to load the entire page contents). Basically, I have found these squid versions both work and load the page successfully but for me, the page is slow to load when using squid compared with going direct. I have investigated this further and the problem may be related to some aspect related to networking on my squid server OS (linux) rather than squid but I am not sure. For those who are interested, please read on ... (a bit long) :-) Regards Paul Freeman The discussion below refers to my investigation using squid 3.1.8. It is running on Ubuntu 10.04LTS and was compiled from a source package created by Amos Jeffries (thanks Amos). The client workstation is running Windows XP SP3. Doing some wireshark packet traces of the traffic leads me to think the slowness is in retrieving two urls: http://www.dushkin.com/web-cgi/olc/nytfeed.pl?DCID=984N=3 http://www.dushkin.com/web-cgi/olc/gencurrentnew.pl?DCID=984N=3 Both the GET requests for these urls get 302 re-direct responses as follows (same order as urls above): http://www.mhcls.com/cls/web-cgi/olc/nytfeed.pl?DCID=984N=3 http://www.mhcls.com/cls/web-cgi/olc/gencurrentnews.pl?DCID=984N=3 Requests to these re-direct urls also receive 302 re-direct responses as follows (same order as urls above): http://www.mhhe.com/cls/?DCID=984N=3 http://www.mhhe.com/cls/?DCID=984N=3 It is this last url (http://www.mhhe.com/cls/?DCID=984N=3) that seems to take a long to retrieve by squid. I originally thought the slowness may have to do with the HTTP/1.1 feature of Transfer-Encoding: chunked as I have come across this in some other work I have been doing recently. This header is included in the www.dushkin.com and www.mhcls.com 302 re-direct responses. I noticed in the header the word chunked is all lower case. This does not appear to be in violation of the HTTP/1.1 spec but some versions of squid use a case sensitive compare for Chunked (capital C) and thus do not match on chunked. IN some instances and squid versions, the Transfer-Encoding: chunked/Chunked header can cause squid to not be able to determine when all the data to fulfil the GET request has been supplied and so it waits. Eventually the web server replying to the GET request will timeout the connection (timeout various depending on the web server but can be of the order of a minute or more), sending a TCP RST. Search the squid-users mailing list for more info on this one. However on further investigation, I don't think this is the case in this instance. For some reason, the squid GET request to www.mhhe.com (IP 12.26.55.139) takes a long time to be completed - approx. 2 minutes. Some data is returned quickly but then there is a period where on my squid server I see a TCP Previous Segment lost then squid server sending Dup ACKs to www.mhhe.com and www.mhhe.com sending TCP Retransmissions for the same segment. The Retransmission RTTs to ACK the one segment are at 0.2,4,8,16,32 and 60 seconds. After that segment has finally been received, the rest of the data is received OK. The reply headers from the GET to www.mhhe.com are as follows: HTTP/1.0 200 OK Server: MHttpd/3.2 (UAI; sparc-solaris2.6; Meta-HTML/5.06) Date: Thu, 30 Sep 2010 00:06:25 GMT Expires: Wed 29 Sep 2010 00:06:25 GMT Last-modified: Thu Sep 2010 00:05:25 GMT Content-length: 13858 Meta-HTML-Engine: MHtppd/3.2 (UAI; sparc-solaris2.6; Meta-HTML/5.06) Content-type: text/html There are two GET requests for the url http://www.mhhe.com/cls/?DCID=984N=3 and each takes approx. 2 minutes to complete which accounts for the approx. 4 minute delay in loading the page. I am not sure what is causing this but it appears at first glance to be related to a networking issue on the host squid server OS. Going directly using the same Workstation/Browser/LAN/Firewall/Internet connection combination does not show the same delay - only approx 29 seconds to load. I still see a TCP Previous Segment lost and the Dup ACKs and TCP Retransmissions when going direct but there are fewer TCP Retransmissions (2-3 compared with 6-7) and hence the quicker reply. The IP address of highered.mcgraw-hill.com is 204.8.133.213 while the IP addresses of www.dushkin.com, www.mhcls.com and www.mhhe.com are the same - 12.26.55.139.
