[squid-users] RE: Squid ICAP TCP Zero Window Size Issues on Solaris
Sorry, I can't attach the trace or logs here, they are too large to mail. Anyone who would like to have a look at them, please send me a mail and I can send them to you. Best Regards Niall PS - ignore the old 'Squid 3.1 ICAP Issue with REQMOD 302' email below - I replied to an old mailing list item and forgot to delete it. -Original Message----- From: Niall O'Cuilinn Sent: 08 April 2011 13:06 To: 'squid-users@squid-cache.org' Cc: Mike Whitty Subject: Squid ICAP TCP Zero Window Size Issues on Solaris Hi all, We have recently been having trouble with large POST requests causing threads to get blocked on our ICAP Server. We are using Squid 3.0 STABLE 15 We have tracked the problem down to TCP Zero Window Size issues when the ICAP Server is trying to write the POST body back to Squid in the REQMOD response. It seems that Squid stops processing the ICAP response but never properly closes the connection. The issue occurs for POST requests to unavailable web servers. Our guess is that Squid determines that the remote host is unavailable and decides not to waste resources processing the ICAP response. We can only reproduce the issue on Solaris. We tried upgrading to Squid 3.1.12 but found a similar but different issue, where for the same large POST requests the TCP Zero Window issue occurs on the connection between the web client and Squid, rather than between the ICAP Server and Squid. We have opened two bugs relating to the issue, bug 3191 for the original 3.0.15 issue and 3190 for the 3.1.12 issue. I attached logs and tcp trace to bug 3190 but had problems uploading the same for bug 3191 - possibly due to the size of the logs (although the smaller trace wont attach for me either. I will reply to this email with the same attachments Any help or advice much appreciated, Thanks Niall -----Original Message- From: Niall O'Cuilinn Sent: 13 April 2010 15:09 To: 'squid-users@squid-cache.org' Subject: Squid 3.1 ICAP Issue with REQMOD 302 Hi, I have recently moved from Squid 3.0 to Squid 3.1. I am trying to integrate it with an ICAP server. I am having a problem where Squid 3.1 is rejecting some responses from the ICAP server which Squid 3.0 accepted. The response in question is a REQMOD response where the ICAP server is returning a HTTP 302 response rather than modifying the original HTTP request. Here is the ICAP request and response: ICAP Request from Squid: REQMOD icap://10.1.1.25:1344/reqmod ICAP/1.0\r\n Host: 10.1.1.25:1344\r\n Date: Mon, 12 Apr 2010 14:25:39 GMT\r\n Encapsulated: req-hdr=0, null-body=398\r\n Allow: 204\r\n \r\n GET http://c.proxy.com/www.test.com/ HTTP/1.1\r\n Host: c.proxy.com\r\n User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3\r\n Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n Accept-Language: en-gb,en;q=0.5\r\n Accept-Encoding: gzip,deflate\r\n Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n Pragma: no-cache\r\n Cache-Control: no-cache\r\n \r\n Response from ICAP Server: ICAP/1.0 200 OK\r\n Date: Mon, 12 Apr 2010 14:25:15 GMT\r\n Connection: keep-alive\r\n ISTag: "ReqModService"\r\n Encapsulated: res-hdr=0,null-body=160\r\n \r\n HTTP/1.x 302 Found\r\n content-type: text/html\r\n location: https://localhost:8443/mib/authentication\r\n \r\n \r\n Squid displays an ICAP error in the browser and states that an illegal response was received from the ICAP server. Any ideas what might be wrong? Although the ICAP server worked correctly with Squid 3.0 I am open to the possibility that the issue is with the ICAP response and that the old Squid was simply more tolerant than v3.1. Thanks in advance, Niall Niall Ó Cuilinn Product Development ChangingWorlds - A Unit of Amdocs Interactive t: +353 1 4401268 | niall.ocuil...@changingworlds.com AMDOCS > CUSTOMER EXPERIENCE SYSTEMS INNOVATION This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at http://www.amdocs.com/email_disclaimer.asp
[squid-users] Squid ICAP TCP Zero Window Size Issues on Solaris
Hi all, We have recently been having trouble with large POST requests causing threads to get blocked on our ICAP Server. We are using Squid 3.0 STABLE 15 We have tracked the problem down to TCP Zero Window Size issues when the ICAP Server is trying to write the POST body back to Squid in the REQMOD response. It seems that Squid stops processing the ICAP response but never properly closes the connection. The issue occurs for POST requests to unavailable web servers. Our guess is that Squid determines that the remote host is unavailable and decides not to waste resources processing the ICAP response. We can only reproduce the issue on Solaris. We tried upgrading to Squid 3.1.12 but found a similar but different issue, where for the same large POST requests the TCP Zero Window issue occurs on the connection between the web client and Squid, rather than between the ICAP Server and Squid. We have opened two bugs relating to the issue, bug 3191 for the original 3.0.15 issue and 3190 for the 3.1.12 issue. I attached logs and tcp trace to bug 3190 but had problems uploading the same for bug 3191 - possibly due to the size of the logs (although the smaller trace wont attach for me either. I will reply to this email with the same attachments Any help or advice much appreciated, Thanks Niall -Original Message- From: Niall O'Cuilinn Sent: 13 April 2010 15:09 To: 'squid-users@squid-cache.org' Subject: Squid 3.1 ICAP Issue with REQMOD 302 Hi, I have recently moved from Squid 3.0 to Squid 3.1. I am trying to integrate it with an ICAP server. I am having a problem where Squid 3.1 is rejecting some responses from the ICAP server which Squid 3.0 accepted. The response in question is a REQMOD response where the ICAP server is returning a HTTP 302 response rather than modifying the original HTTP request. Here is the ICAP request and response: ICAP Request from Squid: REQMOD icap://10.1.1.25:1344/reqmod ICAP/1.0\r\n Host: 10.1.1.25:1344\r\n Date: Mon, 12 Apr 2010 14:25:39 GMT\r\n Encapsulated: req-hdr=0, null-body=398\r\n Allow: 204\r\n \r\n GET http://c.proxy.com/www.test.com/ HTTP/1.1\r\n Host: c.proxy.com\r\n User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3\r\n Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n Accept-Language: en-gb,en;q=0.5\r\n Accept-Encoding: gzip,deflate\r\n Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n Pragma: no-cache\r\n Cache-Control: no-cache\r\n \r\n Response from ICAP Server: ICAP/1.0 200 OK\r\n Date: Mon, 12 Apr 2010 14:25:15 GMT\r\n Connection: keep-alive\r\n ISTag: "ReqModService"\r\n Encapsulated: res-hdr=0,null-body=160\r\n \r\n HTTP/1.x 302 Found\r\n content-type: text/html\r\n location: https://localhost:8443/mib/authentication\r\n \r\n \r\n Squid displays an ICAP error in the browser and states that an illegal response was received from the ICAP server. Any ideas what might be wrong? Although the ICAP server worked correctly with Squid 3.0 I am open to the possibility that the issue is with the ICAP response and that the old Squid was simply more tolerant than v3.1. Thanks in advance, Niall Niall Ó Cuilinn Product Development ChangingWorlds - A Unit of Amdocs Interactive t: +353 1 4401268 | niall.ocuil...@changingworlds.com AMDOCS > CUSTOMER EXPERIENCE SYSTEMS INNOVATION This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at http://www.amdocs.com/email_disclaimer.asp
[squid-users] Squid 3.1 ICAP bug on Solaris
Hi We have found a problem with Squid 3.1 on Solaris With ICAP enabled all pages over 49150 bytes fail to load. Squid returns an ICAP error page. Squid sends an incomplete RESPMOD to the ICAP server. It sends chunks totalling 49150 bytes and then fails to send a final 0 byte chunk. (256 byte preview + 48894 bytes in chunks) We have checked the tcp traffic and the full web page is returned to Squid by the web server. This only occurs on Solaris. If we install Squid 3.1 on Linux it works fine. We have tested on Solaris Sparc and Intel. Both have the same behaviour. We have logged bug 2910 to describe this issue. We have attached tcp dumps and squid logs to the bug. Best Regards Niall P.S. This is the same issue as 'Squid sends incomplete RESPMOD requests to ICAP Server on Solaris'. I wanted to start again since the nature of the issue is a bit clearer now. Niall Ó Cuilinn Product Development ChangingWorlds - A Unit of Amdocs Interactive t: +353 1 4401268 | niall.ocuil...@changingworlds.com AMDOCS > CUSTOMER EXPERIENCE SYSTEMS INNOVATION This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at http://www.amdocs.com/email_disclaimer.asp
Re: [squid-users] Squid 3.1 ICAP Issue with REQMOD 302
Hi I had a look at the null-body values. They correctly match the length of the HTTP 302 response headers block. The extra two bytes is an extra line return. You can see that after the last header there are three '\r\n' line returns. I tried removing one of them but the result was the same. I also turned on more detailed debug logging and found this in the cache.log: -- 2010/04/14 17:03:05.494| HttpReply::sanityCheckStartLine: missing or invalid status number in 'HTTP/1.x 302 Found content-type: text/html location: https://localhost:8443/mib/authentication/checkCookie?backURL=http%3A%2F%2Fc.proxy.com%2Fwww.google.ie ' - I changed the ICAP Server to return 'HTTP/1.0' instead of 'HTTP/1.x' and now it is working. This worked using 'HTTP/1.x' on Squid 3.0. The version I'm using is Squid3.1.1 Thanks Niall This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at http://www.amdocs.com/email_disclaimer.asp
Re: [squid-users] Squid 3.1 ICAP Issue with REQMOD 302
Hi, Just resending the correct request and response: ICAP Request from Squid: REQMOD icap://10.1.1.25:1344/reqmod ICAP/1.0\r\n Host: 10.1.1.25:1344\r\n Date: Mon, 12 Apr 2010 14:25:39 GMT\r\n Encapsulated: req-hdr=0, null-body=398\r\n Allow: 204\r\n \r\n GET http://c.proxy.com/www.test.com/ HTTP/1.1\r\n Host: c.proxy.com\r\n User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3\r\n Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n Accept-Language: en-gb,en;q=0.5\r\n Accept-Encoding: gzip,deflate\r\n Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n Pragma: no-cache\r\n Cache-Control: no-cache\r\n \r\n Response from ICAP Server: ICAP/1.0 200 OK\r\n Date: Mon, 12 Apr 2010 14:25:15 GMT\r\n Connection: keep-alive\r\n ISTag: "ReqModService"\r\n Encapsulated: res-hdr=0,null-body=100\r\n \r\n HTTP/1.x 302 Found\r\n content-type: text/html\r\n location: https://localhost:8443/mib/authentication\r\n \r\n \r\n Niall Ó Cuilinn Product Development ChangingWorlds - A Unit of Amdocs Interactive t: +353 1 4401268 | niall.ocuil...@changingworlds.com AMDOCS > CUSTOMER EXPERIENCE SYSTEMS INNOVATION This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at http://www.amdocs.com/email_disclaimer.asp
RE: [squid-users] Squid 3.1 ICAP Issue with REQMOD 302
Hi Christos Thanks for the reply. Sorry that was my mistake, I removed some sensitive info from the location header URL but forgot to modify the null-body value. It should have read "null-body=100" (I removed 60 chars/bytes). You might be right and it might still be out by two. I will have a look. Have you Squid 3.1 working with ICAP? I am wondering if there are any known issues with ICAP support in v3.1? Thanks Niall Christos Tsantilas wrote: >Niall O'Cuilinn wrote: >> Hi, >> >> I have recently moved from Squid 3.0 to Squid 3.1. I am trying to integrate >> it with an ICAP server. >> >> I am having a problem where Squid 3.1 is rejecting some responses from the >> ICAP server which Squid 3.0 accepted. >> >> The response in question is a REQMOD response where the ICAP server is >> returning a HTTP 302 response rather than modifying the original HTTP >> request. > >Hi Niall, > I believe the Encapsulated header in the ICAP server response is wrong. >The "null-body=160" should be the size of the encapsulated Http headers, >if I am not wrong should be "null-body=102". > >Regards, >Christos > > >> >> Here is the ICAP request and response: >> >> ICAP Request from Squid: >> >> REQMOD icap://10.1.1.25:1344/reqmod ICAP/1.0\r\n >> Host: 10.1.1.25:1344\r\n >> Date: Mon, 12 Apr 2010 14:25:39 GMT\r\n >> Encapsulated: req-hdr=0, null-body=398\r\n >> Allow: 204\r\n >> \r\n >> GET http://c.proxy.com/www.test.com/ HTTP/1.1\r\n >> Host: c.proxy.com\r\n >> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.3) >> Gecko/20100401 Firefox/3.6.3\r\n >> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n >> Accept-Language: en-gb,en;q=0.5\r\n >> Accept-Encoding: gzip,deflate\r\n >> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n >> Pragma: no-cache\r\n >> Cache-Control: no-cache\r\n >> \r\n >> >> Response from ICAP Server: >> >> ICAP/1.0 200 OK\r\n >> Date: Mon, 12 Apr 2010 14:25:15 GMT\r\n >> Connection: keep-alive\r\n >> ISTag: "ReqModService"\r\n >> Encapsulated: res-hdr=0,null-body=160\r\n >> \r\n >> HTTP/1.x 302 Found\r\n >> content-type: text/html\r\n >> location: https://localhost:8443/mib/authentication\r\n >> \r\n >> \r\n >> >> Squid displays an ICAP error in the browser and states that an illegal >> response was received from the ICAP server. >> >> Any ideas what might be wrong? Although the ICAP server worked correctly >> with Squid 3.0 I am open to the possibility that the issue is with the ICAP >> response and that the old Squid was simply more tolerant than v3.1. >> >> Thanks in advance, >> Niall >> >> Niall Ó Cuilinn >> Product Development >> ChangingWorlds - A Unit of Amdocs Interactive >> t: +353 1 4401268 | niall.ocuil...@changingworlds.com >> >> AMDOCS > CUSTOMER EXPERIENCE SYSTEMS INNOVATION >> >> >> This message and the information contained herein is proprietary and >> confidential and subject to the Amdocs policy statement, >> you may review at http://www.amdocs.com/email_disclaimer.asp >>
Re: [squid-users] Squid3 / ICAP question
I think this is because Squid has received at least one bad response from your ICAP server while it was down. As a result it has decided not to talk to your ICAP server for a period of time If you want Squid to ignore outages on your ICAP server and always attempt a request you can set icap_service_failure_limit -1 Setting this to -1 tells Squid to always connect to ICAP, setting a positive number indicates the number of failures allowed before Squid stops communicating with Squid. The period of time that Squid stops talking to ICAP is controlled by 'icap_service_revival_delay', it is a minimum of 30 seconds I think. >I'm using Squid 3.0.STABLE19-1 with c-icap - replacing an existing >"sandwich" setup of squid2-dansguardian-squid2. > >To my great amazement, things seem to work flawlessly. > >But... when I'm restarting c-icap, squid reports: > >--- snip --- >The following error was encountered while trying to retrieve the URL: >http://www.google.de/ig?hl=de >ICAP protocol error. >The system returned: [No Error] >This means that some aspect of the ICAP communication failed. >Some possible problems are: >The ICAP server is not reachable. >An Illegal response was received from the ICAP server. >--- snip --- > >And that's although c-icap is already running again. >Issuing an "squid3 -k reconfigure" solves the problem. > >But why? How can I prevent the need to do that? > >-- >Ralf Hildebrandt > Geschäftsbereich IT | Abteilung Netzwerk > Charité - Universitätsmedizin Berlin > Campus Benjamin Franklin > Hindenburgdamm 30 | D-12203 Berlin > Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 > ralf.hildebra...@charite.de | http://www.charite.de > This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at http://www.amdocs.com/email_disclaimer.asp
[squid-users] Squid 3.1 ICAP Issue with REQMOD 302
Hi, I have recently moved from Squid 3.0 to Squid 3.1. I am trying to integrate it with an ICAP server. I am having a problem where Squid 3.1 is rejecting some responses from the ICAP server which Squid 3.0 accepted. The response in question is a REQMOD response where the ICAP server is returning a HTTP 302 response rather than modifying the original HTTP request. Here is the ICAP request and response: ICAP Request from Squid: REQMOD icap://10.1.1.25:1344/reqmod ICAP/1.0\r\n Host: 10.1.1.25:1344\r\n Date: Mon, 12 Apr 2010 14:25:39 GMT\r\n Encapsulated: req-hdr=0, null-body=398\r\n Allow: 204\r\n \r\n GET http://c.proxy.com/www.test.com/ HTTP/1.1\r\n Host: c.proxy.com\r\n User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3\r\n Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n Accept-Language: en-gb,en;q=0.5\r\n Accept-Encoding: gzip,deflate\r\n Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n Pragma: no-cache\r\n Cache-Control: no-cache\r\n \r\n Response from ICAP Server: ICAP/1.0 200 OK\r\n Date: Mon, 12 Apr 2010 14:25:15 GMT\r\n Connection: keep-alive\r\n ISTag: "ReqModService"\r\n Encapsulated: res-hdr=0,null-body=160\r\n \r\n HTTP/1.x 302 Found\r\n content-type: text/html\r\n location: https://localhost:8443/mib/authentication\r\n \r\n \r\n Squid displays an ICAP error in the browser and states that an illegal response was received from the ICAP server. Any ideas what might be wrong? Although the ICAP server worked correctly with Squid 3.0 I am open to the possibility that the issue is with the ICAP response and that the old Squid was simply more tolerant than v3.1. Thanks in advance, Niall Niall Ó Cuilinn Product Development ChangingWorlds - A Unit of Amdocs Interactive t: +353 1 4401268 | niall.ocuil...@changingworlds.com AMDOCS > CUSTOMER EXPERIENCE SYSTEMS INNOVATION This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at http://www.amdocs.com/email_disclaimer.asp
