[squid-users] Forwarding loop after rebooting.
Hi group, my first post so please be gentle :) I'm a sysadmin who has inherited a small cluster of squid servers, the setup is as follows. 4 x Squid Slave Accelerators that accel a master squid. 1 x Master Squid running a custom made redirect script written in perl that accel a Webserver . 1 x Backend Webserver. Each slave is running 4 versions of squid accelerating separate sites. The master runs 4 instances of squid. The farm is constantly under a fair load - roughly half a million hits a day. The setup works fine, however, recently when the master server was taken down for repair, and brought back up again with the same configuration, it failed to serve content for the busiest instance, and every request returned is with a TCP_DENIED 403 error. The following error was reported in the cache.log 2006/03/18 06:04:52| WARNING: Forwarding loop detected for: GET /folder1/subfolder/subfolder/ HTTP/1.0 If-Modified-Since: Sat, 14 Jan 2006 01:44:45 GMT Host: 192.168.0.10 Accept: */* From: googlebot(at)googlebot.com User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Accept-Encoding: gzip Via: 1.1 slave1.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver) This has happened previously when the server rebooted, it is likely that the master squid service is getting hammered by all slaves as soon as it is brought back into service, could the fact that it's under such heavy load as soon as it starts up be causing a problem in Squid? Squid version:squid-2.5.STABLE10 O/S: 5.8 Generic_117350-12 sun4u sparc SUNW,Ultra-80 I have altered the output to respect privacy of client.
[squid-users] Forwarding loop after rebooting.
Sorry if this a double post. Squid version:squid-2.5.STABLE10 O/S: 5.8 Generic_117350-12 sun4u sparc SUNW,Ultra-80 Hi, I'm a sysadmin who has inherited a small cluster of squid servers, the setup is as follows. 4 x Squid Slave Accelerators that accel a master squid. 1 x Master Squid running a custom made redirect script written in perl that accel a Webserver . 1 x Backend Webserver. Each slave is running 4 versions of squid accelerating separate sites. The master runs 4 instances of squid. The farm is constantly under a fair load - roughly half a million hits a day. The setup works fine, however, recently when the master server was taken down for repair, and brought back up again with the same configuration, it failed to serve content for the busiest instance, and every request returned is with a TCP_DENIED 403 error. The following error was reported in the cache.log 2006/03/18 06:04:52| WARNING: Forwarding loop detected for: GET /folder1/subfolder/subfolder/ HTTP/1.0 If-Modified-Since: Sat, 14 Jan 2006 01:44:45 GMT Host: 192.168.0.10 Accept: */* From: googlebot(at)googlebot.com User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Accept-Encoding: gzip Via: 1.1 slave1.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver), 1.0 master.mydomain.com:80 (webserver/webserver) This has happened previously when the server rebooted, it is likely that the master squid service is getting hammered by all slaves as soon as it is brought back into service, could the fact that it's under such heavy load as soon as it starts up be causing a problem in Squid? I have altered the output to respect privacy of client.
Re: [squid-users] Forwarding loop after rebooting.
Thanks for the reply's, Mark the F.A.Q link you posted was my first point of call :), I understand I am experiencing a loop, and thanks to the documentation have a better understanding of what they are, I'm just unsure why this only happens when the machine is rebooted. Henrik, I will perform further testing against the redirect rules, however what I am finding strange is that the problem only happens after downtime, to resolve the problem I used an alternative redirect_rules file with the same squid.conf file, and the looping errors go away, The first time this happened we ran the 'alternative' redirect rules for a few days, and then my colleague re-introduced the redirect_rule file that failed on reboot, and restarted squid , shutting down cleanly, and waiting for all processes to die off, the site functioned without error, and no looping errors, until the reboot, then the exact same problem occurred. so in short, we have a redirector that works without error up until the falls out of the farm for a short period of time. Thanks again. On 18/03/06, Henrik Nordstrom <[EMAIL PROTECTED]> wrote: > lör 2006-03-18 klockan 13:47 + skrev Mark Stevens: > > > This has happened previously when the server rebooted, it is likely > > that the master squid service is getting hammered by all slaves as > > soon as it is brought back into service, could the fact that it's > > under such heavy load as soon as it starts up be causing a problem in > > Squid? > > No. > > It's by 99.9% a configuration error. > > Forwarding loops occurs when the configuration in how Squid should route > the requests makes Squid send the request to itself. > > Hmm.. you mentioned you are using a redirector to route the requests. If > so then make sure you have not enabled redirector_bypass (defaults off). > Also verify that the redirector is actually working. > > Regards > Henrik > > > > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.2.2 (GNU/Linux) > > iD8DBQBEHC/C516QwDnMM9sRAgEzAJ4mwq/OBh7ua/v8aoi1myF6vGy+mwCfbnhp > qFZqdcXnzB0PZXA77BdI3dE= > =AP0Y > -END PGP SIGNATURE- > > >
[squid-users] Customizable log format patch.
Hi, I'm currently using Squid Stable 2.5.9 on SPARC/Solaris 2.8, in my Live enviroment, but plan upgrade the Squids to Latest version 2.5.13. In my development environment I've built Squid 2.5.13 with the 'Customizable log format'patch obtained from http://devel.squidcache.org/old_projects.html#customlog, everything other than the time offset seem fine, as you suggested in another post I will manually specify the date time, and leave out the offset, hopefully Webtrends won't have any issues with this. However I would be grateful if you could answer the following questions. Will future releases of squid be compatible with the Customizable log patch up until Squid release 3? we now have a requirement to log Referrer and User agent, but am also keen to run the most recent stable version. Will Squid 3 go to a stable release anytime soon? TIA Mark S.
Re: [squid-users] Forwarding loop after rebooting.
Hello again, I've managed to replicate the error in a development environment. My setup in dev is 2 squids accelerating a master squid, that is accelerating a webserver. The 2 child squids are behind a loadbalancer. To reproduce the problem, I shutdown the master squid, and generate HTTP load to the child squids via the load balancer, then after about 5 minutes start up the master squid, here is an example of the response after sending a valid query that worked prior to replication test. HTTP Request generated by wget: Connecting to myurl.mydomain.com[172.23.161.100]:80... connected. HTTP request sent, awaiting response... 1 HTTP/1.0 403 Forbidden 2 Server: squid/2.5.STABLE12 3 Mime-Version: 1.0 4 Date: Sun, 23 Apr 2006 22:24:23 GMT 5 Content-Type: text/html 6 Content-Length: 1101 7 Expires: Sun, 23 Apr 2006 22:24:23 GMT 8 X-Squid-Error: ERR_ACCESS_DENIED 0 9 X-Cache: MISS from master.mydomain.net 10 X-Cache: MISS from master.mydomain.net 11 X-Cache: MISS from sibling1.object1.com 12 Connection: close 22:18:40 ERROR 403: Forbidden. Extract from cache.log: 2006/04/23 23:24:23| The request GET http://myurl.mydomain.com:80/myfolder1/ is ALLOWED, because it matched 'all' 2006/04/23 23:24:23| clientAccessCheck: proxy request denied in accel_only mode 2006/04/23 23:24:23| The request GET http://myurl.mydomain.com/myfolder1/ is DENIED, because it matched 'all' 2006/04/23 23:24:23| storeEntryValidLength: 233 bytes too big; '8E293D7F9154EF3C2032A87976FAFCA1' 2006/04/23 23:24:23| clientReadRequest: FD 215: no data to process ((11) Resource temporarily unavailable) 2006/04/23 23:24:23| The reply for GET http://myurl.mydomain.com/myfolder1/ is ALLOWED, because it matched 'all' Access log extract: 10.1.1.3 - - [23/Apr/2006:23:24:23 +0100] "GET http://myurl.mydomain.com/myfolder1/ HTTP/1.0" 403 1401 TCP_DENIED:NONE 10.1.1.3 - - [23/Apr/2006:23:24:23 +0100] "GET http://myurl.mydomain.com/myfolder1/ HTTP/1.0" 403 1427 TCP_MISS:FIRST_UP_PARENT I have managed to remove the forwarding loop error by instructing squid not to accept requests via itself as recommended, but the content error still exists. My config doesn't contain a negative ttl entry, so I assume it is the default 5 minutes. Any ideas? TIA. Mark. On 18/03/06, Henrik Nordstrom <[EMAIL PROTECTED]> wrote: > lör 2006-03-18 klockan 19:23 + skrev Mark Stevens: > > > I will perform further testing against the redirect rules, however > > what I am finding strange is that the problem only happens after > > downtime, to resolve the problem I used an alternative redirect_rules > > file with the same squid.conf file, and the looping errors go away, > > How your redirector processes it's rules or not is not a Squid > issue/concern. Squid relies on the redirector of your choice to do it's > job. > > Maybe your redirector is relying on some DNS lookups or something else > not yet available at the time you start Squid in the system bootup > procedure? Have seen people bitten by such issues in the past. > > Regards > Henrik > > > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.2.2 (GNU/Linux) > > iD8DBQBEHIgc516QwDnMM9sRAmx4AJ42AEoQYYVnbfdoZfa5JjygWHwXBwCfUE+u > qAf9owU+M+NMy7XW6ceOw28= > =MeSV > -END PGP SIGNATURE- > > >
Re: [squid-users] Forwarding loop after rebooting.
Much thanks for replies. I have blocked http_access to all except child squids to prevent exploitation. I'm still a tad confused to why this problem only happens when the master proxy is down for a short period. Maybe the negative hits were causing it to redirect to itself, and then requests were denied when the child squids expected the proxy to act as a proxy and not just an accelerator. An interesting 'gotcha' considering the setup has been running fine for about 8 months. Thanks again! On 24/04/06, Henrik Nordstrom <[EMAIL PROTECTED]> wrote: > sön 2006-04-23 klockan 23:48 +0100 skrev Mark Stevens: > > > > 2006/04/23 23:24:23| clientAccessCheck: proxy request denied in accel_only > > mode > > This is important... your Squid is used as a peer proxy, but your > configuration does not allow this Squid to be used as a proxy (only > accelerator). > > > Access log extract: > > > > 10.1.1.3 - - [23/Apr/2006:23:24:23 +0100] "GET > > http://myurl.mydomain.com/myfolder1/ HTTP/1.0" 403 1401 > > TCP_DENIED:NONE > > 10.1.1.3 - - [23/Apr/2006:23:24:23 +0100] "GET > > http://myurl.mydomain.com/myfolder1/ HTTP/1.0" 403 1427 > > TCP_MISS:FIRST_UP_PARENT > > Looks to me like your Squid uses itself as parent. > > What cache_peer statements do you have? Do any of these points back to > yourself either directly or indirectly via cache_peer statements at that > peer? > > > Related note: If you have multiple Squids clustered by the same visible > name, make sure each have a unique unique_hostname set. > > Regards > Henrik > > > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.3 (GNU/Linux) > > iD8DBQBETAiz516QwDnMM9sRAn+hAJ9CGC4QjX6NvVEXcs3rLsDGOc7UCgCff1LH > QVV+ANArd02yRSyXBgiNGsM= > =5Ets > -END PGP SIGNATURE- > > >
[squid-users] Custom log patch - Squid version 2.5.10 Stable.
Hi, We want to implement the custom log patch on our existing squids before upgrading. The patch installed with the following errors. patch -p1 < customlog-2_5.patch patching file src/access_log.c patching file src/cache_cf.c Hunk #2 succeeded at 2639 (offset 6 lines). patching file src/cf.data.pre Hunk #1 succeeded at 833 (offset -1 lines). Hunk #2 succeeded at 2511 (offset -10 lines). patching file src/client_side.c Hunk #1 succeeded at 850 (offset -21 lines). Hunk #3 succeeded at 892 (offset -21 lines). Hunk #4 FAILED at 2020. Hunk #5 FAILED at 2054. 2 out of 5 hunks FAILED -- saving rejects to file src/client_side.c.rej patching file src/icp_v2.c patching file src/logfile.c patching file src/protos.h patching file src/structs.h Hunk #3 succeeded at 620 (offset -4 lines). Hunk #5 succeeded at 2204 (offset -11 lines). patching file src/typedefs.h Here is a cut and paste of the src/client_side.c.rej Rejects file. *** *** 2010,2017 http->entry = clientCreateStoreEntry(http, http->request->method, null_request_flags); errorAppendEntry(http->entry, err); - httpReplyDestroy(http->reply); - http->reply = NULL; memFree(buf, MEM_CLIENT_SOCK_BUF); return; } --- 2020,2025 http->entry = clientCreateStoreEntry(http, http->request->method, null_request_flags); errorAppendEntry(http->entry, err); memFree(buf, MEM_CLIENT_SOCK_BUF); return; } *** *** 2046,2053 http->entry = clientCreateStoreEntry(http, http->request->method, null_request_flags); errorAppendEntry(http->entry, err); - httpReplyDestroy(http->reply); - http->reply = NULL; memFree(buf, MEM_CLIENT_SOCK_BUF); return; } --- 2054,2059 http->entry = clientCreateStoreEntry(http, http->request->method, null_request_flags); errorAppendEntry(http->entry, err); memFree(buf, MEM_CLIENT_SOCK_BUF); return; } Anyone seen this before? Thanks in advance. Mark.
Re: [squid-users] Custom log patch - Squid version 2.5.10 Stable.
Hi Hendrik, Squid version: 2.5.10 Stable as in Subject. Patch version: This patch is generated from the customlog-2_5 branch of s2_5 in squid Sun Mar 5 03:14:53 2006 GMT. Thanks. Mark. On 29/06/06, Henrik Nordstrom <[EMAIL PROTECTED]> wrote: tor 2006-06-29 klockan 12:17 +0100 skrev Mark Stevens: > Hi, > > We want to implement the custom log patch on our existing squids > before upgrading. Which Squid version? And which version of the patch? > Hunk #5 FAILED at 2054. > 2 out of 5 hunks FAILED -- saving rejects to file src/client_side.c.rej The patch you have is not for the same version of Squid, or you are applying some other conflicting patch. Hand-edit the missing changes per the instructions in the .rej file and it might work. REgards Henrik -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.3 (GNU/Linux) iD8DBQBEo+wYB5pTNio2V7IRAv/8AJ4tijiQVN7UbaDxhEU0SdmWwbcyVQCgqKEj 7InzAf6l/SF81heLG454Iyk= =g4Mq -END PGP SIGNATURE-
[squid-users] Local benchmarking and Forwarding loops.
Hi, Squid version: 2.5 Stable 5. O/S: Red Hat Enterprise Linux AS release 3 (Taroon) I'm struggling to find a method to run a local benchmark of Squid using Apache Benchmark without getting forwarding errors, I think it's because the test is local, it detects it's own hostname in the Via header. The reason I wanting to test locally is to discover results when bypassing limitations of 100MB Ethernet. I have tried to forge the 'Via' header, but it doesn't appear to make any differences. The test I'm running locally is . ./ab -v 5 -H "Via: notme.domain.net" -c 1 -t 120 http://www.realdomain.net/images/image.gif ab = Apache Benchmark. -v = Verbose lvl 5 -H Header -c = Number of multiple requests to make -t = Amount of time to run test Anyone had any success in running local benchmark against Squid I would love to hear.
Re: [squid-users] Local benchmarking and Forwarding loops.
Hi Henrik and thanks for your response. Is there a method to perform local benchmark of http ops without causing a forwarding loop? TIA. Mark. On 11/10/06, Henrik Nordstrom <[EMAIL PROTECTED]> wrote: ons 2006-10-11 klockan 11:14 +0100 skrev Mark Stevens: > I'm struggling to find a method to run a local benchmark of Squid > using Apache Benchmark without getting forwarding errors, I think it's > because the test is local, it detects it's own hostname in the Via > header. That's how it detects the forwarding loop indeed, but that header is added by Squid. If you see a forwarding loop then you most likely have a loop in your setup, having Squid talk to itself. Regards Henrik
