Re: [squid-users] ICAP Bypassing Causing Performance Issues
On 23/08/11 17:31, Justin Lawler wrote: Thanks Amos - regex pattern we're using is: .*some_url_end.html$ We also have many individual domains which we're bypassing acl bypassIcapRequest dstdomain "/apps/cwapps/squid-3/etc/byPass_ICAP_request.properties" icap_access class_1 deny bypassIcapRequest as time has gone on - we've been adding more URLs to this list also (currently up to 39 URLs) - this won't be doing regular expression matching, but we've seen as time goes on, more and more established connections on ICAP server port. Also CPU usage going up, and we're seeing more 'essential ICAP service is down' errors in the logs. True, that test is faster. "essential ICAP service is down" means the ICAP service is not responding well to the traffic load. So I'm thinking the CPU strain is probably at that end, or possibly some problem with the communication between the two. Retries could be adding to the problem. If it is actually an easily bypassable service try using the bypass configuration option instead of ACLs. That 0 on the service line to 1. That will allow all those requests getting failure messages to bypass the service and get unchanged content straight form the origin. I know 3.0 has ICAP, but regard it ass early days in the implementation. There have been a lot of advances and optimizations added in the later series. Try the newest version you can then contact The Measurement Factory who manage that code. They might be of more help if the problem remains. Traffic has not changed significantly - in fact has maybe gone down. The only change we can really identify is the extra bypassed domains. Does squid parse the properties file for every hit? No the file is compiled into memory structures. Each regex pattern is tested individually though, which is time consuming. Also, we've only been reconfiguring squid when we update this file. Is this enough, or do we need to restart? That is fine. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.14 Beta testers wanted for 3.2.0.10
Re: [squid-users] ICAP Bypassing Causing Performance Issues
On 23/08/11 18:21, Ralf Hildebrandt wrote: * Justin Lawler: Thanks Amos - regex pattern we're using is: .*some_url_end.html$ Wouldn't some_url_end.html$ be more efficient? Yes, a lot more. The .* is redundant and forces look-ahead/behind to happen. Which raises the CPU requirements. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.14 Beta testers wanted for 3.2.0.10
Re: [squid-users] ICAP Bypassing Causing Performance Issues
* Justin Lawler : > Thanks Amos - regex pattern we're using is: > > .*some_url_end.html$ Wouldn't some_url_end.html$ be more efficient?
RE: [squid-users] ICAP Bypassing Causing Performance Issues
Thanks Amos - regex pattern we're using is: .*some_url_end.html$ We also have many individual domains which we're bypassing acl bypassIcapRequest dstdomain "/apps/cwapps/squid-3/etc/byPass_ICAP_request.properties" icap_access class_1 deny bypassIcapRequest as time has gone on - we've been adding more URLs to this list also (currently up to 39 URLs) - this won't be doing regular expression matching, but we've seen as time goes on, more and more established connections on ICAP server port. Also CPU usage going up, and we're seeing more 'essential ICAP service is down' errors in the logs. Traffic has not changed significantly - in fact has maybe gone down. The only change we can really identify is the extra bypassed domains. Does squid parse the properties file for every hit? Also, we've only been reconfiguring squid when we update this file. Is this enough, or do we need to restart? Will look into extra debugging now. Thanks and regards, Justin -Original Message- From: Amos Jeffries [mailto:squ...@treenet.co.nz] Sent: Monday, August 22, 2011 10:29 PM To: squid-users@squid-cache.org Subject: Re: [squid-users] ICAP Bypassing Causing Performance Issues On 23/08/11 00:03, Justin Lawler wrote: > Hi, > > We have had to put in a number of URLs to the squid bypass > > icap_service service_1 reqmod_precache 0 icap://127.0.0.1:1344/reqmod > icap_class class_1 service_1 > > acl bypassIcapRequestURLregex urlpath_regex > "./squid-3/etc/byPass_ICAP_request_URLregex.properties" > icap_access class_1 deny bypassIcapRequestURLregex > > > When we added 4 regular expressions to this file, we started to see the CPU > usage going up quite a bit, and we started to see the number of established > connections from squid to ICAP server double or triple. > > Is this a known issue? Is there a better/more efficient way to bypass ICAP > than above? Other than using other ACL types, no. > > Regular expressions were very simple, just matching end of URLs. a) regex is a bit slow. Did you remember to anchor the ends? and manually aggregate the patterns? avoid extended-regex pattern tricks? b) URLs can be many KB in length. That can make URL regex very CPU intensive. d) routing selection ACLs are run multiple times per request. You can turn on access control debugging (level 28,3) to see how many times those are run and how long they take each test. > > We're running squid 3.0.15 on Solaris 10. > Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.14 Beta testers wanted for 3.2.0.10 This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at http://www.amdocs.com/email_disclaimer.asp
Re: [squid-users] ICAP Bypassing Causing Performance Issues
On 23/08/11 00:03, Justin Lawler wrote: Hi, We have had to put in a number of URLs to the squid bypass icap_service service_1 reqmod_precache 0 icap://127.0.0.1:1344/reqmod icap_class class_1 service_1 acl bypassIcapRequestURLregex urlpath_regex "./squid-3/etc/byPass_ICAP_request_URLregex.properties" icap_access class_1 deny bypassIcapRequestURLregex When we added 4 regular expressions to this file, we started to see the CPU usage going up quite a bit, and we started to see the number of established connections from squid to ICAP server double or triple. Is this a known issue? Is there a better/more efficient way to bypass ICAP than above? Other than using other ACL types, no. Regular expressions were very simple, just matching end of URLs. a) regex is a bit slow. Did you remember to anchor the ends? and manually aggregate the patterns? avoid extended-regex pattern tricks? b) URLs can be many KB in length. That can make URL regex very CPU intensive. d) routing selection ACLs are run multiple times per request. You can turn on access control debugging (level 28,3) to see how many times those are run and how long they take each test. We're running squid 3.0.15 on Solaris 10. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.14 Beta testers wanted for 3.2.0.10
