Re: [squid-users] Large ACL problem
Thanks very much Henrik, The cpu usage problem went way down when i started using dstdomain -Chris >>> Henrik Nordstrom <[EMAIL PROTECTED]> 3/28/2007 3:18:30 PM >>> ons 2007-03-28 klockan 15:48 -0400 skrev Chris Rosset: >> I remember your last email pointed me towards looking at the ACL's, >> sounds like the url_regex are most intensive since they pattern match >> on the whole url? >Its not so much the fact that it needs to match on the whole URL as the >fact that it needs to evaluate each and every pattern you have (all 8K >of them) on the whole URL.. >> and the entries in this acl are all like >> \.100percentcash.com$ >That kind of pattern really really should be placed into a dstdomain acl >as .100percentcash.com >> I tried installing squidguard, but had problems installing so far, >Begin by sorting your data into what must be regex patterns and what >fits better into the structured acls (i.e. dstdomain). >REgards >Henrik
Re: [squid-users] Large ACL problem
Thanks very much for your help! the dstdomain change solved my CPU usage prob. >>> Chris Robertson <[EMAIL PROTECTED]> 3/28/2007 2:48:28 PM >>> Chris Rosset wrote: > Hi Henrik, > > >I remember your last email pointed me towards looking at the ACL's, >> sounds like the url_regex are most intensive since they pattern match >> on the whole url? > >> For more info on what i have >> We have a few ACL's most are not more then a few hundred lines > >> but the one big one we have is 8200 lines, mostly spam, spyware, porn >> sites. > > acl never-allow-url dstdom_regex -i > "/usr/local/squid/etc/FilterLists/never-allow-url" > deny_info ERR_BW_CONTENT_SUPPRESSED never-allow-url > > and the entries in this acl are all like > \.100percentcash.com$ > >acl never-allow-domain dstdomain >"/usr/local/squid/etc/FilterLists/never-allow-domain" >deny-info ERR_BW_CONTENT_SUPPRESSED never-allow-domain >With a file content of... >.100percentcash.com >(etc.) >... and obviously a matching http_access deny rule would perform the >same task, with much lower CPU utilization. >> I tried installing squidguard, but had problems installing so far, >> maybe i could try squirm, jesred or just recompile with >> --enable-gnuregex >> Or any other reccomendation? >> >> Thanks very much >> -Chris >> >Chris
Re: [squid-users] Large ACL problem
ons 2007-03-28 klockan 15:48 -0400 skrev Chris Rosset: > I remember your last email pointed me towards looking at the ACL's, > sounds like the url_regex are most intensive since they pattern match > on the whole url? Its not so much the fact that it needs to match on the whole URL as the fact that it needs to evaluate each and every pattern you have (all 8K of them) on the whole URL.. > and the entries in this acl are all like > \.100percentcash.com$ That kind of pattern really really should be placed into a dstdomain acl as .100percentcash.com > I tried installing squidguard, but had problems installing so far, Begin by sorting your data into what must be regex patterns and what fits better into the structured acls (i.e. dstdomain). REgards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: [squid-users] Large ACL problem
Chris Rosset wrote: Hi Henrik, I remember your last email pointed me towards looking at the ACL's, sounds like the url_regex are most intensive since they pattern match on the whole url? For more info on what i have We have a few ACL's most are not more then a few hundred lines but the one big one we have is 8200 lines, mostly spam, spyware, porn sites. acl never-allow-url dstdom_regex -i "/usr/local/squid/etc/FilterLists/never-allow-url" deny_info ERR_BW_CONTENT_SUPPRESSED never-allow-url and the entries in this acl are all like \.100percentcash.com$ acl never-allow-domain dstdomain "/usr/local/squid/etc/FilterLists/never-allow-domain" deny-info ERR_BW_CONTENT_SUPPRESSED never-allow-domain With a file content of... .100percentcash.com (etc.) ... and obviously a matching http_access deny rule would perform the same task, with much lower CPU utilization. I tried installing squidguard, but had problems installing so far, maybe i could try squirm, jesred or just recompile with --enable-gnuregex Or any other reccomendation? Thanks very much -Chris Chris
Re: [squid-users] Large ACL problem
>>> Henrik Nordstrom <[EMAIL PROTECTED]> 3/28/2007 2:04:40 AM >>> tis 2007-03-27 klockan 18:12 -0400 skrev Chris Rosset: >>> Hello, >>> >>> I am still having a issue with high CPU usage. >>> In a previous post it was suggested that my ACL (which is 8k+ lines) >8k+ lines of what? >for both Squid and SquidGuard it's important you use the correct acl >type for the structured data such as host names, domains etc, and only >use regex patterns as a last resort. >The problem with regex is >a) It's CPU intensive to evaluate as the whole list has to be evaluated >on each request only to find that it doesn't match any of the patterns.. >b) Quite memory hungry. >The other ACL types works much more efficiently thanks to their data >being structured allowing the patterns to be sorted and searched >efficiently. >Regards >Henrik Hi Henrik, I remember your last email pointed me towards looking at the ACL's, sounds like the url_regex are most intensive since they pattern match on the whole url? For more info on what i have We have a few ACL's most are not more then a few hundred lines but the one big one we have is 8200 lines, mostly spam, spyware, porn sites. acl never-allow-url dstdom_regex -i "/usr/local/squid/etc/FilterLists/never-allow-url" deny_info ERR_BW_CONTENT_SUPPRESSED never-allow-url and the entries in this acl are all like \.100percentcash.com$ I tried installing squidguard, but had problems installing so far, maybe i could try squirm, jesred or just recompile with --enable-gnuregex Or any other reccomendation? Thanks very much -Chris
Re: [squid-users] Large ACL problem
tis 2007-03-27 klockan 18:12 -0400 skrev Chris Rosset: > Hello, > > I am still having a issue with high CPU usage. > In a previous post it was suggested that my ACL (which is 8k+ lines) 8k+ lines of what? for both Squid and SquidGuard it's important you use the correct acl type for the structured data such as host names, domains etc, and only use regex patterns as a last resort. The problem with regex is a) It's CPU intensive to evaluate as the whole list has to be evaluated on each request only to find that it doesn't match any of the patterns.. b) Quite memory hungry. The other ACL types works much more efficiently thanks to their data being structured allowing the patterns to be sorted and searched efficiently. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: [squid-users] Large ACL problem
i´m using db 4 !! i will check what´s the exact version ... Alexandre On 3/27/07, Chris Nighswonger <[EMAIL PROTECTED]> wrote: On 3/27/07, Chris Rosset <[EMAIL PROTECTED]> wrote: > So just checking if the --enable-gnuregex might help, or should i go > with squidgard or squirm some other redirector? I'm not sure what you are trying to acl, but Dansguardian works nicely for filtering purposes. Chris -- Sds. Alexandre J. Correa Onda Internet / OPinguim.net http://www.ondainternet.com.br http://www.opinguim.net
Re: [squid-users] Large ACL problem
On 3/27/07, Chris Rosset <[EMAIL PROTECTED]> wrote: So just checking if the --enable-gnuregex might help, or should i go with squidgard or squirm some other redirector? I'm not sure what you are trying to acl, but Dansguardian works nicely for filtering purposes. Chris
Re: [squid-users] Large ACL problem
Alexandre, Mind if i ask what version of Berkeley DB your using? I was installing squidguard last week with 4.1, 4.2 and 4.5 checked the squidguard mailing lists and found out you need a older version So i tried 3.3 but squidguard's complaining it still cant find it even with -with-db=DIR set to the right location. Thanks for your help. -Chris >>> "Alexandre Correa" <[EMAIL PROTECTED]> 3/27/2007 3:41:17 PM >>> >>i´m using squid with squidguard .. about 3.000.000 of expressions.. >>squidguard is working veryyy good.. you can try... but my system is >>Linux.. for solaris.. i believe that you can change some codes (or not :P) On 3/27/07, Chris Rosset <[EMAIL PROTECTED]> wrote: > Hello, > > I am still having a issue with high CPU usage. > In a previous post it was suggested that my ACL (which is 8k+ lines) > could be the culprit. > Also i am going to try a newer squid version, currently i am on 2.5.6 > going to try both > squid-2.5.STABLE14 and a squid2.6.x version > > I noticed in the FAQ that using --enable-gnuregex might help on large > ACL's? > http://wiki.squid-cache.org/SquidFaq/SystemWeirdnesses#head-91b0cf15c4781cd7b3248e3663ac85baef66e70f > > > > but the config --h info seems to indicate it might not. > --enable-gnuregex Compile GNUregex. Unless > you have reason to use this > option, you > should not enable it. This library file >is usually > only required on Windows and very old > Unix boxes > which do not have their own regex library >built in. > > I am running Solaris 8 > uname -a > SunOS sfsysdev4 5.8 Generic_117350-45 sun4u sparc SUNW,Ultra-250 > > So just checking if the --enable-gnuregex might help, or should i go > with squidgard or squirm some other redirector? > > Thanks for your help. > -- Sds. Alexandre J. Correa Onda Internet / OPinguim.net http://www.ondainternet.com.br http://www.opinguim.net
Re: [squid-users] Large ACL problem
i´m using squid with squidguard .. about 3.000.000 of expressions.. squidguard is working veryyy good.. you can try... but my system is Linux.. for solaris.. i believe that you can change some codes (or not :P) On 3/27/07, Chris Rosset <[EMAIL PROTECTED]> wrote: Hello, I am still having a issue with high CPU usage. In a previous post it was suggested that my ACL (which is 8k+ lines) could be the culprit. Also i am going to try a newer squid version, currently i am on 2.5.6 going to try both squid-2.5.STABLE14 and a squid2.6.x version I noticed in the FAQ that using --enable-gnuregex might help on large ACL's? http://wiki.squid-cache.org/SquidFaq/SystemWeirdnesses#head-91b0cf15c4781cd7b3248e3663ac85baef66e70f but the config --h info seems to indicate it might not. --enable-gnuregex Compile GNUregex. Unless you have reason to use this option, you should not enable it. This library file is usually only required on Windows and very old Unix boxes which do not have their own regex library built in. I am running Solaris 8 uname -a SunOS sfsysdev4 5.8 Generic_117350-45 sun4u sparc SUNW,Ultra-250 So just checking if the --enable-gnuregex might help, or should i go with squidgard or squirm some other redirector? Thanks for your help. -- Sds. Alexandre J. Correa Onda Internet / OPinguim.net http://www.ondainternet.com.br http://www.opinguim.net
[squid-users] Large ACL problem
Hello, I am still having a issue with high CPU usage. In a previous post it was suggested that my ACL (which is 8k+ lines) could be the culprit. Also i am going to try a newer squid version, currently i am on 2.5.6 going to try both squid-2.5.STABLE14 and a squid2.6.x version I noticed in the FAQ that using --enable-gnuregex might help on large ACL's? http://wiki.squid-cache.org/SquidFaq/SystemWeirdnesses#head-91b0cf15c4781cd7b3248e3663ac85baef66e70f but the config --h info seems to indicate it might not. --enable-gnuregex Compile GNUregex. Unless you have reason to use this option, you should not enable it. This library file is usually only required on Windows and very old Unix boxes which do not have their own regex library built in. I am running Solaris 8 uname -a SunOS sfsysdev4 5.8 Generic_117350-45 sun4u sparc SUNW,Ultra-250 So just checking if the --enable-gnuregex might help, or should i go with squidgard or squirm some other redirector? Thanks for your help.
