On Sun, 21 Feb 2010 11:43:12 +0000, "J. Webster" <webster_j...@hotmail.com> wrote: > Does this look reasonable?
Mostly. I can just see one operational issue remaining... > I still have the analysis to start with after this point but will use some > linux tools to help with that... > > auth_param basic realm P*****r ProxyServer > auth_param basic credentialsttl 2 hours > auth_param basic program /usr/lib/squid/ncsa_auth /etc/squid/squid_passwd > authenticate_cache_garbage_interval 1 hour > authenticate_ip_ttl 2 hours > #acl all src 0.0.0.0/0.0.0.0 > acl all src all > acl manager proto cache_object > acl localhost src 127.0.0.1 > acl cacheadmin src 88.xxx.xxx.xxx 127.0.0.1 > acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 > acl SSL_ports port 443 > acl Safe_ports port 80 # http > acl Safe_ports port 21 # ftp > acl Safe_ports port 443 # https > acl Safe_ports port 70 # gopher > acl Safe_ports port 210 # wais > acl Safe_ports port 1025-65535 # unregistered ports > acl Safe_ports port 280 # http-mgmt > acl Safe_ports port 488 # gss-http > acl Safe_ports port 591 # filemaker > acl Safe_ports port 777 # multiling http > acl Safe_ports port 1863 # MSN messenger > acl ncsa_users proxy_auth REQUIRED > acl maxuser max_user_ip -s 2 > acl CONNECT method CONNECT > http_access allow manager cacheadmin > http_access deny !Safe_ports > http_access deny CONNECT !SSL_ports > http_access deny to_localhost > http_access deny manager > http_access allow ncsa_users > http_access deny maxuser Um, the maxuser test will not be used yet, because any user who logs in will be accepted by the ncsa_users line. What I'd do here is combine the two: http_access allow !maxuser ncsa_users or if that does not work: http_access allow ncsa_users !maxuser http_access deny !ncsa_users > http_access deny all > icp_access allow all > http_port 8080 > http_port 88.xxx.xxx.xxx:80 > hierarchy_stoplist cgi-bin ? > cache_mem 256MB > maximum_object_size_in_memory 50 KB > cache_replacement_policy heap LFUDA > cache_dir aufs /var/spool/squid 40000 16 256 > maximum_object_size 50 MB > cache_swap_low 90 > cache_swap_high 95 > access_log /var/log/squid/access.log squid > cache_log /var/log/squid/cache.log > buffered_logs on > refresh_pattern ^ftp: 1440 20% 10080 > refresh_pattern ^gopher: 1440 0% 1440 > refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 > refresh_pattern . 0 20% 4320 > quick_abort_min 0 KB > quick_abort_max 0 KB > acl apache rep_header Server ^Apache > broken_vary_encoding allow apache > half_closed_clients off > cache_mgr ***'***.com > cachemgr_passwd ******** all > visible_hostname P*****r ProxyServer > log_icp_queries off > dns_nameservers 208.67.222.222 208.67.220.220 > hosts_file /etc/hosts > memory_pools off > forwarded_for off > client_db off > coredump_dir /var/spool/squid > >> ---------------------------------------- >>> From: webster_j...@hotmail.com >>> To: squ...@treenet.co.nz; squid-users@squid-cache.org >>> Date: Sat, 13 Feb 2010 16:35:29 +0000 >>> Subject: RE: [squid-users] Cache manager analysis >>> >>> >>> Thanks. >>> A few questions on this: >>> (a) when you said this all src all is that meant to be acl src all? >>> (b) Hint 2: if possible, define an ACL or the network ranges where you >>> accept logins. Use it like so >>> The logins are accepted form IP addresses that I never know, it is an >>> external proxy server for geo location so not sure I can do this? logins >>> will only ever by directed to the 88.xxx.xxx.xxx server though? >>> (c) cache_mem 100 MB >>> Bump this up as high as you can go without risking memory swapping. >>> Objects served from RAM are 100x faster than objects not. >>> Where can I view if memeory swapping is happening? >>> (D) maximum_object_size 50 MB >>> Bump this up too. Holding full ISO CDs and windows service packs can >>> boost performance when one is used from the cache. 40GB of disk can >>> store a few. >>> If I increase this, will the server ever try to store streamed video? I >>> had an efficiency problem with the original configuration that came with >>> squid, which meant that streamed video was buffering constantly. Not >>> sure what caused it but with the current config it does not do that. >>> If I increase the cache_mem and max object size do I also need to >>> increase this? >>> maximum_object_size_in_memory 50 KB >>> (E) >>> cache_swap_low 90 >>> cache_swap_high 95 >>> access_log /var/log/squid/access.log squid >>> cache_log /var/log/squid/cache.log >>> buffered_logs on >>> acl QUERY urlpath_regex cgi-bin \? >>> cache deny QUERY >>> >>> Drop the QUERY bits above. It's more than halving the things your Squid >>> can store. >>> Remove the acl and the cache deny? >>> At present, does this stop the cache from storing anything with a ?, ie >>> dynamic pages? >>> What if the same request is made for a dynamic page, will it retrive it >>> from the cache (old page) rather then fetch the new dynamic content? >>> >>> current conf redone below: >>> ---------------------------- >>> auth_param basic realm Proxy server >>> auth_param basic credentialsttl 2 hours >>> auth_param basic program /usr/lib/squid/ncsa_auth >>> /etc/squid/squid_passwd >>> authenticate_cache_garbage_interval 1 hour >>> authenticate_ip_ttl 2 hours >>> #acl all src 0.0.0.0/0.0.0.0 >>> acl src all >>> acl manager proto cache_object >>> acl localhost src 127.0.0.1 >>> acl cacheadmin src 88.xxx.xxx.xxx >>> acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 >>> acl SSL_ports port 443 >>> acl Safe_ports port 80 # http >>> acl Safe_ports port 21 # ftp >>> acl Safe_ports port 443 # https >>> acl Safe_ports port 70 # gopher >>> acl Safe_ports port 210 # wais >>> acl Safe_ports port 1025-65535 # unregistered ports >>> acl Safe_ports port 280 # http-mgmt >>> acl Safe_ports port 488 # gss-http >>> acl Safe_ports port 591 # filemaker >>> acl Safe_ports port 777 # multiling http >>> acl Safe_ports port 1863 # MSN messenger >>> acl ncsa_users proxy_auth REQUIRED >>> acl maxuser max_user_ip -s 2 >>> acl CONNECT method CONNECT >>> http_access allow manager localhost >>> http_access allow manager cacheadmin >>> http_access deny !Safe_ports >>> http_access deny CONNECT !SSL_ports >>> http_access deny to_localhost >>> http_access deny manager >>> http_access allow ncsa_users >>> http_access deny maxuser >>> #http_access allow localhost >>> http_access deny all >>> icp_access allow all >>> http_port 8080 >>> http_port 88.xxx.xxx.xxx:80 >>> hierarchy_stoplist cgi-bin ? >>> cache_mem 100 MB >>> maximum_object_size_in_memory 50 KB >>> cache_replacement_policy heap LFUDA >>> cache_dir aufs /var/spool/squid 40000 16 256 >>> maximum_object_size 50 MB >>> cache_swap_low 90 >>> cache_swap_high 95 >>> access_log /var/log/squid/access.log squid >>> cache_log /var/log/squid/cache.log >>> buffered_logs on >>> #acl QUERY urlpath_regex cgi-bin \? >>> #cache deny QUERY >>> refresh_pattern ^ftp: 1440 20% 10080 >>> refresh_pattern ^gopher: 1440 0% 1440 >>> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 >>> refresh_pattern . 0 20% 4320 >>> quick_abort_min 0 KB >>> quick_abort_max 0 KB >>> acl apache rep_header Server ^Apache >>> broken_vary_encoding allow apache >>> half_closed_clients off >>> cache_mgr a...@aaa.com >>> cachemgr_passwd aaa all >>> visible_hostname ProxyServer >>> log_icp_queries off >>> dns_nameservers 208.67.222.222 208.67.220.220 >>> hosts_file /etc/hosts >>> memory_pools off >>> forwarded_for off >>> client_db off >>> coredump_dir /var/spool/squid >>> >>> ---------------------------------------- >>>> Date: Sat, 13 Feb 2010 18:03:00 +1300 >>>> From: squ...@treenet.co.nz >>>> To: squid-users@squid-cache.org >>>> Subject: Re: [squid-users] Cache manager analysis >>>> >>>> J. Webster wrote: >>>>> What is the best place to start with in cache analysis? >>>>> Would it be cache size, memory object size, IO, etc.? >>>>> I'm looking to optimise the settings for my squid server. >>>> >>>> Step 0) migrate to the latest Squid 2.7 or 3.1 or if possible 2.HEAD >>>> (that one is only nominally beta, it's very stable in reality) >>>> >>>> 1) Start by defining 'optimize' ... are you going to prioritize... >>>> Faster service? >>>> More bandwidth saving? >>>> More client connections? >>>> >>>> 2a) For faster service, look at DNS delays, disk IO delays, maximizing >>>> cacheable objects (dynamic objects etc). >>>> >>>> 2b) For pure bandwidth savings start with a look at object cacheablity. >>>> Check dynamics are being cached, ranges are being fetched in full, etc >>>> >>>> 3) Then profile all the objects stored over a reasonably long period, >>>> looking at size. compare with the age of objects being discarded. >>>> >>>> 3a) tune the storage limits to prioritize the storage locations. giving >>>> priority to RAM, then COSS, then AUFS/diskd. >>>> >>>> 3b) set the storage limits as high as possible to maximize amount of >>>> data stored. anywhere. >>>> >>>> 4) take a good long look at your access controls and in particular the >>>> types speedy/fast/slow. You may get some speed benefits from fixing up >>>> the ordering a bit. regex are killers, remote lookups (helpers, or DNS) >>>> are second worst. >>>> (some performance hints below) >>>> >>>> 5) repeat from (2b) as often as possible. concentrate traffic which >>>> seems to logically be storeable but gets a TCP_MISS anyway. >>>> >>>> Objects served from cache lead to faster service ties for those >>>> objects, >>>> so the speed vs bandwidth are inter-related somewhat. But there is a >>>> tipping point somewhere where tuning one starts to impact the other. >>>> >>>> >>>>> >>>>> Server: about 220GB available for the cache, I'm only using 40000 MB >>>>> at present as in the config below. >>>>> system D2812-A2 >>>>> /0 bus D2812-A2 >>>>> /0/0 memory 110KiB BIOS >>>>> /0/4 processor Intel(R) Core(TM)2 Duo CPU E7300 @ 2.66GHz >>>>> /0/4/5 memory 64KiB L1 cache >>>>> /0/4/6 memory 3MiB L2 cache >>>>> /0/4/0.1 processor Logical CPU >>>>> /0/4/0.2 processor Logical CPU >>>>> /0/7 memory 3MiB L3 cache >>>>> /0/2a memory 1GiB System Memory >>>>> /0/2a/0 memory 1GiB DIMM DDR2 Synchronous 667 MHz (1.5 ns) >>>>> /0/2a/1 memory DIMM DDR2 Synchronous 667 MHz (1.5 ns) [empty] >>>>> /0/2a/2 memory DIMM DDR2 Synchronous 667 MHz (1.5 ns) [empty] >>>>> /0/2a/3 memory DIMM DDR2 Synchronous 667 MHz (1.5 ns) [empty] >>>>> /0/1 processor >>>>> /0/1/0.1 processor Logical CPU >>>>> /0/1/0.2 processor Logical CPU >>>>> >>>>> >>>>> Current squid.conf: >>>>> --------------------- >>>>> auth_param basic realm Proxy server >>>>> auth_param basic credentialsttl 2 hours >>>>> auth_param basic program /usr/lib/squid/ncsa_auth >>>>> /etc/squid/squid_passwd >>>>> authenticate_cache_garbage_interval 1 hour >>>>> authenticate_ip_ttl 2 hours >>>>> acl all src 0.0.0.0/0.0.0.0 >>>> >>>> all src all >>>> >>>>> acl manager proto cache_object >>>>> acl localhost src 127.0.0.1/255.255.255.255 >>>> >>>> acl localhost src 127.0.0.1 >>>> >>>>> acl cacheadmin src 88.xxx.xxx.xxx >>>>> acl to_localhost dst 127.0.0.0/8 >>>> >>>> acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 >>>> >>>>> acl SSL_ports port 443 >>>>> acl Safe_ports port 80 # http >>>>> acl Safe_ports port 21 # ftp >>>>> acl Safe_ports port 443 # https >>>>> acl Safe_ports port 70 # gopher >>>>> acl Safe_ports port 210 # wais >>>>> acl Safe_ports port 1025-65535 # unregistered ports >>>>> acl Safe_ports port 280 # http-mgmt >>>>> acl Safe_ports port 488 # gss-http >>>>> acl Safe_ports port 591 # filemaker >>>>> acl Safe_ports port 777 # multiling http >>>>> acl Safe_ports port 1863 # MSN messenger >>>>> acl ncsa_users proxy_auth REQUIRED >>>>> acl maxuser max_user_ip -s 2 >>>>> acl CONNECT method CONNECT >>>>> http_access allow manager localhost >>>>> http_access allow manager cacheadmin >>>> >>>> Hint: add the localhost IP to the cacheadmin ACL and drop one full set >>>> of "allow manager localhost" tests. >>>> >>>>> http_access deny manager >>>>> http_access allow ncsa_users >>>> >>>> Hint: drop the authentication down ... >>>> >>>>> http_access deny !Safe_ports >>>>> http_access deny CONNECT !SSL_ports >>>>> http_access deny to_localhost >>>> >>>> ... to here. All the attacks against your proxy for bad ports and >>>> sources will be dropped quickly by the security blanket settings. Load >>>> on your auth server will reduce and may speed up it's response time. >>>> >>>> Hint 2: if possible, define an ACL or the network ranges where you >>>> accept logins. Use it like so: >>>> >>>> http_access allow localnet ncsa_users >>>> >>>> ... once again that speeds up the rejections, and helps by reducing >>>> the number of times the slow auth lookup needs checking. >>>> >>>>> http_access deny maxuser >>>>> http_access allow localhost >>>> >>>> If localhost really is allowed to do anything, move it up above the >>>> "to_localhost" one. >>>> Otherwise drop this completely, having the correct auth login details >>>> will permit links from localhost just as easily as from anywhere else. >>>> >>>>> http_access deny all >>>>> icp_access allow all >>>> >>>> Define the networks where peer siblings are trusted. Allwo them and >>>> deny >>>> everything else. >>>> That will reduce a fair bit of load on your Squid trying to service >>>> random ICP requests from the general Internet. >>>> >>>>> http_port 8080 >>>>> http_port 88.xxx.xxx.xxx:80 >>>>> hierarchy_stoplist cgi-bin ? >>>>> cache_mem 100 MB >>>> >>>> Bump this up as high as you can go without risking memory swapping. >>>> Objects served from RAM are 100x faster than objects not. >>>> >>>>> maximum_object_size_in_memory 50 KB >>>>> cache_replacement_policy heap LFUDA >>>>> cache_dir aufs /var/spool/squid 40000 16 256 >>>> >>>> If you pick 2.x squid to upgrade to, add a COSS directory as well. >>>> See the recent threads on optimizing COSS for how to tune that. >>>> >>>>> maximum_object_size 50 MB >>>> >>>> Bump this up too. Holding full ISO CDs and windows service packs can >>>> boost performance when one is used from the cache. 40GB of disk can >>>> store a few. >>>> >>>>> cache_swap_low 90 >>>>> cache_swap_high 95 >>>>> access_log /var/log/squid/access.log squid >>>>> cache_log /var/log/squid/cache.log >>>>> buffered_logs on >>>>> acl QUERY urlpath_regex cgi-bin \? >>>>> cache deny QUERY >>>> >>>> Drop the QUERY bits above. It's more than halving the things your Squid >>>> can store. >>>> >>>>> refresh_pattern ^ftp: 1440 20% 10080 >>>>> refresh_pattern ^gopher: 1440 0% 1440 >>>> >>>> Add right here: >>>> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 >>>> >>>>> refresh_pattern . 0 20% 4320 >>>>> quick_abort_min 0 KB >>>>> quick_abort_max 0 KB >>>>> acl apache rep_header Server ^Apache >>>>> broken_vary_encoding allow apache >>>>> half_closed_clients off >>>>> cache_mgr a...@aaa.com >>>>> cachemgr_passwd aaa all >>>>> visible_hostname ProxyServer >>>>> log_icp_queries off >>>>> dns_nameservers 208.67.222.222 208.67.220.220 >>>>> hosts_file /etc/hosts >>>>> memory_pools off >>>> >>>> Might cause efficiency problems if the underlying malloc is not >>>> optimized. but oh well, up to you. >>>> >>>>> forwarded_for off >>>>> client_db off >>>>> coredump_dir /var/spool/squid >>>>> >>>> >>>> >>>> Amos >>>> -- >>>> Please be using >>>> Current Stable Squid 2.7.STABLE7 or 3.0.STABLE23 >>>> Current Beta Squid 3.1.0.16 >>> >>> _________________________________________________________________ >>> Send us your Hotmail stories and be featured in our newsletter >>> http://clk.atdmt.com/UKM/go/195013117/direct/01/ >> >> _________________________________________________________________ >> Send us your Hotmail stories and be featured in our newsletter >> http://clk.atdmt.com/UKM/go/195013117/direct/01/ > > _________________________________________________________________ > We want to hear all your funny, exciting and crazy Hotmail stories. Tell > us now > http://clk.atdmt.com/UKM/go/195013117/direct/01/
