Re: [squid-users] Trying to improve the Byte Hit Ratio, any tips ?
On Sat, Jan 10, 2009 at 12:43 PM, Vianney Lejeune wrote: >>> >>> By the way, what about the ideal settings for cache_mem, cache size and >>> so >>> on, is there any formula ? Are 2*500 GB HD faster than 1*1TB ? >> >> Yes, as each of those can handle i/o operations concurrently. In >> general, the more disks the better the performance: squid performance >> is usually constrained by the disk head seek times. >> >> See http://wiki.squid-cache.org/SquidFaq/RAID > > Thank you, and what about the formula for cache_mem, cache_size etc ? Everything should be quite well-documented in the FAQ and/or KnowledgeBase. Please refer to that first. -- /kinkie
Re: [squid-users] Trying to improve the Byte Hit Ratio, any tips ?
On Thu, Jan 8, 2009 at 1:26 PM, Vianney Lejeune wrote: >>> >> > cache_mem 250 MB >>> >>> maximum_object_size_in_memory 50 KB >> >> memory, memory, memory. The more you can throw at the problem the more >> objects can be kept and served while hot. Squid with 64-bit can easily >> handle many GBs of memory cache. (at cost of slow shutdown when it saves the >> hottest to disk for the next round.) >> >> >>> cache_replacement_policy heap LFUDA >> >> Been a while since I looked at these, to maximize bytes you want the >> policy that looks at object size as well as 'coldness'. To remove the >> smaller cool objects before the larger equally cool ones. >> >>> cache_dir ufs /data/spool/squid 3 16 256 > > > By the way, what about the ideal settings for cache_mem, cache size and so > on, is there any formula ? Are 2*500 GB HD faster than 1*1TB ? Yes, as each of those can handle i/o operations concurrently. In general, the more disks the better the performance: squid performance is usually constrained by the disk head seek times. See http://wiki.squid-cache.org/SquidFaq/RAID -- /kinkie
Re: [squid-users] Trying to improve the Byte Hit Ratio, any tips ?
> cache_mem 250 MB maximum_object_size_in_memory 50 KB memory, memory, memory. The more you can throw at the problem the more objects can be kept and served while hot. Squid with 64-bit can easily handle many GBs of memory cache. (at cost of slow shutdown when it saves the hottest to disk for the next round.) cache_replacement_policy heap LFUDA Been a while since I looked at these, to maximize bytes you want the policy that looks at object size as well as 'coldness'. To remove the smaller cool objects before the larger equally cool ones. cache_dir ufs /data/spool/squid 3 16 256 By the way, what about the ideal settings for cache_mem, cache size and so on, is there any formula ? Are 2*500 GB HD faster than 1*1TB ?
Re: [squid-users] Trying to improve the Byte Hit Ratio, any tips ?
Hello squid users, I am green when the word squid comes. I have a box running suse 10.3 and i need someone to help me configure squid well. am green at this. Please someone with a few minutes to help me, i will be glad On 1/6/09, Amos Jeffries wrote: > Vianney Lejeune wrote: >> Hello, >> >> I'm trying to improve the Byte Hit Ratio of SquidCache on my >> network. There is 220 computers in the LAN, using internet on a general >> usage basis. The maximum bandwidth is 4Mbps in/out, the total amount of >> data is estimated to be 30 to 60 Gbytes daily. >> >> >> This is the report from cachemgr: >> => >> Average HTTP requests per minute since start:1023.9 >> Average ICP messages per minute since start:0.0 >> Select loop called: 1208577 times, 5.619 ms avg >> Cache information for squid: >> Request Hit Ratios:5min: 37.9%, 60min: 41.1% >> Byte Hit Ratios:5min: 13.2%, 60min: 13.8% (It's quite low, these >> values are usual) >> Request Memory Hit Ratios:5min: 2.0%, 60min: 2.6% (I rebooted >> the server 3 hours ago, this can explain these low values) >> Request Disk Hit Ratios:5min: 41.3%, 60min: 36.3% >> Storage Swap size:27654312 KB >> Storage Mem size:190364 KB >> Mean Object Size:29.65 KB >> Requests given to unlinkd:33035 >> Median Service Times (seconds) 5 min60 min: >> HTTP Requests (All): 0.23230 0.46965 >> Cache Misses: 0.35832 0.72387 >> Cache Hits:0.19742 0.35832 >> Near Hits: 0.20843 0.55240 >> Not-Modified Replies: 0.03829 0.05331 >> DNS Lookups: 0.00094 0.00779 >> ICP Queries: 0.0 0.0 >> <= >> >> This is my squid.conf file: >> => >> >> http_port 3128 transparent >> hierarchy_stoplist cgi-bin ? > >> acl QUERY urlpath_regex cgi-bin \? >> cache deny QUERY > > Without cache peers you can drop the above QEURY acl. > That will raise both hit ratios on semi-dynamic objects. > BUT, see addition to refresh_pattern below... > >> acl apache rep_header Server ^Apache >> broken_vary_encoding allow apache >> maximum_object_size 128 MB > > Re: the above maximum. There may be huge objects going through that can > be cached. > > > cache_mem 250 MB >> maximum_object_size_in_memory 50 KB > > memory, memory, memory. The more you can throw at the problem the more > objects can be kept and served while hot. Squid with 64-bit can easily > handle many GBs of memory cache. (at cost of slow shutdown when it saves > the hottest to disk for the next round.) > >> cache_replacement_policy heap LFUDA > > Been a while since I looked at these, to maximize bytes you want the > policy that looks at object size as well as 'coldness'. To remove the > smaller cool objects before the larger equally cool ones. > >> cache_dir ufs /data/spool/squid 3 16 256 > > Your cache dir is only 30GB. Thats one days traffic or less by your > above statements. For good hit ratios you may need at least 7 days, > preferrably as close to 30 as possible. > > Depending on your OS, AUFS(Linux) or diskd(*BSD) may prove much faster > access than UFS. > >> access_log none >> cache_log none > > The above is generating log file named "none". Would be more useful to > set debug_options ALL,0. If you really don't want to know about the > critical problems that do happen then set filename to /dev/null as well. > >> cache_store_log none >> log_ip_on_direct off >> hosts_file /etc/hosts >> refresh_pattern ^ftp:144020%10080 >> refresh_pattern ^gopher:14400%1440 > > without QUERY acl above, you wil need this right here in the pattern order: > refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 > >> refresh_pattern .020%4320 >> quick_abort_min 0 KB >> quick_abort_max 0 KB >> range_offset_limit 0 KB > > Be careful, but you may want to play at setting these to continue > downloads. (quick_abort -1 KB) > That will cause all partial and restarted downloads to become HIT later. > At risk of some wastage. > >> half_closed_clients off >> shutdown_lifetime 0 seconds >> acl all src 0.0.0.0/0.0.0.0 >> acl manager proto cache_object >> acl localhost src 127.0.0.1/255.255.255.255 >> acl to_localhost dst 127.0.0.0/8 >> acl SSL_ports port 443# https >> acl SSL_ports port 563# snews >> acl SSL_ports port 873# rsync >> acl Safe_ports port 80# http >> acl Safe_ports port 21# ftp >> acl Safe_ports port 443# https >> acl Safe_ports port 70# gopher >> acl Safe_ports port 210# wais >> acl Safe_ports port 1025-65535# unregistered ports >> acl Safe_ports port 280# http-mgmt >> acl Safe_ports port 488# gss-http >> acl Safe_ports port 591# filemaker >> acl Safe_ports port 777# multiling http >> acl Safe_ports port 631# cups >> acl Safe_ports port 873# rsync >> acl Safe_ports port 901# SWAT >> acl purge method PURGE >> acl CONNECT method CONNECT >>
Re: [squid-users] Trying to improve the Byte Hit Ratio, any tips ?
Vianney Lejeune wrote: Hello, I'm trying to improve the Byte Hit Ratio of SquidCache on my network. There is 220 computers in the LAN, using internet on a general usage basis. The maximum bandwidth is 4Mbps in/out, the total amount of data is estimated to be 30 to 60 Gbytes daily. This is the report from cachemgr: => Average HTTP requests per minute since start:1023.9 Average ICP messages per minute since start:0.0 Select loop called: 1208577 times, 5.619 ms avg Cache information for squid: Request Hit Ratios:5min: 37.9%, 60min: 41.1% Byte Hit Ratios:5min: 13.2%, 60min: 13.8% (It's quite low, these values are usual) Request Memory Hit Ratios:5min: 2.0%, 60min: 2.6% (I rebooted the server 3 hours ago, this can explain these low values) Request Disk Hit Ratios:5min: 41.3%, 60min: 36.3% Storage Swap size:27654312 KB Storage Mem size:190364 KB Mean Object Size:29.65 KB Requests given to unlinkd:33035 Median Service Times (seconds) 5 min60 min: HTTP Requests (All): 0.23230 0.46965 Cache Misses: 0.35832 0.72387 Cache Hits:0.19742 0.35832 Near Hits: 0.20843 0.55240 Not-Modified Replies: 0.03829 0.05331 DNS Lookups: 0.00094 0.00779 ICP Queries: 0.0 0.0 <= This is my squid.conf file: => http_port 3128 transparent hierarchy_stoplist cgi-bin ? acl QUERY urlpath_regex cgi-bin \? cache deny QUERY Without cache peers you can drop the above QEURY acl. That will raise both hit ratios on semi-dynamic objects. BUT, see addition to refresh_pattern below... acl apache rep_header Server ^Apache broken_vary_encoding allow apache maximum_object_size 128 MB Re: the above maximum. There may be huge objects going through that can be cached. > cache_mem 250 MB maximum_object_size_in_memory 50 KB memory, memory, memory. The more you can throw at the problem the more objects can be kept and served while hot. Squid with 64-bit can easily handle many GBs of memory cache. (at cost of slow shutdown when it saves the hottest to disk for the next round.) cache_replacement_policy heap LFUDA Been a while since I looked at these, to maximize bytes you want the policy that looks at object size as well as 'coldness'. To remove the smaller cool objects before the larger equally cool ones. cache_dir ufs /data/spool/squid 3 16 256 Your cache dir is only 30GB. Thats one days traffic or less by your above statements. For good hit ratios you may need at least 7 days, preferrably as close to 30 as possible. Depending on your OS, AUFS(Linux) or diskd(*BSD) may prove much faster access than UFS. access_log none cache_log none The above is generating log file named "none". Would be more useful to set debug_options ALL,0. If you really don't want to know about the critical problems that do happen then set filename to /dev/null as well. cache_store_log none log_ip_on_direct off hosts_file /etc/hosts refresh_pattern ^ftp:144020%10080 refresh_pattern ^gopher:14400%1440 without QUERY acl above, you wil need this right here in the pattern order: refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern .020%4320 quick_abort_min 0 KB quick_abort_max 0 KB range_offset_limit 0 KB Be careful, but you may want to play at setting these to continue downloads. (quick_abort -1 KB) That will cause all partial and restarted downloads to become HIT later. At risk of some wastage. half_closed_clients off shutdown_lifetime 0 seconds acl all src 0.0.0.0/0.0.0.0 acl manager proto cache_object acl localhost src 127.0.0.1/255.255.255.255 acl to_localhost dst 127.0.0.0/8 acl SSL_ports port 443# https acl SSL_ports port 563# snews acl SSL_ports port 873# rsync acl Safe_ports port 80# http acl Safe_ports port 21# ftp acl Safe_ports port 443# https acl Safe_ports port 70# gopher acl Safe_ports port 210# wais acl Safe_ports port 1025-65535# unregistered ports acl Safe_ports port 280# http-mgmt acl Safe_ports port 488# gss-http acl Safe_ports port 591# filemaker acl Safe_ports port 777# multiling http acl Safe_ports port 631# cups acl Safe_ports port 873# rsync acl Safe_ports port 901# SWAT acl purge method PURGE acl CONNECT method CONNECT acl ReseauLocal src 10.0.0.0/16 http_access allow manager localhost http_access deny manager http_access allow purge localhost http_access deny purge http_access allow localhost http_access allow ReseauLocal http_access deny all http_reply_access allow all icp_access deny all cache_effective_group proxy httpd_suppress_version_string on via off forwarded_for off log_icp_queries off client_db off coredump_dir /var/spool/squid pipeline_prefetch off <= Do you see something that need to be improved ? Did I miss something?
[squid-users] Trying to improve the Byte Hit Ratio, any tips ?
Hello, I'm trying to improve the Byte Hit Ratio of SquidCache on my network. There is 220 computers in the LAN, using internet on a general usage basis. The maximum bandwidth is 4Mbps in/out, the total amount of data is estimated to be 30 to 60 Gbytes daily. This is the report from cachemgr: => Average HTTP requests per minute since start: 1023.9 Average ICP messages per minute since start:0.0 Select loop called: 1208577 times, 5.619 ms avg Cache information for squid: Request Hit Ratios: 5min: 37.9%, 60min: 41.1% Byte Hit Ratios: 5min: 13.2%, 60min: 13.8% (It's quite low, these values are usual) Request Memory Hit Ratios: 5min: 2.0%, 60min: 2.6% (I rebooted the server 3 hours ago, this can explain these low values) Request Disk Hit Ratios:5min: 41.3%, 60min: 36.3% Storage Swap size: 27654312 KB Storage Mem size: 190364 KB Mean Object Size: 29.65 KB Requests given to unlinkd: 33035 Median Service Times (seconds) 5 min60 min: HTTP Requests (All): 0.23230 0.46965 Cache Misses: 0.35832 0.72387 Cache Hits:0.19742 0.35832 Near Hits: 0.20843 0.55240 Not-Modified Replies: 0.03829 0.05331 DNS Lookups: 0.00094 0.00779 ICP Queries: 0.0 0.0 <= This is my squid.conf file: => http_port 3128 transparent hierarchy_stoplist cgi-bin ? acl QUERY urlpath_regex cgi-bin \? cache deny QUERY acl apache rep_header Server ^Apache broken_vary_encoding allow apache cache_mem 250 MB maximum_object_size 128 MB maximum_object_size_in_memory 50 KB cache_replacement_policy heap LFUDA cache_dir ufs /data/spool/squid 3 16 256 access_log none cache_log none cache_store_log none log_ip_on_direct off hosts_file /etc/hosts refresh_pattern ^ftp: 144020% 10080 refresh_pattern ^gopher:14400% 1440 refresh_pattern . 0 20% 4320 quick_abort_min 0 KB quick_abort_max 0 KB range_offset_limit 0 KB half_closed_clients off shutdown_lifetime 0 seconds acl all src 0.0.0.0/0.0.0.0 acl manager proto cache_object acl localhost src 127.0.0.1/255.255.255.255 acl to_localhost dst 127.0.0.0/8 acl SSL_ports port 443 # https acl SSL_ports port 563 # snews acl SSL_ports port 873 # rsync acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 631 # cups acl Safe_ports port 873 # rsync acl Safe_ports port 901 # SWAT acl purge method PURGE acl CONNECT method CONNECT acl ReseauLocal src 10.0.0.0/16 http_access allow manager localhost http_access deny manager http_access allow purge localhost http_access deny purge http_access allow localhost http_access allow ReseauLocal http_access deny all http_reply_access allow all icp_access deny all cache_effective_group proxy httpd_suppress_version_string on via off forwarded_for off log_icp_queries off client_db off coredump_dir /var/spool/squid pipeline_prefetch off <= Do you see something that need to be improved ? Did I miss something? Regards, Mr Lejeune
