Re: [squid-users] Re: Squid monitoring, access report shows upto 5 % to 7 % cache usage
Thanks Augustus for the email my information is --- [root@proxy squid]# squidclient -h 127.0.0.1 mgr:storedir HTTP/1.0 200 OK Server: squid/3.1.10 Mime-Version: 1.0 Date: Sun, 04 Aug 2013 07:01:30 GMT Content-Type: text/plain Expires: Sun, 04 Aug 2013 07:01:30 GMT Last-Modified: Sun, 04 Aug 2013 07:01:30 GMT X-Cache: MISS from proxy X-Cache-Lookup: MISS from proxy:3128 Via: 1.0 proxy (squid/3.1.10) Connection: close Store Directory Statistics: Store Entries : 13649421 Maximum Swap Size : 58368 KB Current Store Swap Size: 250112280 KB Current Capacity : 43% used, 57% free Store Directory #0 (aufs): /opt/var/spool/squid FS Block Size 4096 Bytes First level subdirectories: 32 Second level subdirectories: 256 Maximum Size: 58368 KB Current Size: 250112280 KB Percent Used: 42.85% Filemap bits in use: 13649213 of 16777216 (81%) Filesystem Space in use: 264249784/854534468 KB (31%) Filesystem Inodes in use: 13657502/54263808 (25%) Flags: SELECTED Removal policy: lru LRU reference age: 44.69 days -- and my squid.conf is as -- always_direct allow all cache_log /opt/var/log/squid/cache.log cache_access_log /opt/var/log/squid/access.log cache_swap_low 90 cache_swap_high 95 acl manager proto cache_object acl localhost src 127.0.0.1/32 ::1 acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1 acl localnet src 172.16.5.0/24 # RFC1918 possible internal network acl localnet src 172.17.0.0/22 # RFC1918 possible internal network acl localnet src 192.168.20.0/24 # RFC1918 possible internal network acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines always_direct allow local-servers acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT http_access allow manager localhost http_access deny manager http_access deny !Safe_ports http_access deny CONNECT !SSL_ports acl ipgroup src 172.16.5.1-172.16.5.255/32 acl ipgroup src 172.17.0.10-172.17.3.254/32 delay_pools 1 delay_class 1 2 delay_parameters 1 256/386 14/18 delay_access 1 allow ipgroup delay_access 1 deny all http_access allow localnet http_access allow localhost http_access allow localnet http_access allow localhost http_access deny all http_port 3128 transparent hierarchy_stoplist cgi-bin ? cache_dir aufs /opt/var/spool/squid 57 32 256 coredump_dir /opt/var/spool/squid maximum_object_size 4 GB refresh_pattern -i \.flv$ 10080 90% 99 ignore-no-cache override-expire ignore-private refresh_pattern -i \.(gif|png|jpg|jpeg|ico)$ 10080 90% 43200 override-expire ignore-no-cache ignore-no-store ignore-private refresh_pattern -i \.(iso|avi|wav|mp3|mp4|mpeg|swf|flv|x-flv)$ 43200 90% 432000 override-expire ignore-no-cache ignore-no-store ignore-private refresh_pattern -i \.(deb|rpm|exe|zip|tar|tgz|ram|rar|bin|ppt|doc|tiff)$ 10080 90% 43200 override-expire ignore-no-cache ignore-no-store ignore-private refresh_pattern -i \.index.(html|htm)$ 0 40% 10080 refresh_pattern -i \.(html|htm|css|js)$ 1440 40% 40320 refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 40% 40320 visible_hostname proxy icap_enable on icap_preview_enable on icap_preview_size 4096 icap_persistent_connections on icap_send_client_ip on icap_send_client_username on icap_service qlproxy1 reqmod_precache bypass=0 icap://127.0.0.1:1344/reqmod icap_service qlproxy2 respmod_precache bypass=0 icap://127.0.0.1:1344/respmod adaptation_access qlproxy1 allow all adaptation_access qlproxy2 allow all Guidance and advice requested Thanks for the reply Joseph John - Original Message - From: babajaga augustus_me...@yahoo.de To: squid-users@squid-cache.org Cc: Sent: Thursday, 1 August 2013 2:11 PM Subject: [squid-users] Re: Squid monitoring, access report shows upto 5 % to 7 % cache usage The relatively low byte-hitrate gives the idea, that somewhere in your squid.conf there is a limitation on the max. objects size to be cached. It might be a good idea, to modify this one, to a larger value. Caus it seems, you still have a lot of disk space available for caching. So you might post your squid.conf here. And, the output of squidclient -h 127.0.0.1 mgr:storedir -- View this message in context:
[squid-users] hit percentage in squid
Hi All Going through the squid docs and mailing list email, I see HIT percentage. Is the HIT percentage is the images which takes from the squid cache when user browse Guidance and advice requested thanks Joseph John
Re: [squid-users] Re: Squid monitoring, access report shows upto 5 % to 7 % cache usage
Hi Amos,Ahmad,Babajaga Thanks for your advice and feed back, I am posting more information --- the HIT,MISS,REFRESH details are cat /opt/var/log/squid/access.log | grep -c HIT 13810283 cat /opt/var/log/squid/access.log | grep -c MISS 57874593 cat /opt/var/log/squid/access.log | grep -c REFRESH 6760966 squidclient info result squidclient -h 127.0.0.1 mgr:info HTTP/1.0 200 OK Server: squid/3.1.10 Mime-Version: 1.0 Date: Thu, 01 Aug 2013 06:10:06 GMT Content-Type: text/plain Expires: Thu, 01 Aug 2013 06:10:06 GMT Last-Modified: Thu, 01 Aug 2013 06:10:06 GMT X-Cache: MISS from proxy X-Cache-Lookup: MISS from proxy:3128 Via: 1.0 proxy (squid/3.1.10) Connection: close Squid Object Cache: Version 3.1.10 Start Time: Wed, 31 Jul 2013 23:08:41 GMT Current Time: Thu, 01 Aug 2013 06:10:06 GMT Connection information for squid: Number of clients accessing cache: 574 Number of HTTP requests received: 508778 Number of ICP messages received: 0 Number of ICP messages sent: 0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 1207.3 Average ICP messages per minute since start: 0.0 Select loop called: 62199585 times, 0.407 ms avg Cache information for squid: Hits as % of all requests: 5min: 21.6%, 60min: 27.1% Hits as % of bytes sent: 5min: 6.6%, 60min: 5.6% Memory hits as % of hit requests: 5min: 5.6%, 60min: 9.0% Disk hits as % of hit requests: 5min: 46.9%, 60min: 47.7% Storage Swap size: 255159268 KB Storage Swap capacity: 43.7% used, 56.3% free Storage Mem size: 259940 KB Storage Mem capacity: 100.1% used, 0.0% free Mean Object Size: 18.29 KB Requests given to unlinkd: 0 Median Service Times (seconds) 5 min 60 min: HTTP Requests (All): 0.35832 0.35832 Cache Misses: 0.39928 0.58309 Cache Hits: 0.00179 0.00179 Near Hits: 0.35832 0.32154 Not-Modified Replies: 0.00091 0.00091 DNS Lookups: 0.31806 0.31806 ICP Queries: 0.0 0.0 Resource usage for squid: UP Time: 25284.735 seconds CPU Time: 3138.918 seconds CPU Usage: 12.41% CPU Usage, 5 minute avg: 20.23% CPU Usage, 60 minute avg: 13.25% Process Data Segment Size via sbrk(): 2959564 KB Maximum Resident Size: 12002784 KB Page faults with physical i/o: 5 Memory usage for squid via mallinfo(): Total space in arena: -1234608 KB Ordinary blocks: -1246103 KB 2902 blks Small blocks: 0 KB 0 blks Holding blocks: 35920 KB 6 blks Free Small blocks: 0 KB Free Ordinary blocks: 11494 KB Total in use: -1210183 KB 101% Total free: 11494 KB 0% Total size: -1198688 KB Memory accounted for: Total accounted: -1713375 KB 143% memPool accounted: 2480929 KB 143% memPool unaccounted: -3679618 KB 0% memPoolAlloc calls: 415798933 memPoolFree calls: 418694853 File descriptor usage for squid: Maximum number of file descriptors: 1024 Largest file desc currently in use: 925 Number of file desc currently in use: 821 Files queued for open: 0 Available number of file descriptors: 203 Reserved number of file descriptors: 100 Store Disk files open: 7 Internal Data Structures: 13948023 StoreEntries 18727 StoreEntries with MemObjects 18556 Hot Object Cache Items 13947830 on-disk objects MySAR results are DATE USERS HOSTS SITES BYTES TRAFFICCache Percent Thursday, 01 August 2013 11 797 13899 36.11G 4% Wednesday, 31 July 2013 42 1024 29862 89.22G 5% Tuesday, 30 July 2013 19 1023 27096 85.24G 5% Monday, 29 July 2013 29 1022 26425 82.55G 5% Sunday, 28 July 2013 37 1024 26324 81.79G 6% Saturday, 27 July 2013 21 1020 26877 86.57G 5% Friday, 26 July 2013 31 1017 24811 105.67G 4% Thursday, 25 July 2013 59 1024 29185 119.15G 4% Wednesday, 24 July 2013 41 1022 27245 106.82G 5% Tuesday, 23 July 2013 25 1022 26785 86.68G 5% Monday, 22 July 2013 29 1019 26609 71.94G 6% Sunday, 21 July 2013 23 1019 24863 79.48G 5% Saturday, 20 July 2013 20 1016 24334 77.44G 5% Friday, 19 July 2013 43 1015 25206 72.18G 5% Thursday, 18 July 2013 18 1016 24187 81.28G
[squid-users] refresh_pattern what are the parameter
Hi All I searched for optimising the cache performance, I came to know that you can cache more types of files using 'refresh_pattern' in squid.conf Can I know what are the parameter which are given to refresh_pattern as options eg such as 'refresh_pattern . 0 20% 4320' Guidance and advice requested thanks Joseph john
Re: [squid-users] Squid monitoring, access report shows upto 5 % to 7 % cache usage
Hi Antony thanks for the reply, I have filed up the corresponding information below - Original Message - From: Antony Stone antony.st...@squid.open.source.it To: squid-users@squid-cache.org squid-users@squid-cache.org Cc: Sent: Monday, 29 July 2013 2:45 PM Subject: Re: [squid-users] Squid monitoring, access report shows upto 5 % to 7 % cache usage On Monday 29 July 2013 at 12:36:05, John Joseph wrote: Hi All For purpose of convincing about the squid cahce, I installed and started monitoring SQUID using mysar http://sourceforge.net/projects/mysar/;; Is this % too low or average how much should a optimized squid setup give. Or should I dig into some more configuration to pump the cache % How large is your cache? It is around 556GB, my cache dir details in squid.conf are cache_dir aufs /opt/var/spool/squid 57 32 256 How many users are going via Squid? At present not more than 250 users, but concurrent users will not be more than 50 users How long have they been using it (how long has Squid been building up its cache)? 3 months How many connection requests do you have per minute/hour/day/whatever makes sense? Can you please advice me on how to check for this parameter How many cache HITs do you see in the log file compared to MISSes (ie: how often are cached objects being requested)? Can you please advice me on how to check for this parameter Any other relevant information you can supply might be helpful in providing an answer. from the mysar, I get the following information, which I am pasting here USERS HOSTS SITES TRAFFIC Cache Percent 12 990 18336 47.71G 5.00% 21 1020 26877 86.57G 5.00% 31 1017 24811 105.67G 4.00% 59 1024 29185 119.15G 4.00% 41 1022 27245 106.82G 5.00% 25 1022 26785 86.68G 5.00% 29 1019 26609 71.94G 6.00% 23 1019 24863 79.48G 5.00% 20 1016 24334 77.44G 5.00% 43 1015 25206 72.18G 5.00% 18 1016 24187 81.28G 5.00% 38 1021 25538 80.74G 6.00% 30 1021 27427 85.19G 5.00% 31 1021 27369 96.13G 5.00% 29 1019 23879 74.26G 5.00% 28 1014 20715 70.90G 5.00% 22 1010 22537 69.59G 5.00% 22 1021 23031 86.52G 4.00% 19 1015 22542 73.21G 5.00% 16 1020 22408 74.67G 5.00% 23 1021 23594 72.99G 5.00% 28 1021 23408 71.97G 6.00% 17 1006 21390 64.28G 5.00% 23 994 22685 61.42G 5.00% 20 1016 25792 71.54G 5.00% 24 1017 25178 74.03G 5.00% 37 1019 29740 83.19G 5.00% 22 1020 25175 77.47G 5.00% Guidance and advice requested. Thanks Joseph John Regards, Antony. -- Users don't know what they want until they see what they get. Please reply to the list; please don't CC me.
[squid-users] Squid monitoring, access report shows upto 5 % to 7 % cache usage
Hi All For purpose of convincing about the squid cahce, I installed and started monitoring SQUID using mysar http://sourceforge.net/projects/mysar/; Is this % too low or average how much should a optimized squid setup give. Or should I dig into some more configuration to pump the cache % thanks Joseph John
Re: [squid-users] Evaluating SQUID performance
HI All My Sincere thanks to Amos,Eliezer,Firas,Henrik Based on your advice, I have decided to try out squidclient calamaris msar web-polygraph Right now trying with squidclient, trying out its options and trying to understand. I have another idea which sprang up, I plan to document my test and write a How to do, aiming not for squid proffesionals, but aiming for people like me, who had just tried out squid Thanks to all for the Advice and tips Great mailing list thanks Joseph John
[squid-users] Evaluating SQUID performance
Hi How could I do a test on squid server and check the performance on the bandwidth saved. Is there any tool for the same. I know squid can save bandwith, but I want to convince others with proof Guidance and Advice requested thanks Joseph John
Re: [squid-users] Fwd: Hye I have some problems with my cache
Thanks for the posting. I was going through the archive to check how to customise mysquid instance. this conf settings cache_swap_low 90 cache_swap_high 95 is for older unused cache What should I do, for keeping or restircitng my cahe_dir for fiexed no of days, ie more than 'n' nof of days it should not be cached thanks Joseph John - Original Message - From: Squidblacklist webmas...@squidblacklist.org To: squid-users@squid-cache.org Cc: Sent: Friday, 22 March 2013 8:14 PM Subject: Re: [squid-users] Fwd: Hye I have some problems with my cache Also I use this directive in my conf to ensure squid will purge older unused cache and not run out of disk space cache_swap_low 90 cache_swap_high 95 Not sure if you need that, but it works for me. On Fri, 22 Mar 2013 07:40:01 -0700 Squidblacklist webmas...@squidblacklist.org wrote: You have your cache set to 1024mb in size, this is very small, it is no wonder it is filling quickly. I would increase that.
Re: [squid-users] Fwd: Hye I have some problems with my cache
Thanks Amos I mistook the definition of cache_swap. I was trying to find a solution in which I do not want my cache server to chache any objects which has been accessed 'n' no of days back So that I save the cache objects which are the latest, and indirecly save space [ I am not sure this is a better way or the correct way] Sorry previous post I was not able to put the question in a proper way Your reply is appreiciated Thanks Joseph John - Original Message - From: Amos Jeffries squ...@treenet.co.nz To: squid-users@squid-cache.org Cc: Sent: Monday, 25 March 2013 3:40 PM Subject: Re: [squid-users] Fwd: Hye I have some problems with my cache On 25/03/2013 11:06 p.m., John Joseph wrote: Thanks for the posting. I was going through the archive to check how to customise mysquid instance. this conf settings cache_swap_low 90 cache_swap_high 95 is for older unused cache What do you mean by older in the above? Those settings determine how much empty cache space Squid should keep available to cover bursts in new cacheable objects. What should I do, for keeping or restircitng my cahe_dir for fiexed no of days, ie more than 'n' nof of days it should not be cached Why would you want to do that? you could end up dropping lots of objects a few minutes or hours before they are needed again, having empty space in the cache is not muh benefit. Amos thanks Joseph John - Original Message - From: Squidblacklist webmas...@squidblacklist.org To: squid-users@squid-cache.org Cc: Sent: Friday, 22 March 2013 8:14 PM Subject: Re: [squid-users] Fwd: Hye I have some problems with my cache Also I use this directive in my conf to ensure squid will purge older unused cache and not run out of disk space cache_swap_low 90 cache_swap_high 95 Not sure if you need that, but it works for me. On Fri, 22 Mar 2013 07:40:01 -0700 Squidblacklist webmas...@squidblacklist.org wrote: You have your cache set to 1024mb in size, this is very small, it is no wonder it is filling quickly. I would increase that.
[squid-users] cache_dir sapce has 600GB free space then also I get message WARNING cache_mem is larger than total disk cache space!
Hi All I am using squid 3.1.10 on CentOS 6.4 64 bit OS. I have configered squid.conf for cache_dir to cache_dir /opt/var/spool/squid . by giving cache_dir ufs /opt/var/spool/squid 100 16 256 As per the documentantion it means I am allowcating 100 MB of cache space. And I have /opt/ partiion free size of 600 GB But when I restart the service I get warining message WARNING cache_mem is larger than total disk cache space! I should not be getting this message, because my cache space (ie /opt/var/spool/squid) has about 600 GB of free space. I have searched the archive and did a google search, all most all the resource say I need to increas the cache space. Here I have 600 GB free space I need guidance on why such message is coming when I restart the service thanks Joseph John
Re: [squid-users] cache_dir sapce has 600GB free space then also I get message WARNING cache_mem is larger than total disk cache space!
Thanks Nichols It worked , I stoped the service and then did squid -z now in my squid.conf I have cache_dir aufs /opt/var/spool/squid 57 32 256 I have one doubt, The file type you have advices is 'aufs' while in the sample configuration it is given 'ufs' , should I keeot 'aufs' or 'ufs' Your advice and email is much appreciated Thanks Joseph John - Original Message - From: Squidblacklist webmas...@squidblacklist.org To: squid-users@squid-cache.org Cc: John Joseph jjk_s...@yahoo.com Sent: Thursday, 21 March 2013 10:11 AM Subject: Re: [squid-users] cache_dir sapce has 600GB free space then also I get message WARNING cache_mem is larger than total disk cache space! Try this instead cache_dir aufs /opt/var/spool/squid 57 32 256 then do squid3 -z rebuild the cache, reload squid. Signed, Fix Nichols http://squidblacklist.org On Wed, 20 Mar 2013 23:07:13 -0700 (PDT) John Joseph jjk_s...@yahoo.com wrote: Hi All I am using squid 3.1.10 on CentOS 6.4 64 bit OS. I have configered squid.conf for cache_dir to cache_dir /opt/var/spool/squid . by giving cache_dir ufs /opt/var/spool/squid 100 16 256 As per the documentantion it means I am allowcating 100 MB of cache space. And I have /opt/ partiion free size of 600 GB But when I restart the service I get warining message WARNING cache_mem is larger than total disk cache space! I should not be getting this message, because my cache space (ie /opt/var/spool/squid) has about 600 GB of free space. I have searched the archive and did a google search, all most all the resource say I need to increas the cache space. Here I have 600 GB free space I need guidance on why such message is coming when I restart the service thanks Joseph John
[squid-users] UFS file system, I am using ext4 , is this a wise choice
Hi I am using ext4 file system for squid3 cache_dir, is this choice advisable. Thanks Joseph John
Re: [squid-users] calculating hardware for 900 users for SQUID cache server
Hi Loic Your feedback was quite useful, I was able to come up with some values after checking your configuration. If by 3 years my users base increase to 4000 no, then will it be a over kill if I go with 32 GB ram. Thanks Joseph John - Original Message - From: Loïc Blot loic.b...@unix-experience.fr To: squid-users@squid-cache.org Cc: Sent: Thursday, 10 January 2013 4:57 PM Subject: Re: [squid-users] calculating hardware for 900 users for SQUID cache server Hello Joseph, I use a Dell R320 (2, because failover), under OpenBSD 5.2 with 16GB RAM and Two Intel PRO/1000 PT (82571EB) (Broadcom 5720 isn't supported). I have 500-600 users/smartphones and 1GB of WAN bandwidth. To improve perfs (and this explain why so many RAM), i move /var/squid/cache to mfs (memory file system), and i use 4G disk cache and 3.5G memory cache. Perfs are very great. My servers are also the main client routers for other traffic. I hope that helps -- Best regards, Loïc BLOT, Engineering UNIX Systems, Security and Networks http://www.unix-experience.fr Le jeudi 10 janvier 2013 à 03:03 -0800, John Joseph a écrit : Hi All I am trying to make the hardware specs for the SQUID cache server. I have around 600 users, they may be using the bandwidth from 500dbps to 3Mbs. Expected annual increase of users will be up to 20%. I would like to size the hardware specs for the server, which will be enough for next coming three years.( user may be up to 900 then) What specs should I go for? For fast r/w I will go for SCSI hard disk, but I am not sure about the amount of RAM, CPU power and harddisk space for disk cache. I would like to request guidance on how to determine the hardware specs Guidance requested. Thanks Joseph John
[squid-users] calculating hardware for 900 users for SQUID cache server
Hi All I am trying to make the hardware specs for the SQUID cache server. I have around 600 users, they may be using the bandwidth from 500dbps to 3Mbs. Expected annual increase of users will be up to 20%. I would like to size the hardware specs for the server, which will be enough for next coming three years.( user may be up to 900 then) What specs should I go for? For fast r/w I will go for SCSI hard disk, but I am not sure about the amount of RAM, CPU power and harddisk space for disk cache. I would like to request guidance on how to determine the hardware specs Guidance requested. Thanks Joseph John
