Re: [squid-users] not caching enough
On 29.09.09 02:20, tookers wrote: Why not make use of some of your RAM for cache_mem? It will make requests for smaller, more frequently requested files a hell of a lot quicker, and it should give you a better hit ratio. mostly because we weren't sure if his RAM will be enough for the cache index... # 3GB process size limit in 32bit, so don't set higher than 1 GB for a very busy cache cache_mem 512 MB maximum_object_size_in_memory 2048 KB I would set those lower, maybue 128M/128K A few refresh_patterns to get images and other static content in cache for longer:- refresh_pattern \.jpg$ 3600 50% 60 ignore-reload refresh_pattern \.gif$ 3600 50% 60 ignore-reload refresh_pattern \.css$ 3600 50% 60 ignore-reload refresh_pattern \.js$ 3600 50% 60 ignore-reload # Cache all html for 5 minutes refresh_pattern \.html$ 300 50% 10 ignore-reload ignore-reload may cause problems for clients. I strongly discourage using that. Better skip those refresh_patterns, for now se can use defaults. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. (R)etry, (A)bort, (C)ancer
Re: [squid-users] not caching enough
Squid version 2.6. This is the apt-get version for ubuntu 8.04. I think you are right about the ignore-reload. Here is my squid.conf that I will put into production at 3pm today. http_port 3128 acl QUERY urlpath_regex cgi-bin \? cache_mem 512 MB# May need to set lower if I run low on RAM maximum_object_size_in_memory 2048 KB# May need to set lower if I run low on RAM maximum_object_size 1 GB cache_dir aufs /cache 50 256 256 redirect_rewrites_host_header off cache_replacement_policy lru acl all src all acl localnet src 10.60.0.0/255.255.0.0 acl localhost src 127.0.0.1 acl to_localhost dst 127.0.0.0/8 0.0.0.0/8 acl Safe_ports port 80 443 210 119 70 21 1025-65535 acl SSL_Ports port 443 acl CONNECT method CONNECT http_access deny !Safe_ports http_access deny CONNECT !SSL_Ports http_access allow localnet http_access allow localhost http_access deny all icp_port 0 refresh_pattern \.jpg$ 3600 50% 60 #ignore-reload refresh_pattern \.gif$ 3600 50% 60 #ignore-reload refresh_pattern \.css$ 3600 50% 60 #ignore-reload refresh_pattern \.js$ 3600 50% 60 #ignore-reload refresh_pattern \.html$ 300 50% 10 #ignore-reload refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 visible_hostname AHSPX01 -- View this message in context: http://www.nabble.com/not-caching-enough-tp25530445p25752421.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] not caching enough
This is great, the proxy is caching about a gig a day. Below is the final and fine tuned squid.conf that I will put into production after school lets out today. administra...@ahspx01:~$ cat /etc/squid/squid.conf http_port 3128 acl QUERY urlpath_regex cgi-bin \? #no_cache deny QUERY cache_mem 512 MB maximum_object_size_in_memory 2048 KB maximum_object_size 1 GB cache_dir aufs /cache 50 256 256 redirect_rewrites_host_header off cache_replacement_policy lru #acl QUERY urlpath_regex cgi-bin \? acl all src all acl localnet src 10.60.0.0/255.255.0.0 acl localhost src 127.0.0.1 acl to_localhost dst 127.0.0.0/8 0.0.0.0/8 acl Safe_ports port 80 443 210 119 70 21 1025-65535 acl SSL_Ports port 443 acl CONNECT method CONNECT http_access allow localnet http_access allow localhost http_access deny !Safe_ports http_access allow localnet http_access allow localhost http_access deny CONNECT http_access deny CONNECT !Safe_Ports http_access deny all icp_port 0 refresh_pattern \.jpg$ 3600 50% 60 ignore-reload refresh_pattern \.gif$ 3600 50% 60 ignore-reload refresh_pattern \.css$ 3600 50% 60 ignore-reload refresh_pattern \.js$ 3600 50% 60 ignore-reload refresh_pattern \.html$ 300 50% 10 ignore-reload refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 #refresh_pattern . 60 50% 10 ignore-reload refrsh_pattern . 0 20% 4320 visible_hostname AHSPX01 -- View this message in context: http://www.nabble.com/not-caching-enough-tp25530445p25704652.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] not caching enough
ant2ne wrote: This is great, the proxy is caching about a gig a day. Below is the final and fine tuned squid.conf that I will put into production after school lets out today. administra...@ahspx01:~$ cat /etc/squid/squid.conf http_port 3128 acl QUERY urlpath_regex cgi-bin \? #no_cache deny QUERY cache_mem 512 MB maximum_object_size_in_memory 2048 KB maximum_object_size 1 GB cache_dir aufs /cache 50 256 256 redirect_rewrites_host_header off cache_replacement_policy lru #acl QUERY urlpath_regex cgi-bin \? acl all src all acl localnet src 10.60.0.0/255.255.0.0 acl localhost src 127.0.0.1 acl to_localhost dst 127.0.0.0/8 0.0.0.0/8 acl Safe_ports port 80 443 210 119 70 21 1025-65535 acl SSL_Ports port 443 acl CONNECT method CONNECT Gah! Your http_access were permitting internal people complete access with no safety limits again. Make the http_access section this: http_access deny !Safe_ports http_access deny CONNECT !SSL_Ports http_access allow localnet http_access allow localhost http_access deny all icp_port 0 refresh_pattern \.jpg$ 3600 50% 60 ignore-reload refresh_pattern \.gif$ 3600 50% 60 ignore-reload refresh_pattern \.css$ 3600 50% 60 ignore-reload refresh_pattern \.js$ 3600 50% 60 ignore-reload refresh_pattern \.html$ 300 50% 10 ignore-reload refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 #refresh_pattern . 60 50% 10 ignore-reload refrsh_pattern . 0 20% 4320 visible_hostname AHSPX01 Amos -- Please be using Current Stable Squid 2.7.STABLE7 or 3.0.STABLE19 Current Beta Squid 3.1.0.14
Re: [squid-users] not caching enough
Thanks for the continued support! You say The CONNECT rule does need to be deny CONNECT !SSL_Ports. - But I dont' see a !SSL_Ports You say I'd shift that one http pattern above up above the ftp pattern. But I dont see http as a refresh pattern. I do see html. Is this what you mean? My currrent squid.conf http_port 3128 acl QUERY urlpath_regex cgi-bin \? #no_cache deny QUERY cache_mem 512 MB maximum_object_size_in_memory 2048 KB maximum_object_size 1 GB cache_dir aufs /cache 50 256 256 redirect_rewrites_host_header off cache_replacement_policy lru #acl QUERY urlpath_regex cgi-bin \? acl all src all acl localnet src 10.60.0.0/255.255.0.0 acl localhost src 127.0.0.1 acl to_localhost dst 127.0.0.0/8 0.0.0.0/8 acl Safe_ports port 80 443 210 119 70 21 1025-65535 acl CONNECT method CONNECT http_access allow localnet http_access allow localhost http_access deny !Safe_ports http_access allow localnet http_access allow localhost http_access deny CONNECT http_access deny CONNECT !Safe_Ports http_access deny all icp_port 0 refresh_pattern \.jpg$ 3600 50% 60 ignore-reload refresh_pattern \.gif$ 3600 50% 60 ignore-reload refresh_pattern \.css$ 3600 50% 60 ignore-reload refresh_pattern \.js$ 3600 50% 60 ignore-reload refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern \.html$ 300 50% 10 ignore-reload refresh_pattern . 60 50% 10 ignore-reload refrsh_pattern . 0 20% 4320 visible_hostname AHSPX01 -- View this message in context: http://www.nabble.com/not-caching-enough-tp25530445p25681166.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] not caching enough
On Wed, 30 Sep 2009 06:35:06 -0700 (PDT), ant2ne tcy...@altonschools.org wrote: Thanks for the continued support! You say The CONNECT rule does need to be deny CONNECT !SSL_Ports. - But I dont' see a !SSL_Ports The ! means NOT in boolean and squid.conf access language. The very next thing I said was With SSL_Ports defining what HTTPS ports are safe to use. ... meaning that you had to add back in the SSL_Ports acl definition which appears to have been lost somewhere. acl SSL_Ports port 443 You say I'd shift that one http pattern above up above the ftp pattern. But I dont see http as a refresh pattern. I do see html. Is this what you mean? Yes that was the one. Best to keep all the file extension patterns together if you can so you don't miss one when changing things later. My currrent squid.conf http_port 3128 acl QUERY urlpath_regex cgi-bin \? #no_cache deny QUERY cache_mem 512 MB maximum_object_size_in_memory 2048 KB maximum_object_size 1 GB cache_dir aufs /cache 50 256 256 redirect_rewrites_host_header off cache_replacement_policy lru #acl QUERY urlpath_regex cgi-bin \? acl all src all acl localnet src 10.60.0.0/255.255.0.0 acl localhost src 127.0.0.1 acl to_localhost dst 127.0.0.0/8 0.0.0.0/8 acl Safe_ports port 80 443 210 119 70 21 1025-65535 acl CONNECT method CONNECT http_access allow localnet http_access allow localhost http_access deny !Safe_ports http_access allow localnet http_access allow localhost http_access deny CONNECT http_access deny CONNECT !Safe_Ports http_access deny all icp_port 0 refresh_pattern \.jpg$ 3600 50% 60 ignore-reload refresh_pattern \.gif$ 3600 50% 60 ignore-reload refresh_pattern \.css$ 3600 50% 60 ignore-reload refresh_pattern \.js$ 3600 50% 60 ignore-reload refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern \.html$ 300 50% 10 ignore-reload refresh_pattern . 60 50% 10 ignore-reload refrsh_pattern . 0 20% 4320 Ah, well both of the above two dot patterns do the same thing. Catching all requested objects not matched by something in the patterns above it. I hadn't read the other email when I wrote that, pick the one you want to use. Note that the 'reload' being ignored is an explicit instruction sent by the browser when the user goes to all the trouble of pressing shift+f5. They usually do this when they have clue AND know that the page and stuff on it is obsolete. There were some very old versions of IE that sent this with each request, but those are fast being replaced with fixed versions nowdays. Amos
Re: [squid-users] not caching enough
Hi there, Why not make use of some of your RAM for cache_mem? It will make requests for smaller, more frequently requested files a hell of a lot quicker, and it should give you a better hit ratio. # 3GB process size limit in 32bit, so don't set higher than 1 GB for a very busy cache cache_mem 512 MB maximum_object_size_in_memory 2048 KB A few refresh_patterns to get images and other static content in cache for longer:- refresh_pattern \.jpg$ 3600 50% 60 ignore-reload refresh_pattern \.gif$ 3600 50% 60 ignore-reload refresh_pattern \.css$ 3600 50% 60 ignore-reload refresh_pattern \.js$ 3600 50% 60 ignore-reload # Cache all html for 5 minutes refresh_pattern \.html$ 300 50% 10 ignore-reload After the first time an image is loaded into cache you should see 'TCP_MEM_HIT' or 'TCP_HIT' in your logs until it expires (after 1 hour), at which point the image will be downloaded from the origin web servers. Give it a go, the refresh patterns should definitely help with caching. ant2ne wrote: I got this 1TB drive and mounted it as /cache. I want to cache everything and anything and keep it until it is outdated. Webmin | Servers | squid | Cache has cache directories set to /cache and Size (MB) set to 50 (the rest set to default) administra...@ahspx01:~$ df -h FilesystemSize Used Avail Use% Mounted on /dev/sda1 72G 1.2G 67G 2% / varrun2.5G 184K 2.5G 1% /var/run varlock 2.5G 0 2.5G 0% /var/lock udev 2.5G 40K 2.5G 1% /dev devshm2.5G 0 2.5G 0% /dev/shm /dev/sdb1 917G 1.1G 870G 1% /cache administra...@ahspx01:~$ ls -l /cache total 5220 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 00 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 01 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 02 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 03 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 04 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 05 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 06 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 07 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 08 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 09 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 0A drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 0B drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 0C drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 0D drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 0E drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 0F drwxr-x--- 2 proxy proxy 16384 2009-07-15 11:01 lost+found -rw-r- 1 proxy proxy 5247120 2009-09-21 14:32 swap.state -rw-r- 1 proxy proxy 0 2009-09-21 06:30 swap.state.last-clean There is currently about 100 computers using this cache proxy. I intend to add another 200. I can use sarg and get reports showing that websites are being cached, so it is working. But it just doesn't seem to be caching enough. Are there file types that are not getting cached that I can turn on? Why wont this cache fill up? -- View this message in context: http://www.nabble.com/not-caching-enough-tp25530445p25659633.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] not caching enough
Thanks for all of the great replies. There is lots of information to digest. I appreciate all of the suggestions. But, Before I got any of these replies, I went ahead and made modifications to my squid.conf to match an example I found on the internet here is my current running squid.conf http_port 3128 icp_port 0 acl QUERY urlpath_regex cgi-bin \? no_cache deny QUERY cache_mem 16 MB cache_dir ufs /cache 50 256 256 redirect_rewrites_host_header off cache_replacement_policy lru acl localnet src 10.60.0.0/255.255.0.0 acl localhost src 127.0.0.1/255.255.255.255 acl Safe_ports port 80 443 210 119 70 21 1025-65535 acl CONNECT method CONNECT acl all src 0.0.0.0/0.0.0.0 http_access allow localnet http_access allow localhost http_access deny !Safe_ports http_access deny CONNECT http_access deny all log_icp_queries off This one seems to be caching. I can refresh webmin system info every few hours and see that /cache is growing in space used. Although, very slowly. Amos Jeffries tookers; I've taken the working squid.conf (above), and applied your suggestions to it (below). Please review this squid.conf (below) and make suggestions to it before I put it into production. http_port 3128 icp_port 0 no_cache deny QUERY cache_mem 512 MB maximum_object_size_in_memory 2048 KB maximum_object_size 1 GB cache_dir ufs /cache 50 256 256 redirect_rewrites_host_header off cache_replacement_policy lru acl QUERY urlpath_regex cgi-bin \? acl all src all acl localnet src 10.60.0.0/255.255.0.0 acl localhost src 127.0.0.1 acl to_localhost dst 127.0.0.0/8 0.0.0.0/8 acl Safe_ports port 80 443 210 119 70 21 1025-65535 acl CONNECT method CONNECT http_access allow localnet http_access allow localhost http_access deny !Safe_ports http_access deny CONNECT http_access deny all icp_access allow our_networks icp_access allow localhost icp_access deny all refresh_pattern \.jpg$ 3600 50% 60 ignore-reload refresh_pattern \.gif$ 3600 50% 60 ignore-reload refresh_pattern \.css$ 3600 50% 60 ignore-reload refresh_pattern \.js$ 3600 50% 60 ignore-reload refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern \.html$ 300 50% 10 ignore-reload THANKS!! -- View this message in context: http://www.nabble.com/not-caching-enough-tp25530445p25668625.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] not caching enough
ant2ne wrote: Thanks for all of the great replies. There is lots of information to digest. I appreciate all of the suggestions. But, Before I got any of these replies, I went ahead and made modifications to my squid.conf to match an example I found on the internet here is my current running squid.conf http_port 3128 icp_port 0 acl QUERY urlpath_regex cgi-bin \? no_cache deny QUERY cache_mem 16 MB cache_dir ufs /cache 50 256 256 redirect_rewrites_host_header off cache_replacement_policy lru acl localnet src 10.60.0.0/255.255.0.0 acl localhost src 127.0.0.1/255.255.255.255 acl Safe_ports port 80 443 210 119 70 21 1025-65535 acl CONNECT method CONNECT acl all src 0.0.0.0/0.0.0.0 http_access allow localnet http_access allow localhost http_access deny !Safe_ports http_access deny CONNECT http_access deny all log_icp_queries off This one seems to be caching. I can refresh webmin system info every few hours and see that /cache is growing in space used. Although, very slowly. Amos Jeffries tookers; I've taken the working squid.conf (above), and applied your suggestions to it (below). Please review this squid.conf (below) and make suggestions to it before I put it into production. http_port 3128 icp_port 0 no_cache deny QUERY cache_mem 512 MB maximum_object_size_in_memory 2048 KB maximum_object_size 1 GB cache_dir ufs /cache 50 256 256 redirect_rewrites_host_header off cache_replacement_policy lru acl QUERY urlpath_regex cgi-bin \? acl all src all acl localnet src 10.60.0.0/255.255.0.0 acl localhost src 127.0.0.1 acl to_localhost dst 127.0.0.0/8 0.0.0.0/8 acl Safe_ports port 80 443 210 119 70 21 1025-65535 acl CONNECT method CONNECT http_access allow localnet http_access allow localhost http_access deny !Safe_ports http_access deny CONNECT http_access deny all icp_access allow our_networks icp_access allow localhost icp_access deny all refresh_pattern \.jpg$ 3600 50% 60 ignore-reload refresh_pattern \.gif$ 3600 50% 60 ignore-reload refresh_pattern \.css$ 3600 50% 60 ignore-reload refresh_pattern \.js$ 3600 50% 60 ignore-reload refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern \.html$ 300 50% 10 ignore-reload THANKS!! Hey there, I don't think you need to include... icp_access allow our_networks icp_access allow localhost icp_access deny all As icp_port 0, icp is essentially disabled, you shouldn't need to specify the icp_access lines. After your Acl CONNECT you need to add http_access deny CONNECT !Safe_Ports You might also want a 'catch-all' refresh_pattern... cache all for 60 seconds if not already matched above. Stick this right at the end of your config. refresh_pattern . 60 50% 10 ignore-reload -- View this message in context: http://www.nabble.com/not-caching-enough-tp25530445p25669291.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] not caching enough
THANKS!!! With those changes I'm looking at... http_port 3128 no_cache deny QUERY cache_mem 512 MB maximum_object_size_in_memory 2048 KB maximum_object_size 1 GB cache_dir ufs /cache 50 256 256 redirect_rewrites_host_header off cache_replacement_policy lru acl QUERY urlpath_regex cgi-bin \? acl all src all acl localnet src 10.60.0.0/255.255.0.0 acl localhost src 127.0.0.1 acl to_localhost dst 127.0.0.0/8 0.0.0.0/8 acl Safe_ports port 80 443 210 119 70 21 1025-65535 acl CONNECT method CONNECT http_access allow localnet http_access allow localhost http_access deny !Safe_ports http_access deny CONNECT http_access deny CONNECT !Safe_Ports http_access deny all icp_port 0 refresh_pattern \.jpg$ 3600 50% 60 ignore-reload refresh_pattern \.gif$ 3600 50% 60 ignore-reload refresh_pattern \.css$ 3600 50% 60 ignore-reload refresh_pattern \.js$ 3600 50% 60 ignore-reload refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern \.html$ 300 50% 10 ignore-reload refresh_pattern . 60 50% 10 ignore-reload visible_hostname AHSPX01 -- View this message in context: http://www.nabble.com/not-caching-enough-tp25530445p25669996.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] not caching enough
Ooops, I need acl QUERY urlpath_regex cgi-bin \? before no_cache deny QUERY -- View this message in context: http://www.nabble.com/not-caching-enough-tp25530445p25670155.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] not caching enough
On Tue, 29 Sep 2009 11:41:16 -0700 (PDT), ant2ne tcy...@altonschools.org wrote: Thanks for all of the great replies. There is lots of information to digest. I appreciate all of the suggestions. But, Before I got any of these replies, I went ahead and made modifications to my squid.conf to match an example I found on the internet here is my current running squid.conf http_port 3128 icp_port 0 acl QUERY urlpath_regex cgi-bin \? no_cache deny QUERY cache_mem 16 MB cache_dir ufs /cache 50 256 256 redirect_rewrites_host_header off cache_replacement_policy lru acl localnet src 10.60.0.0/255.255.0.0 acl localhost src 127.0.0.1/255.255.255.255 acl Safe_ports port 80 443 210 119 70 21 1025-65535 acl CONNECT method CONNECT acl all src 0.0.0.0/0.0.0.0 http_access allow localnet http_access allow localhost http_access deny !Safe_ports http_access deny CONNECT http_access deny all log_icp_queries off This one seems to be caching. I can refresh webmin system info every few hours and see that /cache is growing in space used. Although, very slowly. Amos Jeffries tookers; I've taken the working squid.conf (above), and applied your suggestions to it (below). Please review this squid.conf (below) and make suggestions to it before I put it into production. Okay some more bits still to do... http_port 3128 icp_port 0 no_cache deny QUERY Kill the above line. cache_mem 512 MB maximum_object_size_in_memory 2048 KB maximum_object_size 1 GB cache_dir ufs /cache 50 256 256 If you have a Linux system make that AUFS. If you have one of the BSD systems make that diskd. Those storage types are faster on their OS that plain ufs. Better handling for large caches too. redirect_rewrites_host_header off cache_replacement_policy lru I'd think heap lru might be better. Up to you though. acl QUERY urlpath_regex cgi-bin \? Kill the above line. acl all src all acl localnet src 10.60.0.0/255.255.0.0 acl localhost src 127.0.0.1 acl to_localhost dst 127.0.0.0/8 0.0.0.0/8 acl Safe_ports port 80 443 210 119 70 21 1025-65535 acl CONNECT method CONNECT http_access allow localnet http_access allow localhost The above two lines should really go down http_access deny !Safe_ports http_access deny CONNECT ... here. The CONNECT rule does need to be deny CONNECT !SSL_Ports. With SSL_Ports defining what HTTPS ports are safe to use. CONNECT can be very unsafe when email etc ports are allowed since it opens an anonymous random destination tunnel with no checking on what gets passed through. http_access deny all icp_access allow our_networks icp_access allow localhost icp_access deny all refresh_pattern \.jpg$ 3600 50% 60 ignore-reload refresh_pattern \.gif$ 3600 50% 60 ignore-reload refresh_pattern \.css$ 3600 50% 60 ignore-reload refresh_pattern \.js$ 3600 50% 60 ignore-reload refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern \.html$ 300 50% 10 ignore-reload I'd shift that one http pattern above up above the ftp pattern. Do add back the default . (dot) rule: refresh_pattern . 0 20% 4320 Amos
Re: [squid-users] not caching enough
On 24.09.09 10:45, ant2ne wrote: This web cache proxy server is a dedicated machine running ubuntu 64bit OS (no gui) with 5gigs of RAM and a 1TB drive dedicated to the cache. Only using 500GB currently. (The OS is on a different 80GB drive) Accordign to the math of 10MB of RAM per 1GB of Disk space that would put me right at 5GB of RAM. well, the 5GB of tram will be used only for cache indexes. You may need additional memory for OS, cache etc. or your system may start swapping and get unusable. Yesterday I increased the maximum_object_size value to 32MB from the default of 4MB and haven't noticed any difference in the cache size. I think I'll push it up around 50MB. How much traffic goes through your cache? Note that - there's much of uncacheable content on the web (much of it could be cached if the developers cared) - squid only tried to cache fetched content. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Nothing is fool-proof to a talented fool.
Re: [squid-users] not caching enough
Ok, it has come to my attention that /cache only grwos when I run a report using sarg. So it maybe that my proxy server is working, but only as a proxy and not as a web cache proxy. Below is my squid.conf file with the comment # invert grepped out. Please review and tell me what it is that I need to change to turn this proxy server into a web cache server. below is me squid.conf acl all src 0.0.0.0/0.0.0.0 acl manager proto cache_object acl localhost src 127.0.0.1/255.255.255.255 acl to_localhost dst 127.0.0.0/8 acl purge method PURGE acl CONNECT method CONNECT http_access allow manager localhost http_access deny manager http_access allow purge localhost http_access deny purge http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access deny to_localhost acl our_networks src 10.60.140.0/24 http_access allow our_networks http_access allow localhost http_access allow all http_access deny all icp_access allow all http_port 3128 hierarchy_stoplist cgi-bin ? cache_dir ufs /cache 50 256 256 maximum_object_size 32768 KB access_log /var/log/squid/access.log squid acl QUERY urlpath_regex cgi-bin \? cache deny QUERY refresh_pattern ^ftp: 144020% 10080 refresh_pattern ^gopher:14400% 1440 refresh_pattern . 0 20% 4320 acl apache rep_header Server ^Apache broken_vary_encoding allow apache extension_methods REPORT MERGE MKACTIVITY CHECKOUT hosts_file /etc/hosts coredump_dir /var/spool/squid visible_hostname AHSPX01 -- View this message in context: http://www.nabble.com/not-caching-enough-tp25530445p25645183.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] not caching enough
On Mon, 28 Sep 2009 06:36:33 -0700 (PDT), ant2ne tcy...@altonschools.org wrote: Ok, it has come to my attention that /cache only grwos when I run a report using sarg. So it maybe that my proxy server is working, but only as a proxy and not as a web cache proxy. Below is my squid.conf file with the comment # invert grepped out. Please review and tell me what it is that I need to change to turn this proxy server into a web cache server. below is me squid.conf acl all src 0.0.0.0/0.0.0.0 Please use: acl all src all acl manager proto cache_object acl localhost src 127.0.0.1/255.255.255.255 acl to_localhost dst 127.0.0.0/8 Please instead use: acl localhost src 127.0.0.1 acl to_localhost dst 127.0.0.0/8 0.0.0.0/8 acl purge method PURGE acl CONNECT method CONNECT http_access allow manager localhost http_access deny manager http_access allow purge localhost http_access deny purge http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access deny to_localhost acl our_networks src 10.60.140.0/24 http_access allow our_networks http_access allow localhost http_access allow all So everybody on the entire Internet is allowed to use your cache? not good. http_access deny all icp_access allow all Also not good, for the same reasons. Try: icp_access allow our_networks icp_access allow localhost icp_access deny all http_port 3128 hierarchy_stoplist cgi-bin ? cache_dir ufs /cache 50 256 256 maximum_object_size 32768 KB With sich large disk I would not worry about limiting objects to small sizes. You probably want set that as: maximum_object_size 1 GB ... or 0 KB for 'off' access_log /var/log/squid/access.log squid acl QUERY urlpath_regex cgi-bin \? cache deny QUERY There is a sizable amount of dynamic content which is cacheable these days. Erase the above two lines. refresh_pattern ^ftp: 144020% 10080 refresh_pattern ^gopher: 14400% 1440 Add at this exact spot in the config: refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 acl apache rep_header Server ^Apache broken_vary_encoding allow apache extension_methods REPORT MERGE MKACTIVITY CHECKOUT hosts_file /etc/hosts coredump_dir /var/spool/squid visible_hostname AHSPX01 Amos
Re: [squid-users] not caching enough
: Ok, it has come to my attention that /cache only grwos when I run a : report ... : cache_dir ufs /cache 50 256 256 ...if i can start by asking a silly question: i assume your cache is not yet at the maximum size? The next silly question: have you done any analysis of the squid logs and the types of requests being made through your cache? I know nothing about your usecase but perhaps you've just reached a steady state where everything that can be cached is already being overwritten when it expires. If nothing else a simple grep of your access.log for _HIT/200 should answer your main question... : using sarg. So it maybe that my proxy server is working, but only as a : proxy : and not as a web cache proxy. Below is my squid.conf file with the -Hoss
Re: [squid-users] not caching enough
You said, In which case do RAM check and see how much is used free before trying to cache any more. I'm currently using 205Megs of the 5Gigs of RAM You said, This change requires a full stop of Squid. remove the cache directory and rebuild it with squid -z, then restart Squid. That would explain it. But how do I remove the cache directory 'rm -R /cache'? -- View this message in context: http://www.nabble.com/not-caching-enough-tp25530445p25611950.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] not caching enough
ant2ne wrote: You said, In which case do RAM check and see how much is used free before trying to cache any more. I'm currently using 205Megs of the 5Gigs of RAM You said, This change requires a full stop of Squid. remove the cache directory and rebuild it with squid -z, then restart Squid. That would explain it. But how do I remove the cache directory 'rm -R /cache'? Yes. http://wiki.squid-cache.org/SquidFaq/ClearingTheCache Amos -- Please be using Current Stable Squid 2.7.STABLE7 or 3.0.STABLE19 Current Beta Squid 3.1.0.13
Re: [squid-users] not caching enough
Hey thanks for the input This web cache proxy server is a dedicated machine running ubuntu 64bit OS (no gui) with 5gigs of RAM and a 1TB drive dedicated to the cache. Only using 500GB currently. (The OS is on a different 80GB drive) Accordign to the math of 10MB of RAM per 1GB of Disk space that would put me right at 5GB of RAM. Yesterday I increased the maximum_object_size value to 32MB from the default of 4MB and haven't noticed any difference in the cache size. I think I'll push it up around 50MB. I tried to change my L1 and L2 values from the default 16 256 but it didn't like the change because the squid service didn't restart until I changed it back. I don't think I tried 256 256, so I'll try that next. I will need to read up on and expirement with refresh_pattern -- View this message in context: http://www.nabble.com/not-caching-enough-tp25530445p25578014.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] not caching enough
ant2ne wrote: Hey thanks for the input This web cache proxy server is a dedicated machine running ubuntu 64bit OS (no gui) with 5gigs of RAM and a 1TB drive dedicated to the cache. Only using 500GB currently. (The OS is on a different 80GB drive) Accordign to the math of 10MB of RAM per 1GB of Disk space that would put me right at 5GB of RAM. In which case do RAM check and see how much is used free before trying to cache any more. Yesterday I increased the maximum_object_size value to 32MB from the default of 4MB and haven't noticed any difference in the cache size. I think I'll push it up around 50MB. I tried to change my L1 and L2 values from the default 16 256 but it didn't like the change because the squid service didn't restart until I changed it back. I don't think I tried 256 256, so I'll try that next. This change requires a full stop of Squid. remove the cache directory and rebuild it with squid -z, then restart Squid. I will need to read up on and expirement with refresh_pattern Amos -- Please be using Current Stable Squid 2.7.STABLE7 or 3.0.STABLE19 Current Beta Squid 3.1.0.13
[squid-users] not caching enough
I got this 1TB drive and mounted it as /cache. I want to cache everything and anything and keep it until it is outdated. Webmin | Servers | squid | Cache has cache directories set to /cache and Size (MB) set to 50 (the rest set to default) administra...@ahspx01:~$ df -h FilesystemSize Used Avail Use% Mounted on /dev/sda1 72G 1.2G 67G 2% / varrun2.5G 184K 2.5G 1% /var/run varlock 2.5G 0 2.5G 0% /var/lock udev 2.5G 40K 2.5G 1% /dev devshm2.5G 0 2.5G 0% /dev/shm /dev/sdb1 917G 1.1G 870G 1% /cache administra...@ahspx01:~$ ls -l /cache total 5220 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 00 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 01 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 02 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 03 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 04 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 05 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 06 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 07 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 08 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 09 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 0A drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 0B drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 0C drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 0D drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 0E drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 0F drwxr-x--- 2 proxy proxy 16384 2009-07-15 11:01 lost+found -rw-r- 1 proxy proxy 5247120 2009-09-21 14:32 swap.state -rw-r- 1 proxy proxy 0 2009-09-21 06:30 swap.state.last-clean There is currently about 100 computers using this cache proxy. I intend to add another 200. I can use sarg and get reports showing that websites are being cached, so it is working. But it just doesn't seem to be caching enough. Are there file types that are not getting cached that I can turn on? Why wont this cache fill up? -- View this message in context: http://www.nabble.com/not-caching-enough-tp25530445p25530445.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] not caching enough
ant2ne wrote: I got this 1TB drive and mounted it as /cache. I want to cache everything and anything and keep it until it is outdated. Webmin | Servers | squid | Cache has cache directories set to /cache and Size (MB) set to 50 (the rest set to default) First read http://wiki.squid-cache.org/SquidFaq/SquidMemory. If you insist on trying to fill a 500GB cache, have a look at http://www.squid-cache.org/Doc/config/maximum_object_size/ administra...@ahspx01:~$ df -h FilesystemSize Used Avail Use% Mounted on /dev/sda1 72G 1.2G 67G 2% / varrun2.5G 184K 2.5G 1% /var/run varlock 2.5G 0 2.5G 0% /var/lock udev 2.5G 40K 2.5G 1% /dev devshm2.5G 0 2.5G 0% /dev/shm /dev/sdb1 917G 1.1G 870G 1% /cache administra...@ahspx01:~$ ls -l /cache total 5220 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 00 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 01 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 02 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 03 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 04 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 05 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 06 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 07 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 08 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 09 drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 0A drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 0B drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 0C drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 0D drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 0E drwxr-x--- 258 proxy proxy4096 2009-07-15 14:11 0F drwxr-x--- 2 proxy proxy 16384 2009-07-15 11:01 lost+found -rw-r- 1 proxy proxy 5247120 2009-09-21 14:32 swap.state -rw-r- 1 proxy proxy 0 2009-09-21 06:30 swap.state.last-clean There is currently about 100 computers using this cache proxy. I intend to add another 200. I can use sarg and get reports showing that websites are being cached, so it is working. But it just doesn't seem to be caching enough. Are there file types that are not getting cached that I can turn on? Why wont this cache fill up? Chris
Re: [squid-users] not caching enough
On Mon, 21 Sep 2009 17:07:27 -0800, Chris Robertson crobert...@gci.net wrote: ant2ne wrote: I got this 1TB drive and mounted it as /cache. I want to cache everything and anything and keep it until it is outdated. Webmin | Servers | squid | Cache has cache directories set to /cache and Size (MB) set to 50 (the rest set to default) I suggest you also check the options on your cache_dir line. Ensuring that a fast filesystem mechanism (AUFS, COSS or diskd) is being used. And that the L1/L2 values (default 16 256) are set large enough to cope with 500GB, ie 256 256. First read http://wiki.squid-cache.org/SquidFaq/SquidMemory. If you insist on trying to fill a 500GB cache, have a look at http://www.squid-cache.org/Doc/config/maximum_object_size/ I would also point you in the direction of: http://wiki.squid-cache.org/ConfigExamples/DynamicContent ... which should make a fair few more requests cacheable. You can also use refresh_pattern to extend the time objects without expiry information are stored in cache (but please stay away from the override-* options unless you need them to solve particular file problem, they make your cache disobey HTTP protocol requirements). Amos snip There is currently about 100 computers using this cache proxy. I intend to add another 200. I can use sarg and get reports showing that websites are being cached, so it is working. But it just doesn't seem to be caching enough. Are there file types that are not getting cached that I can turn on? Why wont this cache fill up? Chris Amos
