[squid-users] empty cache
Hi, I've already post with a similar subject but I think it's different. I run squid 24h/24h and my cache directory remains empty. I don't know why no object is cached on disk. Here is a brief of my configuration Squid -v Squid Cache: Version 2.6.STABLE12 configure options: '--enable-storeio=aufs,null,ufs' '--with-large-files' Here some of my squid.conf http_port 3128 transparent cache_mem 100 MB maximum_object_size 25600 KB maximum_object_size_in_memory 24 KB cache_dir aufs /usr/local/squid/var/cache2 2500 16 256 url_rewrite_program /usr/local/bin/squidGuard -c /usr/local/squidGuard/squidGuard.conf url_rewrite_children 80 acl our_networks src 172.16.0.0/12 http_access allow our_networks no_cache deny our_networks ## purger site acl PURGE method PURGE http_access allow PURGE localhost http_access deny PURGE ### http_access allow localhost # And finally deny all other access to this proxy http_access deny all cache_effective_user squid #Default: # wccp_router 0.0.0.0 wccp2_router 172.20.20.11 wccp2_router 172.20.20.12 wccp_version 4 wccp2_forwarding_method 2 wccp2_return_method 2 wccp2_assignment_method 1 wccp2_service standard 0 password=x all the other option is default option Guillaume Chartrand Technicien informatique Cégep régional de Lanaudière Centre administratif, Repentigny (450) 470-0911 poste 7218
RE: [squid-users] empty cache
If it's useful, the right on the cache directory is ls -la /usr/local/squid/var/cache2 total 152 drwxr-xr-x 18 squid squid 4096 Mar 30 23:30 . drwxr-xr-x6 squid squid 4096 Mar 26 11:28 .. drwxr-x--- 258 squid squid 4096 Mar 26 11:28 00 drwxr-x--- 258 squid squid 4096 Mar 26 11:28 01 drwxr-x--- 258 squid squid 4096 Mar 26 11:28 02 drwxr-x--- 258 squid squid 4096 Mar 26 11:28 03 drwxr-x--- 258 squid squid 4096 Mar 26 11:28 04 drwxr-x--- 258 squid squid 4096 Mar 26 11:28 05 drwxr-x--- 258 squid squid 4096 Mar 26 11:28 06 drwxr-x--- 258 squid squid 4096 Mar 26 11:28 07 drwxr-x--- 258 squid squid 4096 Mar 26 11:28 08 drwxr-x--- 258 squid squid 4096 Mar 26 11:28 09 drwxr-x--- 258 squid squid 4096 Mar 26 11:28 0A drwxr-x--- 258 squid squid 4096 Mar 26 11:28 0B drwxr-x--- 258 squid squid 4096 Mar 26 11:28 0C drwxr-x--- 258 squid squid 4096 Mar 26 11:28 0D drwxr-x--- 258 squid squid 4096 Mar 26 11:28 0E drwxr-x--- 258 squid squid 4096 Mar 26 11:28 0F -rw-r-1 squid squid 96 Mar 30 23:30 swap.state -rw-r-1 squid squid0 Mar 30 23:30 swap.state.last-clean Guillaume Chartrand Technicien informatique Cégep régional de Lanaudière Centre administratif, Repentigny (450) 470-0911 poste 7218 -Message d'origine- De : Guillaume Chartrand [mailto:[EMAIL PROTECTED] Envoyé : 31 mars 2008 13:44 À : squid-users@squid-cache.org Objet : [squid-users] empty cache Hi, I've already post with a similar subject but I think it's different. I run squid 24h/24h and my cache directory remains empty. I don't know why no object is cached on disk. Here is a brief of my configuration Squid -v Squid Cache: Version 2.6.STABLE12 configure options: '--enable-storeio=aufs,null,ufs' '--with-large-files' Here some of my squid.conf http_port 3128 transparent cache_mem 100 MB maximum_object_size 25600 KB maximum_object_size_in_memory 24 KB cache_dir aufs /usr/local/squid/var/cache2 2500 16 256 url_rewrite_program /usr/local/bin/squidGuard -c /usr/local/squidGuard/squidGuard.conf url_rewrite_children 80 acl our_networks src 172.16.0.0/12 http_access allow our_networks no_cache deny our_networks ## purger site acl PURGE method PURGE http_access allow PURGE localhost http_access deny PURGE ### http_access allow localhost # And finally deny all other access to this proxy http_access deny all cache_effective_user squid #Default: # wccp_router 0.0.0.0 wccp2_router 172.20.20.11 wccp2_router 172.20.20.12 wccp_version 4 wccp2_forwarding_method 2 wccp2_return_method 2 wccp2_assignment_method 1 wccp2_service standard 0 password=x all the other option is default option Guillaume Chartrand Technicien informatique Cégep régional de Lanaudière Centre administratif, Repentigny (450) 470-0911 poste 7218
RE: [squid-users] TCP_HIT and TCP_MISS
Guillaume Chartrand wrote: Have you solved the forwarding loop? Nope, If I understand the loop is when my squid box try himself to go to the web the routeur redirected to himself. Is that means? That's how I interpret it. If so, I will try to modify my ACL on my router to not redirect packet when it's from my squid box Which might fix the problem... It's a good idea in any case. I modified my ACL in my router to not redirect my squid packet but it's keep saying Forwardind loop detected. What is the output of /usr/local/squid/bin/squidclient -s http://www.nu.nl/img/balkje.gif /usr/local/squid/bin/squidclient http://www.nu.nl/img/balkje.gif; (run on the squid box)? Chris Nothing happens on the command line but in my access.log here the result 1206562805.600570 127.0.0.1 TCP_DENIED/403 1514 GET http://www.nu.nl/img/balkje.gif - NONE/- text/html 1206562805.611 0 127.0.0.1 TCP_DENIED/403 1514 GET http://www.nu.nl/img/balkje.gif - NONE/- text/html Fair enough. You aren't allowing localhost to surf. For this test to work, either copy squidclient to another machine on your network (and use the -h with squidclient to specify your squid host) or allow localhost http_access (http_access allow localhost just above http_access deny all). Chris I modified squid to allow localhost to surf and here is the result in command line HTTP/1.0 200 OK Content-Length: 53 Server: Juniper Networks NitroCache/v1.0 Content-Type: image/gif Last-Modified: Thu, 27 Mar 2008 15:52:16 GMT Expires: Thu, 27 Mar 2008 15:57:16 GMT P3P: CP=NON DSP COR NID CURa ADMa DEVa PSAa PSDa IVAa IVDa OUR IND UNI COM NAV INT STA, policyref=/w3c/p3p.xml Cache-Control: max-age=300 X-Cache: HIT from squid.collanaud.qc.ca Via: 1.1 ilsia006ixs (Juniper Networks Application Acceleration Platform - DX 5.2.4 0), 1.0 squid.collanaud.qc.ca:3128 (squid/2.6.STABLE12) Proxy-Connection: close And here is my access.log 1206626419.158 4887 127.0.0.1 TCP_MISS/200 597 GET http://www.nu.nl/img/balkje.gif - DIRECT/62.69.179.230 image/gif 1206626419.266102 127.0.0.1 TCP_REFRESH_HIT/200 596 GET http://www.nu.nl/img/balkje.gif - DIRECT/62.69.179.230 image/gif 1206626658.440540 127.0.0.1 TCP_REFRESH_MISS/200 597 GET http://www.nu.nl/img/balkje.gif - DIRECT/62.69.184.129 image/gif 1206626658.587135 127.0.0.1 TCP_REFRESH_HIT/200 596 GET http://www.nu.nl/img/balkje.gif - DIRECT/62.69.184.129 image/gif 1206632986.525832 127.0.0.1 TCP_REFRESH_MISS/200 597 GET http://www.nu.nl/img/balkje.gif - DIRECT/62.69.179.231 image/gif 1206632986.645106 127.0.0.1 TCP_REFRESH_HIT/200 596 GET http://www.nu.nl/img/balkje.gif - DIRECT/62.69.179.231 image/gif Guillaume
RE: [squid-users] TCP_HIT and TCP_MISS
-Message d'origine- De : [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] De la part de Chris Robertson Envoyé : 25 mars 2008 15:41 À : squid-users@squid-cache.org Objet : Re: [squid-users] TCP_HIT and TCP_MISS Guillaume Chartrand wrote: -Message d'origine- De : [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] De la part de Chris Robertson Envoyé : 20 mars 2008 21:21 À : squid-users@squid-cache.org Objet : Re: [squid-users] TCP_HIT and TCP_MISS Guillaume Chartrand wrote: I try the solution on the other post to increase file descriptor I have 4096 when I do a ulimit -n I recompile squid Here what I have when I run squid -v Squid Cache: Version 2.6.STABLE12 configure options: No configure options at all? The default (last I checked) did not allow for support of aufs cache dirs. I've just run ./configure Without no other option. Now I look for enable aufs cache dir but I didn't find with option I need to enable. The only option who's near that it's --with-aufs-threads=N_THREADS If it's that option, how many threads should I put. Personally I use --enable-storeio=aufs,null,ufs, with a few other options (with-large-files, enable-snmp, enable-removal-policies=heap,lru). There is a message in the archives (somewhere, I can't find it right now) where Henrik explains that ufs is actually better for a lightly loaded cache (less overhead), but the blocking becomes a factor as the requests per second rise. For me it has become reflexive to suggest using aufs. 2008/03/18 09:11:01| NOTICE: no explicit transparent proxy support enabled. Assuming getsockname() works on intercepted conne ctions 2008/03/18 09:11:01| WARNING: Forwarding loop detected for: Client: 172.20.20.18 http_port: 172.20.20.18:3128 GET http://172.20.20.18:3128/design/motherbd/software/ias/updates.htm HTTP/1.0 Via: 1.0 squid.collanaud.qc.ca:3128 (squid/2.6.STABLE12) X-Forwarded-For: 172.21.132.93 Host: 172.20.20.18:3128 Cache-Control: max-age=259200 Connection: keep-alive So your router is intercepting Squid's traffic and redirecting it back to Squid. That's not so good. In a big way. This was not addressed in your reply. Hopefully it was addressed on your network. I feel it's the real issue. And here is some of my squid.conf # Squid normally listens to port 3128 #http_port 3128 http_port 3128 transparent #Default: # cache_mem 8 MB cache_mem 512 MB #Default: # maximum_object_size 4096 KB maximum_object_size 25600 KB#Default: cache_dir ufs /usr/local/squid/var/cache 2500 16 256 # this one is a symlink to another disk A symlink, or is the other disk mounted here. No matter, but you should probably be using aufs, which you will have to compile support for. It's a symlink only. Interesting. Any reason for using a symlink instead of a mount point? I deleted the symlink and make a mount point cache_dir ufs /usr/local/squid/var/cache2 2500 16 256 So how much memory does this box have? You've dedicated about a GB of RAM for Squid alone (512 cache_mem + (5GB of cache_dir * 0.1)). The machine is on VMware virtual machine and have 2 disk of 40GB each and 1GB of RAM Ugh. My calculation was off by an order of magnitude. 5GB of cache will only take (on average) 50MB of memory. See http://wiki.squid-cache.org/SquidFaq/SquidMemory for more details. But be aware, squid does not put objects from the disk cache BACK into memory. Only objects fetched from the net are put in the memory cache. I've downgrade my cache_mem to 100M cache_mem 100 MB I've modified the cache_dir cache_dir aufs /usr/local/squid/var/cache 2500 16 256 cache_dir aufs /usr/local/squid/var/cache2 2500 16 256 I've make a make clean on my src dir I've deleted all file and directory in my cache directory I make my mount point of my second disk to /usr/local/squid/var/cache2 and deleted my symlink to have direct access I recompile with this line ./configure --enable-storeio=aufs,null,ufs --with-large-files Make Make install I rerun squid with option -z to remake my directory After I run squid normally It's about 12 hours and that squid running and I don't have TCP_HIT I try to access a gif with 2 different browser on the same machine and here What I got in my access.log 1206545448.392 6413 172.20.51.11 TCP_MISS/200 596 GET http://www.nu.nl/img/balkje.gif - DIRECT/62.69.179.208 image/gif 1206545708.067252 172.20.51.11 TCP_MISS/200 596 GET http://www.nu.nl/img/balkje.gif - DIRECT/62.69.184.229 image/gif So I don't know again what I misconfigured Thank
RE: [squid-users] TCP_HIT and TCP_MISS
-Message d'origine- De : [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] De la part de Chris Robertson Envoyé : 20 mars 2008 21:21 À : squid-users@squid-cache.org Objet : Re: [squid-users] TCP_HIT and TCP_MISS Guillaume Chartrand wrote: I try the solution on the other post to increase file descriptor I have 4096 when I do a ulimit -n I recompile squid Here what I have when I run squid -v Squid Cache: Version 2.6.STABLE12 configure options: No configure options at all? The default (last I checked) did not allow for support of aufs cache dirs. I've just run ./configure Without no other option. Now I look for enable aufs cache dir but I didn't find with option I need to enable. The only option who's near that it's --with-aufs-threads=N_THREADS If it's that option, how many threads should I put. 2008/03/18 09:11:01| NOTICE: no explicit transparent proxy support enabled. Assuming getsockname() works on intercepted conne ctions 2008/03/18 09:11:01| WARNING: Forwarding loop detected for: Client: 172.20.20.18 http_port: 172.20.20.18:3128 GET http://172.20.20.18:3128/design/motherbd/software/ias/updates.htm HTTP/1.0 Via: 1.0 squid.collanaud.qc.ca:3128 (squid/2.6.STABLE12) X-Forwarded-For: 172.21.132.93 Host: 172.20.20.18:3128 Cache-Control: max-age=259200 Connection: keep-alive So your router is intercepting Squid's traffic and redirecting it back to Squid. That's not so good. In a big way. And here is some of my squid.conf # Squid normally listens to port 3128 #http_port 3128 http_port 3128 transparent #Default: # cache_mem 8 MB cache_mem 512 MB #Default: # maximum_object_size 4096 KB maximum_object_size 25600 KB#Default: cache_dir ufs /usr/local/squid/var/cache 2500 16 256 # this one is a symlink to another disk A symlink, or is the other disk mounted here. No matter, but you should probably be using aufs, which you will have to compile support for. It's a symlink only. cache_dir ufs /usr/local/squid/var/cache2 2500 16 256 So how much memory does this box have? You've dedicated about a GB of RAM for Squid alone (512 cache_mem + (5GB of cache_dir * 0.1)). The machine is on VMware virtual machine and have 2 disk of 40GB each and 1GB of RAM all the other option is default value and I have some config for wccp And apparently some url_rewriters (unless those are implied by using wccp...). Yes I have SquidGuard configured with squid Chris Thank Guillaume
RE: [squid-users] TCP_HIT and TCP_MISS
I try the solution on the other post to increase file descriptor I have 4096 when I do a ulimit -n I recompile squid Here what I have when I run squid -v Squid Cache: Version 2.6.STABLE12 configure options: Here some cache.log when I restarted after recompile and few test 2008/03/18 09:02:46| Starting Squid Cache version 2.6.STABLE12 for i686-pc-linux-gnu... 2008/03/18 09:02:46| Process ID 10769 2008/03/18 09:02:46| With 4096 file descriptors available 2008/03/18 09:02:46| Using epoll for the IO loop 2008/03/18 09:02:46| Performing DNS Tests... 2008/03/18 09:02:46| Successful DNS name lookup tests... 2008/03/18 09:02:46| DNS Socket created at 0.0.0.0, port 32848, FD 5 2008/03/18 09:02:46| Adding nameserver 172.20.20.3 from /etc/resolv.conf 2008/03/18 09:02:46| helperOpenServers: Starting 10 'squidGuard' processes ... 2008/03/18 09:02:46| Unlinkd pipe opened on FD 20 2008/03/18 09:02:46| Swap maxSize 512 KB, estimated 393846 objects 2008/03/18 09:02:46| Target number of buckets: 19692 2008/03/18 09:02:46| Using 32768 Store buckets 2008/03/18 09:02:46| Max Mem size: 524288 KB 2008/03/18 09:02:46| Max Swap size: 512 KB 2008/03/18 09:02:46| Store logging disabled 2008/03/18 09:02:46| Rebuilding storage in /usr/local/squid/var/cache (CLEAN) 2008/03/18 09:02:46| Rebuilding storage in /usr/local/squid/var/cache2 (CLEAN) 2008/03/18 09:02:46| Using Least Load store dir selection 2008/03/18 09:02:46| Set Current Directory to /usr/local/squid/var/core 2008/03/18 09:02:46| Loaded Icons. 2008/03/18 09:02:46| Accepting transparently proxied HTTP connections at 0.0.0.0, port 3128, FD 23. 2008/03/18 09:02:46| Accepting ICP messages at 0.0.0.0, port 3130, FD 24. 2008/03/18 09:02:46| WCCP Disabled. 2008/03/18 09:02:46| Accepting WCCPv2 messages on port 2048, FD 25. 2008/03/18 09:02:46| Initialising all WCCPv2 lists 2008/03/18 09:02:46| Ready to serve requests. 2008/03/18 09:02:46| Store rebuilding is 3.9% complete 2008/03/18 09:02:49| Done reading /usr/local/squid/var/cache2 swaplog (97993 entries) 2008/03/18 09:02:49| Done reading /usr/local/squid/var/cache swaplog (104485 entries) 2008/03/18 09:02:49| Finished rebuilding storage from disk. 2008/03/18 09:02:49|202478 Entries scanned 2008/03/18 09:02:49| 0 Invalid entries. 2008/03/18 09:02:49| 0 With invalid flags. 2008/03/18 09:02:49|202478 Objects loaded. 2008/03/18 09:02:49| 0 Objects expired. 2008/03/18 09:02:49| 0 Objects cancelled. 2008/03/18 09:02:49| 0 Duplicate URLs purged. 2008/03/18 09:02:49| 0 Swapfile clashes avoided. 2008/03/18 09:02:49| Took 3.1 seconds (65621.5 objects/sec). 2008/03/18 09:02:49| Beginning Validation Procedure 2008/03/18 09:02:49| Completed Validation Procedure 2008/03/18 09:02:49| Validated 202478 Entries 2008/03/18 09:02:49| store_swap_size = 4606476k 2008/03/18 09:02:50| storeLateRelease: released 0 objects 2008/03/18 09:10:29| httpReadReply: Excess data from POST http://login.live.com/login.srf?id=2svc=mailcbid=43577; 2008/03/18 09:11:01| NOTICE: no explicit transparent proxy support enabled. Assuming getsockname() works on intercepted conne ctions 2008/03/18 09:11:01| WARNING: Forwarding loop detected for: Client: 172.20.20.18 http_port: 172.20.20.18:3128 GET http://172.20.20.18:3128/design/motherbd/software/ias/updates.htm HTTP/1.0 Via: 1.0 squid.collanaud.qc.ca:3128 (squid/2.6.STABLE12) X-Forwarded-For: 172.21.132.93 Host: 172.20.20.18:3128 Cache-Control: max-age=259200 Connection: keep-alive And here is some of my squid.conf # Squid normally listens to port 3128 #http_port 3128 http_port 3128 transparent #Default: # cache_mem 8 MB cache_mem 512 MB #Default: # maximum_object_size 4096 KB maximum_object_size 25600 KB#Default: cache_dir ufs /usr/local/squid/var/cache 2500 16 256 # this one is a symlink to another disk cache_dir ufs /usr/local/squid/var/cache2 2500 16 256 all the other option is default value and I have some config for wccp Thank Guillaume Chartrand Technicien informatique Cégep régional de Lanaudière Centre administratif, Repentigny (450) 470-0911 poste 7218 -Message d'origine- De : Manoj_Rajkarnikar [mailto:[EMAIL PROTECTED] Envoyé : 18 mars 2008 00:54 À : Guillaume Chartrand Cc : squid-users@squid-cache.org Objet : Re: [squid-users] TCP_HIT and TCP_MISS On Mon, 17 Mar 2008, Guillaume Chartrand wrote: Hi everybody, I run Squid2.6Stable12 for few months ago and when I look to my access.log I have no TCP_HIT, I just have TCP_MISS, so it's seem to cache nothing. And if I look to the cache.log I have almost just entry like : httpReadReply: Excess data from or WARNING! Your cache is running out of filedescriptors 2008/03/17 07:23:54| WARNING: All url_rewriter processes are busy. 2008/03/17 07:23:54| WARNING: up to 27 pending requests queued Some information from squid -v and squid.conf please.. My configuration is a transparent squid with WCCP router to redirect without client
[squid-users] TCP_HIT and TCP_MISS
Hi everybody, I run Squid2.6Stable12 for few months ago and when I look to my access.log I have no TCP_HIT, I just have TCP_MISS, so it's seem to cache nothing. And if I look to the cache.log I have almost just entry like : httpReadReply: Excess data from or WARNING! Your cache is running out of filedescriptors 2008/03/17 07:23:54| WARNING: All url_rewriter processes are busy. 2008/03/17 07:23:54| WARNING: up to 27 pending requests queued My configuration is a transparent squid with WCCP router to redirect without client configuration. So what I did wrong Guillaume Chartrand Technicien informatique Cégep régional de Lanaudière Centre administratif, Repentigny (450) 470-0911 poste 7218
RE: [squid-users] transparency Squid very slow internet
I don't think it's the problem when I see the results Median Service Times (seconds) 5 min60 min: HTTP Requests (All): 0.05046 0.05331 Cache Misses: 0.05046 0.05331 Cache Hits:0.0 0.0 Near Hits: 0.0 0.0 Not-Modified Replies: 0.0 0.0 DNS Lookups: 0.00094 0.00094 ICP Queries: 0.0 0.0 And for the answer by Amos Jeffries It's usually regex ACL at fault when speed drops noticably. Check that: * ACL are only regex when absoutely necessary (dstdomain, srcdomain, dst, src are all better in most uses). ie acl searchengines dstdomain google.com yahoo.com * limiting regex ACL to only be tested when needed (placing a src netblock ACL ahead of one on the http_access will speed up all requests outside that netblockk). ie http_access allow dodgy_users pornregexes I have just dstdomain, src, dst in my acl. The only acl conatins regex is that line acl msnmessenger url_regex -i gateway.dll Thanks Guillaume Chartrand Technicien informatique Cégep régional de Lanaudière Centre administratif, Repentigny (450) 470-0911 poste 7218 -Message d'origine- De : Thomas Harold [mailto:[EMAIL PROTECTED] Envoyé : 7 mars 2008 11:38 À : Guillaume Chartrand Cc : squid-users@squid-cache.org Objet : Re: [squid-users] transparency Squid very slow internet Guillaume Chartrand wrote: Hi I run squid 2.6.STABLE12 on RHEL3 AS for web-caching and filtering my internet. I use also Squidguard to block some sites. I configure squid to run with WCCP v2 with my cisco router. So all my web-cache traffic is redirected transparently to squid. I don't know why but when I activate the squid it's really decrease my internet speed. It's long to have page loaded, even when it's in my network. I look with the command top and the squid process run only about 2-3 % of CPU and 15% of Memory. I also run iftop and I have about 15 Mb/s Total on my ethernet interface. I don't know where to look in the config to increase the speed. I use about 50% of disk space so it's not so bad Another possible issue is that squid is having to wait on a slow DNS server. Take a look at the mgr:info report: $ /usr/sbin/squidclient mgr:info And look at Median Service Times section. In our case, DNS lookups were in the 5+ second range, due to the primary DNS server being broken. So squid was asking the primary DNS server, waiting 5 seconds, then asking the backup DNS server. A normal squid server will be servicing all requests in under 1 second (depending on your load).
[squid-users] transparency Squid very slow internet
Hi I run squid 2.6.STABLE12 on RHEL3 AS for web-caching and filtering my internet. I use also Squidguard to block some sites. I configure squid to run with WCCP v2 with my cisco router. So all my web-cache traffic is redirected transparently to squid. I don't know why but when I activate the squid it's really decrease my internet speed. It's long to have page loaded, even when it's in my network. I look with the command top and the squid process run only about 2-3 % of CPU and 15% of Memory. I also run iftop and I have about 15 Mb/s Total on my ethernet interface. I don't know where to look in the config to increase the speed. I use about 50% of disk space so it's not so bad Thanks for the help Guillaume Chartrand Technicien informatique Cégep régional de Lanaudière Centre administratif, Repentigny (450) 470-0911 poste 7218
