[squid-users] Redirecting certain sites with a forward option?
I have a few teachers how want a group of students to be forwarded to a page for certain sites. The page would then have forwarding option to view the original site. They are trying to add around 100 URL's to this list. For example, if student tries to access loc.gov site, they would be redirected to our page that would have some guidelines written by teachers, and on the bottom of that page student could click on "Continue to original address, %URL. Squid v 3.1.6 is setup in transparent mode. Is this possible? Here is our config file: http_port 3128 refresh_pattern http://.*\.adobe\.com/ 0 80% 20160 reload-into-ims refresh_pattern http://.*\.windowsupdate\.microsoft\.com/ 0 80% 20160 reload-into-ims refresh_pattern http://office\.microsoft\.com/ 0 80% 20160 reload-into-ims refresh_pattern http://windowsupdate\.microsoft\.com/ 0 80% 20160 reload-into-ims refresh_pattern http://w?xpsp[0-9]\.microsoft\.com/ 0 80% 20160 reload-into-ims refresh_pattern http://w2ksp[0-9]\.microsoft\.com/ 0 80% 20160 reload-into-ims refresh_pattern http://download\.microsoft\.com/ 0 80% 20160 reload-into-ims refresh_pattern http://download\.macromedia\.com/ 0 80% 20160 reload-into-ims refresh_pattern -i \.flv$ 10080 90% 99 ignore-no-cache override-expire ignore-private refresh_pattern -i \.index.(html|htm)$ 0 40% 10080 refresh_pattern -i \.(html|htm|css|js)$ 1440 40% 40320 refresh_pattern -i \.(gif|png|jpg|jpeg|ico)$ 10080 90% 43200 override-expire ignore-no-cache ignore-no-store ignore-private refresh_pattern -i \.(iso|avi|wav|mp3|mp4|mpeg|swf|x-flv)$ 43200 90% 432000 override-expire ignore-no-cache ignore-no-store ignore-private refresh_pattern -i \.(deb|rpm|exe|zip|tar|tgz|ram|rar|bin|ppt|doc|tiff)$ 10080 90% 43200 override-expire ignore-no-cache ignore-no-store ignore-private refresh_pattern ^gopher:14400% 1440 refresh_pattern ^ftp: 144000 20% 1008000 refresh_pattern -i \.index.(html|htm)$ 1440 90% 40320 refresh_pattern -i \.(html|htm|css|js)$ 1440 90% 40320 refresh_pattern (/cgi-bin/|\?) 0 0% 0 acl manager proto cache_object acl localhost src 127.0.0.1/32 acl to_localhost dst 127.0.0.0/8 acl schoolip src 148.110.3.191 acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT http_access allow manager localhost http_access deny manager http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost http_access allow schoolip http_access allow test #http_access allow all # cache mem doesn't take higher value here. we have 8 GB on machine, is 4 GB max? cache_mem 3994 MB maximum_object_size 65536 KB maximum_object_size_in_memory 64 KB memory_replacement_policy heap GDSF cache_replacement_policy heap LFUDA cache_dir aufs /var/spool/squid3 16384 32 512 half_closed_clients off coredump_dir /var/spool/squid3 half_closed_clients off cache_store_log none logfile_rotate 3 ipcache_size 16384 ipcache_low 98 ipcache_high 99 log_fqdn off half_closed_clients off reload_into_ims on httpd_suppress_version_string off
Re: [squid-users] Commercial Squid tweak speeds things up significantly!
I apologize for not seeing the entire thread on this topic. It is quite possible that my Squid is somehow terribly reconfigured and that could be the reason why the other implementation seems faster. I've sent my config file in the previous email. I can change the config and do comparison again to get to the bottom of it... On Tue, Nov 29, 2011 at 1:32 PM, - Mikael - wrote: > Thanks, Amos. I don't understand why their implementation is many > times faster than mine unless my Squid is horribly (mis)configured. > > Is there anything in this config that would slow down Squid? > > http_port 3128 > > refresh_pattern http://.*\.adobe\.com/ 0 80% 20160 reload-into-ims > refresh_pattern http://.*\.windowsupdate\.microsoft\.com/ 0 80% 20160 > reload-into-ims > refresh_pattern http://office\.microsoft\.com/ 0 80% 20160 reload-into-ims > refresh_pattern http://windowsupdate\.microsoft\.com/ 0 80% 20160 > reload-into-ims > refresh_pattern http://w?xpsp[0-9]\.microsoft\.com/ 0 80% 20160 > reload-into-ims > refresh_pattern http://w2ksp[0-9]\.microsoft\.com/ 0 80% 20160 reload-into-ims > refresh_pattern http://download\.microsoft\.com/ 0 80% 20160 reload-into-ims > refresh_pattern http://download\.macromedia\.com/ 0 80% 20160 reload-into-ims > > refresh_pattern -i \.flv$ 10080 90% 99 ignore-no-cache > override-expire ignore-private > refresh_pattern -i \.index.(html|htm)$ 0 40% 10080 > refresh_pattern -i \.(html|htm|css|js)$ 1440 40% 40320 > refresh_pattern -i \.(gif|png|jpg|jpeg|ico)$ 10080 90% 43200 > override-expire ignore-no-cache ignore-no-store ignore-private > refresh_pattern -i \.(iso|avi|wav|mp3|mp4|mpeg|swf|x-flv)$ 43200 90% > 432000 override-expire ignore-no-cache ignore-no-store ignore-private > refresh_pattern -i > \.(deb|rpm|exe|zip|tar|tgz|ram|rar|bin|ppt|doc|tiff)$ 10080 90% 43200 > override-expire ignore-no-cache ignore-no-store ignore-private > > refresh_pattern ^gopher: 1440 0% 1440 > refresh_pattern ^ftp: 144000 20% 1008000 > refresh_pattern -i \.index.(html|htm)$ 1440 90% 40320 > refresh_pattern -i \.(html|htm|css|js)$ 1440 90% 40320 > refresh_pattern (/cgi-bin/|\?) 0 0% 0 > > acl manager proto cache_object > acl localhost src 127.0.0.1/32 > acl to_localhost dst 127.0.0.0/8 > acl schoolip src 10.0.0.1 > > acl SSL_ports port 443 > acl Safe_ports port 80 # http > acl Safe_ports port 21 # ftp > acl Safe_ports port 443 # https > acl Safe_ports port 70 # gopher > acl Safe_ports port 210 # wais > acl Safe_ports port 1025-65535 # unregistered ports > acl Safe_ports port 280 # http-mgmt > acl Safe_ports port 488 # gss-http > acl Safe_ports port 591 # filemaker > acl Safe_ports port 777 # multiling http > acl CONNECT method CONNECT > > http_access allow manager localhost > http_access deny manager > http_access deny !Safe_ports > http_access deny CONNECT !SSL_ports > http_access allow localhost > http_access allow schoolip > http_access allow test > #http_access allow all > > cache_mem 3994 MB > maximum_object_size 65536 KB > maximum_object_size_in_memory 64 KB > memory_replacement_policy heap GDSF > cache_replacement_policy heap LFUDA > cache_dir aufs /var/spool/squid3 16384 32 512 > half_closed_clients off > coredump_dir /var/spool/squid3 > half_closed_clients off > cache_store_log none > cache_access_log none > logfile_rotate 3 > ipcache_size 16384 > ipcache_low 98 > ipcache_high 99 > log_fqdn off > half_closed_clients off > reload_into_ims on > > cache_mgr help@my_domain_name.org > visible_hostname gw.my_domain_name.org > httpd_suppress_version_string off
Re: [squid-users] Commercial Squid tweak speeds things up significantly!
Thanks, Amos. I don't understand why their implementation is many times faster than mine unless my Squid is horribly (mis)configured. Is there anything in this config that would slow down Squid? http_port 3128 refresh_pattern http://.*\.adobe\.com/ 0 80% 20160 reload-into-ims refresh_pattern http://.*\.windowsupdate\.microsoft\.com/ 0 80% 20160 reload-into-ims refresh_pattern http://office\.microsoft\.com/ 0 80% 20160 reload-into-ims refresh_pattern http://windowsupdate\.microsoft\.com/ 0 80% 20160 reload-into-ims refresh_pattern http://w?xpsp[0-9]\.microsoft\.com/ 0 80% 20160 reload-into-ims refresh_pattern http://w2ksp[0-9]\.microsoft\.com/ 0 80% 20160 reload-into-ims refresh_pattern http://download\.microsoft\.com/ 0 80% 20160 reload-into-ims refresh_pattern http://download\.macromedia\.com/ 0 80% 20160 reload-into-ims refresh_pattern -i \.flv$ 10080 90% 99 ignore-no-cache override-expire ignore-private refresh_pattern -i \.index.(html|htm)$ 0 40% 10080 refresh_pattern -i \.(html|htm|css|js)$ 1440 40% 40320 refresh_pattern -i \.(gif|png|jpg|jpeg|ico)$ 10080 90% 43200 override-expire ignore-no-cache ignore-no-store ignore-private refresh_pattern -i \.(iso|avi|wav|mp3|mp4|mpeg|swf|x-flv)$ 43200 90% 432000 override-expire ignore-no-cache ignore-no-store ignore-private refresh_pattern -i \.(deb|rpm|exe|zip|tar|tgz|ram|rar|bin|ppt|doc|tiff)$ 10080 90% 43200 override-expire ignore-no-cache ignore-no-store ignore-private refresh_pattern ^gopher:14400% 1440 refresh_pattern ^ftp: 144000 20% 1008000 refresh_pattern -i \.index.(html|htm)$ 1440 90% 40320 refresh_pattern -i \.(html|htm|css|js)$ 1440 90% 40320 refresh_pattern (/cgi-bin/|\?) 0 0% 0 acl manager proto cache_object acl localhost src 127.0.0.1/32 acl to_localhost dst 127.0.0.0/8 acl schoolip src 10.0.0.1 acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT http_access allow manager localhost http_access deny manager http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost http_access allow schoolip http_access allow test #http_access allow all cache_mem 3994 MB maximum_object_size 65536 KB maximum_object_size_in_memory 64 KB memory_replacement_policy heap GDSF cache_replacement_policy heap LFUDA cache_dir aufs /var/spool/squid3 16384 32 512 half_closed_clients off coredump_dir /var/spool/squid3 half_closed_clients off cache_store_log none cache_access_log none logfile_rotate 3 ipcache_size 16384 ipcache_low 98 ipcache_high 99 log_fqdn off half_closed_clients off reload_into_ims on cache_mgr help@my_domain_name.org visible_hostname gw.my_domain_name.org httpd_suppress_version_string off
Re: [squid-users] Commercial Squid tweak speeds things up significantly!
> That sounds like heap LRU policy, > > it is not fear to compare a squid out of the box with a solution that > has a tunning in parameters. Pesonally, i've develop an algorithm to > save you 30% of bandwith with some tunnings based on statistical > measures. So it is easy, you dont have to waste money. >From my experience high caching percentage and perceived speed gain to end-users could be two completely separate things and are not necessarily related. I'm curious what Amos thinks about the Untagle implementation and if its possible to do miss bypass (or having clients fetch the content) implemented with just config file tweaks. If its just LRU policy than that would be great.
Re: [squid-users] Commercial Squid tweak speeds things up significantly!
> Could you name this product and point at some documentation it has about > this process? Its called Untangle. Its a Debian based distro with Squid based "web caching app". FAQ about caching app with some tech details is available at: wiki.untangle.com/index.php/Web_Cache_FAQs
[squid-users] Commercial Squid tweak speeds things up significantly!
Our school dept wants to buy a commercial proxy (Squid based) which seems to work a whole lot faster than the standard installation (of Squid). The performance difference between the two Squid's seems to be in how commercial Squid implementation is handling a missed object. >From what I understand their Squid implementation caches the content in a normal way, but once a client gets a cache miss, then their Squid allows the client to fetch the record -- apparently Squid doesn't do it for the client which seems to make the big performance difference. Once the object is fetched by the client, Squid intercepts it and stores the object for the other clients. I tested this and that implementation really, really speeds things up. Now I hate to admit this but this commercial product is subscription based and that's the money which I would rather spend on students and teachers, school supplies for them etc. Is it possible to configure Squid that way by modifying config file, or this is more involved than just editing config file? Thanks!
[squid-users] Am I doing this right?
I'm not sure what to make out of these stats. Is there anything here that I should pay attention to and possible correct the config file to do it better? Squid Object Cache: Version 3.1.6 Start Time: Mon, 17 Oct 2011 16:39:04 GMT Current Time: Mon, 17 Oct 2011 19:20:31 GMT Connection information for squid: Number of clients accessing cache: 2 Number of HTTP requests received: 130084 Number of ICP messages received:0 Number of ICP messages sent:0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 805.7 Average ICP messages per minute since start:0.0 Select loop called: 8723042 times, 1.111 ms avg Cache information for squid: Hits as % of all requests: 5min: 12.7%, 60min: 21.4% Hits as % of bytes sent:5min: 4.3%, 60min: 11.5% Memory hits as % of hit requests: 5min: 14.4%, 60min: 26.7% Disk hits as % of hit requests: 5min: 19.9%, 60min: 38.2% Storage Swap size: 1516220 KB Storage Swap capacity: 9.0% used, 91.0% free Storage Mem size: 309964 KB Storage Mem capacity:7.6% used, 92.4% free Mean Object Size: 20.82 KB Requests given to unlinkd: 0 Median Service Times (seconds) 5 min60 min: HTTP Requests (All): 0.08265 0.05331 Cache Misses: 0.09736 0.07409 Cache Hits:0.0 0.0 Near Hits: 0.01745 0.02317 Not-Modified Replies: 0.0 0.0 DNS Lookups: 0.01686 0.01686 ICP Queries: 0.0 0.0 Resource usage for squid: UP Time:9687.457 seconds CPU Time: 219.870 seconds CPU Usage: 2.27% CPU Usage, 5 minute avg:1.39% CPU Usage, 60 minute avg: 1.59% Process Data Segment Size via sbrk(): 409320 KB Maximum Resident Size: 1688048 KB Page faults with physical i/o: 0 Memory usage for squid via mallinfo(): Total space in arena: 409452 KB Ordinary blocks: 409348 KB418 blks Small blocks: 0 KB 0 blks Holding blocks: 27748 KB 12 blks Free Small blocks: 0 KB Free Ordinary blocks: 103 KB Total in use: 437096 KB 100% Total free: 103 KB 0% Total size:437200 KB Memory accounted for: Total accounted: 381533 KB 87% memPool accounted: 381533 KB 87% memPool unaccounted:55666 KB 13% memPoolAlloc calls: 31136974 memPoolFree calls: 31264543 File descriptor usage for squid: Maximum number of file descriptors: 65535 Largest file desc currently in use:429 Number of file desc currently in use: 235 Files queued for open: 0 Available number of file descriptors: 65300 Reserved number of file descriptors: 100 Store Disk files open: 1 Internal Data Structures: 72866 StoreEntries 43376 StoreEntries with MemObjects 43375 Hot Object Cache Items 72839 on-disk objects
Re: [squid-users] Question about configuration directive http_port
What are the benefits of having Squid on the LAN? Our firewall (Sonicwall NSA) explicitly forbids proxies on the LAN for some reason. The firewall will forward all traffic to Squid only if its on public IP address. This is how we are setup right now: (LAN) -> (Sonicwall firewall, NAT, DPI, DHCP) -> (Squid) -> WWW WAN routing is done by the ISP's router that's on site. Latency from LAN to Squid box is <1ms.
[squid-users] Question about configuration directive http_port
What is the best http_port directive to use in this scenario? Squid has a public IP. All clients are behind NAT firewall with public IP. Firewall sends all NATed client traffic to Squid. Squid fetches the site, returns the hit back to the firewall which then forwards it to appropriate clients. Clients are unaware of the squid proxying any traffic. Thanks.
[squid-users] Please help me with refresh statements
Is there a cleaner and more effective way to use these refresh_patterns? This is what I have so far. I would love to cache more of FLV content and specific sites that get lots of traffics, such as yahoo and news.google.com refresh_pattern -i \.flv$ 10080 90% 99 ignore-no-cache override-expire ignore-private refresh_pattern -i \.index.(html|htm)$ 0 40% 10080 refresh_pattern -i \.(html|htm|css|js)$ 1440 40% 40320 refresh_pattern -i \.(gif|png|jpg|jpeg|ico)$ 10080 90% 43200 override-expire ignore-no-cache ignore-no-store ignore-private refresh_pattern -i \.(iso|avi|wav|mp3|mp4|mpeg|swf|x-flv)$ 43200 90% 432000 override-expire ignore-no-cache ignore-no-store ignore-private refresh_pattern -i \.(deb|rpm|exe|zip|tar|tgz|ram|rar|bin|ppt|doc|tiff)$ 10080 90% 43200 override-expire ignore-no-cache ignore-no-store ignore-private refresh_pattern ^gopher:14400% 1440 refresh_pattern ^ftp: 144000 20% 1008000 refresh_pattern (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320
[squid-users] Squid latency?
Two quick questions: I have big latency on the Squid box. It takes about 5 seconds from the time I enter URL address until the page loads. When the page loads it loads very quickly. DNS queries to WAN are <2ms, average ISP tracert and ping latencies are around 20ms. Tracert and ping from LAN to the proxy are <1ms. Is there something in the Squid (an option, log, or util) that would help me figure this out? This is less important but I see lots these messages in the log. Is this something I should be concerned about? 1318650368.497 0 IP-address-removed NONE/400 3371 +?AO tqyU - NONE/- text/html 1318650374.794 0 IP-address-removed NONE/400 3371 +?AO tqyU - NONE/- text/html 1318650381.041 0 IP-address-removed NONE/400 3371 +?AO tqyU - NONE/- text/html 1318650387.434 0 IP-address-removed NONE/400 3371 +?AO tqyU - NONE/- text/html 1318650393.880 0 IP-address-removed NONE/400 3371 +?AO tqyU - NONE/- text/html 1318650400.340 0 IP-address-removed NONE/400 3371 +?AO tqyU - NONE/- text/html 1318650406.872 0 IP-address-removed NONE/400 3371 +?AO tqyU - NONE/- text/html 1318650418.189 0 IP-address-removed NONE/400 3371 +?AO tqyU - NONE/- text/html 1318650424.487 0 IP-address-removed NONE/400 3371 +?AO tqyU - NONE/- text/html 1318650430.685 0 IP-address-removed NONE/400 3371 +?AO tqyU - NONE/- text/html 1318650436.982 0 IP-address-removed NONE/400 3371 +?AO tqyU - NONE/- text/html 1318650443.326 0 IP-address-removed NONE/400 3371 +?AO tqyU - NONE/- text/html Squid Cache: Version 3.1.6 Debian 6.1 Linux 2.6.32-5-686-bigmem 250GB SATA, 8GB memory, 3GHz dual-core
