[squid-users] ICAP service squid 3.3.8
Hi, i am using squid 3.3.8 on Centos 6.4, i am also using qlproxy. Lately seeing these errors every few minutes? any ideas on what can cause this? suspending ICAP service for too many failures essential ICAP service is suspended:icap://127.0.0.1:1344/mw[down,susp,fail11]
Re: [squid-users] Basic questions on transparent/intercept proxy
- Original Message - > From: csn233 > To: "squid-users@squid-cache.org" >To intercept HTTPS traffic, is SSL-bump a must? Even when I only want > to record the CONNECT traffic in access.log just like a normal forward > proxy without decrypting anything? No. But it will log only IPs not the host name or URL. Amm
[squid-users] Squid monitoring, access report shows upto 5 % to 7 % cache usage
Hi All For purpose of convincing about the squid cahce, I installed and started monitoring SQUID using mysar "http://sourceforge.net/projects/mysar/"; Is this % too low or average how much should a optimized squid setup give. Or should I dig into some more configuration to pump the cache % thanks Joseph John
Re: [squid-users] Squid monitoring, access report shows upto 5 % to 7 % cache usage
On Monday 29 July 2013 at 12:36:05, John Joseph wrote: > Hi All > > For purpose of convincing about the squid cahce, I installed and started > monitoring SQUID using mysar "http://sourceforge.net/projects/mysar/"; > > Is this % too low or average how much should a optimized squid setup give. > Or should I dig into some more configuration to pump the cache % How large is your cache? How many users are going via Squid? How long have they been using it (how long has Squid been building up its cache)? How many connection requests do you have per minute/hour/day/whatever makes sense? How many cache HITs do you see in the log file compared to MISSes (ie: how often are cached objects being requested)? Any other relevant information you can supply might be helpful in providing an answer. Regards, Antony. -- Users don't know what they want until they see what they get. Please reply to the list; please don't CC me.
Re: [squid-users] ICAP service squid 3.3.8
On 07/29/2013 12:22 PM, JC Putter wrote: > Hi, i am using squid 3.3.8 on Centos 6.4, i am also using qlproxy. > Lately seeing these errors every few minutes? any ideas on what can > cause this? > > suspending ICAP service for too many failures > essential ICAP service is suspended:icap://127.0.0.1:1344/mw[down,susp,fail11] > What ICAP Service and is it on VM or PM? also is it a 64 bit or 32 bit system? can you give the cache.log output so we can see the problem? Eliezer
[squid-users] 3.4.0.1 Works great.
the basic test that result in: 1375097944.775551 127.0.0.1 TCP_REFRESH_UNMODIFIED/304 355 GET http://www.squid-cache.org/Images/img7.gif - HIER_DIRECT/209.169.10.131 - 1375097950.082763 127.0.0.1 TCP_CLIENT_REFRESH_MISS/200 10345 GET http://www.squid-cache.org/Versions/v3/3.4/ - HIER_DIRECT/209.169.10.131 text/html Shows that the client did an active refresh. All the above is without refresh_patterns at all. Also works on youtube images and other stuff pretty nice.. Youtube videos are pretty large and for me it seems like a weird thing to cache in a cache since they have servers all over the globe doing cache helping the clients view the video nicely.. 1375098872.573 72 127.0.0.1 TCP_REFRESH_UNMODIFIED/304 212 GET http://i1.ytimg.com/vi/HbjUK7i84YQ/default.jpg - HIER_DIRECT/212.199.219.236 - 1375098872.944 0 127.0.0.1 TCP_MEM_HIT/200 628 GET http://r5---sn-nhpax-ua8e.c.youtube.com/crossdomain.xml - HIER_NONE/- text/x-cross-domain-policy 1375098873.084141 127.0.0.1 TCP_MISS/204 424 GET http://s.youtube.com/stream_204? - HIER_DIRECT/212.199.205.241 text/html Eliezer
[squid-users] MX issues? (was Re: Basic questions on transparent/intercept proxy)
Is there some issue with mailing list? (I am assuming Yahoo! mail would not have issue) My past two-three e-mails were delivered much late to list. One even bounced back. (which I resent) The one below was delivered after more than 24hrs or so. It appears only one MX is working. squid-cache.org mail exchanger = 10 squid-cache.org. squid-cache.org mail exchanger = 90 mx2.squid-cache.org. mx2 does not seem to be working. Regards, Amm. - Original Message - > From: Amm > To: "squid-users@squid-cache.org" > Cc: > Sent: Sunday, 28 July 2013 6:41 PM > Subject: Re: [squid-users] Basic questions on transparent/intercept proxy > - Original Message - >> From: csn233 >> To: "squid-users@squid-cache.org" > > >> To intercept HTTPS traffic, is SSL-bump a must? Even when I only want >> to record the CONNECT traffic in access.log just like a normal forward >> proxy without decrypting anything? > > No. But it will log only IPs not the host name or URL. > > Amm >
[squid-users] squid cache-server misses all
hi guys, I have squid-3.1.20 installed on ubuntu-12.10. I didn't change much in squid's default configurations. I just defined "cache_dir" and "nameservers" and enabled "localnet". now when I open any webpages from client, in my access.log I see lots and lots of this message (for almost all contents): ... 1375088677.969 3341 192.168.0.81 TCP_MISS/200 111920 GET http://www.imdb.com/ - DIRECT/72.21.215.52 text/html ... 1375088683.295 479 192.168.0.81 TCP_MISS/200 648 GET http://s.media-imdb.com/twilight/? - DIRECT/72.21.215.52 image/gif I look at squid's cache folder and I see size of cache folder grows (so content is being cached?) but apparently no cached content is being used. I googled but didn't find anything that help me. is there anything I'm missing? Does anyone have a thought or suggestion? thank you.
Re: [squid-users] 3.4.0.1 Works great.
I just realized that my squid.conf for 3.3.8 is unusable with 3.4.0.1 just because of the refresh patterns. Funnily, even the default refresh patterns are seen as bungled. Same config is usable with 3.HEAD-20130717-r12946 Here are mine, which are ALL rejected as bungled - all commented out, but still even the default ones are 'bungled'. # various windows versions #!refresh_pattern http://.*\.windowsupdate\.microsoft\.com/ 0 80% 20160 #!refresh_pattern http://.*\.update\.microsoft\.com/ 0 80% 20160 #!refresh_pattern http://download\.microsoft\.com/ 0 80% 20160 #!refresh_pattern http://windowsupdate\.microsoft\.com/ 0 80% 20160 #!refresh_pattern http://office\.microsoft\.com/ 0 80% 20160 #!refresh_pattern http://w?xpsp[0-9]\.microsoft\.com/ 0 80% 20160 #!refresh_pattern http://w2ksp[0-9]\.microsoft\.com/ 0 80% 20160 #!refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 90% 43200 #!refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 90% 43200 #refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi) 10080 100% 43200 #refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi) 10080 100% 43200 #refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi) 10080 100% 43200 #refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi) 4320 100% 43200 # and some other windows updaters #!refresh_pattern http://download\.macromedia\.com/ 0 80% 20160 #!refresh_pattern ftp://ftp\.nai\.com/ 0 80% 20160 #!refresh_pattern http://ftp\.software\.ibm\.com/ 0 80% 20160 # Others #!refresh_pattern -i \.mp4 1440 90% 43200 #!refresh_pattern -i \.(gif|png|jpg|jpeg|ico)$ 40320 75% 86400 #!refresh_pattern -i \.(iso|avi|wav|mp3|mpeg|swf|flv|x-flv)$ 1440 40% 40320 #!refresh_pattern -i \.(css|js)$ 300 40% 7200 #!refresh_pattern -i \.(html|htm)$ 300 40% 7200 And the default ones: # Add any of your own refresh_pattern entries above these. # refresh_pattern ^ftp: 144020% 10080 refresh_pattern ^gopher:14400% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 [root@jaribu] /opt/squid34/etc# ../sbin/squid -v Squid Cache: Version 3.4.0.1 configure options: '--prefix=/opt/squid34' '--enable-removal-policies=lru heap' '--disable-epoll' '--enable-auth' '--enable-auth-basic=DB NCSA PAM MSNT PAM POP3 SMB SSPI MSNT' '--enable-external-acl-helpers=session unix_group wbinfo_group file_userip' '--enable-auth-ntlm=smb_lm SSPI' '--enable-auth-negotiate=SSPI kerberos' '--with-pthreads' '--enable-storeio=ufs diskd aufs' '--enable-delay-pools' '--enable-snmp' '--with-openssl=/usr' '--enable-forw-via-db' '--enable-cache-digests' '--enable-wccpv2' '--enable-follow-x-forwarded-for' '--with-large-files' '--enable-large-cache-files' '--enable-error_default_language=English' '--enable-esi' '--enable-kqueue' '--enable-icap-client' '--enable-kill-parent-hack' '--enable-ssl' '--enable-leakfinder' '--enable-ssl-crtd' '--enable-url-rewrite-helpers' '--enable-xmalloc-statistics' '--enable-stacktraces' '--enable-zph-qos' '--enable-eui' '--enable-pf-transparent' '--enable-ipf-transparent' --enable-ltdl-convenience [root@jaribu] /opt/squid34/etc# ../sbin/squid -k parse 2013/07/29 18:35:04| Startup: Initializing Authentication Schemes ... 2013/07/29 18:35:04| Startup: Initialized Authentication Scheme 'basic' 2013/07/29 18:35:04| Startup: Initialized Authentication Scheme 'digest' 2013/07/29 18:35:04| Startup: Initialized Authentication Scheme 'negotiate' 2013/07/29 18:35:04| Startup: Initialized Authentication Scheme 'ntlm' 2013/07/29 18:35:04| Startup: Initialized Authentication. 2013/07/29 18:35:04| Processing Configuration File: /opt/squid34/etc/squid.conf (depth 0) 2013/07/29 18:35:04| Processing: http_port 13128 2013/07/29 18:35:04| Processing: icp_port 3130 2013/07/29 18:35:04| Processing: icp_query_timeout 0 2013/07/29 18:35:04| Processing: mcast_icp_query_timeout 2000 2013/07/29 18:35:04| Processing: dead_peer_timeout 10 seconds 2013/07/29 18:35:04| Processing: cache_mem 128 MB 2013/07/29 18:35:04| Processing: maximum_object_size 4194240 KB 2013/07/29 18:35:04| Processing: maximum_object_size_in_memory 1024 KB 2013/07/29 18:35:04| Processing: balance_on_multiple_ip off 2013/07/29 18:35:04| Processing: logformat combined %>a %ui %un [%tl] "%rm %"ru HTTP/%rv" %>Hs %h" "%{User-Agent}>h" %Ss:%Sh 2013/07/29 18:35:04| Processing: cache_dir aufs /usr/local/squid/cache 5120 16 256 2013/07/29 18:35:04| Processing: access_log stdio:/usr/local/squid/logs/access.log squid 2013/07/29 18:35:04| Processing: cache_log /usr/local/squid/logs/cache.log squid 2013/07/29 18:35:04| Processing: cache_store_log none 2013/07/29 18:35:04| Processing: coredump_dir /usr/local/squid/logs 2013/07/29 18:35:04| Processing: pid_filename /usr/local/squid/logs/squid.pid 2013/07/29 18:35:04| Processing: refresh_pattern^ftp: 144020% 10080 2013/07/29 18:35:04| ERROR: A percentage value is missing. FAT
Re: [squid-users] Basic questions on transparent/intercept proxy
On Sun, Jul 28, 2013 at 9:11 PM, Amm wrote: > - Original Message - > >> From: csn233 >> To: "squid-users@squid-cache.org" > >>To intercept HTTPS traffic, is SSL-bump a must? Even when I only want >> to record the CONNECT traffic in access.log just like a normal forward >> proxy without decrypting anything? > > No. But it will log only IPs not the host name or URL. > > Amm No, as in ssl-bump is not a requirement for HTTPS traffic to be logged? Your answer seems to be different from other replies. Can you provide examples of how?
Re: [squid-users] squid cache-server misses all
On 30/07/2013 1:49 a.m., ana any wrote: hi guys, I have squid-3.1.20 installed on ubuntu-12.10. I didn't change much in squid's default configurations. I just defined "cache_dir" and "nameservers" and enabled "localnet". now when I open any webpages from client, in my access.log I see lots and lots of this message (for almost all contents): ... 1375088677.969 3341 192.168.0.81 TCP_MISS/200 111920 GET http://www.imdb.com/ - DIRECT/72.21.215.52 text/html ... 1375088683.295479 192.168.0.81 TCP_MISS/200 648 GET http://s.media-imdb.com/twilight/? - DIRECT/72.21.215.52 image/gif I look at squid's cache folder and I see size of cache folder grows (so content is being cached?) but apparently no cached content is being used. In direct accordance with IMDB cache control rules about what may be done with their website. The first of those URLs presents an index page containing proprietary and copyright infromation under a limited license. It also contains embeded user-specific content. It is marked as *private* information for the individual client and may not be stored by shared caches such as Squid. The second URL is unclear exactly since you have cropped off all the query-string portion which would aid us in identifying the exact problem. Other similar URLs are marked with "no-cache, must-revalidate" and an Expiry timestamp long in the past. To cause HTTP/1.0 software (such as squid-3.1) not to cache or to revalidate on every use, and HTTP/1.1 compliant softwrae (such as squid-3.2 and later) to cache it but revalidate before any use. I googled but didn't find anything that help me. is there anything I'm missing? Does anyone have a thought or suggestion? Best thing to do is upgrade to 3.2 or 3.3. There are refresh_pattern directive options which can fine tune behavioru for unfriendly site. However I don't advise doing anything with those particular URLs due to the embeded user-specific content IMDB adds to their pages (you do not want to be responsible for your users getting each others IMDB account details or notices). If the problem is more widespread in other sites though you may want to share your whole config file (without the # comment lines) and we may be able to find something to help. Amos
Re: [squid-users] Basic questions on transparent/intercept proxy
> From: csn233 >Sent: Monday, 29 July 2013 10:40 PM >Subject: Re: [squid-users] Basic questions on transparent/intercept proxy >On Sun, Jul 28, 2013 at 9:11 PM, Amm wrote: >> - Original Message - >> >>> From: csn233 >>> To: "squid-users@squid-cache.org" >> >>>To intercept HTTPS traffic, is SSL-bump a must? Even when I only want >>> to record the CONNECT traffic in access.log just like a normal forward >>> proxy without decrypting anything? >> >> No. But it will log only IPs not the host name or URL. >> >> Amm >No, as in ssl-bump is not a requirement for HTTPS traffic to be >logged? Your answer seems to be different from other replies. Can you >provide examples of how? I am not sure if I understood your previous question right. I think what others said is right. Here is what I have done. (simplified version) https_port 8081 intercept ssl-bump generate-host-certificates=on cert=/etc/squid/ssl_cert/squid.pem #ssl_bump none all #<--- this line is not required So ssl-bump as a keyword is required on https_port but you dont need ssl_bump ACL line (by default it bumps nothing). Traffic will be logged just as IP. (Not actual hostname) Regards, Amm.
Re: [squid-users] Squid monitoring, access report shows upto 5 % to 7 % cache usage
Hi Antony thanks for the reply, I have filed up the corresponding information below -> Original Message - >From: Antony Stone >To: "squid-users@squid-cache.org" >Cc: >Sent: Monday, 29 July 2013 2:45 PM >Subject: Re: [squid-users] Squid monitoring, access report shows upto 5 % to 7 >% cache usage >On Monday 29 July 2013 at 12:36:05, John Joseph wrote: > >Hi All > > > >For purpose of convincing about the squid cahce, I installed and started > >monitoring SQUID using mysar "http://sourceforge.net/projects/mysar/";; > > >> Is this % too low or average how much should a optimized squid setup give. >> Or should I dig into some more configuration to pump the cache % > How large is your cache? It is around 556GB, my cache dir details in squid.conf are cache_dir aufs /opt/var/spool/squid 57 32 256 > How many users are going via Squid? At present not more than 250 users, but concurrent users will not be more than 50 users > How long have they been using it (how long has Squid been building up its cache)? 3 months > How many connection requests do you have per minute/hour/day/whatever makes sense? Can you please advice me on how to check for this parameter > How many cache HITs do you see in the log file compared to MISSes (ie: how often are cached objects being requested)? Can you please advice me on how to check for this parameter > Any other relevant information you can supply might be helpful in providing > an answer. from the mysar, I get the following information, which I am pasting here USERS HOSTS SITES TRAFFIC Cache Percent 12 990 18336 47.71G 5.00% 21 1020 26877 86.57G 5.00% 31 1017 24811 105.67G 4.00% 59 1024 29185 119.15G 4.00% 41 1022 27245 106.82G 5.00% 25 1022 26785 86.68G 5.00% 29 1019 26609 71.94G 6.00% 23 1019 24863 79.48G 5.00% 20 1016 24334 77.44G 5.00% 43 1015 25206 72.18G 5.00% 18 1016 24187 81.28G 5.00% 38 1021 25538 80.74G 6.00% 30 1021 27427 85.19G 5.00% 31 1021 27369 96.13G 5.00% 29 1019 23879 74.26G 5.00% 28 1014 20715 70.90G 5.00% 22 1010 22537 69.59G 5.00% 22 1021 23031 86.52G 4.00% 19 1015 22542 73.21G 5.00% 16 1020 22408 74.67G 5.00% 23 1021 23594 72.99G 5.00% 28 1021 23408 71.97G 6.00% 17 1006 21390 64.28G 5.00% 23 994 22685 61.42G 5.00% 20 1016 25792 71.54G 5.00% 24 1017 25178 74.03G 5.00% 37 1019 29740 83.19G 5.00% 22 1020 25175 77.47G 5.00% Guidance and advice requested. Thanks Joseph John > Regards, > Antony. -- Users don't know what they want until they see what they get. Please reply to the list; please don't CC me.