Re: [squid-users] Refresh Problem
Hey, What version of squid? (squid -v) also notice that it is expected to be like that if the headers are pointing to this situation. There is a difference between ctrl+f5 to plain old "f5" or at-least it's that way in firefox.. Incomplete pages are internet network problem and not squid. try using some debug_options ALL,2 to see the headers and analyse the reason for this problem. Notice that sometimes nginx can provide plural headers that some cache maintainers can "exploit" and make a non cached reply to be cached for very long time. Eliezer On 08/01/2013 10:16 PM, Gustavo Esquivel wrote: > Hi, > actually i'm using squid proxy and all works fine but i have a little > problem... > sometime the users must to press a couple times F5 button to refresh > the website because they receive not found page or incomplete page. > > anybody have an idea? > thanks a lot!! > > Best Regards, > Gustavo Esquivel >
Re: [squid-users] Refresh Problem
Hi Gustavo, sounds like your internet channel is not reliable enough. Anyway if you think the problem is in squid itself you should consider analysing access.log and cache.log with debug output. Best wishes, Pavel On 08/01/2013 10:16 PM, Gustavo Esquivel wrote: Hi, actually i'm using squid proxy and all works fine but i have a little problem... sometime the users must to press a couple times F5 button to refresh the website because they receive not found page or incomplete page. anybody have an idea? thanks a lot!! Best Regards, Gustavo Esquivel
[squid-users] Refresh Problem
Hi, actually i'm using squid proxy and all works fine but i have a little problem... sometime the users must to press a couple times F5 button to refresh the website because they receive not found page or incomplete page. anybody have an idea? thanks a lot!! Best Regards, Gustavo Esquivel
Fw: [squid-users] Squid 3.4 refresh_pattern bungled config
Hi, Amos. It is possible to make "configuration_includes_quoted_values off " work on refresh_pattern, like he does with acls. Regards. Dzieva. -Mensagem Original- From: Amos Jeffries Sent: Thursday, August 01, 2013 12:42 AM To: squid-users@squid-cache.org Subject: Re: [squid-users] Squid 3.4 refresh_pattern bungled config On 2013-08-01 09:46, Tyler Sweet wrote: Hello, I'm having an odd issue with Squid 3.4 on FreeBSD 9.1, compiled from source. I'm not sure if this is a good place to ask for general support for 3.4 or not, but I figured I would try. squid-dev are working on this right now. 3.4.0.2 will be out as soon as we have a good fix for it. Amos
[squid-users] Squidblacklists Now Commercial ONLY?!
Hello Nichols (Squid Blacklists Team) & Fellow Squid Users, Please enlighten me as I seem to be confused. A couple of weeks back I realized accumulative 401.shtml files in my Squid Blacklists update script folder only to check the website and I must subscribe (and pay). When did this take effect without warning given Squid Users (this forum) contributed to the project?! # Edmonds
Re: [squid-users] ICAP failure when using clamav scan denied reply
On 1/08/2013 9:24 p.m., John Xue wrote: I am using squid 3.2.3 + http://www.squid-cache.org/Versions/v3/3.2/changesets/squid-3.2-11699.patch(Polish: replace several assert(isOpen(fd))) + c-icap 0.1.7 + squidclamav 6.9 + squidGuard 1.4 as default I deny all application/octet-stream reply access, and disable virus scan picture ^.*\.(ico|gif|png|jpg)$ in squidclamav. NP: it would be a better idea to setup an ACL so that those disabled replies are not even sent to the ICAP scanner. Particularly if the problem is an overloaded scanner. It will save you a lot of I/O buffering and parsing work in Squid along with similar resource consumptions and CPU cycles in the scanner. my problem is when user try to access a link that end of .gif but reply content type is application/octet-stream, c-icap will store that content to /var/tmp and keep it, then die in icap process, even I reload icap. my squid log 1375345064.448 6471 1.1.2.3 TCP_DENIED_REPLY/403 9044 GET http://bbs.chinaacc.com/getresource.php?thumb=1&rid=104959 user_Name FIRSTUP_PARENT/1.1.2.2 text/html NOTE: The above request URL does not contain any match for ^.*\.(ico|gif|png|jpg)$ firebug report: GET getresource.php?thumb=1&rid=104959 200 OKbbs.chinaacc.com 3.7 MB1.1.2.2:80008.43s ParamsHeadersResponseCookies Response Headersview source Connectionkeep-alive Content-Dispositioninline; filename="62037b5agw1droqc7t0qeg.gif" Is this what you are looking at when you call it a .gif? This is the content disposition. It is simply a hint for what the browser is to call the file if it were being saved to a file. Content-Encodingnone Content-Length3924554 Content-Typeapplication/octet-stream This is the type of object which is being transferred. As you can see it is not a GIF type. I agree it is probably wrong type assignment by the sender, but we (and definitely Squid) has no way to be certain of that. So in the end what this comes down to is whether squidclamav tool treats content-disposition filename as if it were the URL for deciding the abort/abandon handling. I suspect it is doing the same thing as Squid using just the request URL, since there is no mention of content-disposition in the changelog and that has only recently been standardized as a header. Amos
[squid-users] Re: Squid monitoring, access report shows upto 5 % to 7 % cache usage
The relatively low byte-hitrate gives the idea, that somewhere in your squid.conf there is a limitation on the max. objects size to be cached. It might be a good idea, to modify this one, to a larger value. Caus it seems, you still have a lot of disk space available for caching. So you might post your squid.conf here. And, the output of squidclient -h 127.0.0.1 mgr:storedir -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-monitoring-access-report-shows-upto-5-to-7-cache-usage-tp4661301p4661399.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] Squid 3.4 refresh_pattern bungled config
On 08/01/2013 06:42 AM, Amos Jeffries wrote: > On 2013-08-01 09:46, Tyler Sweet wrote: >> Hello, >> >> I'm having an odd issue with Squid 3.4 on FreeBSD 9.1, compiled from >> source. I'm not sure if this is a good place to ask for general >> support for 3.4 or not, but I figured I would try. > > squid-dev are working on this right now. 3.4.0.2 will be out as soon as > we have a good fix for it. > > Amos The new RPM will be ready once 3.4.0.2 will be out.. Eliezer
Re: [squid-users] Re: question in cache mem in squid 3
On 1/08/2013 9:06 p.m., Ahmad wrote: hi , i have another question here , with my config above with cache_mem is 1000 , now if i chance cache_dir to 150 G instaed of 90 G for the 3 hardsisks above , which result in total about 150* 3 Giga storage . ===> i note that squid wccp service gett flapped every about 40-50 minutes and seems squid to be unstable . but why it affected while im fixing cache_mem to 1000 ??? is my thought correct ??? i mean that im limiting squid_mem to 1 Giga , but i need about memory usage of 150*3*15 = about 7 G shouldn't squid be stable ?? or it is normal result ?? The WCCP flapping is caused by Squid being too busy with something to send out its 10s repeated HERE_I_AM update to the router for a few in a row. If that is triggered by simpy setting large disks it is likely the CPU cycles consumed by disk I/O and/or garbage collection taking up Squid's time. Amos
Re: [squid-users] Re: Squid monitoring, access report shows upto 5 % to 7 % cache usage
On 1/08/2013 6:35 p.m., John Joseph wrote: Hi Amos,Ahmad,Babajaga Thanks for your advice and feed back, I am posting more information --- the HIT,MISS,REFRESH details are cat /opt/var/log/squid/access.log | grep -c "HIT" 13810283 cat /opt/var/log/squid/access.log | grep -c "MISS" 57874593 cat /opt/var/log/squid/access.log | grep -c "REFRESH" 6760966 Doing a rough calculation here I get just under 18%. Which is low but inside the 15-40 range normally seen. squidclient info result squidclient -h 127.0.0.1 mgr:info HTTP/1.0 200 OK Server: squid/3.1.10 Mime-Version: 1.0 Date: Thu, 01 Aug 2013 06:10:06 GMT Content-Type: text/plain Expires: Thu, 01 Aug 2013 06:10:06 GMT Last-Modified: Thu, 01 Aug 2013 06:10:06 GMT X-Cache: MISS from proxy X-Cache-Lookup: MISS from proxy:3128 Via: 1.0 proxy (squid/3.1.10) Connection: close Squid Object Cache: Version 3.1.10 Start Time:Wed, 31 Jul 2013 23:08:41 GMT Current Time:Thu, 01 Aug 2013 06:10:06 GMT Connection information for squid: Number of clients accessing cache:574 Number of HTTP requests received:508778 Number of ICP messages received:0 Number of ICP messages sent:0 Number of queued ICP replies:0 Number of HTCP messages received:0 Number of HTCP messages sent:0 Request failure ratio: 0.00 Average HTTP requests per minute since start:1207.3 Average ICP messages per minute since start:0.0 Select loop called: 62199585 times, 0.407 ms avg Cache information for squid: Hits as % of all requests:5min: 21.6%, 60min: 27.1% Hits as % of bytes sent:5min: 6.6%, 60min: 5.6% Thank you. This explains the whole story. MySAR is reporting your cache bandwidth savings HIT ratio (bytes sent). What you have is a reasonable HIT ratio 21-27% by request count, however it appears that is built from mostly small objects. The larger objects are mostly MISS events. Which drags your byte HIT ratio way down low. However there is more to the story... the log numbers you show above indicate that about 9% of all requests through your cache are REFRESH, which may be recorded as a HIT with no byte count associated even on the largest of objects. With high REFRESH traffic there is also a high amount of IMS traffic. Those two will drag down the HIT ratio for bytes down while also being a good thing - the byte count is down because there actually are less bytes used. NP: Squid does not yet record how much savings is gained from REFRESH or IMS traffic, which would help show this a bit better. Memory hits as % of hit requests:5min: 5.6%, 60min: 9.0% Disk hits as % of hit requests:5min: 46.9%, 60min: 47.7% And the breakdown of where those HITs are coming from shows mostly disk activity, very little memory benefits. MySAR results are DATE USERS HOSTS SITES BYTESTRAFFICCache Percent Thursday, 01 August 2013 11 797 13899 36.11G 4% Wednesday, 31 July 2013 42 1024 29862 89.22G 5% Tuesday, 30 July 2013 19 1023 27096 85.24G 5% Monday, 29 July 201329 1022 26425 82.55G 5% Hmm. These look like your byte-count HIT ratio percentages. Amos
[squid-users] ICAP failure when using clamav scan denied reply
I am using squid 3.2.3 + http://www.squid-cache.org/Versions/v3/3.2/changesets/squid-3.2-11699.patch(Polish: replace several assert(isOpen(fd))) + c-icap 0.1.7 + squidclamav 6.9 + squidGuard 1.4 as default I deny all application/octet-stream reply access, and disable virus scan picture ^.*\.(ico|gif|png|jpg)$ in squidclamav. my problem is when user try to access a link that end of .gif but reply content type is application/octet-stream, c-icap will store that content to /var/tmp and keep it, then die in icap process, even I reload icap. Squid Cache: Version 3.2.3 configure options: '--enable-icmp' '--enable-delay-pools' '--enable-icap-client' '--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-cachemgr-hostname=localhost' '--enable-ssl' '--enable-cache-digests' '--enable-epoll' '--disable-ipfw-transparent' '--disable-ipf-transparent' '--disable-pf-transparent' '--disable-linux-netfilter' '--enable-follow-x-forwarded-for' '--enable-ident-lookups' '--enable-ssl-crtd' '--enable-auth' '--enable-auth-basic=LDAP,NCSA,SMB,MSNT,MSNT-multi-domain' '--enable-auth-ntlm=smb_lm' '--enable-auth-negotiate=kerberos,wrapper' '--enable-external-acl-helpers=kerberos_ldap_group,AD_group,unix_group,wbinfo_group,LDAP_group,file_userip,LM_group' '--with-default-user=squid' '--enable-ltdl-convenience' acl blockmime rep_mime_type application/octet-stream http_reply_access deny blockmime http_reply_access allow all icap_enable on icap_preview_enable on icap_preview_size 4096 icap_persistent_connections on icap_send_client_ip on icap_send_client_username on icap_client_username_header X-Client-Username icap_service squidclamav_req reqmod_precache bypass=0 icap://127.0.0.1:1344/squidclamav icap_service squidclamav_resp respmod_precache bypass=0 icap://127.0.0.1:1344/squidclamav adaptation_access squidclamav_req allow all adaptation_access squidclamav_resp allow all c-icap.conf PidFile /var/run/c-icap/c-icap.pid CommandsSocket /var/run/c-icap/c-icap.ctl Timeout 300 MaxKeepAliveRequests 100 KeepAliveTimeout 600 StartServers 10 MaxServers 20 MinSpareThreads 10 MaxSpareThreads 20 ThreadsPerChild 10 MaxRequestsPerChild 0 Port 1344 User apache Group apache ServerAdmin i@***.com.cn ServerName proxy TmpDir /var/tmp MaxMemObject 1048576 DebugLevel 0 ModulesDir /usr/local/lib/c_icap ServicesDir /usr/local/lib/c_icap TemplateDir /usr/local/share/c_icap/templates/ TemplateDefaultLanguage en LoadMagicFile /usr/local/etc/c-icap.magic RemoteProxyUsers off RemoteProxyUserHeader X-Authenticated-User RemoteProxyUserHeaderEncoded on ServerLog /var/log/c-icap/server.log AccessLog /var/log/c-icap/access.log Service echo srv_echo.so Service squidclamav squidclamav.so squidclamav.conf clamd_local /var/run/clamav/clamd.sock redirect http://proxy/cgi-bin/clwarn.cgi maxsize 5000 timeout 2 logredir 0 dnslookup 1 abort ^.*\.(ico|gif|png|jpg)$ abort ^.*\.(css|xml|xsl|js|html|jsp)$ abort ^.*\.swf$ abortcontent ^image\/.*$ abortcontent ^text\/.*$ abortcontent ^application\/x-javascript$ abortcontent ^video\/x-flv$ abortcontent ^video\/mp4$ abortcontent ^application\/x-shockwave-flash$ abortcontent ^.*application\/x-mms-framed.*$ my squid log 1375345064.448 6471 1.1.2.3 TCP_DENIED_REPLY/403 9044 GET http://bbs.chinaacc.com/getresource.php?thumb=1&rid=104959 user_Name FIRSTUP_PARENT/1.1.2.2 text/html ls -l --time-style=+%s /var/tmp -rw--- 1 apache apache 3924554 1375345064 CI_TMP_bykwF4 lsof /var/tmp/CI_TMP_bykwF4 c-icap 20802 apache 33u REG 253,0 3924554 244479 CI_TMP_bykwF4 lsof -p 20802 c-icap 20802 apache 31u REG253,0 3924554 181742 /var/tmp/CI_TMP_pZJZ3q c-icap 20802 apache 32u REG253,0 3924554 244478 /var/tmp/CI_TMP_nj2kWD c-icap 20802 apache 33u REG253,0 3924554 244479 /var/tmp/CI_TMP_bykwF4 c-icap 20802 apache 35u IPv4 20636425 0t0 TCP localhost:icap->localhost:37850 (ESTABLISHED) firebug report: GET getresource.php?thumb=1&rid=104959 200 OKbbs.chinaacc.com 3.7 MB1.1.2.2:80008.43s ParamsHeadersResponseCookies Response Headersview source Connectionkeep-alive Content-Dispositioninline; filename="62037b5agw1droqc7t0qeg.gif" Content-Encodingnone Content-Length3924554 Content-Typeapplication/octet-stream DateThu, 01 Aug 2013 08:29:43 GMT Last-ModifiedThu, 01 Aug 2013 08:29:43 GMT Proxy-Authentication-InfoNegotiate oYGyMIGvoAMKAQChCwYJKoZIgvcSAQICooGaBIGXYIGUBgkqhkiG9xIBAgICAG+BhDCBgaADAgEFoQMCAQ+idTBzoAMCAReibARqiHRChCBhCm+q94YpjxLaCevHSu6pf+h8c3qgm0klDOgz9hinJRUaR7kq1pwV5+64cHmG146DDjehdQ+AmKcPRxnMsNnjqGr1zcPK2czlMdEDWOCGka7B3jJPGMIJDK6onV1cKqgcUIPosg== Servernginx ViaICAP/1.0 proxy (C-ICAP/0.1.7 SquidClamav/Antivirus service ) X-CacheMISS from webproxy, MISS from proxy X-Cache-LookupMISS from proxy:8001, MISS from proxy:8000 X-Powered-ByPHP/5.2.10 Request Headersview source Accepttext/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Encodinggzip, deflate Acce
Re: [squid-users] Re: question in cache mem in squid 3
On 1/08/2013 8:20 p.m., Ahmad wrote: hi pavel , thanks alot for clarification , i think that after reading the article about memory usage , i think that my squid box is not getting the benefit of my rams (48G) i have 3 hardsisk each one im obtainging 90 G for cache dir , cache_dir aufs /cache1 9 32 256 cache_dir aufs /cache2 9 32 256 cache_dir aufs /cache3 9 32 256 so if i calculate ram dissipation for cache_dir it will be about 15 * (90 * 3) = 4050 M from main memory assume *5G* again my total memory is 48 G also i put cache_mem = 1000 which is *1 G * assume another margin with *5 G at most* so , my memory usage is 5+1+5 = 11 G from 48 G *my question is ,* should i increase cache_mem value or increase cache_dir space Try adding a rock type cache_dir next. That will improve the response speed for under-32KB objects a fair bit. also , should i find a better performance after that ??? Better question is what your performance currently is? Are you certain that memory is the biggest cause of low performance in your proxy? And the answer: again , my machine is being pumped with 350 M with about 2300 Users. By 350M from X users I assume you means 350Mbps ?? When it comes to bandwidth throughput Squid is quite limited by the process of parsing HTTP. This is the main limiter we find in high performance situations. I tend to find one Squid process able to handle 50-150 Mbps reliably well, depending on the request/second rate that exists within that traffic. The more req/sec there are the slower the Mbps rate handled is. If you are already getting 350Mbps out of it, _very_ well done. I suggest your next step (after adding a rock cache_dir) is to attempt enabling SMP support for 3 or 4 workers. Your analysis of the memory needs shows that the box can handle them. Each worker will use the amount you calculated above. Amos
[squid-users] Re: question in cache mem in squid 3
hi , i have another question here , with my config above with cache_mem is 1000 , now if i chance cache_dir to 150 G instaed of 90 G for the 3 hardsisks above , which result in total about 150* 3 Giga storage . ===> i note that squid wccp service gett flapped every about 40-50 minutes and seems squid to be unstable . but why it affected while im fixing cache_mem to 1000 ??? is my thought correct ??? i mean that im limiting squid_mem to 1 Giga , but i need about memory usage of 150*3*15 = about 7 G shouldn't squid be stable ?? or it is normal result ?? regards - Mr.Ahmad -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/question-in-cache-mem-in-squid-3-tp4661365p4661393.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] hit percentage in squid
On 1/08/2013 6:20 p.m., John Joseph wrote: Hi All Going through the squid docs and mailing list email, I see HIT percentage. Is the HIT percentage is the images which takes from the squid cache when user browse It is not related partcularly to images. In HTTP there are no image or files or scripts or anything like that. Just _objects_, which are sent as response payloads. Any object can be stored in cache if the conditions around how it was generated and how it will be updated in future allow it to be stored for any length of time. HIT and MISS are a general concept which also exist in any area which involves a cache of any type. For example a CPUs L1/L2 caches have HIT/MISS events in relation to memory block loading. HIT ratio is simply the mathematical ratio of requests which are a cache HIT (object stored in cache) to those which are a cache MISS (object not in cache) Since Squid does caching for HTTP it has a HIT ratio measurement to evaluate the efficiency of that cache. It is often represented as a % of total traffic and can be measured in either bytes or request count. Amos
[squid-users] Re: question in cache mem in squid 3
hi pavel , thanks alot for clarification , i think that after reading the article about memory usage , i think that my squid box is not getting the benefit of my rams (48G) i have 3 hardsisk each one im obtainging 90 G for cache dir , cache_dir aufs /cache1 9 32 256 cache_dir aufs /cache2 9 32 256 cache_dir aufs /cache3 9 32 256 so if i calculate ram dissipation for cache_dir it will be about 15 * (90 * 3) = 4050 M from main memory assume *5G* again my total memory is 48 G also i put cache_mem = 1000 which is *1 G * assume another margin with *5 G at most* so , my memory usage is 5+1+5 = 11 G from 48 G *my question is ,* should i increase cache_mem value or increase cache_dir space also , should i find a better performance after that ??? again , my machine is being pumped with 350 M with about 2300 Users. -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/question-in-cache-mem-in-squid-3-tp4661365p4661391.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] question in cache mem in squid 3
On 1/08/2013 12:05 p.m., Carlos Defoe wrote: OK, I thought of the cache_dir while I was writing, but in general, the cache_mem is the configuration we can change from, let's say, 8 GB to 32 GB in a minute. What i was asking was: if i have 40 GB of RAM, can I set something like 32 GB to cache_mem, making the box operate with about 39GB of memory effectively used, and leaving only something like 1 GB, or less, for that linux "cache" thing? In a box without a proxy-cache, let's say, a DNS server, linux will not use nothing more than 1GB of RAM, doing no "disk cache". But the fact is that you can put 100GB on a linux box with squid, with only 2 GB of cache_mem, 20 GB of cache_dir, and we will have about 4GB of RAM effectively used, and all the rest of the memory will be "used" as disk cache by linux. There's no real need for that? Can we grow the effective usage, configuring squid, until we have no memory used for "cache" by linux? I suspect so. The kernel should be auto-adjusting its cache usage. Amos
Re: [squid-users] refresh_pattern what are the parameter
On Thursday 01 August 2013 at 09:23:55, John Joseph wrote: > Hi All > I searched for optimising the cache performance, I came to know that you > can cache more types of files using 'refresh_pattern' in squid.conf > Can I know what are the parameter which are given to refresh_pattern as > options eg such as 'refresh_pattern .020%4320' > > Guidance and advice requested http://www.squid-cache.org/Doc/config/refresh_pattern/ might help. Antony. -- "It would appear we have reached the limits of what it is possible to achieve with computer technology, although one should be careful with such statements; they tend to sound pretty silly in five years." - John von Neumann (1949) Please reply to the list; please don't CC me.
[squid-users] refresh_pattern what are the parameter
Hi All I searched for optimising the cache performance, I came to know that you can cache more types of files using 'refresh_pattern' in squid.conf Can I know what are the parameter which are given to refresh_pattern as options eg such as 'refresh_pattern . 0 20% 4320' Guidance and advice requested thanks Joseph john
