[squid-users] leaking memory in squid 3.4.8 and 3.4.7.
Colleagues, squid 3.4.8 and 3.4.7 are leaking memory at the rate of several Mbytes per minute on FreeBSD. Squid 3.4.8 leaks faster than 3.4.7. The settings are more than modest: cache_mem 128 MB cache_dir ufs /webcache/cache 512 16 256 memory_pools off # neither "on" nor "off" have any effect on leaking. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193938 Can I do something about it? There was no such problem with squid-2.7.9. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ Squid-users mailing list Squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
Well you can start with: http://bugs.squid-cache.org/ Which is the place for squid related bugs and specially for the stable versions. The needed information is also here: http://wiki.squid-cache.org/SquidFaq/BugReporting Since it's squid 3.4.7 or 8 you will have manage interface via plain http so take a look at the memory and other info inside the squid-internal-mgr interface. Also to minimize the leak source try to share more information about your usage. Are you using SMP workers or not? What is the mem and info options in the cache manager page data? Did you tried "cache deny all" acl to minimize the source of the leak? Eliezer On 09/28/2014 09:28 AM, Victor Sudakov wrote: Colleagues, squid 3.4.8 and 3.4.7 are leaking memory at the rate of several Mbytes per minute on FreeBSD. Squid 3.4.8 leaks faster than 3.4.7. The settings are more than modest: cache_mem 128 MB cache_dir ufs /webcache/cache 512 16 256 memory_pools off # neither "on" nor "off" have any effect on leaking. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193938 Can I do something about it? There was no such problem with squid-2.7.9. ___ Squid-users mailing list Squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
Hi. On 28.09.2014 12:28, Victor Sudakov wrote: squid 3.4.8 and 3.4.7 are leaking memory at the rate of several Mbytes per minute on FreeBSD. Squid 3.4.8 leaks faster than 3.4.7. The settings are more than modest: cache_mem 128 MB cache_dir ufs /webcache/cache 512 16 256 memory_pools off # neither "on" nor "off" have any effect on leaking. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193938 Can I do something about it? There was no such problem with squid-2.7.9. How many users do you have ? I have 3.4.6 on a panicbox under FreeBSD, I use it for one user (for test purposes). It does have other problems, but it's not leaking memory. In the same time I use 3.3.13 on production. It may have issues in your environment (though it shares them mostly with 3.4 branch anyway), and it's definitely not leaking under hundreds of users. You can comment out the FORBIDDEN line in Makefile and try it out. 3.3 works for me for years on a dozen of installations. Eugene. ___ Squid-users mailing list Squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
Eugene M. Zheganin wrote: > > squid 3.4.8 and 3.4.7 are leaking memory at the rate of several Mbytes > > per minute on FreeBSD. Squid 3.4.8 leaks faster than 3.4.7. > > > > The settings are more than modest: > > > > cache_mem 128 MB > > cache_dir ufs /webcache/cache 512 16 256 > > memory_pools off # neither "on" nor "off" have any effect on leaking. > > > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193938 > > > > Can I do something about it? There was no such problem with squid-2.7.9. > > How many users do you have ? I have 3.4.6 on a panicbox under FreeBSD, I > use it for one user (for test purposes). I have counted 225 unique IPs in the access.log, but some of them are Windows terminal servers. > It does have other problems, but it's not leaking memory. I believe you because 3.4.7 leaks less than 3.4.8. Maybe there is a trend and 3.4.6 leaked less than 3.4.7. > In the same time I use 3.3.13 on production. It may have issues in your > environment (though it shares them mostly with 3.4 branch anyway), and > it's definitely not leaking under hundreds of users. You can comment out > the FORBIDDEN line in Makefile and try it out. 3.3 works for me for > years on a dozen of installations. If I have to stick to some old unsupported version, I'd choose 2.7. It has worked flawlessly until it was deleted from the ports tree and I had to migrate to squid3. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ Squid-users mailing list Squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
On 09/28/2014 12:18 PM, Victor Sudakov wrote: If I have to stick to some old unsupported version, I'd choose 2.7. It has worked flawlessly until it was deleted from the ports tree and I had to migrate to squid3. Victor, Please start by declaring the environment in the form of: is it a intercept or forward proxy?(defined in the browser or using a GW in the middle) In the case you intercept, do you intercept on the squid machine only or you redirect\route the traffic to it? If you intercept share your intercept\pf\IPFW rules. Did you had the chance to watch the cache.log for more information? maybe squid output something to the cache.log? There is too much missing to just declare a memory leak. I have seen something that looks like a memory leak in 3.4.6 and 3.4.7 in the past but it was seen in the memory counters in the cache manager interface which shows exactly to where these bits are leaking to.. If you want\need help consider getting into the #squid irc channel at irc.freenode.net and get some more directions. I am there on and off but once you find me I can try to help you with the basics. Eliezer ___ Squid-users mailing list Squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
Eliezer Croitoru wrote: > On 09/28/2014 12:18 PM, Victor Sudakov wrote: > > If I have to stick to some old unsupported version, I'd choose 2.7. > > It has worked flawlessly until it was deleted from the ports tree and > > I had to migrate to squid3. > > Victor, > Please start by declaring the environment in the form of: > is it a intercept or forward proxy?(defined in the browser or using a GW > in the middle) It's a forward proxy (defined via WPAD). > Did you had the chance to watch the cache.log for more information? > maybe squid output something to the cache.log? I don't see anything suspicious other than multiple lines of the kind 2014/09/28 03:30:07 kid1| WARNING: Closing client connection due to lifetime timeout 2014/09/28 03:30:07 kid1| pp.vk.me:443 2014/09/28 03:30:07 kid1| WARNING: Closing client connection due to lifetime timeout 2014/09/28 03:30:07 kid1| pp.vk.me:443 2014/09/28 03:30:07 kid1| WARNING: Closing client connection due to lifetime timeout 2014/09/28 03:30:07 kid1| vk.com:443 2014/09/28 03:30:17 kid1| WARNING: Closing client connection due to lifetime timeout I can send the whole cache.log if someone cares to look at it. > > There is too much missing to just declare a memory leak. > > I have seen something that looks like a memory leak in 3.4.6 and 3.4.7 > in the past but it was seen in the memory counters in the cache manager > interface which shows exactly to where these bits are leaking to.. I have tried looking at the cachemgr.cgi memory counters but they are too numerous and cryptic for me. I currently monitor squid's RES and SIZE memory usage through top and see it ever growing. Maybe it's not a leak but something else. But squid27 did not behave like that under similar load. > > If you want\need help consider getting into the #squid irc channel at > irc.freenode.net and get some more directions. > I am there on and off but once you find me I can try to help you with > the basics. I'll try to find you there. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ Squid-users mailing list Squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
Eliezer Croitoru wrote: > > Which is the place for squid related bugs and specially for the stable > versions. > The needed information is also here: > http://wiki.squid-cache.org/SquidFaq/BugReporting > > Since it's squid 3.4.7 or 8 you will have manage interface via plain > http so take a look at the memory and other info inside the > squid-internal-mgr interface. > > Also to minimize the leak source try to share more information about > your usage. > Are you using SMP workers or not? How do I figure this out? I compiled squid from the FreeBSD ports collection, don't remember any tunable option concerning SMP workers. > What is the mem and info options in the cache manager page data? There are so many cryptic lines in the cache manager Web interface concerning memory usage. If I only could have someone look at them. I cannot extract any useful information therefrom. > Did you tried "cache deny all" acl to minimize the source of the leak? I suppose this somehow would defeat the purpose of the Squid cache. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ Squid-users mailing list Squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
Hi. On 28.09.2014 21:14, Victor Sudakov wrote: How do I figure this out? I compiled squid from the FreeBSD ports collection, don't remember any tunable option concerning SMP workers. They're off by default. Plus, it's a 3.x feature. If you ported your config from your 2.7 installation, they're off. My opinion - don't turn them on, they're not production ready. Eugene. ___ Squid-users mailing list Squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 29/09/2014 1:41 a.m., Victor Sudakov wrote: > Eliezer Croitoru wrote: >> On 09/28/2014 12:18 PM, Victor Sudakov wrote: >>> If I have to stick to some old unsupported version, I'd choose >>> 2.7. It has worked flawlessly until it was deleted from the >>> ports tree and I had to migrate to squid3. >> >> Victor, Please start by declaring the environment in the form >> of: is it a intercept or forward proxy?(defined in the browser or >> using a GW in the middle) > > It's a forward proxy (defined via WPAD). > >> Did you had the chance to watch the cache.log for more >> information? maybe squid output something to the cache.log? > > I don't see anything suspicious other than multiple lines of the > kind > > 2014/09/28 03:30:07 kid1| WARNING: Closing client connection due to > lifetime timeout 2014/09/28 03:30:07 kid1| pp.vk.me:443 > 2014/09/28 03:30:07 kid1| WARNING: Closing client connection due to > lifetime timeout 2014/09/28 03:30:07 kid1| pp.vk.me:443 > 2014/09/28 03:30:07 kid1| WARNING: Closing client connection due to > lifetime timeout 2014/09/28 03:30:07 kid1| vk.com:443 > 2014/09/28 03:30:17 kid1| WARNING: Closing client connection due to > lifetime timeout > > I can send the whole cache.log if someone cares to look at it. > >> >> There is too much missing to just declare a memory leak. >> >> I have seen something that looks like a memory leak in 3.4.6 and >> 3.4.7 in the past but it was seen in the memory counters in the >> cache manager interface which shows exactly to where these bits >> are leaking to.. > > I have tried looking at the cachemgr.cgi memory counters but they > are too numerous and cryptic for me. Would you mind posting a copy of that mgr report? perhapse we can see something unusual. > > I currently monitor squid's RES and SIZE memory usage through top > and see it ever growing. Maybe it's not a leak but something else. > But squid27 did not behave like that under similar load. > Squid-3 does use more memory overall than Squid-2. The latest versions in particular use extra memory for transient (in-progress) transaction processing which was previously passing through cache_mem and limited by that. >> >> If you want\need help consider getting into the #squid irc >> channel at irc.freenode.net and get some more directions. I am >> there on and off but once you find me I can try to help you with >> the basics. > > I'll try to find you there. > Also, are you using HTTPS interception? there are known memory leaks in OpenSSL. Amos -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (MingW32) iQEcBAEBAgAGBQJUKKprAAoJELJo5wb/XPRjEAYH/AgKrcy1UJUfyMdO3N30HX/R GT1jh0DzcKGADsktIQ5oindGPjC7GAeWhccV3EAJPwBFySQ/HzXo5+SlolNlrGBi Jgbrj+hfyoIWEIfKsEs+DuhVMtSFxTGtJmh6OVCMBH6hN5xzoqXXrl9o0Hxy85ky N6+3VRvaMnFHCvenwJXgct3dGFUSeqQy6qrQcGVbrQrdamFyNRNDgFVk6WJD0Gpt 5faS9zjQQuSUFVakeoEdV0vHb0yFPAo1VKhRyB9HKncHlh493+Ke52puDa2ZsDho FuQ5GzU5lryMOw7pkQcWU+atL11DGesxoTLvFFk/do6OJri5RY+U0CA7/9Pztu8= =A2yd -END PGP SIGNATURE- ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
Victor Sudakov wrote: > > > Did you tried "cache deny all" acl to minimize the source of the leak? > > I suppose this somehow would defeat the purpose of the Squid cache. I have tried "cache deny all" and squid is still growing. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 29/09/2014 1:40 p.m., Amos Jeffries wrote: > On 29/09/2014 1:41 a.m., Victor Sudakov wrote: >> Eliezer Croitoru wrote: >>> On 09/28/2014 12:18 PM, Victor Sudakov wrote: If I have to stick to some old unsupported version, I'd choose 2.7. It has worked flawlessly until it was deleted from the ports tree and I had to migrate to squid3. > > > Also, are you using HTTPS interception? there are known memory > leaks in OpenSSL. > Also, what does your squid.conf contain? (without comments or cachemgr_passwd lines) Amos -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (MingW32) iQEcBAEBAgAGBQJUKN4tAAoJELJo5wb/XPRjVSEH/Ah1s/D+B9Q+r7JssOUF6jss vPjy1QgLK/HBNzVUJHEUgT931V4S+P84GFoIGxKcIrgwFkojGG1SC7ZeXdaW3BF8 Ut0TfBh7u1UX+7Mu1Tn1OPMS0x6mmoMnIRVuAJ53y1uLfIbE7McFuQ7S3KhpcpZ8 o8IYY23YO1jsQskToHpHYqd8IKd8AHzZwJmgGeOk3dEEWoEhXTLRh0tRTykvkZjc qqumwIgIzIbT6acjXgEWmYQsABApkVyIKcQrlzwT83VKVnWR4YxkevBi2ke+ZdYc wHQKOHPXFx83O6SpGBQXbTNQoWcgjO8OhgACDCNxDSxOnWOzod4rduSa0FYSdm4= =n1e+ -END PGP SIGNATURE- ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
Amos Jeffries wrote: > If I have to stick to some old unsupported version, I'd > choose 2.7. It has worked flawlessly until it was deleted > from the ports tree and I had to migrate to squid3. > > > > > > Also, are you using HTTPS interception? there are known memory > > leaks in OpenSSL. No, God forbid. No interception of any kind. > > > > Also, what does your squid.conf contain? (without comments or > cachemgr_passwd lines) auth_param ntlm program /root/bin/ntlm_auth SIBPTUS/MSG01-SIBPTUS SIBPTUS/DC01-SIBPTUS auth_param ntlm children 100 auth_param ntlm keep_alive on acl localnet src 10.0.0.0/8 # RFC1918 possible internal network acl localnet src 172.16.0.0/12 # RFC1918 possible internal network acl localnet src 192.168.0.0/16 # RFC1918 possible internal network acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines acl SSL_ports port 443 9010 8082 4443 9443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 4443 # https Tomskstat acl Safe_ports port 9010 # CGP management acl Safe_ports port 8082 # ecolog acl CONNECT method CONNECT acl sibptus src 10.14.128.0/20 acl password proxy_auth_regex -i "/usr/local/etc/squid/users.txt" acl domain_users proxy_auth REQUIRED acl workhours time MTWH 9:00-18:00 acl workhours time F 9:00-16:45 acl lunch time 13:00-14:00 acl private dst 10.0.0.0/8 acl private dst 172.16.0.0/12 acl private dst 192.168.0.0/16 acl business dstdomain "/usr/local/etc/squid/whitelist.txt" acl pogoda url_regex -i ^http://pics.rbc.ru/img/grinf/elections3.gif$ acl pogoda url_regex -i ^http://informer.gismeteo.ru/29430-10.GIF$ acl ident_privileged ident_regex -i "/usr/local/etc/squid/ident.txt" acl badsites dstdomain "/usr/local/etc/squid/badsites.txt" acl illegal dstdomain "/usr/local/etc/squid/illegal.txt" http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost manager http_access deny manager http_access deny badsites http_access deny private http_access allow pogoda http_access allow sibptus business http_access allow sibptus ident_privileged http_access allow sibptus lunch domain_users http_access allow sibptus !workhours domain_users http_access allow sibptus password http_access allow localhost http_access deny all ident_lookup_access allow sibptus http_port 3128 http_port 8080 tcp_outgoing_address 10.14.140.9 cache_mem 128 MB cache_dir ufs /webcache/cache 512 16 256 access_log stdio:/webcache/logs/access.log pid_filename /webcache/squid.pid cache_log /webcache/logs/cache.log coredump_dir /var/squid/cache/squid ftp_user sq...@sibptus.ru refresh_pattern ^ftp: 144020% 10080 refresh_pattern ^gopher:14400% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 shutdown_lifetime 10 seconds cache_mgr n...@sibptus.ru visible_hostname proxy.sibptus.transneft.ru delay_pools 1 delay_class 1 1# pool 1 is a class 1 pool delay_access 1 allow all delay_parameters 1 1024000/1024000 append_domain .sibptus.transneft.ru memory_pools off -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 29/09/2014 8:16 p.m., Victor Sudakov wrote: > Amos Jeffries wrote: >>> >>> I have tried looking at the cachemgr.cgi memory counters but >>> they are too numerous and cryptic for me. >> >> Would you mind posting a copy of that mgr report? perhapse we can >> see something unusual. >> > > I am attaching two reports. The first one was taken right after > the squid restart. The second it several hours later when squid is > already 843M in size (610M RES). > You have 200 MB of RAM locked up in IDENT lookups. Probably http://bugs.squid-cache.org/show_bug.cgi?id=3803 Amos -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (MingW32) iQEcBAEBAgAGBQJUKRu3AAoJELJo5wb/XPRjfxoH/0VEUG3xatAK9qI5d+Kvv7od AmBCXuCJd6uM/a1WLv6ZrliN752+ohacKaog0k1ov7trPiap4xtxDJugKjbmQOu8 8U06Mb1xpAstwLG9sra6jZjaIX69n+LEo8/xdfWJqB0yfFM+ipn7pWKxKZCB9L8Z 0e4/xeKHPMF1m/eZ2UsEY26Loo+IGl1+j4SYcyrl7q9zjdSnKDJeIA81wlXLOfFt UOKbn75ZCk711srJ1BPi5723KgjvzMW9Tt5VWtqlHxfzjTOXmRMVJ6dUFj36moeX SsoS/CPUFgGIp4at+rOH7ocrcIRXTs252crJa6SuQGKvfpFDWkvQ8TrSjIZY5k8= =aYsu -END PGP SIGNATURE- ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
On 28.09.14 08:34, Eliezer Croitoru wrote: Also to minimize the leak source try to share more information about your usage. Are you using SMP workers or not? What is the mem and info options in the cache manager page data? Did you tried "cache deny all" acl to minimize the source of the leak? For what its worth, I'm also seeing a big memory leak on multiple servers under Squid 3.4.6: I have two basic Squid configurations - one of them is a plain caching proxy and is _not_ leaking. The other does no caching, but is doing ICAP, external ACLs, SSL bumping and TPROXY and leaks like a sieve (several gigabytes a day). I _think_ I have narrowed it down to something ICAP related and I'm currently valgrinding. Unfortunately I can't seem to get the valgrind instrumentation to work so I'm having to do without (I compile --with-valgrind-debug but "squidclient mgr:mem" doesn't produce a valgrind report). I have noticed that there are a _lot_ of memory errors reported by valgrind though (uninitialised memory). Pretty much all of the leaky memory is showing up as unaccounted: Memory accounted for: Total accounted:84306 KB 3% memPool accounted: 84306 KB 3% memPool unaccounted: 2917158 KB 97% I am using SMP workers, but turning that off doesn't fix the issue. -- - Steve Hill Technical Director Opendium Limited http://www.opendium.com Direct contacts: Instant messager: xmpp:st...@opendium.com Email:st...@opendium.com Phone:sip:st...@opendium.com Sales / enquiries contacts: Email:sa...@opendium.com Phone:+44-844-9791439 / sip:sa...@opendium.com Support contacts: Email:supp...@opendium.com Phone:+44-844-4844916 / sip:supp...@opendium.com ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
Hey Steve, Can you share the basic cache manager requests statistics and the up time for the service? (mgr:info) This would give us a basic idea of the load\requests needed to reproduce it. Eliezer On 09/29/2014 12:04 PM, Steve Hill wrote: For what its worth, I'm also seeing a big memory leak on multiple servers under Squid 3.4.6: I have two basic Squid configurations - one of them is a plain caching proxy and is _not_ leaking. The other does no caching, but is doing ICAP, external ACLs, SSL bumping and TPROXY and leaks like a sieve (several gigabytes a day). I _think_ I have narrowed it down to something ICAP related and I'm currently valgrinding. Unfortunately I can't seem to get the valgrind instrumentation to work so I'm having to do without (I compile --with-valgrind-debug but "squidclient mgr:mem" doesn't produce a valgrind report). I have noticed that there are a _lot_ of memory errors reported by valgrind though (uninitialised memory). Pretty much all of the leaky memory is showing up as unaccounted: Memory accounted for: Total accounted:84306 KB 3% memPool accounted: 84306 KB 3% memPool unaccounted: 2917158 KB 97% I am using SMP workers, but turning that off doesn't fix the issue. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
> > Can you share the basic cache manager requests statistics and the up > time for the service? > (mgr:info) > > This would give us a basic idea of the load\requests needed to reproduce it. I am not Steve but in case you care to look at my statistics as well, I am attaching it. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru Title: CacheMgr@localhost: info Cache Manager menu Squid Object Cache: Version 3.4.8 Build Info: Start Time:Mon, 29 Sep 2014 15:24:54 GMT Current Time:Tue, 30 Sep 2014 02:09:51 GMT Connection information for squid: Number of clients accessing cache: 161 Number of HTTP requests received: 152054 Number of ICP messages received: 0 Number of ICP messages sent: 0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 235.8 Average ICP messages per minute since start: 0.0 Select loop called: 4664551 times, 8.296 ms avg Cache information for squid: Hits as % of all requests: 5min: 4.2%, 60min: 5.2% Hits as % of bytes sent: 5min: 29.0%, 60min: 21.8% Memory hits as % of hit requests: 5min: 48.0%, 60min: 46.6% Disk hits as % of hit requests: 5min: 20.5%, 60min: 18.7% Storage Swap size: 471844 KB Storage Swap capacity: 90.0% used, 10.0% free Storage Mem size: 130284 KB Storage Mem capacity: 99.4% used, 0.6% free Mean Object Size: 23.06 KB Requests given to unlinkd: 9490 Median Service Times (seconds) 5 min60 min: HTTP Requests (All): 0.07409 0.06640 Cache Misses: 0.19742 0.13498 Cache Hits:0.00865 0.00463 Near Hits: 0.12106 0.35832 Not-Modified Replies: 0.00865 0.00379 DNS Lookups: 0.06963 0.06364 ICP Queries: 0.0 0.0 Resource usage for squid: UP Time: 38697.179 seconds CPU Time: 16971.890 seconds CPU Usage: 43.86% CPU Usage, 5 minute avg: 42.01% CPU Usage, 60 minute avg: 30.28% Maximum Resident Size: -1832976 KB Page faults with physical i/o: 4507 Memory accounted for: Total accounted: 403163 KB memPoolAlloc calls:39 memPoolFree calls: 193148820 File descriptor usage for squid: Maximum number of file descriptors: 16384 Largest file desc currently in use:735 Number of file desc currently in use: 599 Files queued for open: 0 Available number of file descriptors: 15785 Reserved number of file descriptors: 100 Store Disk files open: 0 Internal Data Structures: 20577 StoreEntries 9061 StoreEntries with MemObjects 8988 Hot Object Cache Items 20465 on-disk objects Generated Tue, 30 Sep 2014 02:09:46 GMT, by cachemgr.cgi/3@fs01-sibptus.sibptus.transneft.ru ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
Amos Jeffries wrote: > >>> > >>> I have tried looking at the cachemgr.cgi memory counters but > >>> they are too numerous and cryptic for me. > >> > >> Would you mind posting a copy of that mgr report? perhapse we can > >> see something unusual. > >> > > > > I am attaching two reports. The first one was taken right after > > the squid restart. The second it several hours later when squid is > > already 843M in size (610M RES). > > > > You have 200 MB of RAM locked up in IDENT lookups. > > Probably http://bugs.squid-cache.org/show_bug.cgi?id=3803 I have temporarily disabled IDENT-related acls. Squid still grows in memory, but more slowly. So IDENT was certainly one of the major causes of the leak. Could we make another iteration of looking at the cachemgr.cgi memory counters? -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 09/30/2014 05:11 AM, Victor Sudakov wrote: >> >> Can you share the basic cache manager requests statistics and the >> up time for the service? (mgr:info) >> >> This would give us a basic idea of the load\requests needed to >> reproduce it. > > I am not Steve but in case you care to look at my statistics as > well, I am attaching it. > OK. The basic info states that the server is up for about 10+ hours. and passed in this period of time: 152054 request which is about 230-253 per minute. The cache is holding more then 20k objects and maximizing it's ram memory usage (less 0.6%). Considering that you are using 128MB for ram storage the service should be considered 128MB used as more then normal. In the log it states that the accounted memory: 403163 KB which is about 390MB in usage. 390 -128 = 262 so the only issue is about 250-260 MB which are being used and not for cache. the basic usage of squid is about 3MB if i'm not wrong but there are operational needs such as buffers and in transit stuff. So 255 MB to think about.. 500 FD which means about 300-350 connections doubles 64kb. then it's about 13-14 MB for in transit stuff. So there is about 241 MB which needs to be justified. These are the basics. Eliezer -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEcBAEBAgAGBQJUKogaAAoJENxnfXtQ8ZQU/pUIAIh2Q1687fXSTsiUCO0jlnyO NOD7F250EcppFwrQ4GPzDvRiNWUhzJSVzncAzJWlgQbWmoCnsvwB0C7MUN6bSOBf ek0qmJi1m/xz1ePVFI1u1k/PfC4t2FTC8XM2zICSQKSkHIlV06EPNd5Km954bIYP C7kzSl9CHl34v3179S22rEt1hej4Q3FEVayvoATrrPdrmyWLE2lICJYFxqGEIPK4 ivzxdcpdVovR2AwCHeDqXrTB/wTrYaZz1TQWUZScCmPibd1oqGXd3gHefE1aOMwS ZsuBUZLfiTmaezBBeEqq8P3oWLIBh2mU1m4JogUbDixkrLpi4GaIFk+TIp9CTNw= =e8AJ -END PGP SIGNATURE- ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 09/30/2014 08:33 AM, Victor Sudakov wrote: > Attaching two cachemgr reports: right after squid restart and > several hours later (grown to 816M in SIZE). I am not very good at calculating what I do not know how to yet! What I do understand is that "Cumulative allocated volume: 22.198 GB" means that this is what was used ever by squid for this uptime. Since I am not such a good calculator and the data exists already lets lets try and see the next output of: mgr:mem mgr:info and "top -n 1 -b|egrep "proxy|squid|PID"" Which should give a nice output of the squid process memory consumption. Thanks, Eliezer -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEcBAEBAgAGBQJUKo5YAAoJENxnfXtQ8ZQUaA4H/2wEiismAH8PnrrkH6BdZ4Y/ m64kSlM2uyVB+dOdNqaH2Ftm+QLRQxKzFvaWp6dmA5Ih3C8rxt1tBD4ZAW9BwXJp SVynocW44iXr/8aboIjmph4ojo9PVSuiZ87XrRB+jHC+K1YUpnNOWdmSYVEE/apx xWt9wONllmHUqISy4C1CFhbZxeyweGsRMYzwWzgR76gDDTmw7oCZsp48H8s/aE7c qchNNWCKmXqtHKyhFPobJpRvy4uAt5Xx8RxaHva46qF3EEWRDWM1Hcl4c26YkPVw 88hylHuz1JB4LCeG0g4Kxcs4AOnEHgqfXWtgpPwGevoHq64ytiVnBGI/o8bp/Dc= =kI7u -END PGP SIGNATURE- ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Eliezer Croitoru wrote: > > Attaching two cachemgr reports: right after squid restart and > > several hours later (grown to 816M in SIZE). > I am not very good at calculating what I do not know how to yet! > What I do understand is that "Cumulative allocated volume: 22.198 GB" > means that this is what was used ever by squid for this uptime. > Since I am not such a good calculator and the data exists already lets > lets try and see the next output of: > mgr:mem > mgr:info I have already restarted squid several times since I obtained the reports attached to the previous mail. The mgr:mem and mgr:info info reports don't have to be from the same process, do they? And is there a way to obtain mgr:info etc from the command line, not from a browser? > and "top -n 1 -b|egrep "proxy|squid|PID"" > Which should give a nice output of the squid process memory consumption. - -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEcBAEBAgAGBQJUKqD+AAoJEA2k8lmbXsY0IP4H/2IjJUN4Gi7h0L58W3RvEthQ 8qkR7KoAKQHpeJ/BAQLehUM5W3M4DInJvxezfjrJIo6tS1UgzsoB9Gv012YcHU7J JwQi/8zofnw4kjLeXnEJrIh4yTl5u46zQQARG+iyDjd1oRr4AMtO8zu84+gz1oc8 18Dsyjt5g9HV2ErsCkmSmKpIIVo/sU/PBj3Wsa4g386FCBMB7UkzRP3TrnwqFbgc vrEZ3xKfZHm7AB16L7gURw64kp1Rs26rdcm4JVnkzlMrYziAZmWfZcQeeJILAFmM wLwOdQLj19OFpxS5wfo/L1yux3oLcBxLS15Rdw8w+RoYptWFPeSY/pfD5Er4Mzo= =Tjgd -END PGP SIGNATURE- ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 1/10/2014 1:24 a.m., Victor Sudakov wrote: > Eliezer Croitoru wrote: >>> Attaching two cachemgr reports: right after squid restart and >>> several hours later (grown to 816M in SIZE). >> I am not very good at calculating what I do not know how to yet! >> What I do understand is that "Cumulative allocated volume: 22.198 >> GB" means that this is what was used ever by squid for this >> uptime. Since I am not such a good calculator and the data exists >> already lets lets try and see the next output of: mgr:mem >> mgr:info > > I have already restarted squid several times since I obtained the > reports attached to the previous mail. The mgr:mem and mgr:info > info reports don't have to be from the same process, do they? Yes, its preferrable that they are from the same process as close to the same time as possible. > > And is there a way to obtain mgr:info etc from the command line, > not from a browser? The squidclient tool can be used to pull manager reports directly and quickly. Like so: squidclient mgr:info >squid_mgr_info.txt squidclient mgr:mem >squid_mgr_memory.tsv Since you expressed some confusion about reading the mem report here are some basics that should help: The memory report table is a TSV format. The .tsv file generated above can be opened by most any spreadsheet tools (Excell, OpenOffice Calc, etc). You just select "TXT formatted" or "Comma Separated Value" as the file type then select Tab as separator (de-select comma / CSV) if asked what the format type is. That makes it easier to view and play with the data. In your case with pooling disabled the relevant details are in the section of columns titled "In Use". The "Chunked" and "Allocations Saved" sections are garbage when pools are off. The report is sorted by which objects have been allocated most memory. So if you are familar or can find out what the named objects are used for then it may (or not) be easy to spot something odd. eg. in your first report I saw the IdentStateData was in hundreds of MB with 50K active requests for just 150 clients. *very* odd given your config should make no more than 3 IDENT lookups per client. Most objects names should give away ther meaning. The unusual one you should expect to be first is mem_node. Which is the name for a 4KB chunk of cache_mem and in-transit objects. There is some bytes of meta data making it slightly bigger than 4KB, which on some OS means the memory page uses 8KB per object without telling Squid. This can be a hidden gotcha and why Squid sometimes uses slightly over 2x what the mem_node column says is allocated to cache_mem. :-( > >> and "top -n 1 -b|egrep "proxy|squid|PID"" Which should give a >> nice output of the squid process memory consumption. Be careful noting which column is resident memory and which is virtual memory in that output. Each helper spawned is allocated virtual memory equal to the total resident size of its squid parent process (including cache_mem and any shared memory usage) at the time of its spawning. It can show very huge numbers even while Squid goes nowhere near using any of it. HTH Amos -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (MingW32) iQEcBAEBAgAGBQJUKqvAAAoJELJo5wb/XPRj2rEH/2ThUFxwWX5JgXDH/Tpp/QLc YYROxf65XpSRMUVvyygl1AVzgZxwO3XK6HQGz/bzlRWBIQFbLYuCucWLk0RBBy2Q 4CHdS90SOML4fjCUpxTXvtbH8f6jp+MNH6ULEvmCpDtmAg1o6NQD86mO/nlddfHU /TexsrkvGMNWluMGDi/dhUJY2XHvAbMfd3l3dGpmzmDu2uPM8LwDDV26nJ07657j BIxG1kue+sAD0LdK+DQlegedo4zu4aCy8H355eGskhUht56dx0dMniTe0q8nxP+x y/cuiGOEjjfHwNmLAXZbSVjdMZc4V2sk9mhyXQ7CAQ4CAD3e0VxrXqsPwLh4lwE= =/mkc -END PGP SIGNATURE- ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 About the memory page size: getconf PAGESIZE getconf PAGE_SIZE One of the above should provide the basic information. Waiting for some more info. Eliezer On 09/30/2014 04:10 PM, Amos Jeffries wrote: > The unusual one you should expect to be first is mem_node. Which > is the name for a 4KB chunk of cache_mem and in-transit objects. > There is some bytes of meta data making it slightly bigger than > 4KB, which on some OS means the memory page uses 8KB per object > without telling Squid. This can be a hidden gotcha and why Squid > sometimes uses slightly over 2x what the mem_node column says is > allocated to cache_mem. :-( > > >>> > and "top -n 1 -b|egrep "proxy|squid|PID"" Which should give > a nice output of the squid process memory consumption. > Be careful noting which column is resident memory and which is > virtual memory in that output. Each helper spawned is allocated > virtual memory equal to the total resident size of its squid parent > process (including cache_mem and any shared memory usage) at the > time of its spawning. It can show very huge numbers even while > Squid goes nowhere near using any of it. > > HTH Amos -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEcBAEBAgAGBQJUKq/wAAoJENxnfXtQ8ZQU70IH/0PUDwHg1aXQZcPpMb1FA9oM 0cFu5AIZOf5c4a5VcgvYRLo0mq7yrQnQ21D2cggs5U/OqB7/7UwDUleYqyc+qNN/ vZug02WaWO/GlNztzUPHXUsuLBqAO3ULHXc9BcC7h/M3IB4hjHeiwH5YsAKx17aA ktc97witmsVzU0/LUN9B4Ptyhb5vDq/jVfbrn5JkETPUO3lreuw1EzxTKSpf5eKU VqrSOFo+tAnMWoN3WK5Wn1ZCY7HlC+Rwevb6wjtBJtQU6SW4k7T075soLbHyWprS mwW1srOmc4KZ1ZDtxkhN3NBKfr92dWzUxxJPTr1WpTTYMg9MaCvIHY9I+TDzLY8= =0BTf -END PGP SIGNATURE- ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 29/09/2014 10:04 p.m., Steve Hill wrote: > On 28.09.14 08:34, Eliezer Croitoru wrote: > >> Also to minimize the leak source try to share more information >> about your usage. Are you using SMP workers or not? What is the >> mem and info options in the cache manager page data? Did you >> tried "cache deny all" acl to minimize the source of the leak? > > For what its worth, I'm also seeing a big memory leak on multiple > servers under Squid 3.4.6: > > I have two basic Squid configurations - one of them is a plain > caching proxy and is _not_ leaking. The other does no caching, but > is doing ICAP, external ACLs, SSL bumping and TPROXY and leaks like > a sieve (several gigabytes a day). > > I _think_ I have narrowed it down to something ICAP related and > I'm currently valgrinding. Unfortunately I can't seem to get the > valgrind instrumentation to work so I'm having to do without (I > compile --with-valgrind-debug but "squidclient mgr:mem" doesn't > produce a valgrind report). I have noticed that there are a _lot_ > of memory errors reported by valgrind though (uninitialised > memory). IIRC the valgrind report is strangely at the end of mgr:info rather than mgr:mem. It also only lists details if there is *real* leaked memory. With one small exception all the reports of memory "leaks" in 3.2/3.3/3.4 series are in fact memory over-allocations. The difference is that over-allocated memory is still referenced by some part of Squid, so it *does not* show up in a leak finder like valgrind. For example; most of Victors "leak" was in fact thousands of hung asynchronous code chains/threads waiting for some network I/O which could never happen, or if it does not for some very long time. Still perfectly accessible to Squid via the index of active IDENT lookups (ie not leaked), but should have been explicitly released already. > > Pretty much all of the leaky memory is showing up as unaccounted: > Memory accounted for: Total accounted:84306 KB 3% memPool > accounted: 84306 KB 3% memPool unaccounted: 2917158 KB > 97% > > I am using SMP workers, but turning that off doesn't fix the > issue. > If you are using a 64-bit OS then the unaccounted numbers there are bogus. They are based on mallinfo() lookup results which suffer from 32-bit wrap issues. Use the OS command line to get a memory report from top or similar instead. HTH Amos -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (MingW32) iQEcBAEBAgAGBQJUKrp/AAoJELJo5wb/XPRjfgEH/jMS0eZ0y/rZ4k4nZAprDCmr +iJrBOR7/7yObQ4FEZlX793qofky760x35VuHvVantzl0oy2CNk2rCmluI85WOTn i4WYfCvU/1J5ECp9PnodcslldgFM8TXcG5yuXhj2Ro36Prs0RYuXvwC7cuf4ASd9 BuoP/RCbNjlJFwHoHEcXM35OLMU0dshnSZ6YnesLOZODRUz9qNJK/x1Q0xeXAXBC LjRatV2IEr8O5iV6BqTHQDVHOAUuG/m35dIeruyeg6r3pLngtR8PYxWfLsdTVxfP UUjN4EoVozpyuIytvB4/PYSKDkFf8AtQbEnznXtnuyaWZR49/E8kpSp4bd61fyg= =x7XO -END PGP SIGNATURE- ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
On 29.09.14 13:39, Eliezer Croitoru wrote: Hey Steve, Can you share the basic cache manager requests statistics and the up time for the service? (mgr:info) This is with 8 workers and was restarted this morning, about 6 hours ago. As you can see, it's using about 5GB at the moment - as mentioned, there is no caching enabled for this configuration so there doesn't seem much reason for it to grow so large. Squid Object Cache: Version 3.4.6 Build Info: Start Time: Tue, 30 Sep 2014 08:33:01 GMT Current Time: Tue, 30 Sep 2014 14:14:28 GMT Connection information for squid: Number of clients accessing cache: 2545 Number of HTTP requests received: 926815 Number of ICP messages received:0 Number of ICP messages sent:0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 2714.3 Average ICP messages per minute since start:0.0 Select loop called: 33693001 times, 5.219 ms avg Cache information for squid: Hits as % of all requests: 5min: 0.0%, 60min: 0.0% Hits as % of bytes sent:5min: 3.5%, 60min: 4.4% Memory hits as % of hit requests: 5min: 0.0%, 60min: 0.0% Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.0% Storage Swap size: 0 KB Storage Swap capacity: 0.0% used, 0.0% free Storage Mem size: 2168 KB Storage Mem capacity:0.0% used, 0.0% free Mean Object Size: 0.00 KB Requests given to unlinkd: 0 Median Service Times (seconds) 5 min60 min: HTTP Requests (All): 0.06557 0.06988 Cache Misses: 0.06629 0.07171 Cache Hits:0.0 0.0 Near Hits: 0.0 0.0 Not-Modified Replies: 0.0 0.0 DNS Lookups: 0.00012 0.0 ICP Queries: 0.0 0.0 Resource usage for squid: UP Time:20487.255 seconds CPU Time: 3177.711 seconds CPU Usage: 15.51% CPU Usage, 5 minute avg:11.44% CPU Usage, 60 minute avg: 12.40% Maximum Resident Size: 19930960 KB Page faults with physical i/o: 6 Memory usage for squid via mallinfo(): Total space in arena: 5014812 KB Ordinary blocks: 5004965 KB 2620 blks Small blocks: 0 KB 0 blks Holding blocks: 82080 KB 48 blks Free Small blocks: 0 KB Free Ordinary blocks:9847 KB Total in use:9847 KB 0% Total free: 9847 KB 0% Total size:5096892 KB Memory accounted for: Total accounted: 447132 KB 9% memPool accounted: 447132 KB 9% memPool unaccounted: 4649760 KB 91% memPoolAlloc calls: 528851610 memPoolFree calls: 531832922 File descriptor usage for squid: Maximum number of file descriptors: 131072 Largest file desc currently in use:209 Number of file desc currently in use: 754 Files queued for open: 0 Available number of file descriptors: 130318 Reserved number of file descriptors: 800 Store Disk files open: 0 Internal Data Structures: 510 StoreEntries 510 StoreEntries with MemObjects 408 Hot Object Cache Items 0 on-disk objects For comparison, almost all of the http (but none of the https) requests that go through the proxy above are then sent through a second proxy which does do some caching, but no fancy stuff like ICAP or ssl bumping - the caching proxy has been running for 18 days and has a far smaller process size (again, 8 workers): Squid Object Cache: Version 3.4.6 Build Info: Start Time: Fri, 12 Sep 2014 08:48:50 GMT Current Time: Tue, 30 Sep 2014 14:18:35 GMT Connection information for squid: Number of clients accessing cache: 1 Number of HTTP requests received: 19496669 Number of ICP messages received:0 Number of ICP messages sent:0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 742.7 Average ICP messages per minute since start:0.0 Select loop called: 763795228 times, 2.062 ms avg Cache information for squid: Hits as % of all requests: 5min: 13.1%, 60min: 12.8% Hits as % of bytes sent:5min: 7.0%, 60min: 4.7% Memory hits as % of hit requests: 5min: 0.0%, 60min: 0.0% Disk hits as % of hit requests: 5min: 16.4%, 60min: 11
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
On 30.09.14 15:13, Amos Jeffries wrote: IIRC the valgrind report is strangely at the end of mgr:info rather than mgr:mem. In that case the wiki is wrong. :) Anyway, it doesn't show up on either of them for me so I guess you might be right that it isn't "real" leaked memory. (I still thought I'd see some indication that it was invoking the valgrind report though, even if it contained no data) With one small exception all the reports of memory "leaks" in 3.2/3.3/3.4 series are in fact memory over-allocations. The difference is that over-allocated memory is still referenced by some part of Squid, so it *does not* show up in a leak finder like valgrind. Do you have any advice for finding out what memory is still referenced in large amounts? Since this is unaccounted memory, it presumably doesn't appear in mgr:mem (at least, I can't see anything obvious in there)? I'm trying to figure out if there's a way of convincing valgrind to dump info about all the currently allocated memory while the program is still running - there would be a lot of legitimate stuff in the report, but hopefully a few hundred MB of memory that shouldn't be there would stick out like a sore thumb. If you are using a 64-bit OS then the unaccounted numbers there are bogus. They are based on mallinfo() lookup results which suffer from 32-bit wrap issues. Use the OS command line to get a memory report from top or similar instead. In this case I don't believe the data is bogus. I have 8 workers, which top shows as: 26249 squid 20 0 871m 782m 5348 S 0.7 10.0 8:44.81 squid 26245 squid 20 0 732m 644m 5344 S 0.3 8.3 8:00.77 squid 26244 squid 20 0 706m 617m 5348 S 1.0 7.9 7:42.29 squid 26250 squid 20 0 699m 613m 5348 S 3.6 7.9 4:43.49 squid 26246 squid 20 0 699m 612m 5348 S 2.3 7.9 6:12.78 squid 26251 squid 20 0 662m 576m 5348 S 0.7 7.4 5:45.11 squid 26248 squid 20 0 649m 564m 5348 S 2.3 7.2 9:22.91 squid 26247 squid 20 0 603m 518m 5348 S 1.3 6.6 4:45.47 squid Adding the sizes in the "virt" column gives me 5621MB. The combined RSS is 4926MB. Process 26250 has just 2MB in swap, the rest have no swap at all. I guess the difference between the RSS and Virt totals can probably be almost entirely accounted for by the executables. mallinfo() says there is about 4897MB in the arena - close enough to sum of the RSS that I'm inclined to believe it. Since no one process exceeds 2GB, mallinfo() is probably trustworthy in this case. The accounted for memory is 440MB - potentially higher than I would expect for a Squid with caching disabled, but certainly low enough to not be an immediate concern. But that gives us about 4.4GB of memory unaccounted for (by subtracting 440MB from the sum of the RSS as shown by top). 4.4GB of unaccounted memory after running for just 6 hours is a significant amount, and if left to their own devices Squid continues to eat all the memory, leaving the machine thrashing swap. I count 531 sockets in netstat, so my guess is that it isn't just data associated with some open sockets: netstat -apn|egrep '26249|26245|26244|26250|26246|26251|26248|26247' -c 531 -- - Steve ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 30/09/2014 6:33 p.m., Victor Sudakov wrote: > Victor Sudakov wrote: >> Amos Jeffries wrote: >>> >>> You have 200 MB of RAM locked up in IDENT lookups. >>> >>> Probably http://bugs.squid-cache.org/show_bug.cgi?id=3803 >> >> I have temporarily disabled IDENT-related acls. Squid still grows >> in memory, but more slowly. So IDENT was certainly one of the >> major causes of the leak. >> >> Could we make another iteration of looking at the cachemgr.cgi >> memory counters? >> > > Attaching two cachemgr reports: right after squid restart and > several hours later (grown to 816M in SIZE). > These are still showing over 47K IDENT lookups. Probably you did not disable all the uses of IDENT. You need to both set "ident_access deny all" and remove use of ident type ACLs. Before trying that though, I attached a new patch to bug report 3803. Can you apply that and see if it works? NP: For your request rate of 150 req/sec I expect the mgr:mem "cbdata IdentStateData (17)" to show less than 3K in "(#) Allocated" ( 12152 in "(KB) Allocated"). Amos -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (MingW32) iQEcBAEBAgAGBQJUKsUhAAoJELJo5wb/XPRjxBwH/i9ee1HQtS2cwsYrNNqd2yG8 cdlvpfceHXY+iS19HL/hOPRDFUN9VfTh9LxWdk7wCASmIFd8he5I5e8Fpgu0/+80 HhpXkXgTMTUljKc6A7Vs+M6RUalp1MdWqcebzzuh7xMTXDzKaD1pky3xOsFcPuqy umoDC7fFukhuCWVkBXBv/IvC8z3oUUbUD8oLVUD5PVqW5X3/RDYWW4L7LosxDt92 4rKW6uDJs+ytYlRYfZ43I1AEHlXMbvAxClFtVttG91I5b2uBzKasWiI9OQCyVVht uAnm7LlQvHnT72JchbyqRPPaeESyFRpNGPlFFNv7mi67EAs2N6cMGDbJeaRLWV4= =lAX8 -END PGP SIGNATURE- ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 1/10/2014 3:59 a.m., Steve Hill wrote: > > On 30.09.14 15:13, Amos Jeffries wrote: > >> IIRC the valgrind report is strangely at the end of mgr:info >> rather than mgr:mem. > > In that case the wiki is wrong. :) Anyway, it doesn't show up on > either of them for me so I guess you might be right that it isn't > "real" leaked memory. (I still thought I'd see some indication > that it was invoking the valgrind report though, even if it > contained no data) > >> With one small exception all the reports of memory "leaks" in >> 3.2/3.3/3.4 series are in fact memory over-allocations. The >> difference is that over-allocated memory is still referenced by >> some part of Squid, so it *does not* show up in a leak finder >> like valgrind. > > Do you have any advice for finding out what memory is still > referenced in large amounts? Since this is unaccounted memory, it > presumably doesn't appear in mgr:mem (at least, I can't see > anything obvious in there)? > > I'm trying to figure out if there's a way of convincing valgrind to > dump info about all the currently allocated memory while the > program is still running - there would be a lot of legitimate stuff > in the report, but hopefully a few hundred MB of memory that > shouldn't be there would stick out like a sore thumb. That would be lovely. If you can find it I'd like to know too :-) A third option is using a ALL,9 cache.log tracer. We have begun instrumenting class contstructors and destructors with debug statements to record their lifetimes regardless of whether they are accounted. A lot are not yet done though. The bzr repo contains a debug script at scripts/find-alive.pl which correlates constructors to destructors in that log. It is useful for scanning a cache.log while Squid is still running, crashed, or paused in debugger to show what Jobs/requests etc are currently active. > >> If you are using a 64-bit OS then the unaccounted numbers there >> are bogus. They are based on mallinfo() lookup results which >> suffer from 32-bit wrap issues. Use the OS command line to get a >> memory report from top or similar instead. > > In this case I don't believe the data is bogus. I have 8 workers, > which top shows as: 26249 squid 20 0 871m 782m 5348 S 0.7 > 10.0 8:44.81 squid 26245 squid 20 0 732m 644m 5344 S 0.3 > 8.3 8:00.77 squid 26244 squid 20 0 706m 617m 5348 S 1.0 > 7.9 7:42.29 squid 26250 squid 20 0 699m 613m 5348 S 3.6 > 7.9 4:43.49 squid 26246 squid 20 0 699m 612m 5348 S 2.3 > 7.9 6:12.78 squid 26251 squid 20 0 662m 576m 5348 S 0.7 > 7.4 5:45.11 squid 26248 squid 20 0 649m 564m 5348 S 2.3 > 7.2 9:22.91 squid 26247 squid 20 0 603m 518m 5348 S 1.3 > 6.6 4:45.47 squid > > Adding the sizes in the "virt" column gives me 5621MB. The > combined RSS is 4926MB. Process 26250 has just 2MB in swap, the > rest have no swap at all. I guess the difference between the RSS > and Virt totals can probably be almost entirely accounted for by > the executables. > > mallinfo() says there is about 4897MB in the arena - close enough > to sum of the RSS that I'm inclined to believe it. Since no one > process exceeds 2GB, mallinfo() is probably trustworthy in this > case. > > The accounted for memory is 440MB - potentially higher than I > would expect for a Squid with caching disabled, but certainly low > enough to not be an immediate concern. But that gives us about > 4.4GB of memory unaccounted for (by subtracting 440MB from the sum > of the RSS as shown by top). > > 4.4GB of unaccounted memory after running for just 6 hours is a > significant amount, and if left to their own devices Squid > continues to eat all the memory, leaving the machine thrashing > swap. > > I count 531 sockets in netstat, so my guess is that it isn't just > data associated with some open sockets: netstat -apn|egrep > '26249|26245|26244|26250|26246|26251|26248|26247' -c 531 > > -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (MingW32) iQEcBAEBAgAGBQJUKsh9AAoJELJo5wb/XPRjUnAIALX5yfqY/XOVOcH90OXUree2 sHDJdCB5zlr4EO94qvichnkUm5bCMlg78dkPv8Nwyd43nG4SUUBNt4nh2dvZxYHn 8scHWFA1ZMsbl3k3dQ3eEHNCm+PfugvBo34ZokkMpSR8mLZhh9vqxXDhh9rwj6ez K8mKnYMbNI/z/DOHhJxcM1NdnbYIyO84EUQfb9RGaJaKb8LDNLt4j6yCEeKa2Z8N YcnPdA9VZZ3RCpImq0Py8U0kZbCPPBRs+cJAvUIgaAO080ZTrbF+bLZ+dHe4W7hA JMxXUD4lfEy2HVja2+jfKBcJQwVCDwBK0tcu76bQ64WIrfuIIc/NyIOBwEhPT1I= =cLQ3 -END PGP SIGNATURE- ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
Amos Jeffries wrote: > >>> > >>> You have 200 MB of RAM locked up in IDENT lookups. > >>> > >>> Probably http://bugs.squid-cache.org/show_bug.cgi?id=3803 > >> > >> I have temporarily disabled IDENT-related acls. Squid still grows > >> in memory, but more slowly. So IDENT was certainly one of the > >> major causes of the leak. > >> > >> Could we make another iteration of looking at the cachemgr.cgi > >> memory counters? > >> > > > > Attaching two cachemgr reports: right after squid restart and > > several hours later (grown to 816M in SIZE). > > > > These are still showing over 47K IDENT lookups. > > Probably you did not disable all the uses of IDENT. You need to both > set "ident_access deny all" and remove use of ident type ACLs. Bingo! After setting "ident_access deny all" squid does not grow infinitely any more. However, it remains a major CPU hog. I am attaching the output of mgr:info, mgr:mem and top as requested. === last pid: 81001; load averages: 1.49, 1.48, 1.32 up 12+14:20:2112:58:39 180 processes: 2 running, 178 sleeping Mem: 478M Active, 244M Inact, 189M Wired, 43M Cache, 107M Buf, 2640K Free Swap: 2548M Total, 67M Used, 2481M Free, 2% Inuse PID USERNAME THR PRI NICE SIZERES STATE C TIME WCPU COMMAND 39322 squid1 1170 352M 323M CPU00 150:05 95.36% squid 84342 squid1 440 5700K 4140K sbwait 1 0:09 0.00% ntlm_auth 84529 squid1 440 4676K 2272K sbwait 1 0:03 0.00% ntlm_auth 39325 squid1 440 4508K 1452K piperd 1 0:03 0.00% unlinkd 84724 squid1 440 3652K 1892K sbwait 1 0:02 0.00% ntlm_auth 84725 squid1 440 3652K 1740K sbwait 1 0:01 0.00% ntlm_auth 84726 squid1 440 3652K 1632K sbwait 1 0:01 0.00% ntlm_auth 93981 squid1 440 3652K 1556K sbwait 0 0:01 0.00% ntlm_auth 12271 squid1 440 3652K 1500K sbwait 0 0:01 0.00% ntlm_auth 12272 squid1 440 3652K 1464K sbwait 1 0:00 0.00% ntlm_auth 12273 squid1 440 3652K 1436K sbwait 0 0:00 0.00% ntlm_auth 12274 squid1 440 3652K 1408K sbwait 1 0:00 0.00% ntlm_auth 13032 squid1 440 3652K 1392K sbwait 1 0:00 0.00% ntlm_auth 13033 squid1 440 3652K 1384K sbwait 1 0:00 0.00% ntlm_auth 14750 squid1 440 3652K 1376K sbwait 1 0:00 0.00% ntlm_auth 14751 squid1 440 3652K 1364K sbwait 0 0:00 0.00% ntlm_auth 14752 squid1 440 3652K 1356K sbwait 1 0:00 0.00% ntlm_auth 18076 squid1 440 3652K 1344K sbwait 1 0:00 0.00% ntlm_auth HTTP/1.1 200 OK Server: squid/3.4.8 Mime-Version: 1.0 Date: Wed, 01 Oct 2014 06:00:28 GMT Content-Type: text/plain Expires: Wed, 01 Oct 2014 06:00:28 GMT Last-Modified: Wed, 01 Oct 2014 06:00:28 GMT X-Cache: MISS from proxy.sibptus.transneft.ru X-Cache-Lookup: MISS from proxy.sibptus.transneft.ru:3128 Via: 1.1 proxy.sibptus.transneft.ru (squid/3.4.8) Connection: close Squid Object Cache: Version 3.4.8 Build Info: Start Time: Tue, 30 Sep 2014 15:56:38 GMT Current Time: Wed, 01 Oct 2014 06:00:28 GMT Connection information for squid: Number of clients accessing cache: 170 Number of HTTP requests received: 375550 Number of ICP messages received:0 Number of ICP messages sent:0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 445.1 Average ICP messages per minute since start:0.0 Select loop called: 32559927 times, 1.555 ms avg Cache information for squid: Hits as % of all requests: 5min: 3.2%, 60min: 4.1% Hits as % of bytes sent:5min: 22.1%, 60min: 11.3% Memory hits as % of hit requests: 5min: 7.6%, 60min: 20.8% Disk hits as % of hit requests: 5min: 18.8%, 60min: 20.1% Storage Swap size: 471890 KB Storage Swap capacity: 90.0% used, 10.0% free Storage Mem size: 130484 KB Storage Mem capacity: 99.6% used, 0.4% free Mean Object Size: 23.64 KB Requests given to unlinkd: 65340 Median Service Times (seconds) 5 min60 min: HTTP Requests (All): 1.00114 0.44492 Cache Misses: 2.13280 1.00114 Cache Hits:0.94847 0.05951 Near Hits: 1.11539 0.68577 Not-Modified Replies: 0.52331 0.08265 DNS Lookups: 0.30421 0.12472 ICP Queries: 0.0 0.0 Resource usage for squid: UP Time:50629.945 seconds CPU Time: 9112.184 seconds CPU Usage: 18.00% CPU Usage, 5 minute avg:
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 1/10/2014 8:45 p.m., Victor Sudakov wrote: > Amos Jeffries wrote: > > You have 200 MB of RAM locked up in IDENT lookups. > > Probably http://bugs.squid-cache.org/show_bug.cgi?id=3803 I have temporarily disabled IDENT-related acls. Squid still grows in memory, but more slowly. So IDENT was certainly one of the major causes of the leak. Could we make another iteration of looking at the cachemgr.cgi memory counters? >>> >>> Attaching two cachemgr reports: right after squid restart and >>> several hours later (grown to 816M in SIZE). >>> >> >> These are still showing over 47K IDENT lookups. >> >> Probably you did not disable all the uses of IDENT. You need to >> both set "ident_access deny all" and remove use of ident type >> ACLs. > > Bingo! After setting "ident_access deny all" squid does not grow > infinitely any more. However, it remains a major CPU hog. > Yay. Any news on the bug patch? Note that from the same "CPU hog" cycles you are now getting around 2x the HTTP traffic throughput. > > Start Time: Mon, 29 Sep 2014 15:24:54 GMT Current Time: Tue, 30 Sep > 2014 02:09:51 GMT Connection information for squid: Number of > clients accessing cache: 161 Number of HTTP requests received: > 152054 Average HTTP requests per minute since start: 235.8 Select > loop called: 4664551 times, 8.296 ms avg > Start Time: Tue, 30 Sep 2014 15:56:38 GMT Current Time: Wed, 01 Oct > 2014 06:00:28 GMT Connection information for squid: Number of > clients accessing cache: 170 Number of HTTP requests received: > 375550 Average HTTP requests per minute since start: 445.1 Select > loop called: 32559927 times, 1.555 ms avg You have the delay pools feature configured. It is a wasteful consumer of CPU cycles. Also NTLM authentication is used, that doubles the HTTP request overheads on each new TCP connection. There are two things you can do to further improve performance: 1) converting from NTLM to Kerberos authentication. 2) moving the delay pools limitation into kernel QoS systems. Amos -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (MingW32) iQEcBAEBAgAGBQJUK7dsAAoJELJo5wb/XPRjJqQIAICPcvQbP+pMvmZk5SnqHUvU vrHLXcK1+WHhw+29UvYTZUQaTf8f7r3oWSfQS4mm56YrekMiFFJSeb+QMTh3nRdK ij3dcgdk/cT1ziQJgyh7i2AyzpC8iAofC5I8MTP247qzV14s0ZdmtRpyCbPYeRL5 JqCsMW+X/7dWbVvNhSjb0J57Po2M4Fo0RWFuPIhU/gOP6jLyesbqgm1CaGXSYvyh kVjwZuaNKjCL1bV18vPPMJXW6Kgl6p4cK+X/v8aw00Eb2EcYsu8ieaLXvAehlsuB xViLWe1KPuJOawJ6uaGULkoGF+a6hc+3DbJ+iLHRFNdW1OHvKWJGgOs8M5jiIHw= =Z7vQ -END PGP SIGNATURE- ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
On 29.09.14 10:04, Steve Hill wrote:
I _think_ I have narrowed it down to something ICAP related
Looks like I was wrong - it actually seems to be external ACL related.
I have an external ACL defined as:
external_acl_type preauth cache=0 children-max=1 concurrency=100 ttl=0
negative_ttl=0 %SRC %>{User-Agent} %URI %METHOD /usr/sbin/squid-preauth
The inclusion of %URI means that it's going to be called a lot, even
with caching, but in this case I've turned caching off. As far as I can
see in the code, if cache=0 or (ttl=0 and negative_ttl=0), it doesn't
touch the cache at all so my guess is that this isn't a problem with the
caching code.
I'm testing this with Siege and consistently seeing "Total size"
increasing by about 51MB and "memPool unaccounted" increasing by about
14MB after 20,000 requests from a fresh start (so, 2.6K per request and
0.7K/request respectively). If I disable the external ACL then I see
growths of about 10MB and 1MB respectively.
Although this isn't especially consistent with the stats from a
production system that I tested yesterday, which showed about
5.5K/request (total) and 5K/request (unaccounted) over 926815 requests.
--
- Steve
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
I have an external ACL defined as:
external_acl_type preauth cache=0 children-max=1 concurrency=100 ttl=0
negative_ttl=0 %SRC %>{User-Agent} %URI %METHOD /usr/sbin/squid-preauth
It is well known that external ACLs with ttl=0 and cache=0 leak RAM: I
had this problem and discovered the reason only recently, but Amos knows
it since 2011:
https://www.mail-archive.com/squid-dev@squid-cache.org/msg14809.html
I recently opened a bug about this, that I will update now:
http://bugs.squid-cache.org/show_bug.cgi?id=4088
A workaround that we use and may be applicable in your case is to use
ttl=1 cache=1.
Regards,
Bergonz
--
Ing. Michele Bergonzoni - Laboratori Guglielmo Marconi S.p.a.
Phone:+39-051-6781926 e-mail: berg...@labs.it
alt.advanced.networks.design.configure.operate
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 2/10/2014 1:19 a.m., Michele Bergonzoni wrote:
>> I have an external ACL defined as: external_acl_type preauth
>> cache=0 children-max=1 concurrency=100 ttl=0 negative_ttl=0 %SRC
>> %>{User-Agent} %URI %METHOD /usr/sbin/squid-preauth
>
> It is well known that external ACLs with ttl=0 and cache=0 leak
> RAM: I had this problem and discovered the reason only recently,
> but Amos knows it since 2011:
>
> https://www.mail-archive.com/squid-dev@squid-cache.org/msg14809.html
>
>
If you read the next post in that thread you will see the referenced
patch was dropped. IIRC it always produced a non-match result in some
configs.
An updated variant is present in 3.4 which still produces a useful
result in all cases. The leak is probably from that.
> I recently opened a bug about this, that I will update now:
>
> http://bugs.squid-cache.org/show_bug.cgi?id=4088
>
> A workaround that we use and may be applicable in your case is to
> use ttl=1 cache=1.
>
Thank you for the reminder. I will start work on this next.
Amos
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (MingW32)
iQEcBAEBAgAGBQJUK/miAAoJELJo5wb/XPRjTeMH/jxUyS/7Bn14nYC3zkR6O4Oi
LPjCSKVxsOpjyr0577bmvYwmCEC6KEOO5cwoPUXNPys4OuZ8w4nQjo98Jsh1yF4n
/atpPWS+qOFXwHgvgU1SPlxbl3hB44jQCuOQPqgtX7ZQff4c4z7EbH03Ieq2Y9pq
6S8jo+jtAN4p9kRgnhgLy1uMM4mP3uM4mFKimkY3f5seQynL02+xzFamqEpxzoUU
87cieyHRy4okicC2VvAg33/ErpXJLAfYswF+pbZcOqxnerbI08mXK9HeCkXuATUB
v1z4HEJKlZijJFUr9DfZ4d9ara/abQUiVs8gr+kDBRDF51i3F235VxWDdGSHjxU=
=DPXu
-END PGP SIGNATURE-
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
On 01.10.14 13:19, Michele Bergonzoni wrote:
I have an external ACL defined as:
external_acl_type preauth cache=0 children-max=1 concurrency=100 ttl=0
negative_ttl=0 %SRC %>{User-Agent} %URI %METHOD /usr/sbin/squid-preauth
It is well known that external ACLs with ttl=0 and cache=0 leak RAM: I
had this problem and discovered the reason only recently, but Amos knows
it since 2011:
Hmm, I've changed my ttl, negative_ttl and cache to 1 and Squid nolonger
seems to grow under test conditions... Unfortunately it looks like its
still growing on the production system. I guess my tests aren't
reproducing the same issue - back to the drawing board. :(
--
--
- Steve Hill
Technical Director
Opendium Limited http://www.opendium.com
Direct contacts:
Instant messager: xmpp:st...@opendium.com
Email:st...@opendium.com
Phone:sip:st...@opendium.com
Sales / enquiries contacts:
Email:sa...@opendium.com
Phone:+44-844-9791439 / sip:sa...@opendium.com
Support contacts:
Email:supp...@opendium.com
Phone:+44-844-4844916 / sip:supp...@opendium.com
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
Amos Jeffries wrote: [dd] > > Bingo! After setting "ident_access deny all" squid does not grow > > infinitely any more. However, it remains a major CPU hog. > > > > Yay. Any news on the bug patch? Will try during the weekend. I can live without IDENT lookups for a while, they are not very important, just convenient. > > Note that from the same "CPU hog" cycles you are now getting around 2x > the HTTP traffic throughput. I have found out that the major CPU hog is the NTLM authenticator. After I disabled the NTLM helper, there is no high CPU utilization. Which brings the next question, please see below :) > > You have the delay pools feature configured. It is a wasteful consumer > of CPU cycles. > 2) moving the delay pools limitation into kernel QoS systems. 1. I am planning to use the delay pool to restrict bandwidth differently to different users. The kernerl QoS system (ipfw pipes in my case) cannot do that for non-local users. 2. Delay pools worked fine in squid27, never a problem. I don't see a reason why they should become a problem in squid3. > Also NTLM authentication is used, that doubles the HTTP > request overheads on each new TCP connection. > 1) converting from NTLM to Kerberos authentication. I have tried to setup Kerberos (negotiate) authentication, but all I see is Internet Explorer asking users for their login/password. I am pretty sure that I have setup the server part correctly. At least when I do the following: kinit -t /usr/local/etc/squid/squid.keytab HTTP/proxy.sibptus.transneft.ru I obtain the TGT issued to HTTP/proxy.sibptus.transneft...@sibptus.transneft.ru My squid.keytab contains: Vno Type Principal 0 arcfour-hmac-md5 HTTP/proxy.sibptus.transneft...@sibptus.transneft.ru To me, this means the Kerberos server part is correct. I don't know for the present how to debug it further. Any Kerberos gurus? Below is a bit of debug from negotiate_kerberos_auth negotiate_kerberos_auth.cc(212): pid=96295 :2014/10/03 15:45:53 kid1| Took 0.41 seconds (80933.38 objects/sec). 2014/10/03 15:45:53| negotiate_kerberos_auth: INFO: Starting version 3.0.4sq 2014/10/03 15:45:53 kid1| Beginning Validation Procedure 2014/10/03 15:45:53 kid1| Completed Validation Procedure 2014/10/03 15:45:53 kid1| Validated 33380 Entries 2014/10/03 15:45:53 kid1| store_swap_size = 878994.00 KB negotiate_kerberos_auth.cc(258): pid=96289 :2014/10/03 15:45:53| negotiate_kerberos_auth: DEBUG: Got 'YR TlRMTVNTUAABl4II4gAGAbEdDw==' from squid (length: 59). negotiate_kerberos_auth.cc(311): pid=96289 :2014/10/03 15:45:53| negotiate_kerberos_auth: DEBUG: Decode 'TlRMTVNTUAABl4II4gAGAbEdDw==' (decoded length: 40). negotiate_kerberos_auth.cc(321): pid=96289 :2014/10/03 15:45:53| negotiate_kerberos_auth: WARNING: received type 1 NTLM token negotiate_kerberos_auth.cc(258): pid=96290 :2014/10/03 15:45:53| negotiate_kerberos_auth: DEBUG: Got 'YR TlRMTVNTUAABl4II4gAGAbEdDw==' from squid (length: 59). negotiate_kerberos_auth.cc(311): pid=96290 :2014/10/03 15:45:53| negotiate_kerberos_auth: DEBUG: Decode 'TlRMTVNTUAABl4II4gAGAbEdDw==' (decoded length: 40). negotiate_kerberos_auth.cc(321): pid=96290 :2014/10/03 15:45:53| negotiate_kerberos_auth: WARNING: received type 1 NTLM token negotiate_kerberos_auth.cc(258): pid=96292 :2014/10/03 15:45:53| negotiate_kerberos_auth: DEBUG: Got 'YR TlRMTVNTUAABl4II4gAGAbEdDw==' from squid (length: 59). negotiate_kerberos_auth.cc(311): pid=96292 :2014/10/03 15:45:53| negotiate_kerberos_auth: DEBUG: Decode 'TlRMTVNTUAABl4II4gAGAbEdDw==' (decoded length: 40). negotiate_kerberos_auth.cc(321): pid=96292 :2014/10/03 15:45:53| negotiate_kerberos_auth: WARNING: received type 1 NTLM token negotiate_kerberos_auth.cc(258): pid=96287 :2014/10/03 15:45:53| negotiate_kerberos_auth: DEBUG: Got 'YR TlRMTVNTUAABl4II4gAGAbEdDw==' from squid (length: 59). negotiate_kerberos_auth.cc(311): pid=96287 :2014/10/03 15:45:53| negotiate_kerberos_auth: DEBUG: Decode 'TlRMTVNTUAABl4II4gAGAbEdDw==' (decoded length: 40). negotiate_kerberos_auth.cc(321): pid=96287 :2014/10/03 15:45:53| negotiate_kerberos_auth: WARNING: received type 1 NTLM token negotiate_kerberos_auth.cc(258): pid=96293 :2014/10/03 15:45:53| negotiate_kerberos_auth: DEBUG: Got 'YR TlRMTVNTUAABl4II4gAGAbEdDw==' from squid (length: 59). negotiate_kerberos_auth.cc(311): pid=96293 :2014/10/03 15:45:53| negotiate_kerberos_auth: DEBUG: Decode 'TlRMTVNTUAABl4II4gAGAbEdDw==' (decoded length: 40). negotiate_kerberos_auth.cc(321): pid=96293 :2014/10/03 15:45:53| negotiate_kerberos_auth: WARNING: received type 1 NTLM token negotiate_kerberos_auth.cc(258): pid=96294 :2014/10/03 15:45:53| negotiate_kerberos_auth: DEBUG: Got 'Y
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
Amos Jeffries wrote: > There are two things you can do to further improve performance: > 1) converting from NTLM to Kerberos authentication. Does Kerberos proxy authentication work with Firefox (Windows) at all? Success stories and recipes, anyone? NTLM did work both with Firefox and MSIE and even Chrome. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
Hello Victor, If you see 'received type 1 NTLM token' message it means your IE was not able to use the Kerberos auth and have chosen NTLM instead. Please ensure you are browsing from *domain joined* machine and using NTLM/Kerberos authentication wrapper as described in http://wiki.squid-cache.org/ConfigExamples/Authenticate/WindowsActiveDirectory#Install_negotiate_wrapper. If you do not want to use NTLM then probably our humble guide will be of any use - http://docs.diladele.com/administrator_guide_3_4/installation_and_removal/active_directory/index.html Best regards, Rafael From: squid-users on behalf of Victor Sudakov Sent: Friday, October 3, 2014 1:51 PM To: Amos Jeffries Cc: squid-users@lists.squid-cache.org Subject: Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7. Amos Jeffries wrote: > There are two things you can do to further improve performance: > 1) converting from NTLM to Kerberos authentication. Does Kerberos proxy authentication work with Firefox (Windows) at all? Success stories and recipes, anyone? NTLM did work both with Firefox and MSIE and even Chrome. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
Rafael Akchurin wrote: > > Does Kerberos proxy authentication work with Firefox (Windows) at all? > > Success stories and recipes, anyone? > > If you see 'received type 1 NTLM token' message it means your IE was > not able to use the Kerberos auth and have chosen NTLM instead. Any ideas how I can figure out the cause thereof? I understand that the browser should be requesting a ticket for the HTTP/proxy.sibptus.transneft...@sibptus.transneft.ru service from the domain controller. How can I find out why it is not requesting it or not receiving it? > Please ensure you are browsing from *domain joined* machine I certainly am. > and > using NTLM/Kerberos authentication wrapper as described in > http://wiki.squid-cache.org/ConfigExamples/Authenticate/WindowsActiveDirectory#Install_negotiate_wrapper. My objective is to enable Kerberos proxy authentication from domain joined Windows machines in MSIE, Firefox and possible Chrome. Why and when do I need this wrapper? Is squid's own negotiate_kerberos_auth plugin not good enough for my purpose? > > If you do not want to use NTLM I most certainly don't want to use NTLM. I want to use Kerberos with MSIE and Firefox. It is possible at all, especially with the latter? > then probably our humble guide will > be of any use - > http://docs.diladele.com/administrator_guide_3_4/installation_and_removal/active_directory/index.html I have basically done all that already, except the Basic auth and LDAP groups stuff. I don't want those for the present. Tell me please, after implementing your guide, do you enjoy Kerberos proxy authentication in Firefox, or does it fall back to Basic auth? -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
I believe I do (but you made me doubt:) Can you please check if your browser is set to use FQDN of proxy not proxy's IP address. Raf From: Victor Sudakov Sent: Friday, October 3, 2014 4:46 PM To: Rafael Akchurin Cc: Amos Jeffries; squid-users@lists.squid-cache.org Subject: Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7. Rafael Akchurin wrote: > > Does Kerberos proxy authentication work with Firefox (Windows) at all? > > Success stories and recipes, anyone? > > If you see 'received type 1 NTLM token' message it means your IE was > not able to use the Kerberos auth and have chosen NTLM instead. Any ideas how I can figure out the cause thereof? I understand that the browser should be requesting a ticket for the HTTP/proxy.sibptus.transneft...@sibptus.transneft.ru service from the domain controller. How can I find out why it is not requesting it or not receiving it? > Please ensure you are browsing from *domain joined* machine I certainly am. > and > using NTLM/Kerberos authentication wrapper as described in > http://wiki.squid-cache.org/ConfigExamples/Authenticate/WindowsActiveDirectory#Install_negotiate_wrapper. My objective is to enable Kerberos proxy authentication from domain joined Windows machines in MSIE, Firefox and possible Chrome. Why and when do I need this wrapper? Is squid's own negotiate_kerberos_auth plugin not good enough for my purpose? > > If you do not want to use NTLM I most certainly don't want to use NTLM. I want to use Kerberos with MSIE and Firefox. It is possible at all, especially with the latter? > then probably our humble guide will > be of any use - > http://docs.diladele.com/administrator_guide_3_4/installation_and_removal/active_directory/index.html I have basically done all that already, except the Basic auth and LDAP groups stuff. I don't want those for the present. Tell me please, after implementing your guide, do you enjoy Kerberos proxy authentication in Firefox, or does it fall back to Basic auth? -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
Rafael Akchurin wrote: > > Tell me please, after implementing your guide, do you enjoy Kerberos > > proxy authentication in Firefox, or does it fall back to Basic auth? > I believe I do (but you made me doubt:) Can you please doublecheck which method your Firefox is using? All the howtos I have seen so far suggest a fallback method of basic authentication. I suppose that is what is actually being used in many cases. > > Can you please check if your browser is set to use FQDN of proxy not > proxy's IP address. I am absolutely sure of that. I am not new to Kerberos in general. I have experience in setting up a Heimdal realm and trust relations between a Heimdal realm and a MS Windows 2000 domain (yes, we still have a Windows 2000 AD). I have configured Kerberos authentication in POP3, IMAP, CVS, SVN, SSH (Unix and Centrify PuTTY). So the chances of me making some trivial mistake like DNS misconfiguration or an unsynchronized clock are rather small. Though of course we all make mistakes sometimes. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
On 01.10.14 13:54, Amos Jeffries wrote: I recently opened a bug about this, that I will update now: http://bugs.squid-cache.org/show_bug.cgi?id=4088 Thank you for the reminder. I will start work on this next. I'm afraid the patch you added to that bug report doesn't work for me (in fact, it makes things worse). I've updated the ticket with details. -- - Steve ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
On 30.09.14 16:13, Amos Jeffries wrote: I'm trying to figure out if there's a way of convincing valgrind to dump info about all the currently allocated memory while the program is still running - there would be a lot of legitimate stuff in the report, but hopefully a few hundred MB of memory that shouldn't be there would stick out like a sore thumb. That would be lovely. If you can find it I'd like to know too :-) su -s /bin/bash squid valgrind --vgdb=full --error-limit=no --tool=memcheck --leak-check=full --show-reachable=yes --leak-resolution=high --num-callers=40 squid -f /etc/squid/squid.conf -N Do whatever you need to do to trigger a leak, then in another window: su -s /bin/bash squid gdb squid At the gdb prompt: target remote | vgdb set logging file /tmp/leaks.txt set logging on monitor leak_check full reachable any This will dump out all the currently allocated memory to the console (and to /tmp/leaks.txt). -- - Steve ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 8/10/2014 4:40 a.m., Steve Hill wrote: > On 30.09.14 16:13, Amos Jeffries wrote: > >>> I'm trying to figure out if there's a way of convincing >>> valgrind to dump info about all the currently allocated memory >>> while the program is still running - there would be a lot of >>> legitimate stuff in the report, but hopefully a few hundred MB >>> of memory that shouldn't be there would stick out like a sore >>> thumb. >> >> That would be lovely. If you can find it I'd like to know too >> :-) > > su -s /bin/bash squid valgrind --vgdb=full --error-limit=no > --tool=memcheck --leak-check=full --show-reachable=yes > --leak-resolution=high --num-callers=40 squid -f > /etc/squid/squid.conf -N > > Do whatever you need to do to trigger a leak, then in another > window: su -s /bin/bash squid gdb squid > > At the gdb prompt: target remote | vgdb set logging file > /tmp/leaks.txt set logging on monitor leak_check full reachable > any > > This will dump out all the currently allocated memory to the > console (and to /tmp/leaks.txt). > Thanks. New patch added to bug 4088. Please see if it resolves the external_acl_type leak. Amos -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (MingW32) iQEcBAEBAgAGBQJUNUSgAAoJELJo5wb/XPRjUTEH/A95Za6+QjzMwn2kSIkmHF2E FzEOPmC0t73lsHmVJptFnblsA2sKWgDokg+NRM1LISdLtx7EZ9aRN+8cQBBdCBnN AXuSz0nWUkI2ugPw6A5YOX70YEwwmkXiG52honyvmUsC0wyj5aBiAeNRZLfy5ZVo V1uSccJTeavrGPKgB5xrFVVEyF5oSMwxeM3Zu+lukgWKWHcIeSyFV2IJkvpG/Q/E U4JdLcWu3PjoZGQzHPIVGn0t+EaN9qVRDyq+p6E/weLvUhQqSi33qGJIoiZz7dbv 9zSEKegDv6+OGw4P8uEVzxebWSTCcr7NbYk/cUzMfg7fCp9zYed3MZL235O8JcM= =SlNr -END PGP SIGNATURE- ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] leaking memory in squid 3.4.8 and 3.4.7.
On 08.10.14 15:05, Amos Jeffries wrote: New patch added to bug 4088. Please see if it resolves the external_acl_type leak. Seems to fix the problem - thank you! -- - Steve Hill Technical Director Opendium Limited http://www.opendium.com Direct contacts: Instant messager: xmpp:st...@opendium.com Email:st...@opendium.com Phone:sip:st...@opendium.com Sales / enquiries contacts: Email:sa...@opendium.com Phone:+44-1792-824568 / sip:sa...@opendium.com Support contacts: Email:supp...@opendium.com Phone:+44-1792-825748 / sip:supp...@opendium.com ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
