Re: [squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64
Hi. On 12.01.2015 19:06, Amos Jeffries wrote: I am confident that those types of leaks do not exist at al in Squid 3.4. These rounds of mmory exhaustion problems are caused by pseudo-leaks, where Squid incorrectly holds onto memory (has not forgotten it though) far longer than it should be. Could you please clarify for me what is the Long Strings pool and how can I manage it's size ? After start the largest consuming pool is the mem_node one, but it usually stops increasing after a few days (somewhere around the cache_memory border - don't know if it's it, or just a coincedence). Long Strings, however, keep raising and raising, and after some days it becomes the largest one. I'm using the following settings: cache_mem 512 MB cache_dir diskd /var/squid/cache 1100 16 256 after few days SNMP reports that the clients amount is around 1700. Thanks. Eugene. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64
Hi. On 09.01.2015 06:12, Amos Jeffries wrote: Grand total: = 9.5 GB of RAM just for Squid. .. then there is whatever memory the helper programs, other software on the server and operating system all need. I'm now also having a strong impression that squid is leaking memory. Now, when 3.4.x is able to handle hundreds of users during several hours I notice that it's memory usage is constantly increasing. My patience always ends at the point of 1.5 Gigs memory usage, where server memory starts to be exhausted (squid is running with lots of other stuff) and I restart it. This is happening on exactly the same config the 3.3.13 was running, so ... I have cache_mem set to 512 megs, diskd, medium sized cache_dir and lots of users. Is something changed drastically in 3.4.x comparing to the 3.3.13, or is it, as it seems, a memory leak ? Thanks. Eugene. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yep. Memory leaking - if it really it - will be occurs on all platforms. If not - this is OS-specific issue. libc, malloc library problem. But not squid itself. 12.01.2015 18:06, Eugene M. Zheganin пишет: Hi. On 12.01.2015 16:41, Eugene M. Zheganin wrote: I'm now also having a strong impression that squid is leaking memory. Now, when 3.4.x is able to handle hundreds of users during several hours I notice that it's memory usage is constantly increasing. My patience always ends at the point of 1.5 Gigs memory usage, where server memory starts to be exhausted (squid is running with lots of other stuff) and I restart it. This is happening on exactly the same config the 3.3.13 was running, so ... I have cache_mem set to 512 megs, diskd, medium sized cache_dir and lots of users. Is something changed drastically in 3.4.x comparing to the 3.3.13, or is it, as it seems, a memory leak ? Squid 3.4 on FreeBSD is by default compiling with the --enable-debug-cbdata option and when 45th log selector is at it's default 1, cache.log is filling with CBData memory leaking alarms. Here is the list for the last 40 minutes, sorted with the occurrence count: 104136 Checklist.cc:160 81438 Checklist.cc:187 177226 Checklist.cc:320 84861 Checklist.cc:45 89151 CommCalls.cc:21 22069 DiskIO/DiskDaemon/DiskdIOStrategy.cc:353 120 UserRequest.cc:166 29 UserRequest.cc:172 55814 clientStream.cc:235 5966 client_side_reply.cc:93 4516 client_side_request.cc:134 5568 dns_internal.cc:1131 4859 dns_internal.cc:1140 86 event.cc:90 7770 external_acl.cc:1426 1548 fqdncache.cc:340 7467 helper.cc:856 39905 ipcache.cc:353 11880 store.cc:1611 181959 store_client.cc:154 256951 store_client.cc:337 6835 ufs/UFSStoreState.cc:333 are those all false alarms ? Thanks. Eugene. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBAgAGBQJUs7rJAAoJENNXIZxhPexGmk0IAJmEmfJ1aBLL9DlrrnHM95JU 8VeGgsQi/wVpAjS2z1JA5eDJZ6WY5tBycVkJsKK0SBaHXrFjTHQkEUuy4iFQLpkV q8xZ4Y/X0rKJ6ayy6XSHaEd4wznlthteCTI5ImTW9fiL7PXfW7mci+o2g6lUPNa7 edtep7gp04ICmkLdq1F6P5InxksoLpc1iavV281SRowPDv151TFlZ5cn0A3fmqIv J/Pi19ss3vabiU3VXEvhiA5duxtx2lIs+BMZpU2Q3L9nQhvUf2pa8xMRBRF95RCd 8Pagb7Exzc/0/2JIjt8oCeV60Rr7xOUcwoxOXRC/4EBxzTWBH7FOkrnmBpVGNnM= =NJuF -END PGP SIGNATURE- ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yep. Memory leaking - if it really it - will be occurs on all platforms. If not - this is OS-specific issue. libc, malloc library problem. But not squid itself. 12.01.2015 18:06, Eugene M. Zheganin пишет: -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBAgAGBQJUs7rFAAoJENNXIZxhPexG7EYIAKfrJU2JU6gulc11PMnuNrq1 P0lMm5WYTh2joRj6j3mCsiMR8FyolOQCxT298wY3/zXlY4bvluuwwqJ9hW4JiuMn RGXK5iJvGg8zr0yABiDoNXFLgUMVdW8NqibSfecRds7ZZkjEhnn8tUO+2jU03ZBy dZzg7TavNOeRextBJCaknr4IKwvoQWQsiparTF91wJSg9YfQ7oHsWellTlbI7uPC r/2opE2nOtKF+PEbspmzXgzt76RBe1xNM4dWikbeZOPzXvg0n7Iwbhd6TSTfWLS0 Wb4HAPB7qVJ52Nx3lYjFYHrIMqKClrj+ETQVJp7CKOZCxP25jMyF+F1Oa9d9ZxE= =mVwt -END PGP SIGNATURE- ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Looks like an OS-specific issue. I don't see any memory leaking on my boxes (running Solaris 10, yes ;)). Moreover, helpers is corrrectly got an release memory. 12.01.2015 17:41, Eugene M. Zheganin пишет: Hi. On 09.01.2015 06:12, Amos Jeffries wrote: Grand total: = 9.5 GB of RAM just for Squid. .. then there is whatever memory the helper programs, other software on the server and operating system all need. I'm now also having a strong impression that squid is leaking memory. Now, when 3.4.x is able to handle hundreds of users during several hours I notice that it's memory usage is constantly increasing. My patience always ends at the point of 1.5 Gigs memory usage, where server memory starts to be exhausted (squid is running with lots of other stuff) and I restart it. This is happening on exactly the same config the 3.3.13 was running, so ... I have cache_mem set to 512 megs, diskd, medium sized cache_dir and lots of users. Is something changed drastically in 3.4.x comparing to the 3.3.13, or is it, as it seems, a memory leak ? Thanks. Eugene. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBAgAGBQJUs7R6AAoJENNXIZxhPexGrKwH/1T9k9zGwgEqQeg/6+u1z1bV kShvT7TOVNGXHMXWEka2NWjn/o973nHRAUbwBd6MUMsRSd0o3hOBYnVByYAI/6UM X/CmZpADcTMS/WSAFIrSlqj/Ml1HOafOQcmMrxw6h5jJ9qoO/O8oPHGiBpiAIjGh eMtwX0qiyfx+Xy8ncYUial/JtQPm3jsxBuCofBHatqeAA9vPyng+a+e/C4MKILX/ D4EbzCGd8CBzH8vKGkPIwUKbXJ3j79yf7ve+u+YREX/DuJ68uroHJEOo8lNnHr+/ cVoJ70C3Nju0ZYE2Dme8kYJ7764k+K4sdlD10mBXroMxqgBoUZTqIl+Tx/G8o5c= =E/KG -END PGP SIGNATURE- ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64
Hi. On 12.01.2015 16:41, Eugene M. Zheganin wrote: I'm now also having a strong impression that squid is leaking memory. Now, when 3.4.x is able to handle hundreds of users during several hours I notice that it's memory usage is constantly increasing. My patience always ends at the point of 1.5 Gigs memory usage, where server memory starts to be exhausted (squid is running with lots of other stuff) and I restart it. This is happening on exactly the same config the 3.3.13 was running, so ... I have cache_mem set to 512 megs, diskd, medium sized cache_dir and lots of users. Is something changed drastically in 3.4.x comparing to the 3.3.13, or is it, as it seems, a memory leak ? Squid 3.4 on FreeBSD is by default compiling with the --enable-debug-cbdata option and when 45th log selector is at it's default 1, cache.log is filling with CBData memory leaking alarms. Here is the list for the last 40 minutes, sorted with the occurrence count: 104136 Checklist.cc:160 81438 Checklist.cc:187 177226 Checklist.cc:320 84861 Checklist.cc:45 89151 CommCalls.cc:21 22069 DiskIO/DiskDaemon/DiskdIOStrategy.cc:353 120 UserRequest.cc:166 29 UserRequest.cc:172 55814 clientStream.cc:235 5966 client_side_reply.cc:93 4516 client_side_request.cc:134 5568 dns_internal.cc:1131 4859 dns_internal.cc:1140 86 event.cc:90 7770 external_acl.cc:1426 1548 fqdncache.cc:340 7467 helper.cc:856 39905 ipcache.cc:353 11880 store.cc:1611 181959 store_client.cc:154 256951 store_client.cc:337 6835 ufs/UFSStoreState.cc:333 are those all false alarms ? Thanks. Eugene. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 9/01/2015 8:10 a.m., Doug Sampson wrote: Hi, I have the similar problem on FreeBSD 10.1-STABLE #1 r275861 with squid-3.4.10. I also applied MEMPOOLS=1 when starting squid. I experience the process slowing down and unacceptable performance. Squid is configured to use kerberos and ntlm authentication and lap group authentication. other settings: ~14 MB for Squid process memory. cache_replacement_policy heap LFUDA cache_mem 4096 MB + 4096 MB for RAM cache + 60 MB for RAM cache index maximum_object_size 32 MB cache_dir diskd /usr/local/squid/cache 32768 32 256 + 540 MB for disk cache index + (600 * 16 * 0.5) MB for each active client connection state. == 4800 MB NP: modern web browsers open up to 8 parallel connections to load a page (happy eyeballs makes that 16 TCP sockets) per client == ~8 MB per active client. Grand total: = 9.5 GB of RAM just for Squid. .. then there is whatever memory the helper programs, other software on the server and operating system all need. I have seen the following errors in cache.log: FATAL: Received Segment Violation...dying. FATAL: Received Bus Error...dying. need a backtrace to tell what the Segment Violation is about. Bus Error is your CPU or RAM futzing up. The OS or hardware is broken. Quite possibly this is the result of runnign out of RAM and swap space. after this the squid restarts. The system has 10GB of memory and is working as internal cache for ~600 users. Please point me in the right direction. I have no problem running squid33-3.3.13 on FreeBSD 9.3-STABLE #0 r270210. Thank you very much. Regards, lk Man, I empathize with you. Have you tried running Squid 3.4.x on FreeBSD 9.3? Sometimes I wonder if it's FreeBSD 10.x that's causing the issue... I tried the shell variable MEMPOOLS=1 and that quickly made the situation a lot worse. Swap space would get filled up very quickly and the system would slow down quickly before crashing. All MEMPOOLS=1 does is make Squid internally recycle the memory that gets free'd. Re-using it for other objects of the same type that are allocated right after its free'd. So new memory only gets allocated once the recycling pool for an object type is empty, everything allocated so far actively in use. The main point of using MEMPOOLS=1 is a debugging aid to track down leaks, by a) reduced OS allocator calls making frequent ones obvious, and b) the mgr:mem report more clearly listing what objects are being free'd (chunked pool contains recycled objects) and which are not (pool always empty). Steve Hill has come up with some patches that resolve memory issues with the new 3.4 helper annotations feature when using Negotiate or NTMLM auth helpers. For authenticated clients with long connection times or high traffic volumes the state can accumulate up to quite large amounts of memory. see the Debugging slow proxy thread of earlier this week. HTH Amos -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (MingW32) iQEcBAEBAgAGBQJUrysEAAoJELJo5wb/XPRjFFMH/RNhVzPWfFixDDarpHUFQMoE 6QLkaD5wwQfRp3xfBW/QfZ9vvu04zptFOvNr4766jGsh9RSGZdNZwgPc9pCADDrv Xs0HCO1VDMxGoBjFfaS2XJ5DLX2zQdbYlZKb0yghxXSYzg0ZZEhmsKO8r1Exp8j7 KO95yP3z/8agmwXbU2sJ1esqRC7IfW2sF/DtU8cPTzUf0cKEyjGoCbVrNAN1qKUH jLf1iw8sNr3xGwFG/WmQmKpYTsXInSp4GmwXgCSQ0T5DCLVDtWuvnXpobxR8iuSW zIX/CDKTe58lfVP4EXdzJNBgiuEiVZaRccynbk2L8HRvJ8kt9Pk50P5tWXM5FMw= =lv6G -END PGP SIGNATURE- ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64
Doug Sampson do...@dawnsign.com writes: Hi, I have the similar problem on FreeBSD 10.1-STABLE #1 r275861 with squid-3.4.10. I also applied MEMPOOLS=1 when starting squid. I experience the process slowing down and unacceptable performance. Squid is configured to use kerberos and ntlm authentication and lap group authentication. other settings: cache_replacement_policy heap LFUDA cache_mem 4096 MB maximum_object_size 32 MB cache_dir diskd /usr/local/squid/cache 32768 32 256 I have seen the following errors in cache.log: FATAL: Received Segment Violation...dying. FATAL: Received Bus Error...dying. after this the squid restarts. The system has 10GB of memory and is working as internal cache for ~600 users. Please point me in the right direction. I have no problem running squid33-3.3.13 on FreeBSD 9.3-STABLE #0 r270210. Thank you very much. Regards, lk Man, I empathize with you. Have you tried running Squid 3.4.x on FreeBSD 9.3? Sometimes I wonder if it's FreeBSD 10.x that's causing the issue... I am running FreeBSD 9.3-STABLE #1 r274084 with Squid 3.4.9 as a cache in DMZ without authentication. There are no problems. lk I tried the shell variable MEMPOOLS=1 and that quickly made the situation a lot worse. Swap space would get filled up very quickly and the system would slow down quickly before crashing. Any other ideas? ~Doug ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64
Hi, I have the similar problem on FreeBSD 10.1-STABLE #1 r275861 with squid-3.4.10. I also applied MEMPOOLS=1 when starting squid. I experience the process slowing down and unacceptable performance. Squid is configured to use kerberos and ntlm authentication and lap group authentication. other settings: cache_replacement_policy heap LFUDA cache_mem 4096 MB maximum_object_size 32 MB cache_dir diskd /usr/local/squid/cache 32768 32 256 I have seen the following errors in cache.log: FATAL: Received Segment Violation...dying. FATAL: Received Bus Error...dying. after this the squid restarts. The system has 10GB of memory and is working as internal cache for ~600 users. Please point me in the right direction. I have no problem running squid33-3.3.13 on FreeBSD 9.3-STABLE #0 r270210. Thank you very much. Regards, lk Man, I empathize with you. Have you tried running Squid 3.4.x on FreeBSD 9.3? Sometimes I wonder if it's FreeBSD 10.x that's causing the issue... I tried the shell variable MEMPOOLS=1 and that quickly made the situation a lot worse. Swap space would get filled up very quickly and the system would slow down quickly before crashing. Any other ideas? ~Doug ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I mean, cache_mem is too big. First, 600 users is not very big installation to do huge memory cache. 1-2 Gb will be ehough. With fast disk cache. Second - 4 Gb ram is near 32-bit address limit. Possible 32-bit library or somewhat meets this limit. Is OS and Squid completely 64-bit? 09.01.2015 1:10, Doug Sampson пишет: Hi, I have the similar problem on FreeBSD 10.1-STABLE #1 r275861 with squid-3.4.10. I also applied MEMPOOLS=1 when starting squid. I experience the process slowing down and unacceptable performance. Squid is configured to use kerberos and ntlm authentication and lap group authentication. other settings: cache_replacement_policy heap LFUDA cache_mem 4096 MB maximum_object_size 32 MB cache_dir diskd /usr/local/squid/cache 32768 32 256 I have seen the following errors in cache.log: FATAL: Received Segment Violation...dying. FATAL: Received Bus Error...dying. after this the squid restarts. The system has 10GB of memory and is working as internal cache for ~600 users. Please point me in the right direction. I have no problem running squid33-3.3.13 on FreeBSD 9.3-STABLE #0 r270210. Thank you very much. Regards, lk Man, I empathize with you. Have you tried running Squid 3.4.x on FreeBSD 9.3? Sometimes I wonder if it's FreeBSD 10.x that's causing the issue... I tried the shell variable MEMPOOLS=1 and that quickly made the situation a lot worse. Swap space would get filled up very quickly and the system would slow down quickly before crashing. Any other ideas? ~Doug ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBAgAGBQJUrvaqAAoJENNXIZxhPexGTigIAKCNyOeAD03un/vxFINUN2Yb 0cZLNzSjIDkzhepXkfQo2yazcTSoGoJ5QFSwMj9yljFCve2JjTj1O6KT4JxbOzvG vMpEqY6KjuCsENU8vm6lBq+D7PmtZw20V+nofA1V3y1L9CGCuygVTe33FuWNIVwA VQKmhV+0IaG8vIdHgKtYKUK9n4zV48DAPurpjTf4UotzD0+BYef/BOatrQ1HGp9/ z6cmqB9FGIxOk5MZIgeer/3bjkYng7Q60zm7pHnMFhb265+0bsMzkvGl/PMvzKib ASRdTqy+rZiE/Y48KfcjTjY4Eyvk69U3I5iuu7xU+CvawMsBhifRn79A7e0eHTQ= =gLo3 -END PGP SIGNATURE- ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64
On 26/11/2014 8:59 a.m., Doug Sampson wrote: Thanks, Amos, for your pointers. I've commented out all the fresh_patterns lines appearing above the last two lines. I also have dropped diskd in favor of using aufs exclusively, taking out the min-size parameter. I've commented out the diskd_program support option. In the previous version of squid (2.7) I had split the cache_dir into two types with great success using coss and aufs. Previously I had only aufs and performance wasn't where I wanted it. Apparently coss is no longer supported in the 3.x version of squid atop FreeBSD. COSS has been replaced with Rock storage type in Squid-3. They should be used in roughly similar ways in terms of traffic optimization. The pathname for the cache swap logs have been fixed. Apparently this came from a squid.conf example that I copied in parts. Would this be the reason why we are seeing the error messages in /var/log/messages regarding swapping mentioned in my original post? No. I think that is coming out of the OS kernel memory management. It uses the term swap as well in regards to disk backed virtual memory. If your system is swapping (using that disk backed swap memory) while running Squid then you will get terrible performance as a matter of course since the Squid cache index and cache_meme is often very large in RAM and accessed often. The hierarchy_stoplist line has been stripped out as you say it is deprecated. The mem .TSV file is attached herewith. Currently I have the cache_dir located on the OS disk and all of the cache logging files on a second drive. Is this the optimal setup of cache-dir and logs? I would do it the other way around. Logs are appended with a small amount of data each transaction, whereas the main cache_dir has a relativey large % of the bandwidth throughput being written out to it constantly (less % in recent Squid, but still a lot). The dik most likely to die early is the one holding cache_dir. I'm still running into the issue of being out of available space in the swapfile on my system. I've attached another TSV file indicating the various type of memory being in use and whatnot. Is there anything in there that screams out? Amos, you said earlier that it was the OS system that needed to be tuned. Are there any references to where I can fine-tune it for Squid usage? I looked here http://oss.org.cn/man/newsoft/squid/Squid_FAQ/FAQ-8.html#how-much-ram and I'm unable to figure out a way to decrease the amount of memory that I could use. I tried limiting cache_mem to 1344 MB from some higher value but that didn't work. What are some of the methods that FreeBSD 10.0 users using to limit the use of memory that Squid uses? Sort of fixing the memory leaks, it looks like I need to consider the possibility of restarting the Squid service on a regular basis (i.e. at least once a week) in order to enable Squid to perform at an acceptable level and to avoid clogging /var/log/messages with such messages as follows: ... +swap_pager_getswapspace(15): failed +swap_pager_getswapspace(2): failed +swap_pager_getswapspace(16): failed +swap_pager_getswapspace(16): failed ... Is this a common practice among squid admins to restart squid periodically? Thank you! ~Doug Current memory usage: Pool Obj SizeChunksAllocatedIn UseIdleAllocations SavedRate (bytes)KB/ch obj/ch(#) used free part %Frag (#) (KB) high (KB) high (hrs) %Tot(#) (KB) high (KB) high (hrs) %alloc(#) (KB) high (KB)(#) %cnt %vol(#)/sec mem_node 4136292390 1180982 1336651 4.76 86.464 292191 1180178 1336651 4.76 99.932 199 804 16173 1641718 1.322 21.088 0.001 Short Strings 401701206 66454 73289 4.76 4.865 1699725 66396 73289 4.76 99.913 1481 58 250 37674459 30.347 4.680 0.029 HttpHeaderEntry56796862 43579 47925 4.76 3.191 796197 43543 47925 4.76 99.917 665 37 163 10727306 8.641 1.866 0.008 HttpReply 28074302 20317 22720 4.76 1.487 74275 20310 22720 4.76 99.964 27 8 135 754702 0.608 0.656 0.001 MemObject 24074278 17409 19473 4.76 1.275 74273 17408 19473 4.76 99.993 5 2 121 187591 0.151 0.140 0.000 StoreEntry10479949 8120 9020 4.76 0.594 79944 8120 9020 4.76 99.994 5 1 53 198382 0.160 0.064 0.000 HttpHdrCc 9650739 4757 5279 4.76 0.348 50725 4756 5279 4.76 99.972 14 2 41 602296 0.485 0.180 0.000 Medium Strings12837543 4693 4889 3.29 0.344 37426 4679 4889 3.29 99.688 117 15 48 2579153 2.078 1.025 0.002 cbdata MemBuf (9) 6474339 4647 5197 4.76 0.340 74310 4645 5197 4.76 99.961 29 2 31 1332078 1.073 0.265 0.001 Long Strings 5124170 2085 2085 0.04 0.153 4105 2053 2085 0.04 98.441 65 33 130 389656 0.314 0.620 0.000 LRU policy node2474220 1740 1946 4.76 0.127 74217 1740 1946 4.76 99.996 3 1 12 29622 0.024 0.002 0.000 4KB Strings 4096394 1576 1576
Re: [squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64
Maybe your problem is related to sysctl mib tuning about swap/overcommit etc. I did not observed memory leak with squid 3.4.4, but FB 10 do swap frequently than old version. Simon ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64
I used the ugly tuning: set vm.defer_swapspace_pageouts to 1. But it is may caused some issue when the physic memory is really exhausted. I have no much time to investigate the right way, but I think maybe vm.swap_idle_threshold1/vm.swap_idle_threshold2 or vm.overcommit etc. maybe harmful. Simon 在 14/12/1 23:16, Doug Sampson 写道: Maybe your problem is related to sysctl mib tuning about swap/overcommit etc. I did not observed memory leak with squid 3.4.4, but FB 10 do swap frequently than old version. Could you elaborate a bit more? That went over my head. What could I do in terms of tuning the system? ~Doug ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64
On 25/11/2014 9:06 a.m., Doug Sampson wrote: Recently due to squid 2.7 being EOL'ed, we migrated our squid server to version 3.4.9 on a FreeBSD 10.0-RELEASE running on 64-bit hardware. We started seeing paging file being swapped out eventually running out of available memory. From the time squid gets started it usually takes about two days before we see these entries in /var/log/messages as follows: +swap_pager_getswapspace(16): failed +swap_pager_getswapspace(16): failed +swap_pager_getswapspace(16): failed +swap_pager_getswapspace(12): failed +swap_pager_getswapspace(16): failed +swap_pager_getswapspace(12): failed +swap_pager_getswapspace(6): failed +swap_pager_getswapspace(16): failed Looking at the 'top' results, I see that the swap file has been totally exhausted. Memory used by squid hovers around 2.3GB out of the total 3GB of system memory. I am not sure what is causing these memory leaks. After rebooting, squid-internal-mgr/info shows the following statistics: Squid Object Cache: Version 3.4.9 Build Info: Start Time: Mon, 24 Nov 2014 18:39:08 GMT Current Time: Mon, 24 Nov 2014 19:39:13 GMT Connection information for squid: Number of clients accessing cache: 18 Number of HTTP requests received:10589 Number of ICP messages received: 0 Number of ICP messages sent: 0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start:176.2 Average ICP messages per minute since start: 0.0 Select loop called: 763993 times, 4.719 ms avg Cache information for squid: Hits as % of all requests: 5min: 3.2%, 60min: 17.0% Hits as % of bytes sent: 5min: 2.0%, 60min: 6.7% Memory hits as % of hit requests: 5min: 0.0%, 60min: 37.2% Disk hits as % of hit requests: 5min: 22.2%, 60min: 33.2% Storage Swap size: 7361088 KB Storage Swap capacity: 58.5% used, 41.5% free Storage Mem size: 54348 KB Storage Mem capacity: 3.9% used, 96.1% free Mean Object Size:23.63 KB Requests given to unlinkd: 1 Median Service Times (seconds) 5 min60 min: HTTP Requests (All): 0.10857 0.19742 Cache Misses: 0.10857 0.32154 Cache Hits:0.08265 0.01387 Near Hits: 0.15048 0.12106 Not-Modified Replies: 0.00091 0.00091 DNS Lookups: 0.05078 0.05078 ICP Queries: 0.0 0.0 Resource usage for squid: UP Time: 3605.384 seconds CPU Time: 42.671 seconds CPU Usage: 1.18% CPU Usage, 5 minute avg: 0.72% CPU Usage, 60 minute avg: 1.17% Maximum Resident Size: 845040 KB Page faults with physical i/o: 20 Memory accounted for: Total accounted: 105900 KB memPoolAlloc calls: 2673353 memPoolFree calls:2676487 File descriptor usage for squid: Maximum number of file descriptors: 87516 Largest file desc currently in use:310 Number of file desc currently in use: 198 Files queued for open: 0 Available number of file descriptors: 87318 Reserved number of file descriptors: 100 Store Disk files open: 0 Internal Data Structures: 311543 StoreEntries 4421 StoreEntries with MemObjects 4416 Hot Object Cache Items 311453 on-disk objects I will post another one tomorrow that will indicate growing memory/swapfile consumption. Here is my squid.conf: # OPTIONS FOR AUTHENTICATION # - # 1st four lines for auth_param basic children 5 auth_param basic realm Squid proxy-caching web server auth_param basic credentialsttl 2 hours auth_param basic casesensitive off # next three lines for kerberos authentication (needed to use usernames) # used in conjunction with acl auth proxy_auth line below #auth_param negotiate program /usr/local/libexec/squid/negotiate_kerberos_auth -i #auth_param negotiate children 50 startup=10 idle=5 #auth_param negotiate keep_alive on # ACCESS CONTROLS # - # Example rule allowing access from your local networks. # Adapt to list your (internal) IP networks from where browsing # should be allowed #acl manager proto cache_object acl manager url_regex -i ^cache_object:// /squid-internal-mgr/ acl adminhost src 192.168.1.149 acl localnet src 192.168.1.0/24 # RFC1918 possible internal network acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines acl webserver src 198.168.1.35 acl some_big_clients src 192.168.1.149/32 #CI53 # We want to limit downloads of these type of files # Put this all in one line acl magic_words url_regex -i ftp .exe .mp3 .vqf .tar.gz .gz .rpm .zip .rar .avi .mpeg .mpe .mpg .qt .ram .rm .iso .raw .wav .dmg .mp4
[squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64
Recently due to squid 2.7 being EOL'ed, we migrated our squid server to version 3.4.9 on a FreeBSD 10.0-RELEASE running on 64-bit hardware. We started seeing paging file being swapped out eventually running out of available memory. From the time squid gets started it usually takes about two days before we see these entries in /var/log/messages as follows: +swap_pager_getswapspace(16): failed +swap_pager_getswapspace(16): failed +swap_pager_getswapspace(16): failed +swap_pager_getswapspace(12): failed +swap_pager_getswapspace(16): failed +swap_pager_getswapspace(12): failed +swap_pager_getswapspace(6): failed +swap_pager_getswapspace(16): failed Looking at the 'top' results, I see that the swap file has been totally exhausted. Memory used by squid hovers around 2.3GB out of the total 3GB of system memory. I am not sure what is causing these memory leaks. After rebooting, squid-internal-mgr/info shows the following statistics: Squid Object Cache: Version 3.4.9 Build Info: Start Time: Mon, 24 Nov 2014 18:39:08 GMT Current Time: Mon, 24 Nov 2014 19:39:13 GMT Connection information for squid: Number of clients accessing cache: 18 Number of HTTP requests received: 10589 Number of ICP messages received:0 Number of ICP messages sent:0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 176.2 Average ICP messages per minute since start:0.0 Select loop called: 763993 times, 4.719 ms avg Cache information for squid: Hits as % of all requests: 5min: 3.2%, 60min: 17.0% Hits as % of bytes sent:5min: 2.0%, 60min: 6.7% Memory hits as % of hit requests: 5min: 0.0%, 60min: 37.2% Disk hits as % of hit requests: 5min: 22.2%, 60min: 33.2% Storage Swap size: 7361088 KB Storage Swap capacity: 58.5% used, 41.5% free Storage Mem size: 54348 KB Storage Mem capacity:3.9% used, 96.1% free Mean Object Size: 23.63 KB Requests given to unlinkd: 1 Median Service Times (seconds) 5 min60 min: HTTP Requests (All): 0.10857 0.19742 Cache Misses: 0.10857 0.32154 Cache Hits:0.08265 0.01387 Near Hits: 0.15048 0.12106 Not-Modified Replies: 0.00091 0.00091 DNS Lookups: 0.05078 0.05078 ICP Queries: 0.0 0.0 Resource usage for squid: UP Time:3605.384 seconds CPU Time: 42.671 seconds CPU Usage: 1.18% CPU Usage, 5 minute avg:0.72% CPU Usage, 60 minute avg: 1.17% Maximum Resident Size: 845040 KB Page faults with physical i/o: 20 Memory accounted for: Total accounted: 105900 KB memPoolAlloc calls: 2673353 memPoolFree calls:2676487 File descriptor usage for squid: Maximum number of file descriptors: 87516 Largest file desc currently in use:310 Number of file desc currently in use: 198 Files queued for open: 0 Available number of file descriptors: 87318 Reserved number of file descriptors: 100 Store Disk files open: 0 Internal Data Structures: 311543 StoreEntries 4421 StoreEntries with MemObjects 4416 Hot Object Cache Items 311453 on-disk objects I will post another one tomorrow that will indicate growing memory/swapfile consumption. Here is my squid.conf: # OPTIONS FOR AUTHENTICATION # - # 1st four lines for auth_param basic children 5 auth_param basic realm Squid proxy-caching web server auth_param basic credentialsttl 2 hours auth_param basic casesensitive off # next three lines for kerberos authentication (needed to use usernames) # used in conjunction with acl auth proxy_auth line below #auth_param negotiate program /usr/local/libexec/squid/negotiate_kerberos_auth -i #auth_param negotiate children 50 startup=10 idle=5 #auth_param negotiate keep_alive on # ACCESS CONTROLS # - # Example rule allowing access from your local networks. # Adapt to list your (internal) IP networks from where browsing # should be allowed #acl manager proto cache_object acl manager url_regex -i ^cache_object:// /squid-internal-mgr/ acl adminhost src 192.168.1.149 acl localnet src 192.168.1.0/24 # RFC1918 possible internal network acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines acl webserver src 198.168.1.35 acl some_big_clients
Re: [squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 25/11/2014 9:06 a.m., Doug Sampson wrote: Recently due to squid 2.7 being EOL'ed, we migrated our squid server to version 3.4.9 on a FreeBSD 10.0-RELEASE running on 64-bit hardware. We started seeing paging file being swapped out eventually running out of available memory. From the time squid gets started it usually takes about two days before we see these entries in /var/log/messages as follows: +swap_pager_getswapspace(16): failed +swap_pager_getswapspace(16): failed +swap_pager_getswapspace(16): failed +swap_pager_getswapspace(12): failed +swap_pager_getswapspace(16): failed +swap_pager_getswapspace(12): failed +swap_pager_getswapspace(6): failed +swap_pager_getswapspace(16): failed Looking at the 'top' results, I see that the swap file has been totally exhausted. Memory used by squid hovers around 2.3GB out of the total 3GB of system memory. I am not sure what is causing these memory leaks. After rebooting, squid-internal-mgr/info shows the following statistics: Squid Object Cache: Version 3.4.9 Build Info: Start Time: Mon, 24 Nov 2014 18:39:08 GMT Current Time: Mon, 24 Nov 2014 19:39:13 GMT Connection information for squid: Number of clients accessing cache:18 Number of HTTP requests received:10589 Number of ICP messages received:0 Number of ICP messages sent: 0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 176.2 Average ICP messages per minute since start: 0.0 Select loop called: 763993 times, 4.719 ms avg Cache information for squid: Hits as % of all requests: 5min: 3.2%, 60min: 17.0% Hits as % of bytes sent: 5min: 2.0%, 60min: 6.7% Memory hits as % of hit requests:5min: 0.0%, 60min: 37.2% Disk hits as % of hit requests: 5min: 22.2%, 60min: 33.2% Storage Swap size:7361088 KB Storage Swap capacity: 58.5% used, 41.5% free Storage Mem size:54348 KB Storage Mem capacity: 3.9% used, 96.1% free Mean Object Size: 23.63 KB Requests given to unlinkd: 1 Median Service Times (seconds) 5 min60 min: HTTP Requests (All): 0.10857 0.19742 Cache Misses: 0.10857 0.32154 Cache Hits:0.08265 0.01387 Near Hits: 0.15048 0.12106 Not-Modified Replies: 0.00091 0.00091 DNS Lookups: 0.05078 0.05078 ICP Queries: 0.0 0.0 Resource usage for squid: UP Time:3605.384 seconds CPU Time: 42.671 seconds CPU Usage: 1.18% CPU Usage, 5 minute avg: 0.72% CPU Usage, 60 minute avg: 1.17% Maximum Resident Size: 845040 KB Page faults with physical i/o: 20 Memory accounted for: Total accounted: 105900 KB memPoolAlloc calls: 2673353 memPoolFree calls:2676487 File descriptor usage for squid: Maximum number of file descriptors: 87516 Largest file desc currently in use:310 Number of file desc currently in use: 198 Files queued for open: 0 Available number of file descriptors: 87318 Reserved number of file descriptors: 100 Store Disk files open: 0 Internal Data Structures: 311543 StoreEntries 4421 StoreEntries with MemObjects 4416 Hot Object Cache Items 311453 on-disk objects I will post another one tomorrow that will indicate growing memory/swapfile consumption. Here is my squid.conf: # OPTIONS FOR AUTHENTICATION # - # 1st four lines for auth_param basic children 5 auth_param basic realm Squid proxy-caching web server auth_param basic credentialsttl 2 hours auth_param basic casesensitive off # next three lines for kerberos authentication (needed to use usernames) # used in conjunction with acl auth proxy_auth line below #auth_param negotiate program /usr/local/libexec/squid/negotiate_kerberos_auth -i #auth_param negotiate children 50 startup=10 idle=5 #auth_param negotiate keep_alive on # ACCESS CONTROLS # - # Example rule allowing access from your local networks. # Adapt to list your (internal) IP networks from where browsing # should be allowed #acl manager proto cache_object acl manager url_regex -i ^cache_object:// /squid-internal-mgr/ acl adminhost src 192.168.1.149 acl localnet src 192.168.1.0/24 # RFC1918 possible internal network acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines acl webserver src 198.168.1.35 acl some_big_clients src 192.168.1.149/32 #CI53 # We want to limit downloads of these type of files # Put this all in one line acl magic_words url_regex -i ftp .exe .mp3 .vqf .tar.gz .gz .rpm .zip .rar .avi .mpeg .mpe .mpg .qt .ram .rm .iso .raw .wav .dmg .mp4 .img # We
