Re: [squid-users] Squid memory usage
On 31/05/2013 3:00 p.m., Nathan Hoad wrote: Fantastic explanation! Thanks heaps Amos. Would it make sense for this to go onto wiki.squid-cache.org somewhere? Yes. Francesco and I are trying to document all of the manager reports in there. http://wiki.squid-cache.org/Features/CacheManager#Available_Reports But, it is slow going with just the two of us documenting in spare time. (hint, hint, anyone?) Amos -- Nathan Hoad Software Developer www.getoffmalawn.com On Wed, May 29, 2013 at 7:00 PM, Amos Jeffries wrote: On 29/05/2013 2:19 p.m., Nathan Hoad wrote: On Tue, May 28, 2013 at 3:23 PM, Amos Jeffries wrote: On 28/05/2013 3:59 p.m., Nathan Hoad wrote: I take it you are referring to the 2.0g resident size? That is what I'm referring to, yes - the resident size has increased to 2.5g since my previous mail, virtual to 2.6g. 1GB is within the reasonable use limit for a fully loaded Squid under peak traffic. The resident size reported is usually the biggest "ever" size of memory usage by the process. FWIW: The memory report shows about 324MB being tracked by Squid as currently in use for other things than cache_mem with 550 active clients doing 117 transactions at present. The client transaction related pools show that the current values are 1/3 of peak traffic, so 3x 360MB ==> ~1GB under peak traffic appears entirely possible for your Squid. Out of interest, how did you come to the 324MB? I'd be interested in learning how to read the output a bit better :) Okay. (for anyone reading the report is a TSV format [open in Libre Calc or Excel as Tab-separated columns]). The final row of the big table is Totals of all rows above. I took the 1360314 from Allocated section "(KB)" column [~1360 MB] and subtracted 1GB / 1024MB. That is the current Total memory usage Squid is aware of either in active use or waiting re-allocation, minus what you said cache_mem was configured to. NP: Before subtracting I did a quick check of the mem_node (ie cache_mem memory 'pages') there is ~1025 MB. Enough for the full 1024 MB cache_mem and some extra MB of items in-transit right now that are not cacheable - which use mem_nodes as well. Using a bit of developer inside-knowledge I identify in the Allocated section "(#)" column the main state objects which are allocated one for each client connection: * "cbdata ConnStateData" shows 2749 - at 1 per currently open client TCP connection. * "cbdata ClientHttpRequest " shows 550 - at 1 each per client HTTP request. ++ sorry I got that wrong earlier myself. * "cbdata ClientReplyContext" also shows 550 - at 1 each per currently underway client HTTP responses. These objects also give me the details to estimate current versus peak traffic memory requirements. For example: cbdata ClientReplyContext shows 550 current allocated, using 35269 KB, with a highest-ever allocation of 100933 KB - roughy 3x the current memory usage. The difference between Allocated "(KB)" column and "high (KB)" columns shows us how much is allocated now versus the highest ever allocated. A leak usually shows up as both those columns being nearly identical values, although it is possible that 1:10 objects leaks or something weird like that which can hide it. I may have to change my reading a bit. Thinking about that column meaning now I see the Total value for allocations "high (KB)" only shows 1462831 KB - its not very accurate, but does show that if all objects were reaching their max at the same time it would still be ~500MB short of that 2GB the system reports. Another option that usually adds fuzz to these numbers is spawning of helpers - the fork() used for that allocates the child process a whole duplicate of the current memory space usage of the parent process. Effectively doubling the OS-reported memory values from whatever the reality is. Like Alex said, leaks show up as an ever-increasing value in these numbers somewhere. Regular snapshots of that report and the system values taken across a week or two should be able to show if there is anything constantly growing at a regular rate. Amos
Re: [squid-users] Squid memory usage
On Thu, May 30, 2013 at 12:53 AM, Alex Rousskov wrote: > In a steady state, memory usage should not "increase over time through > general use". A steady state includes such preconditions as > > * no memory pools or sufficient traffic to fill memory pools > * no caching or sufficient traffic to fill all caches > * steady peak loads (as opposed to getting more and more load over time) > > Sure, memory usage will fluctuate with load, but the overall trend in a > steady state should not be "up", it should be "flat". In other words, > there will always be memory usage peaks and valleys but neither should > be increasing with time in a steady state. If they do, there is a leak > somewhere. Ah, I follow now! Thanks very much for the explanation. I've started logging the stats every hour so I'll see how those go over the next few days. Nathan. > > > Cheers, > > Alex. >
Re: [squid-users] Squid memory usage
Fantastic explanation! Thanks heaps Amos. Would it make sense for this to go onto wiki.squid-cache.org somewhere? -- Nathan Hoad Software Developer www.getoffmalawn.com On Wed, May 29, 2013 at 7:00 PM, Amos Jeffries wrote: > On 29/05/2013 2:19 p.m., Nathan Hoad wrote: >> >> On Tue, May 28, 2013 at 3:23 PM, Amos Jeffries >> wrote: >>> >>> On 28/05/2013 3:59 p.m., Nathan Hoad wrote: >>> >>> I take it you are referring to the 2.0g resident size? >> >> That is what I'm referring to, yes - the resident size has increased >> to 2.5g since my previous mail, virtual to 2.6g. >> >>> 1GB is within the reasonable use limit for a fully loaded Squid under >>> peak >>> traffic. The resident size reported is usually the biggest "ever" size of >>> memory usage by the process. >>> >>> FWIW: The memory report shows about 324MB being tracked by Squid as >>> currently in use for other things than cache_mem with 550 active clients >>> doing 117 transactions at present. The client transaction related pools >>> show >>> that the current values are 1/3 of peak traffic, so 3x 360MB ==> ~1GB >>> under >>> peak traffic appears entirely possible for your Squid. >> >> Out of interest, how did you come to the 324MB? I'd be interested in >> learning how to read the output a bit better :) > > > Okay. (for anyone reading the report is a TSV format [open in Libre Calc or > Excel as Tab-separated columns]). > > The final row of the big table is Totals of all rows above. I took the > 1360314 from Allocated section "(KB)" column [~1360 MB] and subtracted 1GB / > 1024MB. That is the current Total memory usage Squid is aware of either in > active use or waiting re-allocation, minus what you said cache_mem was > configured to. > > NP: Before subtracting I did a quick check of the mem_node (ie cache_mem > memory 'pages') there is ~1025 MB. Enough for the full 1024 MB cache_mem and > some extra MB of items in-transit right now that are not cacheable - which > use mem_nodes as well. > > > Using a bit of developer inside-knowledge I identify in the Allocated > section "(#)" column the main state objects which are allocated one for each > client connection: > * "cbdata ConnStateData" shows 2749 - at 1 per currently open client TCP > connection. > * "cbdata ClientHttpRequest " shows 550 - at 1 each per client HTTP request. > ++ sorry I got that wrong earlier myself. > * "cbdata ClientReplyContext" also shows 550 - at 1 each per currently > underway client HTTP responses. > > These objects also give me the details to estimate current versus peak > traffic memory requirements. For example: > cbdata ClientReplyContext shows 550 current allocated, using 35269 KB, with > a highest-ever allocation of 100933 KB - roughy 3x the current memory usage. > > > The difference between Allocated "(KB)" column and "high (KB)" columns shows > us how much is allocated now versus the highest ever allocated. A leak > usually shows up as both those columns being nearly identical values, > although it is possible that 1:10 objects leaks or something weird like that > which can hide it. > > I may have to change my reading a bit. Thinking about that column meaning > now I see the Total value for allocations "high (KB)" only shows 1462831 KB > - its not very accurate, but does show that if all objects were reaching > their max at the same time it would still be ~500MB short of that 2GB the > system reports. Another option that usually adds fuzz to these numbers is > spawning of helpers - the fork() used for that allocates the child process a > whole duplicate of the current memory space usage of the parent process. > Effectively doubling the OS-reported memory values from whatever the reality > is. > > > Like Alex said, leaks show up as an ever-increasing value in these numbers > somewhere. Regular snapshots of that report and the system values taken > across a week or two should be able to show if there is anything constantly > growing at a regular rate. > > Amos
Re: [squid-users] Squid memory usage
On 05/28/2013 08:19 PM, Nathan Hoad wrote: > On Wed, May 29, 2013 at 1:56 AM, Alex Rousskov wrote: >> Memory leaks increase memory usage over time. Does that happen in your >> environment? If you do not know, you may want to start logging Squid >> memory usage every hour or so. > I am happy to start doing this, but given that memory usage would > increase over time through general use anyway, I'm unsure how I could > differentiate between expected memory usage increases and a memory > leak. In a steady state, memory usage should not "increase over time through general use". A steady state includes such preconditions as * no memory pools or sufficient traffic to fill memory pools * no caching or sufficient traffic to fill all caches * steady peak loads (as opposed to getting more and more load over time) Sure, memory usage will fluctuate with load, but the overall trend in a steady state should not be "up", it should be "flat". In other words, there will always be memory usage peaks and valleys but neither should be increasing with time in a steady state. If they do, there is a leak somewhere. Cheers, Alex.
Re: [squid-users] Squid memory usage
On 29/05/2013 2:19 p.m., Nathan Hoad wrote: On Tue, May 28, 2013 at 3:23 PM, Amos Jeffries wrote: On 28/05/2013 3:59 p.m., Nathan Hoad wrote: I take it you are referring to the 2.0g resident size? That is what I'm referring to, yes - the resident size has increased to 2.5g since my previous mail, virtual to 2.6g. 1GB is within the reasonable use limit for a fully loaded Squid under peak traffic. The resident size reported is usually the biggest "ever" size of memory usage by the process. FWIW: The memory report shows about 324MB being tracked by Squid as currently in use for other things than cache_mem with 550 active clients doing 117 transactions at present. The client transaction related pools show that the current values are 1/3 of peak traffic, so 3x 360MB ==> ~1GB under peak traffic appears entirely possible for your Squid. Out of interest, how did you come to the 324MB? I'd be interested in learning how to read the output a bit better :) Okay. (for anyone reading the report is a TSV format [open in Libre Calc or Excel as Tab-separated columns]). The final row of the big table is Totals of all rows above. I took the 1360314 from Allocated section "(KB)" column [~1360 MB] and subtracted 1GB / 1024MB. That is the current Total memory usage Squid is aware of either in active use or waiting re-allocation, minus what you said cache_mem was configured to. NP: Before subtracting I did a quick check of the mem_node (ie cache_mem memory 'pages') there is ~1025 MB. Enough for the full 1024 MB cache_mem and some extra MB of items in-transit right now that are not cacheable - which use mem_nodes as well. Using a bit of developer inside-knowledge I identify in the Allocated section "(#)" column the main state objects which are allocated one for each client connection: * "cbdata ConnStateData" shows 2749 - at 1 per currently open client TCP connection. * "cbdata ClientHttpRequest " shows 550 - at 1 each per client HTTP request. ++ sorry I got that wrong earlier myself. * "cbdata ClientReplyContext" also shows 550 - at 1 each per currently underway client HTTP responses. These objects also give me the details to estimate current versus peak traffic memory requirements. For example: cbdata ClientReplyContext shows 550 current allocated, using 35269 KB, with a highest-ever allocation of 100933 KB - roughy 3x the current memory usage. The difference between Allocated "(KB)" column and "high (KB)" columns shows us how much is allocated now versus the highest ever allocated. A leak usually shows up as both those columns being nearly identical values, although it is possible that 1:10 objects leaks or something weird like that which can hide it. I may have to change my reading a bit. Thinking about that column meaning now I see the Total value for allocations "high (KB)" only shows 1462831 KB - its not very accurate, but does show that if all objects were reaching their max at the same time it would still be ~500MB short of that 2GB the system reports. Another option that usually adds fuzz to these numbers is spawning of helpers - the fork() used for that allocates the child process a whole duplicate of the current memory space usage of the parent process. Effectively doubling the OS-reported memory values from whatever the reality is. Like Alex said, leaks show up as an ever-increasing value in these numbers somewhere. Regular snapshots of that report and the system values taken across a week or two should be able to show if there is anything constantly growing at a regular rate. Amos
Re: [squid-users] Squid memory usage
On Tue, May 28, 2013 at 3:23 PM, Amos Jeffries wrote: > On 28/05/2013 3:59 p.m., Nathan Hoad wrote: > > I take it you are referring to the 2.0g resident size? That is what I'm referring to, yes - the resident size has increased to 2.5g since my previous mail, virtual to 2.6g. > > 1GB is within the reasonable use limit for a fully loaded Squid under peak > traffic. The resident size reported is usually the biggest "ever" size of > memory usage by the process. > > FWIW: The memory report shows about 324MB being tracked by Squid as > currently in use for other things than cache_mem with 550 active clients > doing 117 transactions at present. The client transaction related pools show > that the current values are 1/3 of peak traffic, so 3x 360MB ==> ~1GB under > peak traffic appears entirely possible for your Squid. Out of interest, how did you come to the 324MB? I'd be interested in learning how to read the output a bit better :) > > HTH > Amos On Wed, May 29, 2013 at 1:56 AM, Alex Rousskov wrote: > On 05/27/2013 09:59 PM, Nathan Hoad wrote: > > > Memory leaks increase memory usage over time. Does that happen in your > environment? If you do not know, you may want to start logging Squid > memory usage every hour or so. I am happy to start doing this, but given that memory usage would increase over time through general use anyway, I'm unsure how I could differentiate between expected memory usage increases and a memory leak. Nathan.
Re: [squid-users] Squid memory usage
On 05/27/2013 09:59 PM, Nathan Hoad wrote: > I'm running Squid 3.2.9, and I am seeing huge memory and CPU usage on > busy sites. The CPU usage is expected due to the level of traffic, but > the memory usage not so much. > > PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND > 15631 squid 25 0 2073m 2.0g 5464 R 87.4 20.3 230:11.58 > /usr/sbin/squid -N -f /etc/squid/squid.conf > > cache_mem is set to 1024 MB. I would not expect Squid to need an > additional GB of memory to manage its internals. > > The output of the manager's mem page can be downloaded here: > > http://www.getoffmalawn.com/static/squidmgr.mem > > What can I do to reduce this memory usage? It looks like a memory leak to me. Memory leaks increase memory usage over time. Does that happen in your environment? If you do not know, you may want to start logging Squid memory usage every hour or so. Please note that the increase may come in bursts corresponding to intervals of high usage or unusual traffic. The memory consumption may even go down between those bursts. Thank you, Alex.
Re: [squid-users] Squid memory usage
On 28/05/2013 3:59 p.m., Nathan Hoad wrote: I'm running Squid 3.2.9, and I am seeing huge memory and CPU usage on busy sites. The CPU usage is expected due to the level of traffic, but the memory usage not so much. PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 15631 squid 25 0 2073m 2.0g 5464 R 87.4 20.3 230:11.58 /usr/sbin/squid -N -f /etc/squid/squid.conf cache_mem is set to 1024 MB. I would not expect Squid to need an additional GB of memory to manage its internals. I take it you are referring to the 2.0g resident size? The output of the manager's mem page can be downloaded here: http://www.getoffmalawn.com/static/squidmgr.mem What can I do to reduce this memory usage? It looks like a memory leak to me. 1GB is within the reasonable use limit for a fully loaded Squid under peak traffic. The resident size reported is usually the biggest "ever" size of memory usage by the process. FWIW: The memory report shows about 324MB being tracked by Squid as currently in use for other things than cache_mem with 550 active clients doing 117 transactions at present. The client transaction related pools show that the current values are 1/3 of peak traffic, so 3x 360MB ==> ~1GB under peak traffic appears entirely possible for your Squid. HTH Amos
Re: [squid-users] Squid memory usage
On 8/8/2012 6:49 PM, Hugo Deprez wrote: Hello, since I changed the configuration memory usage is growing slowy. Now squid is using 17% of 4GB Eliezer, I am not sure to understand. But I am using two VM, active/passive setup with a corosync VIP. I remembered someone talking here about a VM that was replicated and caused squid to leak memory. so I asked if it's a VM that was cloned or not to verify it. for me it still odd that a cloned VM will cause such a thing so I will just say it's seems like a bogus alarm set by someone by false assumption. just curios about this cluster setup you have there: can you give some more details about it? (email me directly) I am working on a cluster setup of squid tproxy balanced on a routing level. Thanks, Elizer I will consider upgrading one member of the cluster to 3.1.20 (squeeze packages). DO you think this will sole the issue ? Regards, On 8 August 2012 05:30, Amos Jeffries wrote: On 08.08.2012 02:35, Simon Roscic wrote: > I think this is probably: http://bugs.squid-cache.org/show_bug.cgi?id=3605 Can you start with the cachemgr memory usage report and confirm whether the same FwdServer excessive memory usage is seen? Amos -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations eliezer ngtech.co.il
Re: [squid-users] Squid memory usage
Hello, in my case it does not look like FwdServer memory useage is excessive (if i interpreted it correctly). I added my findings to the bug 3605, also i attached the output from squidclient mgr:mem and squidclient mgr:info to bug 3605. kind regards, Simon On 2012-08-08 05:30, Amos Jeffries wrote: I think this is probably: http://bugs.squid-cache.org/show_bug.cgi?id=3605 Can you start with the cachemgr memory usage report and confirm whether the same FwdServer excessive memory usage is seen? Amos
Re: [squid-users] Squid memory usage
Hello, since I changed the configuration memory usage is growing slowy. Now squid is using 17% of 4GB Eliezer, I am not sure to understand. But I am using two VM, active/passive setup with a corosync VIP. I will consider upgrading one member of the cluster to 3.1.20 (squeeze packages). DO you think this will sole the issue ? Regards, On 8 August 2012 05:30, Amos Jeffries wrote: > On 08.08.2012 02:35, Simon Roscic wrote: >> >> Hello, >> >> I am experiencing the same Problem as Hugo Deprez reported and i >> think it is a memory leak. >> >> In my case i can confirm this memory leakage for Squid releases >> 3.2.0.18 and 3.2.0.19. >> We have a few hundred users using our Squid Proxy VM (4 Cores/4 GB >> RAM/Ubuntu 12.04/x64) and during work hours memory useage is >> increasing quite quickly (e.g. in 1 hour 15% increase of RAM useage by >> Squid). >> After a few hours my Squid Proxy VM begins to swap: >> >> Mem: 4049728k total, 3943484k used, 106244k free, 7836k buffers >> Swap: 2097148k total, 826108k used, 1271040k free, 117900k cached >> >> PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND >> 26177 proxy 20 0 4159m 3.4g 2160 S3 87.0 29:20.81 squid3 >> >> also adding: >> >>> Can you try tuning these options? >>> memory_pools off >>> memory_pools_limit 1 MB >> >> >> as suggested by Drunkard Zhang does not help. >> >> ... >> >> So, how can we help to track down this problem? > > > I think this is probably: > http://bugs.squid-cache.org/show_bug.cgi?id=3605 > > Can you start with the cachemgr memory usage report and confirm whether the > same FwdServer excessive memory usage is seen? > > > Amos >
Re: [squid-users] Squid memory usage
On 08.08.2012 02:35, Simon Roscic wrote: Hello, I am experiencing the same Problem as Hugo Deprez reported and i think it is a memory leak. In my case i can confirm this memory leakage for Squid releases 3.2.0.18 and 3.2.0.19. We have a few hundred users using our Squid Proxy VM (4 Cores/4 GB RAM/Ubuntu 12.04/x64) and during work hours memory useage is increasing quite quickly (e.g. in 1 hour 15% increase of RAM useage by Squid). After a few hours my Squid Proxy VM begins to swap: Mem: 4049728k total, 3943484k used, 106244k free, 7836k buffers Swap: 2097148k total, 826108k used, 1271040k free, 117900k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 26177 proxy 20 0 4159m 3.4g 2160 S3 87.0 29:20.81 squid3 also adding: Can you try tuning these options? memory_pools off memory_pools_limit 1 MB as suggested by Drunkard Zhang does not help. ... So, how can we help to track down this problem? I think this is probably: http://bugs.squid-cache.org/show_bug.cgi?id=3605 Can you start with the cachemgr memory usage report and confirm whether the same FwdServer excessive memory usage is seen? Amos
Re: [squid-users] Squid memory usage
Hello, I am experiencing the same Problem as Hugo Deprez reported and i think it is a memory leak. In my case i can confirm this memory leakage for Squid releases 3.2.0.18 and 3.2.0.19. We have a few hundred users using our Squid Proxy VM (4 Cores/4 GB RAM/Ubuntu 12.04/x64) and during work hours memory useage is increasing quite quickly (e.g. in 1 hour 15% increase of RAM useage by Squid). After a few hours my Squid Proxy VM begins to swap: Mem: 4049728k total, 3943484k used, 106244k free, 7836k buffers Swap: 2097148k total, 826108k used, 1271040k free, 117900k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 26177 proxy 20 0 4159m 3.4g 2160 S3 87.0 29:20.81 squid3 also adding: Can you try tuning these options? memory_pools off memory_pools_limit 1 MB as suggested by Drunkard Zhang does not help. ... So, how can we help to track down this problem? kind regards, Simon Roscic
RE: [squid-users] Squid memory usage
> Date: Fri, 3 Aug 2012 14:16:29 +0200 > From: hugo.dep...@gmail.com > To: squid-users@squid-cache.org > Subject: [squid-users] Squid memory usage > > Dear community, > > I am running squid3 on Linux Debian squeeze.(3.1.6). > > I encounter a suddenly a high memory usage on my virtual machine don't > really know why. > Looking at the cacti memory graph is showing a memory jump from 1.5 Gb > to 4GB and then ther server started to swap. > > For information the virtual machine has 4Gb of RAM. > > Here is the settings of squid.conf : > > cache_dir ufs /var/spool/squid3 100 16 256 > cache_mem 100 MB > > hierarchy_stoplist cgi-bin ? > refresh_pattern ^ftp: 1440 20% 10080 > refresh_pattern ^gopher: 1440 0% 1440 > refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 > refresh_pattern . 0 20% 4320 > > > my squid3 process is using: 81% of my RAM. So arround 3,2Gb of memory. > > proxy 25889 0.6 81.1 3937744 3299616 ? S Aug02 9:34 > (squid) -YC -f /etc/squid3/squid.conf > > I am currently having arround 50 users using it. > > > I did have a look at the FAQ > (http://wiki.squid-cache.org/SquidFaq/SquidMemory#how-much-ram), but I > didn't find any tips for my situation in it. > > > Have you got any idea ? How can I troubleshoot this ? > > Thanks ! Upgrade to a newer version. I had the same issues with a memory leak on icons. It was fixed 4-5 months ago. Jenny
Re: [squid-users] Squid memory usage
On 8/3/2012 3:16 PM, Hugo Deprez wrote: Dear community, I am running squid3 on Linux Debian squeeze.(3.1.6). I encounter a suddenly a high memory usage on my virtual machine don't really know why. Looking at the cacti memory graph is showing a memory jump from 1.5 Gb to 4GB and then ther server started to swap. For information the virtual machine has 4Gb of RAM. Here is the settings of squid.conf : cache_dir ufs /var/spool/squid3 100 16 256 cache_mem 100 MB hierarchy_stoplist cgi-bin ? refresh_pattern ^ftp: 144020% 10080 refresh_pattern ^gopher:14400% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 my squid3 process is using: 81% of my RAM. So arround 3,2Gb of memory. proxy25889 0.6 81.1 3937744 3299616 ? SAug02 9:34 (squid) -YC -f /etc/squid3/squid.conf I am currently having arround 50 users using it. I did have a look at the FAQ (http://wiki.squid-cache.org/SquidFaq/SquidMemory#how-much-ram), but I didn't find any tips for my situation in it. Have you got any idea ? How can I troubleshoot this ? Thanks ! is this a replicated VM by any chance? -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations eliezer ngtech.co.il
Re: [squid-users] Squid memory usage
I will try this option. I found a bug report which seems similar to my issue : http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=664051 but seems the only solution is to upgrade to a newer squid version... :/ On 3 August 2012 14:47, Drunkard Zhang wrote: > 2012/8/3 Hugo Deprez : >> Dear community, >> >> I am running squid3 on Linux Debian squeeze.(3.1.6). >> >> I encounter a suddenly a high memory usage on my virtual machine don't >> really know why. >> Looking at the cacti memory graph is showing a memory jump from 1.5 Gb >> to 4GB and then ther server started to swap. >> >> For information the virtual machine has 4Gb of RAM. >> >> Here is the settings of squid.conf : >> >> cache_dir ufs /var/spool/squid3 100 16 256 >> cache_mem 100 MB >> > > Can you try tuning these options? > memory_pools off > memory_pools_limit 1 MB > > >> hierarchy_stoplist cgi-bin ? >> refresh_pattern ^ftp: 144020% 10080 >> refresh_pattern ^gopher:14400% 1440 >> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 >> refresh_pattern . 0 20% 4320 >> >> >> my squid3 process is using: 81% of my RAM. So arround 3,2Gb of memory. >> >> proxy25889 0.6 81.1 3937744 3299616 ? SAug02 9:34 >> (squid) -YC -f /etc/squid3/squid.conf >> >> I am currently having arround 50 users using it. >> >> >> I did have a look at the FAQ >> (http://wiki.squid-cache.org/SquidFaq/SquidMemory#how-much-ram), but I >> didn't find any tips for my situation in it. >> >> >> Have you got any idea ? How can I troubleshoot this ? >> >> Thanks ! > > > > -- > 张绍文 > gongfan...@gmail.com > zhan...@gwbnsh.net.cn > 18601633785
Re: [squid-users] Squid memory usage
2012/8/3 Hugo Deprez : > Dear community, > > I am running squid3 on Linux Debian squeeze.(3.1.6). > > I encounter a suddenly a high memory usage on my virtual machine don't > really know why. > Looking at the cacti memory graph is showing a memory jump from 1.5 Gb > to 4GB and then ther server started to swap. > > For information the virtual machine has 4Gb of RAM. > > Here is the settings of squid.conf : > > cache_dir ufs /var/spool/squid3 100 16 256 > cache_mem 100 MB > Can you try tuning these options? memory_pools off memory_pools_limit 1 MB > hierarchy_stoplist cgi-bin ? > refresh_pattern ^ftp: 144020% 10080 > refresh_pattern ^gopher:14400% 1440 > refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 > refresh_pattern . 0 20% 4320 > > > my squid3 process is using: 81% of my RAM. So arround 3,2Gb of memory. > > proxy25889 0.6 81.1 3937744 3299616 ? SAug02 9:34 > (squid) -YC -f /etc/squid3/squid.conf > > I am currently having arround 50 users using it. > > > I did have a look at the FAQ > (http://wiki.squid-cache.org/SquidFaq/SquidMemory#how-much-ram), but I > didn't find any tips for my situation in it. > > > Have you got any idea ? How can I troubleshoot this ? > > Thanks ! -- 张绍文 gongfan...@gmail.com zhan...@gwbnsh.net.cn 18601633785
Re: [squid-users] Squid memory usage
Henrik, I read FAQ and implemented almost most of the suggestion to reduce memory usage. I am not much concern about memory usage as there plenty of available memory but the issue is CPU usage goes high up to 100% and slows down squid response once squid grows beyond allocated cache_mem size . Does that mean squid is spending most of time in releasing the objects from cache ? Most of the objects stored in cache has TTL of 1 hour. Following are few lines from squid.conf file. http_port 0.0.0.0:80 accel defaultsite=s1.xyz.com vhost protocol=http cache_peer 10.0.0.175 Parent 80 0 no-query round-robin originserver monitorurl=http://10.0.0.175:80/healthcheck.gif cache_peer 10.0.0.177 Parent 80 0 no-query round-robin originserver monitorurl=http://10.0.0.177:80/healthcheck.gif cache_peer 10.0.0.179 Parent 80 0 no-query round-robin originserver monitorurl=http://10.0.0.179:80/healthcheck.gif cache_peer 10.0.0.181 Parent 80 0 no-query round-robin originserver monitorurl=http://10.0.0.181:80/healthcheck.gif dead_peer_timeout 10 seconds hierarchy_stoplist cgi-bin hierarchy_stoplist ? cache_mem 4294967296 bytes maximum_object_size_in_memory 1048576 bytes memory_replacement_policy lru cache_replacement_policy lru cache_dir null /empty cache_swap_low 60 cache_swap_high 80 refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern (cgi-bin|\?) 0 0% 0 refresh_pattern . 1800 20% 3600 Regards Nitesh On Sat, Nov 8, 2008 at 2:06 AM, Henrik Nordstrom <[EMAIL PROTECTED]> wrote: > Have you read the faq section on memory usage? > > > > On fre, 2008-11-07 at 20:02 +0530, nitesh naik wrote: >> Henrik / Amos, >> >> Do you all think I should reduce cache_mem to lesser value ? Squid >> stops responding as memory usage of squid grows upto 12GB. I have >> allocate 8 GB cache_mem. >> >> We are using 64 bit machine running on Suse 10.1. >> >> Regards >> Nitesh >> >> >> On Thu, Nov 6, 2008 at 11:35 PM, nitesh naik <[EMAIL PROTECTED]> wrote: >> > Thanks everyone for your reply. >> > >> > I went through all these docs and also compiled squid with dmalloc >> > option and disabled memory_pool. Squid memory usage grows upto 12GB+ >> > and squid stops responding when we try to rotate logs using squid -k >> > rotate. >> > >> > I want squid up and running all the time even if its memory usage >> > grows double the allocate cache_mem value. >> > >> > Regards >> > Nitesh >> > On Thu, Nov 6, 2008 at 3:58 PM, Adam Carter <[EMAIL PROTECTED]> wrote: >> >>> Squid memory usage grows beyond allocate cache_mem size of 8 GB. >> >> >> >> http://wiki.squid-cache.org/SquidFaq/SquidMemory >> >> >> > >
Re: [squid-users] Squid memory usage
Have you read the faq section on memory usage? On fre, 2008-11-07 at 20:02 +0530, nitesh naik wrote: > Henrik / Amos, > > Do you all think I should reduce cache_mem to lesser value ? Squid > stops responding as memory usage of squid grows upto 12GB. I have > allocate 8 GB cache_mem. > > We are using 64 bit machine running on Suse 10.1. > > Regards > Nitesh > > > On Thu, Nov 6, 2008 at 11:35 PM, nitesh naik <[EMAIL PROTECTED]> wrote: > > Thanks everyone for your reply. > > > > I went through all these docs and also compiled squid with dmalloc > > option and disabled memory_pool. Squid memory usage grows upto 12GB+ > > and squid stops responding when we try to rotate logs using squid -k > > rotate. > > > > I want squid up and running all the time even if its memory usage > > grows double the allocate cache_mem value. > > > > Regards > > Nitesh > > On Thu, Nov 6, 2008 at 3:58 PM, Adam Carter <[EMAIL PROTECTED]> wrote: > >>> Squid memory usage grows beyond allocate cache_mem size of 8 GB. > >> > >> http://wiki.squid-cache.org/SquidFaq/SquidMemory > >> > > signature.asc Description: This is a digitally signed message part
Re: [squid-users] Squid memory usage
Henrik / Amos, Do you all think I should reduce cache_mem to lesser value ? Squid stops responding as memory usage of squid grows upto 12GB. I have allocate 8 GB cache_mem. We are using 64 bit machine running on Suse 10.1. Regards Nitesh On Thu, Nov 6, 2008 at 11:35 PM, nitesh naik <[EMAIL PROTECTED]> wrote: > Thanks everyone for your reply. > > I went through all these docs and also compiled squid with dmalloc > option and disabled memory_pool. Squid memory usage grows upto 12GB+ > and squid stops responding when we try to rotate logs using squid -k > rotate. > > I want squid up and running all the time even if its memory usage > grows double the allocate cache_mem value. > > Regards > Nitesh > On Thu, Nov 6, 2008 at 3:58 PM, Adam Carter <[EMAIL PROTECTED]> wrote: >>> Squid memory usage grows beyond allocate cache_mem size of 8 GB. >> >> http://wiki.squid-cache.org/SquidFaq/SquidMemory >> >
Re: [squid-users] Squid memory usage
Thanks everyone for your reply. I went through all these docs and also compiled squid with dmalloc option and disabled memory_pool. Squid memory usage grows upto 12GB+ and squid stops responding when we try to rotate logs using squid -k rotate. I want squid up and running all the time even if its memory usage grows double the allocate cache_mem value. Regards Nitesh On Thu, Nov 6, 2008 at 3:58 PM, Adam Carter <[EMAIL PROTECTED]> wrote: >> Squid memory usage grows beyond allocate cache_mem size of 8 GB. > > http://wiki.squid-cache.org/SquidFaq/SquidMemory >
Re: [squid-users] Squid memory usage
nitesh naik wrote: Hi All, Squid memory usage grows beyond allocate cache_mem size of 8 GB. Total physical memory available on machine is 20 GB. Does that mean there is memory leak and I should replace malloc library and compile squid ? cache_mem is the size of the in-memory object cache squid uses for very popular objects. Squid itself uses memory for many other things which make the total larger. Amos I am using squid2.6 without using disk for caching. Memory usage for squid via mallinfo(): Total space in arena: -52916 KB Ordinary blocks: -82192 KB 215707 blks Small blocks: 0 KB 0 blks Holding blocks: 11404 KB 3 blks Free Small blocks: 0 KB Free Ordinary blocks: 29275 KB Total in use: -70788 KB 171% Total free: 29275 KB -70% Total size:-41512 KB Memory accounted for: Total accounted: 11256849 KB memPoolAlloc calls: 870846623 memPoolFree calls: 819677123 Regards Nitesh -- Please be using Current Stable Squid 2.7.STABLE5 or 3.0.STABLE10 Current Beta Squid 3.1.0.1
Re: [squid-users] squid memory usage and SNMP
On tis, 2008-10-07 at 18:22 -0300, Leonardo Rodrigues Magalhães wrote: > with SNMP i can already grab the 'Total Accounted' value but i > would like the real allocated memory by squid, which seems to be the > 'Total in use' value. I do not think these are exposed in the MIB yet.. Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Squid Memory usage on Fedora Core 3
On Thu, 13 Oct 2005, Daniel H. Latshaw wrote: when i reinstalled to the newer version i added on item at a time from my old squid.conf and restarted squid to see where my problem was. i don't believe it was in the 2.5.6 version, it was my acl. as i added that acl back in, my memory usage jumped back to the 1GB mark. Any idea why that particular acl would do that? What kind of ACL are you using? Regards Henrik
RE: [squid-users] Squid Memory usage on Fedora Core 3
> -Original Message- > From: Daniel H. Latshaw [mailto:[EMAIL PROTECTED] > Sent: Thursday, October 13, 2005 6:31 AM > To: squid-users@squid-cache.org > Subject: Re: [squid-users] Squid Memory usage on Fedora Core 3 > > > On Thu, 2005-10-13 at 00:06 +0200, Henrik Nordstrom wrote: > > On Wed, 12 Oct 2005, Daniel Latshaw wrote: > > > > > When i start squid, there is a process owned by root that is using 376mb > > > ram. there is also a process owned by squid starting at 377mb ram and > > > growing to 723mb ram and beyond. > > > > Do you have any very large ACLs defined? > > Yes, that is exactly what the problem is. I was using the > sa-blacklist-current-domains list. the list itself is only > ~2mb or so. > > > > Memory usage for squid via mallinfo(): > > > Total in use: 969957 KB 100% > > > Memory accounted for: > > > Total accounted: 7415 KB > > > > This does not look good... > > > It is down to using about 8mb now. > > > > Which exact Squid version is this? > > > i was using the 2.5.stable6 that comes with fedora core 3. i > have since > removed that and am using 2.5.stable11 from fedora's site. > > when i reinstalled to the newer version i added on item at a time from > my old squid.conf and restarted squid to see where my problem was. i > don't believe it was in the 2.5.6 version, it was my acl. as i added > that acl back in, my memory usage jumped back to the 1GB mark. > > Any idea why that particular acl would do that? > My guess would be that you were using url_regex (or dstdom_regex). The sa-blacklist-current-domains that I found looks to be a listing of domains, which would be more compatible with a dstdomain acl. You might want to adjust the list so every line starts with a period (.) for subdomain matching. > Thanks! > > dan > > > Regards > > Henrik > > > > Chris
Re: [squid-users] Squid Memory usage on Fedora Core 3
On Thu, 2005-10-13 at 00:06 +0200, Henrik Nordstrom wrote: > On Wed, 12 Oct 2005, Daniel Latshaw wrote: > > > When i start squid, there is a process owned by root that is using 376mb > > ram. there is also a process owned by squid starting at 377mb ram and > > growing to 723mb ram and beyond. > > Do you have any very large ACLs defined? Yes, that is exactly what the problem is. I was using the sa-blacklist-current-domains list. the list itself is only ~2mb or so. > > Memory usage for squid via mallinfo(): > > Total in use: 969957 KB 100% > > Memory accounted for: > > Total accounted: 7415 KB > > This does not look good... > It is down to using about 8mb now. > > Which exact Squid version is this? > i was using the 2.5.stable6 that comes with fedora core 3. i have since removed that and am using 2.5.stable11 from fedora's site. when i reinstalled to the newer version i added on item at a time from my old squid.conf and restarted squid to see where my problem was. i don't believe it was in the 2.5.6 version, it was my acl. as i added that acl back in, my memory usage jumped back to the 1GB mark. Any idea why that particular acl would do that? Thanks! dan > Regards > Henrik >
Re: [squid-users] Squid Memory usage on Fedora Core 3
On Wed, 12 Oct 2005, Daniel Latshaw wrote: When i start squid, there is a process owned by root that is using 376mb ram. there is also a process owned by squid starting at 377mb ram and growing to 723mb ram and beyond. Do you have any very large ACLs defined? Memory usage for squid via mallinfo(): Total in use: 969957 KB 100% Memory accounted for: Total accounted: 7415 KB This does not look good... Which exact Squid version is this? Regards Henrik
