Re: [gentoo-user] Memory Usage
Hello all, Grant wrote: Also, I've noticed in top that when my server's 2GB of memory is filled, it uses a small amount of swap (~24k) before it frees some up. The Swap: 24k then remains. Is that normal? Yes I have also notice that after a heavy ram use I get this tiny space used on my swap. And it remains even after my ram has been half freed. My box has 1GB of ram and the small amount I sometime get on swap is 164KB and it can stay like that for 2 weeks, until i reboot. But I assume rebooting is not a great solution for a server :=) Writing to disk is too expensive as Pablo Antonio suggested, so once swap has been used it won't be cleaned until next use, but how to be sure it is cleaned at next use ? Anyway if you want to have stats on the usage of your memory you can use 'free' but for a long-period log I would recommend 'vmstat' as it puts everything on one line so you could gnuplot it later. vmstat | grep -v [a-z] memory.log As Daniel Iliev suggested, there is the configurable sysctl for swap control solutions. The variables vm.swappiness and the vm.swap_token_timeout which control the trashing behavior. I didn't really find much info on vm.swap_token_timeout. The man is not very extensive. Has Anyone more info on configuration of the file /etc/sysctl.conf ? especially for : vm.swappiness vm.swap_token_timeout If we get a solution or just some more info, it would be nice to have a wiki page about it, wouldn't it ? ;) Have a Good Day, Red. + MAGELLIUM @ http://www.magellium.fr/ + -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Memory Usage
Redouane Boumghar wrote: Hello all, Grant wrote: Also, I've noticed in top that when my server's 2GB of memory is filled, it uses a small amount of swap (~24k) before it frees some up. The Swap: 24k then remains. Is that normal? Yes I have also notice that after a heavy ram use I get this tiny space used on my swap. And it remains even after my ram has been half freed. My box has 1GB of ram and the small amount I sometime get on swap is 164KB and it can stay like that for 2 weeks, until i reboot. But I assume rebooting is not a great solution for a server :=) Writing to disk is too expensive as Pablo Antonio suggested, so once swap has been used it won't be cleaned until next use, but how to be sure it is cleaned at next use ? From /etc/conf.d/rc: # RC_SWAP_ERASE controls erasing of swap partitions at shutdown. # Useful for all those paranoid peeps to nuke their memory. RC_SWAP_ERASE=yes Anyway if you want to have stats on the usage of your memory you can use 'free' but for a long-period log I would recommend 'vmstat' as it puts everything on one line so you could gnuplot it later. vmstat | grep -v [a-z] memory.log As Daniel Iliev suggested, there is the configurable sysctl for swap control solutions. The variables vm.swappiness and the vm.swap_token_timeout which control the trashing behavior. I didn't really find much info on vm.swap_token_timeout. The man is not very extensive. Has Anyone more info on configuration of the file /etc/sysctl.conf ? especially for : vm.swappiness vm.swap_token_timeout You can put this in your /etc/sysctl.conf file: vm.swappiness = 30 That is what I have in mine. It does work though. I have changed and you can see the difference. If we get a solution or just some more info, it would be nice to have a wiki page about it, wouldn't it ? ;) Have a Good Day, Red. + MAGELLIUM @ http://www.magellium.fr/ + -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Memory Usage
On 15 December 2006 13:53, Redouane Boumghar wrote: Hello all, Grant wrote: Also, I've noticed in top that when my server's 2GB of memory is filled, it uses a small amount of swap (~24k) before it frees some up. The Swap: 24k then remains. Is that normal? Yes I have also notice that after a heavy ram use I get this tiny space used on my swap. And it remains even after my ram has been half freed. My box has 1GB of ram and the small amount I sometime get on swap is 164KB and it can stay like that for 2 weeks, until i reboot. But I assume rebooting is not a great solution for a server :=) Writing to disk is too expensive as Pablo Antonio suggested, so once swap has been used it won't be cleaned until next use, but how to be sure it is cleaned at next use ? That is *not* the reason. Your memory is organised in pages. How much memory a page is, 4KB, 8Kb,..., depends on your memory model and the architecture of your box. Let's assume your systems needs more memory than you have ram installed. So your memory manager searches for pages that haven't been used for a while. Let's assume further that you are running cupsd but haven't printed after booting yet. So it's highly likely that those memory pages occupied by cupsd are swapped out (or actually paged out because linux doesn't do swapping but paging). After a while, your memory usage decreases. Why would cupsd be paged in again unless you start printing? Since cupsd isn't used it stays where it is - in your swap partition. I used cupsd here just as an example. It's the same for many other daemons that are there to provide a service. If you don't use that service the daemon is a very good candidate for being paged out. It isn't a candidate for being paged in again at all as long as you don't use the service. Other good candidates are code parts in applications that are executed exactly once, at start time. Say, you keep your browser open all the time. The initialisation code has been executed when you open the browser first, never thereafter. So it's a very good candidate for being paged out if it fills at least one memory page. It will never be paged in again because it isn't used any more. When you close your browser that code isn't paged in either but simply deleted from swap space. So once your system has started to use swap space, some memory pages will always stay in your swap space because things are paged in again only if they are used. Uwe -- Mark Twain: I rather decline two drinks than a German adjective. http://www.SysEx.com.na -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Memory Usage
Hi, Thanks a lot for all this lightening Have a good day, Red. Uwe Thiem wrote: [snip] So once your system has started to use swap space, some memory pages will always stay in your swap space because things are paged in again only if they are used. -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Memory Usage
Redouane Boumghar wrote: The man is not very extensive. Has Anyone more info on configuration of the file /etc/sysctl.conf ? especially for : vm.swappiness vm.swap_token_timeout Actually using sysctl is the same as reading/writing values from/in the files found in /proc. For example putting vm.swappiness = 50 in /etc/sysctl.conf and executing sysctl -p afterwards is the same as echo 50 /proc/sys/vm/swappiness. The deference is that sysctl is being issued at every boot and it restores the values as thy are set in sysctl.conf by the administrator. Otherwise the kernel uses the default values. While swap_token_timeout is documented in two files [1],[2] from the kernel-source-code/Documentation directory, it appears that swappiness isn't documented at all. At least grep -rin swappiness Documentation/ didn't give any results for gentoo-sources-2.6.18-r4 I've read (don't remember where) there are patches to make kernel change its swappiness value automatically, depending on the memory usage for the particular moment. [1] /usr/src/linux/Documentation/sysctl/vm.txt [2] /usr/src/linux/Documentation/filesystems/proc.txt -- Best regards, Daniel -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Memory Usage
Hi, On Fri, 15 Dec 2006 17:26:11 +0200 Daniel Iliev [EMAIL PROTECTED] wrote: I've read (don't remember where) there are patches to make kernel change its swappiness value automatically, depending on the memory usage for the particular moment. That would be the ck-patchset. Gentoo has it in ck-sources. -hwh -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Memory Usage
Hans-Werner Hilse wrote: Hi, On Fri, 15 Dec 2006 17:26:11 +0200 Daniel Iliev [EMAIL PROTECTED] wrote: I've read (don't remember where) there are patches to make kernel change its swappiness value automatically, depending on the memory usage for the particular moment. That would be the ck-patchset. Gentoo has it in ck-sources. -hwh Most probably. I was using ck-sources for a while. ;-) -- Best regards, Daniel -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Memory Usage
On Thu, 14 Dec 2006 09:48:25 -0800, Grant wrote: From what I understand, Linux memory isn't freed up until it is full. Is there a way to find out how much memory is actively in use? The free command. $ free total used free sharedbuffers cached Mem: 1028164 928764 99400 0 28228 468768 -/+ buffers/cache: 431768 596396 Swap: 1556168 2104761345692 -- Neil Bothwick Thesaurus: ancient reptile with an excellent vocabulary signature.asc Description: PGP signature
Re: [gentoo-user] Memory Usage
From what I understand, Linux memory isn't freed up until it is full. Is there a way to find out how much memory is actively in use? The free command. $ free total used free sharedbuffers cached Mem: 1028164 928764 99400 0 28228 468768 -/+ buffers/cache: 431768 596396 Swap: 1556168 2104761345692 That looks like the exact same info top gives me. I have 2GB in my server and I'm trying to figure out if I could get away with 1GB without swapping. Is there a tool that can help me figure that out? - Grant -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Memory Usage
Grant wrote: From what I understand, Linux memory isn't freed up until it is full. Is there a way to find out how much memory is actively in use? The free command. $ free total used free sharedbuffers cached Mem: 1028164 928764 99400 0 28228 468768 -/+ buffers/cache: 431768 596396 Swap: 1556168 2104761345692 That looks like the exact same info top gives me. I have 2GB in my server and I'm trying to figure out if I could get away with 1GB without swapping. Is there a tool that can help me figure that out? - Grant Well here is my free: [EMAIL PROTECTED] / # free total used free sharedbuffers cached Mem: 1034792 977788 57004 0 243928 169800 -/+ buffers/cache: 564060 470732 Swap: 976712 32 976680 [EMAIL PROTECTED] / # If you drop to 1GB what you will loose is space for buffers and cached data. It may be a small amount of slow down but depending on what your server is used for, it may make a difference. Computing memory seems to depend on the program being used. Gkrellm in my opinion does it pretty good and accurate. Who knows which one is really accurate though. If all you have is a console, I would use top and take the amount of memory in use then deduct the buffers and cache. It may not be perfect though. Someone correct me if I am wrong here. Hope that helps. Dale :-) :-) :-) -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Memory Usage
On Thu, 14 Dec 2006 10:49:34 -0800, Grant wrote: $ free total used free sharedbuffers cached Mem: 1028164 928764 99400 0 28228 468768 -/+ buffers/cache: 431768 596396 Swap: 1556168 2104761345692 That looks like the exact same info top gives me. I have 2GB in my server and I'm trying to figure out if I could get away with 1GB without swapping. Is there a tool that can help me figure that out? The second line shows the usage without caches/buffers. If you want to see what your system would be like with 1G, use the mem=1024 option on the kernel line in GRUB. Run the system for a while and monitor swap usage. -- Neil Bothwick Everybody needs a little love sometime; stop hacking and fall in love! signature.asc Description: PGP signature
Re: [gentoo-user] Memory Usage
Neil Bothwick wrote: On Thu, 14 Dec 2006 10:49:34 -0800, Grant wrote: $ free total used free sharedbuffers cached Mem: 1028164 928764 99400 0 28228 468768 -/+ buffers/cache: 431768 596396 Swap: 1556168 2104761345692 That looks like the exact same info top gives me. I have 2GB in my server and I'm trying to figure out if I could get away with 1GB without swapping. Is there a tool that can help me figure that out? The second line shows the usage without caches/buffers. If you want to see what your system would be like with 1G, use the mem=1024 option on the kernel line in GRUB. Run the system for a while and monitor swap usage. The last time I tried mem=xxx it didn't work. It was several years ago on a vanilla kernel 2.6.x. I'm not sure if this kernel parameter is valid anymore. So here is an additional idea for wasting 1GB of RAM: create a 1G file in /dev/shm/ ;-) Anyway to control the swapping one could use the sysctl utility providing a value between 0 and 100 in /etc/sysctl.conf for vm.swappiness. 0 means almost no swapping 100 means swap almost everything. To disable swapping at all, of course there is swapoff -a ;-) HTH -- Best regards, Daniel -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Memory Usage
From what I understand, Linux memory isn't freed up until it is full. Is there a way to find out how much memory is actively in use? The free command. $ free total used free sharedbuffers cached Mem: 1028164 928764 99400 0 28228 468768 -/+ buffers/cache: 431768 596396 Swap: 1556168 2104761345692 That looks like the exact same info top gives me. I have 2GB in my server and I'm trying to figure out if I could get away with 1GB without swapping. Is there a tool that can help me figure that out? - Grant Well here is my free: [EMAIL PROTECTED] / # free total used free sharedbuffers cached Mem: 1034792 977788 57004 0 243928 169800 -/+ buffers/cache: 564060 470732 Swap: 976712 32 976680 [EMAIL PROTECTED] / # If you drop to 1GB what you will loose is space for buffers and cached data. It may be a small amount of slow down but depending on what your server is used for, it may make a difference. My server is mainly used for apache2 with mod_perl. I would think that cache comes in handy. Will a web server pretty much always find something more to cache, or can you add memory to the point where everything that can be cached is cached? - Grant -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Memory Usage
On 09:48 Thu 14 Dec , Grant wrote: [snip] The Swap: 24k then remains. Is that normal? If the question is whether it is normal that the swap space is not freed even when it's not being used anymore, the answer would be yes. Writing to disk is too expensive, so I think the kernel does free the swap space only when it really needs that space. Someone correct me if I'm wrong though. -- Pablo Antonio (AKA crazy2k) http://www.pablo-a.com.ar/ -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Memory Usage
The Swap: 24k then remains. Is that normal? If the question is whether it is normal that the swap space is not freed even when it's not being used anymore, the answer would be yes. I'm wondering if it's normal for the system to use a small amount of swap before it frees memory for the first time. - Grant -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Memory Usage
Grant wrote: The Swap: 24k then remains. Is that normal? If the question is whether it is normal that the swap space is not freed even when it's not being used anymore, the answer would be yes. I'm wondering if it's normal for the system to use a small amount of swap before it frees memory for the first time. - Grant It has on every machine I have regardless of how much memory it has or uses. It may take a little bit and it may be just a 1MB or so but it always seems to use a little. This is my report with uptime included: [EMAIL PROTECTED] / # uptime 19:14:04 up 2 days, 23:13, 1 user, load average: 1.45, 1.35, 1.24 [EMAIL PROTECTED] / # free total used free sharedbuffers cached Mem: 1034792 886964 147828 0 266296 203008 -/+ buffers/cache: 417660 617132 Swap: 976712 32 976680 [EMAIL PROTECTED] / # I think it is using 32KBs. It's not much but it is using a little bit. Thing is, I still have a 147MBs of memory that is not in use at all. Go figure. Dale :-) :-) :-) -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Memory Usage
My server is mainly used for apache2 with mod_perl. I would think that cache comes in handy. Will a web server pretty much always find something more to cache, or can you add memory to the point where everything that can be cached is cached? - Grant I have read a few articles on how Linux manages memory. The thing I get from it is this, Linux likes to have data in memory first, then it uses swap because it is more organized. The last place it wants data is on the hard drive. It is all about speed of access. Linux likes to be really fast. I guess if you have 16GBs of ram and it only access say 10GBs of data, then it will eventually have everything in memory and not access the drive much at all. Same can be said for memory plus swap except that swap is on the drive of course. I will say this, I have 1GB of ram on mine. I run a full install of KDE and I have had 100 or more pictures open with Gimp and have never ran out of memory. The most swap I have ever used is about 200MBs and that is with it set to use a lot of swap. I think it was set at 80 or so. With Linux, 1GB is a lot of ram. The only exception being some heavy games or video stuff. If you have it, I would leave it unless you know you need it somewhere else. That's just my opinion and is subject to change. The thing is, it's memory in a hosted machine and I think I'm paying like $35/month for the extra gigabyte. I should probably do some testing or just have them remove the memory for a month and see how I like it. - Grant -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Memory Usage
Grant wrote: The thing is, it's memory in a hosted machine and I think I'm paying like $35/month for the extra gigabyte. I should probably do some testing or just have them remove the memory for a month and see how I like it. - Grant That does change things. In my opinion, I would do like you are planning and try it for a month or so and see how it does. Dale :-) :-) :-) -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Memory Usage
On Thu, 14 Dec 2006 19:08:35 -0600 Dale wrote: Grant wrote: My server is mainly used for apache2 with mod_perl. I would think that cache comes in handy. Will a web server pretty much always find something more to cache, or can you add memory to the point where everything that can be cached is cached? - Grant I have read a few articles on how Linux manages memory. The thing I get from it is this, Linux likes to have data in memory first, then it uses swap because it is more organized. The last place it wants data is on the hard drive. It is all about speed of access. Linux likes to be really fast. I guess if you have 16GBs of ram and it only access say 10GBs of data, then it will eventually have everything in memory and not access the drive much at all. Same can be said for memory plus swap except that swap is on the drive of course. I will say this, I have 1GB of ram on mine. I run a full install of KDE and I have had 100 or more pictures open with Gimp and have never ran out of memory. The most swap I have ever used is about 200MBs and that is with it set to use a lot of swap. I think it was set at 80 or so. With Linux, 1GB is a lot of ram. The only exception being some heavy games or video stuff. Gaming and video aren't the only reasons to need lots of memory. I upgraded my workstation from 512MB to 1GB so I could run BackupPC. Since BackupPC uses rsync to copy files from the internal HD to an external HD and since the machine has approx 3,000,000 files a lot of ram is used :- The additional ram is also useful for vmware. Just my $.02 David -- gentoo-user@gentoo.org mailing list
