Re: bdflush or others affecting disk cache
Because it's more efficient to use swap space to hold stuff from RAM that is not currently being used. But that means the Kernel is making the assumption it can cache the swap data more efficiently than just putting that data in RAM as the software requests? I'd take a whack at this and would think that cache misses would result in lower performance at this? Linux will happily shift process memory into swap to make more room for buffers. Why keep 100M worth of not-currently-active daemon in RAM when there is a process trying to buffer the whole disk? I agree... but the swap space usage is constantly changing... so I guess that means VM is making a poor decision as to what is not-currently-active... swapping out stuff that then needs to be read back or written, causing the disk thrashing? Wouldn't it be far, FAR faster for the system to reduce the cache by about 100Mb or so instead of swapping that 100Mb to disk? And note that the swap No. It is faster to use that memory for buffers if the system is being disk bound. Well, it is being disk bound because it is constantly using swap... causing it to be disk bound... causing the system to increase cache size... causing more swap usage... etc. Anyone see this before? - Original Message - From: Donovan Baarda [EMAIL PROTECTED] To: Jason Lim [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Monday, April 19, 2004 08:28 AM Subject: Re: bdflush or others affecting disk cache On Mon, 2004-04-19 at 09:31, Jason Lim wrote: Hi all, I've been banging my head on this one for a while now on a 2.4.20 system. [...] The problem is that swap usage can grow to 100Mb... yet the buffers and cache remain at astoundingly high levels. I can actually see memory to cache and buffers increasing and at the same time see it increasing swap usage! What I don't get is why the system... with about 700Mb in cache and 70Mb in buffers, is using swap space at all. [...] Because it's more efficient to use swap space to hold stuff from RAM that is not currently being used. Linux will happily shift process memory into swap to make more room for buffers. Why keep 100M worth of not-currently-active daemon in RAM when there is a process trying to buffer the whole disk? Wouldn't it be far, FAR faster for the system to reduce the cache by about 100Mb or so instead of swapping that 100Mb to disk? And note that the swap No. It is faster to use that memory for buffers if the system is being disk bound. usage is constantly fluctuating, so you can see the performance problem this is causing. Any ideas?! The VM management code in Linux is something that is constantly getting tweaked and re-written. 2.4.20 is quite old now, and it wouldn't surprise me if the current 2.4.26 kernel has had the VM significantly improved since then. The performance hits you are seeing are probably because a process is walking through the disk. The 2.4.20 VM system may not be handling it as gracefully as it could, but I bet there is a process doing heaps of disk reads that is triggering it. -- Donovan Baarda [EMAIL PROTECTED] http://minkirri.apana.org.au/~abo/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: bdflush or others affecting disk cache
Followup: interesting results. I've now tried removing the swap altogther (swapoff) and the server appears to be running much smoother and faster. Here is the new top info: 212 processes: 210 sleeping, 2 running, 0 zombie, 0 stopped CPU states: 8.4% user, 32.2% system, 0.9% nice, 58.2% idle Mem: 1027212K av, 1015440K used, 11772K free, 0K shrd, 186196K buff Swap: 0K av, 0K used, 0K free 370588K cached by the way, most of the processes are httpd and mysql (this is a hosting server). I'm somewhat concerned about having no swap though... any side-effects of running with no swap? As expected, most of the swapped data reverted to RAM by reducing the cache size (by approximately the amount that was used by swap). Hope someone can shed some light on this. I'm looking at the results, but can't understand why it is swapping so aggressively... to the point that it is running itself out of RAM for active programs to increase cache size. Jas - Original Message - From: Jason Lim [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Monday, 19 April, 2004 7:31 AM Subject: bdflush or others affecting disk cache Hi all, I've been banging my head on this one for a while now on a 2.4.20 system. Here is the output of top: Mem: 1027212K av, 1018600K used,8612K free, 0K shrd, 70728K buff Swap: 2097136K av, 35556K used, 2061580K free 690140K cached and the output of free: total used free sharedbuffers cached Mem: 10272121016256 10956 0 71528 683956 -/+ buffers/cache: 260772 766440 Swap: 2097136 346922062444 The problem is that swap usage can grow to 100Mb... yet the buffers and cache remain at astoundingly high levels. I can actually see memory to cache and buffers increasing and at the same time see it increasing swap usage! What I don't get is why the system... with about 700Mb in cache and 70Mb in buffers, is using swap space at all. I've searched high and low on Google... using phrases like linux kernel proc cache, buffers, bdflush, etc. but I still can't explain this. Wouldn't it be far, FAR faster for the system to reduce the cache by about 100Mb or so instead of swapping that 100Mb to disk? And note that the swap usage is constantly fluctuating, so you can see the performance problem this is causing. Any ideas?! Thanks in advance. Jas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: bdflush or others affecting disk cache
I'm going to have to disagree with the above poster. This VM behavior is not ideal and is really counter-productive. 2.4.x saw lot's of VM work to improve performance over broad ranges of work-load. The problems occur when changes are made for corner-cases and some more mainstream workloads suffer. anyway, not to belabor the point here, but 2.4 has seen almost constant improvement in VM (and scheduler as well). I didn't see performance improve to acceptable levels until about 2.4.23/24. You will want to upgrade your kernel to the latest (2.4.26 as I write this) and you should see a vast improvement in VM behavior. on your question of running w/o swap space I will answer: NOT ON YOUR LIFE! you should *never* run any kind of server w/o swap unless you don't mind processes randomly dying because OOM killer decides they should go for the sake of the system... so, for the sake of your sanity (and the security of your system) upgrade to 2.4.26 and re-enable swap! good luck, Dave On Mon, Apr 19, 2004 at 08:27:35PM +0800 or thereabouts, Jason Lim wrote: Followup: interesting results. I've now tried removing the swap altogther (swapoff) and the server appears to be running much smoother and faster. Here is the new top info: 212 processes: 210 sleeping, 2 running, 0 zombie, 0 stopped CPU states: 8.4% user, 32.2% system, 0.9% nice, 58.2% idle Mem: 1027212K av, 1015440K used, 11772K free, 0K shrd, 186196K buff Swap: 0K av, 0K used, 0K free 370588K cached by the way, most of the processes are httpd and mysql (this is a hosting server). I'm somewhat concerned about having no swap though... any side-effects of running with no swap? As expected, most of the swapped data reverted to RAM by reducing the cache size (by approximately the amount that was used by swap). Hope someone can shed some light on this. I'm looking at the results, but can't understand why it is swapping so aggressively... to the point that it is running itself out of RAM for active programs to increase cache size. Jas - Original Message - From: Jason Lim [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Monday, 19 April, 2004 7:31 AM Subject: bdflush or others affecting disk cache Hi all, I've been banging my head on this one for a while now on a 2.4.20 system. Here is the output of top: Mem: 1027212K av, 1018600K used,8612K free, 0K shrd, 70728K buff Swap: 2097136K av, 35556K used, 2061580K free 690140K cached and the output of free: total used free sharedbuffers cached Mem: 10272121016256 10956 0 71528 683956 -/+ buffers/cache: 260772 766440 Swap: 2097136 346922062444 The problem is that swap usage can grow to 100Mb... yet the buffers and cache remain at astoundingly high levels. I can actually see memory to cache and buffers increasing and at the same time see it increasing swap usage! What I don't get is why the system... with about 700Mb in cache and 70Mb in buffers, is using swap space at all. I've searched high and low on Google... using phrases like linux kernel proc cache, buffers, bdflush, etc. but I still can't explain this. Wouldn't it be far, FAR faster for the system to reduce the cache by about 100Mb or so instead of swapping that 100Mb to disk? And note that the swap usage is constantly fluctuating, so you can see the performance problem this is causing. Any ideas?! Thanks in advance. Jas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED] -- *** David Wilk System Administrator Community Internet Access, Inc. [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: bdflush or others affecting disk cache
On Tue, 2004-04-20 at 03:55, David Wilk wrote: I'm going to have to disagree with the above poster. This VM behavior is not ideal and is really counter-productive. 2.4.x saw lot's of VM work to improve performance over broad ranges of work-load. The problems occur when changes are made for corner-cases and some more mainstream workloads suffer. [...] In my defence, I did say that linux VM has a history VM fixes and rewrites in attempts to best tune behaviour. I did suggest that the problem he was seeing was the 2.4.20 VM not doing a good job. I totally agree that most of the linux VM problems have been caused by trying to be too smart. I'm not an expert, but every time I've read threads on it I've thought Man, haven't you guys ever heard of KISS. However, I think that it does make sense to swap out dead code to make room for buffer space. You just need to make sure it's done right :-) on your question of running w/o swap space I will answer: NOT ON YOUR LIFE! you should *never* run any kind of server w/o swap unless you don't mind processes randomly dying because OOM killer decides they should go for the sake of the system... Even in systems with with bucket-loads of RAM, adding swap can make your system faster. dead code can be swapped out, making room for more buffer space. (ducks to avoid argument about VM tuning :-) so, for the sake of your sanity (and the security of your system) upgrade to 2.4.26 and re-enable swap! Yeah. There are also security problems with 2.4.20 that have been fixed in 2.4.26 -- Donovan Baarda [EMAIL PROTECTED] http://minkirri.apana.org.au/~abo/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: bdflush or others affecting disk cache
Because it's more efficient to use swap space to hold stuff from RAM that is not currently being used. But that means the Kernel is making the assumption it can cache the swap data more efficiently than just putting that data in RAM as the software requests? I'd take a whack at this and would think that cache misses would result in lower performance at this? Linux will happily shift process memory into swap to make more room for buffers. Why keep 100M worth of not-currently-active daemon in RAM when there is a process trying to buffer the whole disk? I agree... but the swap space usage is constantly changing... so I guess that means VM is making a poor decision as to what is not-currently-active... swapping out stuff that then needs to be read back or written, causing the disk thrashing? Wouldn't it be far, FAR faster for the system to reduce the cache by about 100Mb or so instead of swapping that 100Mb to disk? And note that the swap No. It is faster to use that memory for buffers if the system is being disk bound. Well, it is being disk bound because it is constantly using swap... causing it to be disk bound... causing the system to increase cache size... causing more swap usage... etc. Anyone see this before? - Original Message - From: Donovan Baarda [EMAIL PROTECTED] To: Jason Lim [EMAIL PROTECTED] Cc: debian-isp@lists.debian.org Sent: Monday, April 19, 2004 08:28 AM Subject: Re: bdflush or others affecting disk cache On Mon, 2004-04-19 at 09:31, Jason Lim wrote: Hi all, I've been banging my head on this one for a while now on a 2.4.20 system. [...] The problem is that swap usage can grow to 100Mb... yet the buffers and cache remain at astoundingly high levels. I can actually see memory to cache and buffers increasing and at the same time see it increasing swap usage! What I don't get is why the system... with about 700Mb in cache and 70Mb in buffers, is using swap space at all. [...] Because it's more efficient to use swap space to hold stuff from RAM that is not currently being used. Linux will happily shift process memory into swap to make more room for buffers. Why keep 100M worth of not-currently-active daemon in RAM when there is a process trying to buffer the whole disk? Wouldn't it be far, FAR faster for the system to reduce the cache by about 100Mb or so instead of swapping that 100Mb to disk? And note that the swap No. It is faster to use that memory for buffers if the system is being disk bound. usage is constantly fluctuating, so you can see the performance problem this is causing. Any ideas?! The VM management code in Linux is something that is constantly getting tweaked and re-written. 2.4.20 is quite old now, and it wouldn't surprise me if the current 2.4.26 kernel has had the VM significantly improved since then. The performance hits you are seeing are probably because a process is walking through the disk. The 2.4.20 VM system may not be handling it as gracefully as it could, but I bet there is a process doing heaps of disk reads that is triggering it. -- Donovan Baarda [EMAIL PROTECTED] http://minkirri.apana.org.au/~abo/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: bdflush or others affecting disk cache
Followup: interesting results. I've now tried removing the swap altogther (swapoff) and the server appears to be running much smoother and faster. Here is the new top info: 212 processes: 210 sleeping, 2 running, 0 zombie, 0 stopped CPU states: 8.4% user, 32.2% system, 0.9% nice, 58.2% idle Mem: 1027212K av, 1015440K used, 11772K free, 0K shrd, 186196K buff Swap: 0K av, 0K used, 0K free 370588K cached by the way, most of the processes are httpd and mysql (this is a hosting server). I'm somewhat concerned about having no swap though... any side-effects of running with no swap? As expected, most of the swapped data reverted to RAM by reducing the cache size (by approximately the amount that was used by swap). Hope someone can shed some light on this. I'm looking at the results, but can't understand why it is swapping so aggressively... to the point that it is running itself out of RAM for active programs to increase cache size. Jas - Original Message - From: Jason Lim [EMAIL PROTECTED] To: debian-isp@lists.debian.org Sent: Monday, 19 April, 2004 7:31 AM Subject: bdflush or others affecting disk cache Hi all, I've been banging my head on this one for a while now on a 2.4.20 system. Here is the output of top: Mem: 1027212K av, 1018600K used,8612K free, 0K shrd, 70728K buff Swap: 2097136K av, 35556K used, 2061580K free 690140K cached and the output of free: total used free sharedbuffers cached Mem: 10272121016256 10956 0 71528 683956 -/+ buffers/cache: 260772 766440 Swap: 2097136 346922062444 The problem is that swap usage can grow to 100Mb... yet the buffers and cache remain at astoundingly high levels. I can actually see memory to cache and buffers increasing and at the same time see it increasing swap usage! What I don't get is why the system... with about 700Mb in cache and 70Mb in buffers, is using swap space at all. I've searched high and low on Google... using phrases like linux kernel proc cache, buffers, bdflush, etc. but I still can't explain this. Wouldn't it be far, FAR faster for the system to reduce the cache by about 100Mb or so instead of swapping that 100Mb to disk? And note that the swap usage is constantly fluctuating, so you can see the performance problem this is causing. Any ideas?! Thanks in advance. Jas
Re: bdflush or others affecting disk cache
I'm going to have to disagree with the above poster. This VM behavior is not ideal and is really counter-productive. 2.4.x saw lot's of VM work to improve performance over broad ranges of work-load. The problems occur when changes are made for corner-cases and some more mainstream workloads suffer. anyway, not to belabor the point here, but 2.4 has seen almost constant improvement in VM (and scheduler as well). I didn't see performance improve to acceptable levels until about 2.4.23/24. You will want to upgrade your kernel to the latest (2.4.26 as I write this) and you should see a vast improvement in VM behavior. on your question of running w/o swap space I will answer: NOT ON YOUR LIFE! you should *never* run any kind of server w/o swap unless you don't mind processes randomly dying because OOM killer decides they should go for the sake of the system... so, for the sake of your sanity (and the security of your system) upgrade to 2.4.26 and re-enable swap! good luck, Dave On Mon, Apr 19, 2004 at 08:27:35PM +0800 or thereabouts, Jason Lim wrote: Followup: interesting results. I've now tried removing the swap altogther (swapoff) and the server appears to be running much smoother and faster. Here is the new top info: 212 processes: 210 sleeping, 2 running, 0 zombie, 0 stopped CPU states: 8.4% user, 32.2% system, 0.9% nice, 58.2% idle Mem: 1027212K av, 1015440K used, 11772K free, 0K shrd, 186196K buff Swap: 0K av, 0K used, 0K free 370588K cached by the way, most of the processes are httpd and mysql (this is a hosting server). I'm somewhat concerned about having no swap though... any side-effects of running with no swap? As expected, most of the swapped data reverted to RAM by reducing the cache size (by approximately the amount that was used by swap). Hope someone can shed some light on this. I'm looking at the results, but can't understand why it is swapping so aggressively... to the point that it is running itself out of RAM for active programs to increase cache size. Jas - Original Message - From: Jason Lim [EMAIL PROTECTED] To: debian-isp@lists.debian.org Sent: Monday, 19 April, 2004 7:31 AM Subject: bdflush or others affecting disk cache Hi all, I've been banging my head on this one for a while now on a 2.4.20 system. Here is the output of top: Mem: 1027212K av, 1018600K used,8612K free, 0K shrd, 70728K buff Swap: 2097136K av, 35556K used, 2061580K free 690140K cached and the output of free: total used free sharedbuffers cached Mem: 10272121016256 10956 0 71528 683956 -/+ buffers/cache: 260772 766440 Swap: 2097136 346922062444 The problem is that swap usage can grow to 100Mb... yet the buffers and cache remain at astoundingly high levels. I can actually see memory to cache and buffers increasing and at the same time see it increasing swap usage! What I don't get is why the system... with about 700Mb in cache and 70Mb in buffers, is using swap space at all. I've searched high and low on Google... using phrases like linux kernel proc cache, buffers, bdflush, etc. but I still can't explain this. Wouldn't it be far, FAR faster for the system to reduce the cache by about 100Mb or so instead of swapping that 100Mb to disk? And note that the swap usage is constantly fluctuating, so you can see the performance problem this is causing. Any ideas?! Thanks in advance. Jas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED] -- *** David Wilk System Administrator Community Internet Access, Inc. [EMAIL PROTECTED]
Re: bdflush or others affecting disk cache
On Mon, 2004-04-19 at 09:31, Jason Lim wrote: Hi all, I've been banging my head on this one for a while now on a 2.4.20 system. [...] The problem is that swap usage can grow to 100Mb... yet the buffers and cache remain at astoundingly high levels. I can actually see memory to cache and buffers increasing and at the same time see it increasing swap usage! What I don't get is why the system... with about 700Mb in cache and 70Mb in buffers, is using swap space at all. [...] Because it's more efficient to use swap space to hold stuff from RAM that is not currently being used. Linux will happily shift process memory into swap to make more room for buffers. Why keep 100M worth of not-currently-active daemon in RAM when there is a process trying to buffer the whole disk? Wouldn't it be far, FAR faster for the system to reduce the cache by about 100Mb or so instead of swapping that 100Mb to disk? And note that the swap No. It is faster to use that memory for buffers if the system is being disk bound. usage is constantly fluctuating, so you can see the performance problem this is causing. Any ideas?! The VM management code in Linux is something that is constantly getting tweaked and re-written. 2.4.20 is quite old now, and it wouldn't surprise me if the current 2.4.26 kernel has had the VM significantly improved since then. The performance hits you are seeing are probably because a process is walking through the disk. The 2.4.20 VM system may not be handling it as gracefully as it could, but I bet there is a process doing heaps of disk reads that is triggering it. -- Donovan Baarda [EMAIL PROTECTED] http://minkirri.apana.org.au/~abo/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
