Re: Swap? (was Re: [CinCVS] constant hangs on HDV project)
On 2/13/07, Kevin Brosius [EMAIL PROTECTED] wrote: On 2007-02-12 20:30, Dan Streetman wrote: On 2/11/07, Kevin Brosius [EMAIL PROTECTED] wrote: On 2007-02-05 20:24, Dan Streetman wrote: On 2/5/07, Nicolas Maufrais [EMAIL PROTECTED] wrote: Heroine Virtual Ltd recommends disabling swap when a lot of memory is installed. This is not a good idea under any circumstances. Disabling swap will never help anything or improve performance, ever. It will only cause the system to start killing processes when you use up all system memory. Do not disable swap. Hi Dan, This will only be true for kernels with the OOM killer running, right? It's irrelevant; whether you have it or not, if you have swap disabled and run out of real memory, then real bad things will happen. The only reason I know this is because I used to be able to build an xfree86 source tree, and a second compile could sometimes run without disk access. That tree was on the order of 100M at the time. I could imagine that disk cache might have been weighted poorly in the paging algorithm at some point. If that was half the mem, then the swap comments above might be related to that. Remember, we are talking about Cin usage, which would be disk intensive for render writes/reads, along with page hungry for effects processing. The mix may be different for single app usage than the test case you suggest below. The disk caching is interesting, because with the advent of reiser, and slightly newer kernels, I noticed that disk caching was nothing like ext2. It would not cache an entire 100M source code true when accessed with reiser, although that may have been an artifact of slightly newer kernels. I do not recall if I tested that or not. So, you think disk caching has no impact? Possible, I haven't tried testing on a 4G machine. Disc caching is irrelevant to swapping. Linux will use _all_ free (real) memory for disc caching, and reduce the cache as programs request more and more memory to actually use. When the actual memory used by system programs approaches real memory limits, the cached memory will approach 0. How much sense does it make to swap out disc cache? ;-) Maybe you would like to try it yourself like I suggested? I'll make it easy, I've attached a very small trivial program that will quickly eat up all your system memory. Run it first with swap enabled and then without. Compile with gcc -o memory-eater memory-eater.c If you want to bring disc caching into it, do: dd if=/dev/zero of=/tmp/big-file bs=1024k count=4k ; memory-eater note the 4k is 4G of ram, if you have less just use a smaller number, as the disc cache obviously can't exceed real system memory :) I also attached 4 images from when I did this on my system, which hopefully will show you that what I am saying is true; swapping does not happen until real system memory is used up, and disc caching is completely irrelevant to swapping. (#1 is after disc cache reaches real system memory - no swap used, #2 is when real used memory reaches/slightly exceeds real system memory - no cache, small swap, #3 is showing continued swap usage - no cache, #4 is after killing the program) And a final note, when you run that program with swap disabled, the kernel's OOM killer will probably kill only that program. You can disable the OOM killer (at least, man malloc says you can...) by doing echo 2 /proc/sys/vm/overcommit_memory, so if you want you can try it without swap _and_ without the OOM killer I'd be interested to hear what happens there! ;-) memory-eater.c Description: Binary data memory-eater1.gif Description: GIF image memory-eater2.gif Description: GIF image memory-eater3.gif Description: GIF image memory-eater4.gif Description: GIF image
Re: Swap? (was Re: [CinCVS] constant hangs on HDV project)
There is some understatement in that word probably. I ran into the OOM Killer recently on a 2.4 kernel (unrelated to cinelerra), and found that when you're out of virtual memory pages, the next process which needs a page is the one killed. It will be cinelerra if you're lucky. If not, it might be X... or worse. As the man said, ... real bad things will happen. Dan Streetman wrote: And a final note, when you run that program with swap disabled, the kernel's OOM killer will probably kill only that program. ... ___ Cinelerra mailing list Cinelerra@skolelinux.no https://init.linpro.no/mailman/skolelinux.no/listinfo/cinelerra
Re: Swap? (was Re: [CinCVS] constant hangs on HDV project)
On 2007-02-12 20:30, Dan Streetman wrote: On 2/11/07, Kevin Brosius [EMAIL PROTECTED] wrote: On 2007-02-05 20:24, Dan Streetman wrote: On 2/5/07, Nicolas Maufrais [EMAIL PROTECTED] wrote: Heroine Virtual Ltd recommends disabling swap when a lot of memory is installed. This is not a good idea under any circumstances. Disabling swap will never help anything or improve performance, ever. It will only cause the system to start killing processes when you use up all system memory. Do not disable swap. Hi Dan, This will only be true for kernels with the OOM killer running, right? Is running the right adjective? I think the OOM killer is part of the memory allocation subsystem, not a kernel process, right? So ever since it was added to the kernel it is there. I don't think you can build a kernel without it these days, can you? But if you mean kernels before the OOM killer was added, sure...that was a looong time ago tho. Well, I thought it could be disabled. There's been discussion of that on lkml a few times. But I'm not up to date. But that's beside the point. If you run out of memory, either the kernel kills picks a single program to kill and let everything else in the system live (i.e. with OOM killer), or _everything_ on the system dies a slow and painful death (i.e. without OOM killer). Here's the link to comments from Cinelerra's author about swap usage: http://cvs.cinelerra.org/docs/wiki/doku.php?id=english_manual:cinelerra_cv_en_20 I've seen system behavior that makes me believe his comments about disk caching using system memory (back in ext2 days.) Maybe this is no longer true for newer kernels. But I suspect your 'never help' comment might be overzealous. ;) Would you say newer kernels with current versions of filesystems no longer exhibit this behavior? Or is something else going on here? Ok, maybe never help is a bit extreme... ;-) Recent kernels do not start swapping pages out when memory is 1/2 full (as stated in that FAQ). I'm typing this from a system with 1G of memory, 900M of which is in use right now, and swap is 588K. I've been using Linux (Slackware) since 1997, and I don't ever remember that behavior. But hey, I can't say for sure that it has never been true. There very well may be some very old kernels that swap overzealously like that...I can only say I've never seen it in ~10 years of Linux use. 1997... yeah, about the same here. ext2 used to heavily cache files. The only reason I know this is because I used to be able to build an xfree86 source tree, and a second compile could sometimes run without disk access. That tree was on the order of 100M at the time. I could imagine that disk cache might have been weighted poorly in the paging algorithm at some point. If that was half the mem, then the swap comments above might be related to that. Remember, we are talking about Cin usage, which would be disk intensive for render writes/reads, along with page hungry for effects processing. The mix may be different for single app usage than the test case you suggest below. The disk caching is interesting, because with the advent of reiser, and slightly newer kernels, I noticed that disk caching was nothing like ext2. It would not cache an entire 100M source code true when accessed with reiser, although that may have been an artifact of slightly newer kernels. I do not recall if I tested that or not. I would recommend 2 things, first get a program that constantly monitors system stats (I run xosview 100% of the time and have for many years...xosview.sf.net). Then second, (with swap enabled) start up cinelerra and/or some other large programs and start eating up memory. See at what point swapping starts and how much memory is swapped. See what happens when program memory requirements exceed system memory, i.e. used system memory + swap used exceed total system memory. Then shut everything down and disable swap, and start up those programs again and eat up more than system memory and see what happens (hint: don't be doing anything important when you try this :-). Until system memory ran out, was the no-swap case any better than the swap case? Was swap significantly used before system memory usage run up to near 100%? What happened when you ran out of memory and had no swap? So, you think disk caching has no impact? Possible, I haven't tried testing on a 4G machine. I suspect everyone will find that the system behaves identically until memory usage gets very close to total available system memory, and when memory usage exceeds available system memory the swap/no-swap cases are very different. But if the test shows no-swap is better for you, by all means don't use swap... -- Kevin ___ Cinelerra mailing list Cinelerra@skolelinux.no https://init.linpro.no/mailman/skolelinux.no/listinfo/cinelerra
Re: Swap? (was Re: [CinCVS] constant hangs on HDV project)
On 2/11/07, Kevin Brosius [EMAIL PROTECTED] wrote: On 2007-02-05 20:24, Dan Streetman wrote: On 2/5/07, Nicolas Maufrais [EMAIL PROTECTED] wrote: Heroine Virtual Ltd recommends disabling swap when a lot of memory is installed. This is not a good idea under any circumstances. Disabling swap will never help anything or improve performance, ever. It will only cause the system to start killing processes when you use up all system memory. Do not disable swap. Hi Dan, This will only be true for kernels with the OOM killer running, right? Is running the right adjective? I think the OOM killer is part of the memory allocation subsystem, not a kernel process, right? So ever since it was added to the kernel it is there. I don't think you can build a kernel without it these days, can you? But if you mean kernels before the OOM killer was added, sure...that was a looong time ago tho. But that's beside the point. If you run out of memory, either the kernel kills picks a single program to kill and let everything else in the system live (i.e. with OOM killer), or _everything_ on the system dies a slow and painful death (i.e. without OOM killer). Here's the link to comments from Cinelerra's author about swap usage: http://cvs.cinelerra.org/docs/wiki/doku.php?id=english_manual:cinelerra_cv_en_20#disabling_swap_space I've seen system behavior that makes me believe his comments about disk caching using system memory (back in ext2 days.) Maybe this is no longer true for newer kernels. But I suspect your 'never help' comment might be overzealous. ;) Would you say newer kernels with current versions of filesystems no longer exhibit this behavior? Or is something else going on here? Ok, maybe never help is a bit extreme... ;-) Recent kernels do not start swapping pages out when memory is 1/2 full (as stated in that FAQ). I'm typing this from a system with 1G of memory, 900M of which is in use right now, and swap is 588K. I've been using Linux (Slackware) since 1997, and I don't ever remember that behavior. But hey, I can't say for sure that it has never been true. There very well may be some very old kernels that swap overzealously like that...I can only say I've never seen it in ~10 years of Linux use. I would recommend 2 things, first get a program that constantly monitors system stats (I run xosview 100% of the time and have for many years...xosview.sf.net). Then second, (with swap enabled) start up cinelerra and/or some other large programs and start eating up memory. See at what point swapping starts and how much memory is swapped. See what happens when program memory requirements exceed system memory, i.e. used system memory + swap used exceed total system memory. Then shut everything down and disable swap, and start up those programs again and eat up more than system memory and see what happens (hint: don't be doing anything important when you try this :-). Until system memory ran out, was the no-swap case any better than the swap case? Was swap significantly used before system memory usage run up to near 100%? What happened when you ran out of memory and had no swap? I suspect everyone will find that the system behaves identically until memory usage gets very close to total available system memory, and when memory usage exceeds available system memory the swap/no-swap cases are very different. But if the test shows no-swap is better for you, by all means don't use swap... ___ Cinelerra mailing list Cinelerra@skolelinux.no https://init.linpro.no/mailman/skolelinux.no/listinfo/cinelerra
Swap? (was Re: [CinCVS] constant hangs on HDV project)
On 2007-02-05 20:24, Dan Streetman wrote: On 2/5/07, Nicolas Maufrais [EMAIL PROTECTED] wrote: Heroine Virtual Ltd recommends disabling swap when a lot of memory is installed. This is not a good idea under any circumstances. Disabling swap will never help anything or improve performance, ever. It will only cause the system to start killing processes when you use up all system memory. Do not disable swap. Hi Dan, This will only be true for kernels with the OOM killer running, right? Here's the link to comments from Cinelerra's author about swap usage: http://cvs.cinelerra.org/docs/wiki/doku.php?id=english_manual:cinelerra_cv_en_20#disabling_swap_space I've seen system behavior that makes me believe his comments about disk caching using system memory (back in ext2 days.) Maybe this is no longer true for newer kernels. But I suspect your 'never help' comment might be overzealous. ;) Would you say newer kernels with current versions of filesystems no longer exhibit this behavior? Or is something else going on here? Thanks, -- Kevin ___ Cinelerra mailing list Cinelerra@skolelinux.no https://init.linpro.no/mailman/skolelinux.no/listinfo/cinelerra
