Re: VM Report was:Re: Break 2.4 VM in five easy steps
Mike Galbraith <[EMAIL PROTECTED]> writes: > On Fri, 8 Jun 2001, John Stoffel wrote: > > > Mike> OK, riddle me this. If this test is a crummy test, just how is > > Mike> it that I was able to warn Rik in advance that when 2.4.5 was > > Mike> released, he should expect complaints? How did I _know_ that? > > Mike> The answer is that I fiddle with Rik's code a lot, and I test > > Mike> with this test because it tells me a lot. It may not tell you > > Mike> anything, but it does me. > > > > I never said it was a crummy test, please do not read more into my > > words than was written. What I was trying to get across is that just > > one test (such as a compile of the kernel) isn't perfect at showing > > where the problems are with the VM sub-system. > > Hmm... > > Tobias> Could you please explain what is good about this test? I > Tobias> understand that it will stress the VM, but will it do so in a > Tobias> realistic and relevant way? > > I agree, this isn't really a good test case. I'd rather see what > > happens when you fire up a gimp session to edit an image which is > *almost* the size of RAM, or even just 50% the size of ram. Then how > does that affect your other processes that are running at the same > time? > > ...but anyway, yes it just one test from any number of possibles. One great test that I'm using regularly to see what's goin' on, is at http://lxr.linux.no/. It is a cool utility to cross reference your Linux kernel source tree, and in the mean time eat gobs of memory, do lots of I/O, and burn many CPU cycles (all at the same time). Ideal test, if you ask me and if anybody has the time, it would be nice to see different timing numbers when run on different kernels. Just make sure you run it on the same kernel tree to make reproducable results. It has three passes, and the third one is the most interesting one (use vmstat 1 to see why). When run with 64MB RAM configuration, it would swap heavily, with 128MB somewhat, and at 192MB maybe not (depending on the other applications running at the same time). Try it, it is a nice utility, and a great test. :) -- Zlatko - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
> reads the RTC device. The patched RTC driver can then > measure the elapsed time between the interrupt and the > read from userspace. Voila: latency. interesting, but I'm not sure there's much advantage over doing it entirely in user-space with the normal /dev/rtc: http://brain.mcmaster.ca/~hahn/realfeel.c it just prints out the raw time difference from when rtc should have woken up the program. you can do your own histogram; for summary purposes, something like stdev is probably best. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
reads the RTC device. The patched RTC driver can then measure the elapsed time between the interrupt and the read from userspace. Voila: latency. interesting, but I'm not sure there's much advantage over doing it entirely in user-space with the normal /dev/rtc: http://brain.mcmaster.ca/~hahn/realfeel.c it just prints out the raw time difference from when rtc should have woken up the program. you can do your own histogram; for summary purposes, something like stdev is probably best. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
Mike Galbraith [EMAIL PROTECTED] writes: On Fri, 8 Jun 2001, John Stoffel wrote: Mike OK, riddle me this. If this test is a crummy test, just how is Mike it that I was able to warn Rik in advance that when 2.4.5 was Mike released, he should expect complaints? How did I _know_ that? Mike The answer is that I fiddle with Rik's code a lot, and I test Mike with this test because it tells me a lot. It may not tell you Mike anything, but it does me. I never said it was a crummy test, please do not read more into my words than was written. What I was trying to get across is that just one test (such as a compile of the kernel) isn't perfect at showing where the problems are with the VM sub-system. Hmm... Tobias Could you please explain what is good about this test? I Tobias understand that it will stress the VM, but will it do so in a Tobias realistic and relevant way? I agree, this isn't really a good test case. I'd rather see what happens when you fire up a gimp session to edit an image which is *almost* the size of RAM, or even just 50% the size of ram. Then how does that affect your other processes that are running at the same time? ...but anyway, yes it just one test from any number of possibles. One great test that I'm using regularly to see what's goin' on, is at http://lxr.linux.no/. It is a cool utility to cross reference your Linux kernel source tree, and in the mean time eat gobs of memory, do lots of I/O, and burn many CPU cycles (all at the same time). Ideal test, if you ask me and if anybody has the time, it would be nice to see different timing numbers when run on different kernels. Just make sure you run it on the same kernel tree to make reproducable results. It has three passes, and the third one is the most interesting one (use vmstat 1 to see why). When run with 64MB RAM configuration, it would swap heavily, with 128MB somewhat, and at 192MB maybe not (depending on the other applications running at the same time). Try it, it is a nice utility, and a great test. :) -- Zlatko - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Sat, 9 Jun 2001, Jonathan Morton wrote: > >> On the subject of Mike Galbraith's kernel compilation test, how much > >> physical RAM does he have for his machine, what type of CPU is it, and what > >> (approximate) type of device does he use for swap? I'll see if I can > >> partially duplicate his results at this end. So far all my tests have been > >> done with a fast CPU - perhaps I should try the P166/MMX or even try > >> loading linux-pmac onto my 8100. > > > >It's a PIII/500 with one ide disk. > > ...with how much RAM? That's the important bit. Duh! :) I'm a dipstick. 128mb. -Mike - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, Marcelo Tosatti wrote: > On Fri, 8 Jun 2001, John Stoffel wrote: > > > More importantly, a *repeatable* set of tests is what is needed to > > test the VM and get consistent results from run to run, so you can see > > how your changes are impacting performance. The kernel compile > > doesn't really have any one process grow to a large fraction of > > memory, so dropping in a compile which *does* is a good thing. > > I agree with you. > > Mike, I'm sure you have noticed that stock kernel gives much better > results than mine or Jonathan's patch. I noticed that Jonathan brought back waiting.. that (among others) made me very interested. > Now the stock kernel gives us crappy interactivity compared to my patch. > (Note: my patch still does not gives me the interactivity I want under > high VM loads, but I hope to get there soon). (And that's why) Among other things (yes, I do love throughput) I've poked at the interactivity problem. I can't improve it anymore without doing some strategic waiting :( I used to be able to help it a little by doing a careful roll-up in scrub size as load builds.. trying to smooth the transition from latency oriented to hammer down throughput. > BTW, we are talking with the OSDL (http://www.osdlab.org) guys about a > possibility to setup a test system which would run a different variety of > benchmarks to give us results of different kinds of workloads. If that > ever happens, we'll probably get rid of most of this testing problems. Excellent! -Mike - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, Tobias Ringstrom wrote: > On Fri, 8 Jun 2001, Mike Galbraith wrote: > > On Fri, 8 Jun 2001, Tobias Ringstrom wrote: > > > On Fri, 8 Jun 2001, Mike Galbraith wrote: > > > > I gave this a shot at my favorite vm beater test (make -j30 bzImage) > > > > while testing some other stuff today. > > > > > > Could you please explain what is good about this test? I understand that > > > it will stress the VM, but will it do so in a realistic and relevant way? > > > > Can you explain what is bad about this test? ;) It spins the same VM wheels > > I think a load of ~30 is quit uncommon, and therefor it is unclear to me > that it would be a test that would be repesentative of most normal loads. It's not supposed to be repesentative. It's supposed to take the box rapidly (but not instantly) from idle through lo->medium->high and maintain solid throughput. > > as any other load does. What's the difference if I have a bunch of httpd > > allocating or a bunch of cc1/as/ld? This load has a modest cachable data > > set and is compute bound.. and above all gives very repeatable results. > > Not a big difference. The difference I was thinking abount is the > difference between spawning lots of processes allocating, using and > freeing lots of memory, compared to a case where you have a few processes > touching a lot of already allocated pages in some pattern. I was > wondering whether optimizing for your case would be good or bad for the > other case. I know, I know, I should do more testing myself. And I > should probably not ask you, since you really really like your test, > and you will probably just say yes... ;-) It's not a matter of optimizing for my case.. that would be horrible. It's a matter of is the vm capable of rapid and correct responses. > At home, I'm running a couple of computers. One of them is a slow > computer running Linux, serving mail, NFS, SMB, etc. I'm usually logged > in on a couple of virtual consoles. On this machine, I do not mind if all > shells, daemons and other idle processes are beeing swapped out in favor > of disk cache for the NFS and SMB serving. In fact, that is a very good > thing, and I want it that way. > > Another maching is my desktop machine. When using this maching, I really > hate when my emacsen, browsers, xterms, etc are swapped out just to give > me some stupid disk cache for my xmms or compilations. I do not care if a > kernel compile is a little slower as long as my applications are snappy. > > How could Linux predict this? It is a matter of taste, IMHO. I have no idea. It would be _wonderful_ if it could detect interactive tasks and give them preferencial treatment. > > I use it to watch reaction to surge. I watch for the vm to build to a > > solid maximum throughput without thrashing. That's the portion of VM > > that I'm interested in, so that's what I test. Besides :) I simply don't > > have the hardware to try to simulate hairy chested server loads. There > > are lots of folks with hairy chested boxes.. they should test that stuff. > > Agreed. More testing is needed. Now if we would have those knobs and > wheels to turn, we could perhaps also tune our systems to behave as we > like them, and submit that as well. Right now you need to be a kernel > hacker, and see through all the magic with shm, mmap, a bunch of caches, > page lists, etc. I'd give a lot for a nice picture (or state diagram) > showing the lifetime of a page, but I have not found such a picture > anywhere. Besides, the VM seems to change every new release anyway. > > > I've been repeating ~this test since 2.0 times, and have noticed a 1:1 > > relationship. When I notice that my box is ~happy doing this load test, > > I also notice very few VM gripes hitting the list. > > Ok, but as you say, we need more tests. > > > > Isn't the interesting case when you have a number of processes using lots > > > of memory, but only a part of all that memory is beeing actively used, and > > > that memory fits in RAM. In that case, the VM should make sure that the > > > not used memory is swapped out. In RAM you should have the used memory, > > > but also disk cache if there is any RAM left. Does the current VM handle > > > this case fine yet? IMHO, this is the case most people care about. It is > > > definately the case I care about, at least. :-) > > > > The interesting case is _every_ case. Try seeing my particular test as > > a simulation of a small classroom box with 30 students compiling their > > assignments and it'll suddenly become quite realistic. You'll notice > > by the numbers I post that I was very careful to not overload the box in > > a rediculous manner when selecting the total size of the job.. it's just > > a heavily loaded box. This test does not overload my IO resources, so > > it tests the VM's ability to choose and move the right stuff at the right > > time to get the job done with a minimum of additional overhead. > > I did not understand
Re: VM Report was:Re: Break 2.4 VM in five easy steps
>> On the subject of Mike Galbraith's kernel compilation test, how much >> physical RAM does he have for his machine, what type of CPU is it, and what >> (approximate) type of device does he use for swap? I'll see if I can >> partially duplicate his results at this end. So far all my tests have been >> done with a fast CPU - perhaps I should try the P166/MMX or even try >> loading linux-pmac onto my 8100. > >It's a PIII/500 with one ide disk. ...with how much RAM? That's the important bit. -- from: Jonathan "Chromatix" Morton mail: [EMAIL PROTECTED] (not for attachments) The key to knowledge is not to rely on people to teach you it. GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*) - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Sat, 9 Jun 2001, Jonathan Morton wrote: > On the subject of Mike Galbraith's kernel compilation test, how much > physical RAM does he have for his machine, what type of CPU is it, and what > (approximate) type of device does he use for swap? I'll see if I can > partially duplicate his results at this end. So far all my tests have been > done with a fast CPU - perhaps I should try the P166/MMX or even try > loading linux-pmac onto my 8100. It's a PIII/500 with one ide disk. -Mike - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, Mike Galbraith wrote: > On Fri, 8 Jun 2001, John Stoffel wrote: > > I agree, this isn't really a good test case. I'd rather see what > > happens when you fire up a gimp session to edit an image which is > > *almost* the size of RAM, or even just 50% the size of ram. > > OK, riddle me this. If this test is a crummy test, just how is it Personally, I'd like to see BOTH of these tests, and many many more. Preferably, handed to the VM hackers in various colourful graphs that allow even severely undercaffeinated hackers to see how things changed for the good or the bad between kernel revisions. cheers, Rik -- Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose... http://www.surriel.com/ http://distro.conectiva.com/ Send all your spam to [EMAIL PROTECTED] (spam digging piggy) - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
Jonathan Morton wrote: > > [ Re-entering discussion after too long a day and a long sleep... ] > > >> There is the problem in terms of some people want pure interactive > >> performance, while others are looking for throughput over all else, > >> but those are both extremes of the spectrum. Though I suspect > >> raw throughput is the less wanted (in terms of numbers of systems) > >> than keeping interactive response good during VM pressure. > > > >And this raises a very very important point: raw throughtput wins > >enterprise-like benchmarks, and the enterprise people are the ones who pay > >most of hackers here. (including me and Rik) > > Very true. As well as the fact that interactivity is much harder to > measure. The question is, what is interactivity (from the kernel's > perspective)? It usually means small(ish) processes with intermittent > working-set and CPU requirements. These types of process can safely be > swapped out when not immediately in use, but the kernel has to be able to > page them in quite quickly when needed. Doing that under heavy load is > very non-trivial. For the low-latency stuff, latency can be defined as the worst-case time to schedule a userspace process in response to an interrupt. That metric is also appropriate in this case, (latency equals interactivity), although here you don't need to be so fanatical about the *worst case*. A few scheduling blips here are less fatal. I have tools to measure latency (aka interactivity). At http://www.uow.edu.au/~andrewm/linux/schedlat.html#downloads there is a kernel patch called `rtc-debug' which causes the PC RTC to generate a stream of interrupts. A user-space task called `amlat' responds to those interrupts and reads the RTC device. The patched RTC driver can then measure the elapsed time between the interrupt and the read from userspace. Voila: latency. When you close the RTC device (by killing amlat), the RTC driver will print out a histogram of the latencies. `amlat' at present gives itself SCHED_RR policy and runs under mlockall() - for your testing you'll need to delete those lines. So. Simple apply rtc-debug, run `amlat' and kill it when you've finished the workload. The challenge will be to relate the latency histogram to human-perceived interactivity. I'm not sure of the best way of doing that. Perhaps monitor the 90th percentile, and aim to keep it below 100 milliseconds. Also, `amlat' should do a bit of disk I/O as well. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
[ Re-entering discussion after too long a day and a long sleep... ] >> There is the problem in terms of some people want pure interactive >> performance, while others are looking for throughput over all else, >> but those are both extremes of the spectrum. Though I suspect >> raw throughput is the less wanted (in terms of numbers of systems) >> than keeping interactive response good during VM pressure. > >And this raises a very very important point: raw throughtput wins >enterprise-like benchmarks, and the enterprise people are the ones who pay >most of hackers here. (including me and Rik) Very true. As well as the fact that interactivity is much harder to measure. The question is, what is interactivity (from the kernel's perspective)? It usually means small(ish) processes with intermittent working-set and CPU requirements. These types of process can safely be swapped out when not immediately in use, but the kernel has to be able to page them in quite quickly when needed. Doing that under heavy load is very non-trivial. It can also mean multimedia applications with a continuous (maybe small) working set, a continuous but not 100% CPU usage, and the special property that the user WILL notice if this process gets swapped out even briefly. mpg123 and XMMS fall into this category, and I sometimes tried running these alongside my compilation tests to see how they fared. I think I had it going fairly well towards the end, with mpg123 stuttering relatively rarely and briefly while VM load was high. On the subject of Mike Galbraith's kernel compilation test, how much physical RAM does he have for his machine, what type of CPU is it, and what (approximate) type of device does he use for swap? I'll see if I can partially duplicate his results at this end. So far all my tests have been done with a fast CPU - perhaps I should try the P166/MMX or even try loading linux-pmac onto my 8100. -- from: Jonathan "Chromatix" Morton mail: [EMAIL PROTECTED] (not for attachments) The key to knowledge is not to rely on people to teach you it. GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*) - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, John Stoffel wrote: > > Marcelo> Now the stock kernel gives us crappy interactivity compared > Marcelo> to my patch. (Note: my patch still does not gives me the > Marcelo> interactivity I want under high VM loads, but I hope to get > Marcelo> there soon). > > This raises the important question, how can we objectively measure > interactive response in the kernel and relate it to the user's > perceived interactive response? If we could come up with some sort of > testing system that would show us this, it would help alot, since we > could just have people run tests in a more automatic and repeatable > manner. > > And I think it would also help us automatically tune the Kernel, since > it would have a knowledge of it's own performance. > > There is the problem in terms of some people want pure interactive > performance, while others are looking for throughput over all else, > but those are both extremes of the spectrum. Though I suspect > raw throughput is the less wanted (in terms of numbers of systems) > than keeping interactive response good during VM pressure. And this raises a very very important point: raw throughtput wins enterprise-like benchmarks, and the enterprise people are the ones who pay most of hackers here. (including me and Rik) We have to be careful about that. > I have zero knowledge of how we could do this, but giving the kernel > some counters, even if only for use during debugging runs, which would > give us some objective feedback on performance would be a big win. > > Having people just send in reports of "I ran X,Y,Z and it was slow" > doesn't help us, since it's so hard to re-create their environment so > you can run tests against it. Lets wait for some test system to be set up (eg the OSDL thing). Once thats done, I'm sure we will find out some way of doing it. Well, good weekend for you too. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
Marcelo> Now the stock kernel gives us crappy interactivity compared Marcelo> to my patch. (Note: my patch still does not gives me the Marcelo> interactivity I want under high VM loads, but I hope to get Marcelo> there soon). This raises the important question, how can we objectively measure interactive response in the kernel and relate it to the user's perceived interactive response? If we could come up with some sort of testing system that would show us this, it would help alot, since we could just have people run tests in a more automatic and repeatable manner. And I think it would also help us automatically tune the Kernel, since it would have a knowledge of it's own performance. There is the problem in terms of some people want pure interactive performance, while others are looking for throughput over all else, but those are both extremes of the spectrum. Though I suspect raw throughput is the less wanted (in terms of numbers of systems) than keeping interactive response good during VM pressure. I have zero knowledge of how we could do this, but giving the kernel some counters, even if only for use during debugging runs, which would give us some objective feedback on performance would be a big win. Having people just send in reports of "I ran X,Y,Z and it was slow" doesn't help us, since it's so hard to re-create their environment so you can run tests against it. Anyway, enjoy the weekend all. John John Stoffel - Senior Unix Systems Administrator - Lucent Technologies [EMAIL PROTECTED] - http://www.lucent.com - 978-952-7548 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, John Stoffel wrote: > > Mike> OK, riddle me this. If this test is a crummy test, just how is > Mike> it that I was able to warn Rik in advance that when 2.4.5 was > Mike> released, he should expect complaints? How did I _know_ that? > Mike> The answer is that I fiddle with Rik's code a lot, and I test > Mike> with this test because it tells me a lot. It may not tell you > Mike> anything, but it does me. > > I never said it was a crummy test, please do not read more into my > words than was written. What I was trying to get across is that just > one test (such as a compile of the kernel) isn't perfect at showing > where the problems are with the VM sub-system. > > Jonathan Morton has been using another large compile to also test the > sub-system, and it includes a compile which puts a large, single > process pressure on the VM. I consider this to be a more > representative test of how the VM deals with pressure. > > The kernel compile is an ok test of basic VM handling, but from what > I've been hearing on linux-kernel and linux-mm is that the VM goes to > crap when you have a mix of stuff running, and one (or more) processes > starts up or grows much larger and starts impacting the system > performance. > > I'm also not knocking your contributions to this discussion, so stop > being so touchy. I was trying to contribute and say (albeit poorly) > that a *mix* of tests is needed to test the VM. > > More importantly, a *repeatable* set of tests is what is needed to > test the VM and get consistent results from run to run, so you can see > how your changes are impacting performance. The kernel compile > doesn't really have any one process grow to a large fraction of > memory, so dropping in a compile which *does* is a good thing. I agree with you. Mike, I'm sure you have noticed that stock kernel gives much better results than mine or Jonathan's patch. Now the stock kernel gives us crappy interactivity compared to my patch. (Note: my patch still does not gives me the interactivity I want under high VM loads, but I hope to get there soon). BTW, we are talking with the OSDL (http://www.osdlab.org) guys about a possibility to setup a test system which would run a different variety of benchmarks to give us results of different kinds of workloads. If that ever happens, we'll probably get rid of most of this testing problems. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, Mike Galbraith wrote: > On Fri, 8 Jun 2001, Tobias Ringstrom wrote: > > On Fri, 8 Jun 2001, Mike Galbraith wrote: > > > I gave this a shot at my favorite vm beater test (make -j30 bzImage) > > > while testing some other stuff today. > > > > Could you please explain what is good about this test? I understand that > > it will stress the VM, but will it do so in a realistic and relevant way? > > Can you explain what is bad about this test? ;) It spins the same VM wheels I think a load of ~30 is quit uncommon, and therefor it is unclear to me that it would be a test that would be repesentative of most normal loads. > as any other load does. What's the difference if I have a bunch of httpd > allocating or a bunch of cc1/as/ld? This load has a modest cachable data > set and is compute bound.. and above all gives very repeatable results. Not a big difference. The difference I was thinking abount is the difference between spawning lots of processes allocating, using and freeing lots of memory, compared to a case where you have a few processes touching a lot of already allocated pages in some pattern. I was wondering whether optimizing for your case would be good or bad for the other case. I know, I know, I should do more testing myself. And I should probably not ask you, since you really really like your test, and you will probably just say yes... ;-) At home, I'm running a couple of computers. One of them is a slow computer running Linux, serving mail, NFS, SMB, etc. I'm usually logged in on a couple of virtual consoles. On this machine, I do not mind if all shells, daemons and other idle processes are beeing swapped out in favor of disk cache for the NFS and SMB serving. In fact, that is a very good thing, and I want it that way. Another maching is my desktop machine. When using this maching, I really hate when my emacsen, browsers, xterms, etc are swapped out just to give me some stupid disk cache for my xmms or compilations. I do not care if a kernel compile is a little slower as long as my applications are snappy. How could Linux predict this? It is a matter of taste, IMHO. > I use it to watch reaction to surge. I watch for the vm to build to a > solid maximum throughput without thrashing. That's the portion of VM > that I'm interested in, so that's what I test. Besides :) I simply don't > have the hardware to try to simulate hairy chested server loads. There > are lots of folks with hairy chested boxes.. they should test that stuff. Agreed. More testing is needed. Now if we would have those knobs and wheels to turn, we could perhaps also tune our systems to behave as we like them, and submit that as well. Right now you need to be a kernel hacker, and see through all the magic with shm, mmap, a bunch of caches, page lists, etc. I'd give a lot for a nice picture (or state diagram) showing the lifetime of a page, but I have not found such a picture anywhere. Besides, the VM seems to change every new release anyway. > I've been repeating ~this test since 2.0 times, and have noticed a 1:1 > relationship. When I notice that my box is ~happy doing this load test, > I also notice very few VM gripes hitting the list. Ok, but as you say, we need more tests. > > Isn't the interesting case when you have a number of processes using lots > > of memory, but only a part of all that memory is beeing actively used, and > > that memory fits in RAM. In that case, the VM should make sure that the > > not used memory is swapped out. In RAM you should have the used memory, > > but also disk cache if there is any RAM left. Does the current VM handle > > this case fine yet? IMHO, this is the case most people care about. It is > > definately the case I care about, at least. :-) > > The interesting case is _every_ case. Try seeing my particular test as > a simulation of a small classroom box with 30 students compiling their > assignments and it'll suddenly become quite realistic. You'll notice > by the numbers I post that I was very careful to not overload the box in > a rediculous manner when selecting the total size of the job.. it's just > a heavily loaded box. This test does not overload my IO resources, so > it tests the VM's ability to choose and move the right stuff at the right > time to get the job done with a minimum of additional overhead. I did not understand those numbers when I saw them the first time. Now, I must say that your test does not look as silly as it did before. > The current VM handles things generally well imho, but has problems > regulating itself under load. My test load hits the VM right in it's > weakest point (not _that_ weak, but..) by starting at zero and building > rapidly to max.. and keeping it _right there_. > > > I'm not saying that it's a completely uninteresting case when your active > > memory is bigger than you RAM of course, but perhaps there should be other > > algorithms handling that case, such as putting some of the
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, John Stoffel wrote: > Mike> OK, riddle me this. If this test is a crummy test, just how is > Mike> it that I was able to warn Rik in advance that when 2.4.5 was > Mike> released, he should expect complaints? How did I _know_ that? > Mike> The answer is that I fiddle with Rik's code a lot, and I test > Mike> with this test because it tells me a lot. It may not tell you > Mike> anything, but it does me. > > I never said it was a crummy test, please do not read more into my > words than was written. What I was trying to get across is that just > one test (such as a compile of the kernel) isn't perfect at showing > where the problems are with the VM sub-system. Hmm... Tobias> Could you please explain what is good about this test? I Tobias> understand that it will stress the VM, but will it do so in a Tobias> realistic and relevant way? I agree, this isn't really a good test case. I'd rather see what happens when you fire up a gimp session to edit an image which is *almost* the size of RAM, or even just 50% the size of ram. Then how does that affect your other processes that are running at the same time? ...but anyway, yes it just one test from any number of possibles. > Jonathan Morton has been using another large compile to also test the > sub-system, and it includes a compile which puts a large, single > process pressure on the VM. I consider this to be a more > representative test of how the VM deals with pressure. What does 'more representative' mean given that the VM must react to every situation it runs into? > The kernel compile is an ok test of basic VM handling, but from what Now we're communicating. I never said it was more than that ;-) > I've been hearing on linux-kernel and linux-mm is that the VM goes to > crap when you have a mix of stuff running, and one (or more) processes > starts up or grows much larger and starts impacting the system > performance. > > I'm also not knocking your contributions to this discussion, so stop > being so touchy. I was trying to contribute and say (albeit poorly) > that a *mix* of tests is needed to test the VM. Yes, more people need to test. I don't need to do all of those other tests (no have right toys), more people need to do repeatable tests. > More importantly, a *repeatable* set of tests is what is needed to > test the VM and get consistent results from run to run, so you can see > how your changes are impacting performance. The kernel compile > doesn't really have any one process grow to a large fraction of > memory, so dropping in a compile which *does* is a good thing. I know I'm only watching basic functionality. I'm watching basic functionality with one very consistant test run very consistantly. -Mike - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
Mike> OK, riddle me this. If this test is a crummy test, just how is Mike> it that I was able to warn Rik in advance that when 2.4.5 was Mike> released, he should expect complaints? How did I _know_ that? Mike> The answer is that I fiddle with Rik's code a lot, and I test Mike> with this test because it tells me a lot. It may not tell you Mike> anything, but it does me. I never said it was a crummy test, please do not read more into my words than was written. What I was trying to get across is that just one test (such as a compile of the kernel) isn't perfect at showing where the problems are with the VM sub-system. Jonathan Morton has been using another large compile to also test the sub-system, and it includes a compile which puts a large, single process pressure on the VM. I consider this to be a more representative test of how the VM deals with pressure. The kernel compile is an ok test of basic VM handling, but from what I've been hearing on linux-kernel and linux-mm is that the VM goes to crap when you have a mix of stuff running, and one (or more) processes starts up or grows much larger and starts impacting the system performance. I'm also not knocking your contributions to this discussion, so stop being so touchy. I was trying to contribute and say (albeit poorly) that a *mix* of tests is needed to test the VM. More importantly, a *repeatable* set of tests is what is needed to test the VM and get consistent results from run to run, so you can see how your changes are impacting performance. The kernel compile doesn't really have any one process grow to a large fraction of memory, so dropping in a compile which *does* is a good thing. John John Stoffel - Senior Unix Systems Administrator - Lucent Technologies [EMAIL PROTECTED] - http://www.lucent.com - 978-952-7548 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, John Stoffel wrote: > > "Tobias" == Tobias Ringstrom <[EMAIL PROTECTED]> writes: > > Tobias> On Fri, 8 Jun 2001, Mike Galbraith wrote: > > >> I gave this a shot at my favorite vm beater test (make -j30 bzImage) > >> while testing some other stuff today. > > Tobias> Could you please explain what is good about this test? I > Tobias> understand that it will stress the VM, but will it do so in a > Tobias> realistic and relevant way? > > I agree, this isn't really a good test case. I'd rather see what > happens when you fire up a gimp session to edit an image which is > *almost* the size of RAM, or even just 50% the size of ram. Then how > does that affect your other processes that are running at the same > time? OK, riddle me this. If this test is a crummy test, just how is it that I was able to warn Rik in advance that when 2.4.5 was released, he should expect complaints? How did I _know_ that? The answer is that I fiddle with Rik's code a lot, and I test with this test because it tells me a lot. It may not tell you anything, but it does me. -Mike - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, Tobias Ringstrom wrote: > On Fri, 8 Jun 2001, Mike Galbraith wrote: > > I gave this a shot at my favorite vm beater test (make -j30 bzImage) > > while testing some other stuff today. > > Could you please explain what is good about this test? I understand that > it will stress the VM, but will it do so in a realistic and relevant way? Can you explain what is bad about this test? ;) It spins the same VM wheels as any other load does. What's the difference if I have a bunch of httpd allocating or a bunch of cc1/as/ld? This load has a modest cachable data set and is compute bound.. and above all gives very repeatable results. I use it to watch reaction to surge. I watch for the vm to build to a solid maximum throughput without thrashing. That's the portion of VM that I'm interested in, so that's what I test. Besides :) I simply don't have the hardware to try to simulate hairy chested server loads. There are lots of folks with hairy chested boxes.. they should test that stuff. I've been repeating ~this test since 2.0 times, and have noticed a 1:1 relationship. When I notice that my box is ~happy doing this load test, I also notice very few VM gripes hitting the list. > Isn't the interesting case when you have a number of processes using lots > of memory, but only a part of all that memory is beeing actively used, and > that memory fits in RAM. In that case, the VM should make sure that the > not used memory is swapped out. In RAM you should have the used memory, > but also disk cache if there is any RAM left. Does the current VM handle > this case fine yet? IMHO, this is the case most people care about. It is > definately the case I care about, at least. :-) The interesting case is _every_ case. Try seeing my particular test as a simulation of a small classroom box with 30 students compiling their assignments and it'll suddenly become quite realistic. You'll notice by the numbers I post that I was very careful to not overload the box in a rediculous manner when selecting the total size of the job.. it's just a heavily loaded box. This test does not overload my IO resources, so it tests the VM's ability to choose and move the right stuff at the right time to get the job done with a minimum of additional overhead. The current VM handles things generally well imho, but has problems regulating itself under load. My test load hits the VM right in it's weakest point (not _that_ weak, but..) by starting at zero and building rapidly to max.. and keeping it _right there_. > I'm not saying that it's a completely uninteresting case when your active > memory is bigger than you RAM of course, but perhaps there should be other > algorithms handling that case, such as putting some of the swapping > processes to sleep for some time, especially if you have lots of processes > competing for the memory. I may be wrong, but it seems to me that your > testcase falls into this second category (also known as thrashing). Thrashing? Let's look some numbers. (not the ugly ones, the ~ok ones;) real9m12.198s make -j 30 bzImage user7m41.290s sys 0m34.840s user : 0:07:47.69 76.8% page in : 452632 nice : 0:00:00.00 0.0% page out: 399847 system: 0:01:17.08 12.7% swap in :75338 idle : 0:01:03.97 10.5% swap out:88291 real8m6.994s make bzImage user7m34.350s sys 0m26.550s user : 0:07:37.52 78.4% page in :90546 nice : 0:00:00.00 0.0% page out:18164 system: 0:01:26.13 14.8% swap in :1 idle : 0:00:39.69 6.8% swap out:0 ...look at cpu utilization. One minute +tiny change to complete the large job vs the small (VM footprint) job. The box is not thrashing, it's working it's little silicon butt off. What I'm testing is the VM's ability to handle load without thrashing so badly that it loses throughput bigtime, stalls itself whatever.. it's ability to regulate itself. I consider a minute and a half to be ~acceptable, a minute to be good, and 30 seconds to be excellent. That's just my own little VM performance thermometer. > An at last, a humble request: Every problem I've had with the VM has been > that it either swapped out too many processes and used too much cache, or > the other way around. I'd really enjoy a way to tune this behaviour, if > possible. Tunables aren't really practical in VM (imho). If there were a dozen knobs, you'd have to turn a dozen knobs a dozen times a day. VM has to be self regulating. In case you can't tell (the length of this reply) I like my fovorite little generic throughput test a LOT :-) -Mike - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
> "Tobias" == Tobias Ringstrom <[EMAIL PROTECTED]> writes: Tobias> On Fri, 8 Jun 2001, Mike Galbraith wrote: >> I gave this a shot at my favorite vm beater test (make -j30 bzImage) >> while testing some other stuff today. Tobias> Could you please explain what is good about this test? I Tobias> understand that it will stress the VM, but will it do so in a Tobias> realistic and relevant way? I agree, this isn't really a good test case. I'd rather see what happens when you fire up a gimp session to edit an image which is *almost* the size of RAM, or even just 50% the size of ram. Then how does that affect your other processes that are running at the same time? This testing could even be automated with the script-foo stuff to get consistent results across runs, which is the prime requirement of any sort of testing. On another issue, in swap.c we have two defines for buffer_mem and page_cache, but the first maxes out at 60%, while the cache maxes out at 75%. Shouldn't they both be lower numbers? Or at least equally sized? I've set my page_cache maximum to be 60, I'll be trying to test it over the weekend, but good weather will keep me outside doing other stuff... Thanks, John John Stoffel - Senior Unix Systems Administrator - Lucent Technologies [EMAIL PROTECTED] - http://www.lucent.com - 978-952-7548 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, Mike Galbraith wrote: > I gave this a shot at my favorite vm beater test (make -j30 bzImage) > while testing some other stuff today. Could you please explain what is good about this test? I understand that it will stress the VM, but will it do so in a realistic and relevant way? Isn't the interesting case when you have a number of processes using lots of memory, but only a part of all that memory is beeing actively used, and that memory fits in RAM. In that case, the VM should make sure that the not used memory is swapped out. In RAM you should have the used memory, but also disk cache if there is any RAM left. Does the current VM handle this case fine yet? IMHO, this is the case most people care about. It is definately the case I care about, at least. :-) I'm not saying that it's a completely uninteresting case when your active memory is bigger than you RAM of course, but perhaps there should be other algorithms handling that case, such as putting some of the swapping processes to sleep for some time, especially if you have lots of processes competing for the memory. I may be wrong, but it seems to me that your testcase falls into this second category (also known as thrashing). An at last, a humble request: Every problem I've had with the VM has been that it either swapped out too many processes and used too much cache, or the other way around. I'd really enjoy a way to tune this behaviour, if possible. /Tobias - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, Jonathan Morton wrote: > http://www.chromatix.uklinux.net/linux-patches/vm-update-2.patch > > Try this. I can't guarantee it's SMP-safe yet (I'm leaving the gurus to > that, but they haven't told me about any errors in the past hour so I'm > assuming they aren't going to find anything glaringly wrong...), but you > might like to see if your performance improves with it. It also fixes the > OOM-killer bug, which you refer to above. > > Some measurements, from my own box (1GHz Athlon, 256Mb RAM): > > For the following benchmarks, physical memory availability was reduced > according to the parameter in the left column. The benchmark is the > wall-clock time taken to compile MySQL. > > mem= 2.4.5 earlier tweaks now > 48M 8m30s 6m30s 5m58s > 32M unknown 2h15m 12m34s > > The following was performed with all 256Mb RAM available. This is > compilation of MySQL using make -j 15. > > kernel: 2.4.5 now > time: 6m30s 6m15s > peak swap:190M70M > > For the following test, the 256Mb swap partition on my IDE drive was > disabled and replaced with a 1Gb swapfile on my Ultra160 SCSI drive. This > is compilation of MySQL using make -j 20. > > kernel: 2.4.5 now > time: 7m20s 6m30s > peak swap:370M254M > > Draw your own conclusions. :) (ok;) Hi, I gave this a shot at my favorite vm beater test (make -j30 bzImage) while testing some other stuff today. seven identical runs, six slightly different kernels plus yours. real11m23.522s 2.4.5.vm-update-2 user7m59.170s sys 0m37.030s user : 0:08:07.06 65.6% page in : 642402 nice : 0:00:00.00 0.0% page out: 676820 system: 0:02:09.44 17.4% swap in : 105965 idle : 0:02:05.66 16.9% swap out: 162603 real10m9.512s 2.4.5.virgin user7m55.520s sys 0m35.460s user : 0:08:02.66 72.2% page in : 535186 nice : 0:00:00.00 0.0% page out: 377992 system: 0:01:37.78 14.6% swap in :99445 idle : 0:01:28.14 13.2% swap out:81926 real10m48.939s 2.4.5.virgin+reclaim.marcelo user7m54.960s sys 0m36.240s user : 0:08:02.33 68.0% page in : 566239 nice : 0:00:00.00 0.0% page out: 431874 system: 0:01:56.02 16.4% swap in : 108633 idle : 0:01:50.61 15.6% swap out:96415 real9m54.466s 2.4.5.virgin+reclaim.mike (icky 'bleeder valve') user7m57.370s sys 0m35.890s user : 0:08:04.74 74.1% page in : 527678 nice : 0:00:00.00 0.0% page out: 405259 system: 0:01:12.01 11.0% swap in :98616 idle : 0:01:37.47 14.9% swap out:91492 real9m12.198s 2.4.5.tweak user7m41.290s sys 0m34.840s user : 0:07:47.69 76.8% page in : 452632 nice : 0:00:00.00 0.0% page out: 399847 system: 0:01:17.08 12.7% swap in :75338 idle : 0:01:03.97 10.5% swap out:88291 real9m41.563s 2.4.5.tweak+reclaim.marcelo user7m59.880s sys 0m34.690s user : 0:08:07.22 73.4% page in : 515433 nice : 0:00:00.00 0.0% page out: 545762 system: 0:01:35.34 14.4% swap in :88425 idle : 0:01:21.11 12.2% swap out: 125967 real9m47.682s 2.4.5.tweak+reclaim.mike user8m2.190s sys 0m34.550s user : 0:08:09.57 75.7% page in : 513166 nice : 0:00:00.00 0.0% page out: 473539 system: 0:01:20.27 12.4% swap in :83127 idle : 0:01:16.89 11.9% swap out: 108886 Conclusion: Your patch hits the cache too hard and pays through the nose for doing so.. at least under this hefty weight load it does. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, Jonathan Morton wrote: http://www.chromatix.uklinux.net/linux-patches/vm-update-2.patch Try this. I can't guarantee it's SMP-safe yet (I'm leaving the gurus to that, but they haven't told me about any errors in the past hour so I'm assuming they aren't going to find anything glaringly wrong...), but you might like to see if your performance improves with it. It also fixes the OOM-killer bug, which you refer to above. Some measurements, from my own box (1GHz Athlon, 256Mb RAM): For the following benchmarks, physical memory availability was reduced according to the parameter in the left column. The benchmark is the wall-clock time taken to compile MySQL. mem= 2.4.5 earlier tweaks now 48M 8m30s 6m30s 5m58s 32M unknown 2h15m 12m34s The following was performed with all 256Mb RAM available. This is compilation of MySQL using make -j 15. kernel: 2.4.5 now time: 6m30s 6m15s peak swap:190M70M For the following test, the 256Mb swap partition on my IDE drive was disabled and replaced with a 1Gb swapfile on my Ultra160 SCSI drive. This is compilation of MySQL using make -j 20. kernel: 2.4.5 now time: 7m20s 6m30s peak swap:370M254M Draw your own conclusions. :) (ok;) Hi, I gave this a shot at my favorite vm beater test (make -j30 bzImage) while testing some other stuff today. seven identical runs, six slightly different kernels plus yours. real11m23.522s 2.4.5.vm-update-2 user7m59.170s sys 0m37.030s user : 0:08:07.06 65.6% page in : 642402 nice : 0:00:00.00 0.0% page out: 676820 system: 0:02:09.44 17.4% swap in : 105965 idle : 0:02:05.66 16.9% swap out: 162603 real10m9.512s 2.4.5.virgin user7m55.520s sys 0m35.460s user : 0:08:02.66 72.2% page in : 535186 nice : 0:00:00.00 0.0% page out: 377992 system: 0:01:37.78 14.6% swap in :99445 idle : 0:01:28.14 13.2% swap out:81926 real10m48.939s 2.4.5.virgin+reclaim.marcelo user7m54.960s sys 0m36.240s user : 0:08:02.33 68.0% page in : 566239 nice : 0:00:00.00 0.0% page out: 431874 system: 0:01:56.02 16.4% swap in : 108633 idle : 0:01:50.61 15.6% swap out:96415 real9m54.466s 2.4.5.virgin+reclaim.mike (icky 'bleeder valve') user7m57.370s sys 0m35.890s user : 0:08:04.74 74.1% page in : 527678 nice : 0:00:00.00 0.0% page out: 405259 system: 0:01:12.01 11.0% swap in :98616 idle : 0:01:37.47 14.9% swap out:91492 real9m12.198s 2.4.5.tweak user7m41.290s sys 0m34.840s user : 0:07:47.69 76.8% page in : 452632 nice : 0:00:00.00 0.0% page out: 399847 system: 0:01:17.08 12.7% swap in :75338 idle : 0:01:03.97 10.5% swap out:88291 real9m41.563s 2.4.5.tweak+reclaim.marcelo user7m59.880s sys 0m34.690s user : 0:08:07.22 73.4% page in : 515433 nice : 0:00:00.00 0.0% page out: 545762 system: 0:01:35.34 14.4% swap in :88425 idle : 0:01:21.11 12.2% swap out: 125967 real9m47.682s 2.4.5.tweak+reclaim.mike user8m2.190s sys 0m34.550s user : 0:08:09.57 75.7% page in : 513166 nice : 0:00:00.00 0.0% page out: 473539 system: 0:01:20.27 12.4% swap in :83127 idle : 0:01:16.89 11.9% swap out: 108886 Conclusion: Your patch hits the cache too hard and pays through the nose for doing so.. at least under this hefty weight load it does. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, Mike Galbraith wrote: I gave this a shot at my favorite vm beater test (make -j30 bzImage) while testing some other stuff today. Could you please explain what is good about this test? I understand that it will stress the VM, but will it do so in a realistic and relevant way? Isn't the interesting case when you have a number of processes using lots of memory, but only a part of all that memory is beeing actively used, and that memory fits in RAM. In that case, the VM should make sure that the not used memory is swapped out. In RAM you should have the used memory, but also disk cache if there is any RAM left. Does the current VM handle this case fine yet? IMHO, this is the case most people care about. It is definately the case I care about, at least. :-) I'm not saying that it's a completely uninteresting case when your active memory is bigger than you RAM of course, but perhaps there should be other algorithms handling that case, such as putting some of the swapping processes to sleep for some time, especially if you have lots of processes competing for the memory. I may be wrong, but it seems to me that your testcase falls into this second category (also known as thrashing). An at last, a humble request: Every problem I've had with the VM has been that it either swapped out too many processes and used too much cache, or the other way around. I'd really enjoy a way to tune this behaviour, if possible. /Tobias - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
Tobias == Tobias Ringstrom [EMAIL PROTECTED] writes: Tobias On Fri, 8 Jun 2001, Mike Galbraith wrote: I gave this a shot at my favorite vm beater test (make -j30 bzImage) while testing some other stuff today. Tobias Could you please explain what is good about this test? I Tobias understand that it will stress the VM, but will it do so in a Tobias realistic and relevant way? I agree, this isn't really a good test case. I'd rather see what happens when you fire up a gimp session to edit an image which is *almost* the size of RAM, or even just 50% the size of ram. Then how does that affect your other processes that are running at the same time? This testing could even be automated with the script-foo stuff to get consistent results across runs, which is the prime requirement of any sort of testing. On another issue, in swap.c we have two defines for buffer_mem and page_cache, but the first maxes out at 60%, while the cache maxes out at 75%. Shouldn't they both be lower numbers? Or at least equally sized? I've set my page_cache maximum to be 60, I'll be trying to test it over the weekend, but good weather will keep me outside doing other stuff... Thanks, John John Stoffel - Senior Unix Systems Administrator - Lucent Technologies [EMAIL PROTECTED] - http://www.lucent.com - 978-952-7548 - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, Tobias Ringstrom wrote: On Fri, 8 Jun 2001, Mike Galbraith wrote: I gave this a shot at my favorite vm beater test (make -j30 bzImage) while testing some other stuff today. Could you please explain what is good about this test? I understand that it will stress the VM, but will it do so in a realistic and relevant way? Can you explain what is bad about this test? ;) It spins the same VM wheels as any other load does. What's the difference if I have a bunch of httpd allocating or a bunch of cc1/as/ld? This load has a modest cachable data set and is compute bound.. and above all gives very repeatable results. I use it to watch reaction to surge. I watch for the vm to build to a solid maximum throughput without thrashing. That's the portion of VM that I'm interested in, so that's what I test. Besides :) I simply don't have the hardware to try to simulate hairy chested server loads. There are lots of folks with hairy chested boxes.. they should test that stuff. I've been repeating ~this test since 2.0 times, and have noticed a 1:1 relationship. When I notice that my box is ~happy doing this load test, I also notice very few VM gripes hitting the list. Isn't the interesting case when you have a number of processes using lots of memory, but only a part of all that memory is beeing actively used, and that memory fits in RAM. In that case, the VM should make sure that the not used memory is swapped out. In RAM you should have the used memory, but also disk cache if there is any RAM left. Does the current VM handle this case fine yet? IMHO, this is the case most people care about. It is definately the case I care about, at least. :-) The interesting case is _every_ case. Try seeing my particular test as a simulation of a small classroom box with 30 students compiling their assignments and it'll suddenly become quite realistic. You'll notice by the numbers I post that I was very careful to not overload the box in a rediculous manner when selecting the total size of the job.. it's just a heavily loaded box. This test does not overload my IO resources, so it tests the VM's ability to choose and move the right stuff at the right time to get the job done with a minimum of additional overhead. The current VM handles things generally well imho, but has problems regulating itself under load. My test load hits the VM right in it's weakest point (not _that_ weak, but..) by starting at zero and building rapidly to max.. and keeping it _right there_. I'm not saying that it's a completely uninteresting case when your active memory is bigger than you RAM of course, but perhaps there should be other algorithms handling that case, such as putting some of the swapping processes to sleep for some time, especially if you have lots of processes competing for the memory. I may be wrong, but it seems to me that your testcase falls into this second category (also known as thrashing). Thrashing? Let's look some numbers. (not the ugly ones, the ~ok ones;) real9m12.198s make -j 30 bzImage user7m41.290s sys 0m34.840s user : 0:07:47.69 76.8% page in : 452632 nice : 0:00:00.00 0.0% page out: 399847 system: 0:01:17.08 12.7% swap in :75338 idle : 0:01:03.97 10.5% swap out:88291 real8m6.994s make bzImage user7m34.350s sys 0m26.550s user : 0:07:37.52 78.4% page in :90546 nice : 0:00:00.00 0.0% page out:18164 system: 0:01:26.13 14.8% swap in :1 idle : 0:00:39.69 6.8% swap out:0 ...look at cpu utilization. One minute +tiny change to complete the large job vs the small (VM footprint) job. The box is not thrashing, it's working it's little silicon butt off. What I'm testing is the VM's ability to handle load without thrashing so badly that it loses throughput bigtime, stalls itself whatever.. it's ability to regulate itself. I consider a minute and a half to be ~acceptable, a minute to be good, and 30 seconds to be excellent. That's just my own little VM performance thermometer. An at last, a humble request: Every problem I've had with the VM has been that it either swapped out too many processes and used too much cache, or the other way around. I'd really enjoy a way to tune this behaviour, if possible. Tunables aren't really practical in VM (imho). If there were a dozen knobs, you'd have to turn a dozen knobs a dozen times a day. VM has to be self regulating. In case you can't tell (the length of this reply) I like my fovorite little generic throughput test a LOT :-) -Mike - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, John Stoffel wrote: Tobias == Tobias Ringstrom [EMAIL PROTECTED] writes: Tobias On Fri, 8 Jun 2001, Mike Galbraith wrote: I gave this a shot at my favorite vm beater test (make -j30 bzImage) while testing some other stuff today. Tobias Could you please explain what is good about this test? I Tobias understand that it will stress the VM, but will it do so in a Tobias realistic and relevant way? I agree, this isn't really a good test case. I'd rather see what happens when you fire up a gimp session to edit an image which is *almost* the size of RAM, or even just 50% the size of ram. Then how does that affect your other processes that are running at the same time? OK, riddle me this. If this test is a crummy test, just how is it that I was able to warn Rik in advance that when 2.4.5 was released, he should expect complaints? How did I _know_ that? The answer is that I fiddle with Rik's code a lot, and I test with this test because it tells me a lot. It may not tell you anything, but it does me. -Mike - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
Mike OK, riddle me this. If this test is a crummy test, just how is Mike it that I was able to warn Rik in advance that when 2.4.5 was Mike released, he should expect complaints? How did I _know_ that? Mike The answer is that I fiddle with Rik's code a lot, and I test Mike with this test because it tells me a lot. It may not tell you Mike anything, but it does me. I never said it was a crummy test, please do not read more into my words than was written. What I was trying to get across is that just one test (such as a compile of the kernel) isn't perfect at showing where the problems are with the VM sub-system. Jonathan Morton has been using another large compile to also test the sub-system, and it includes a compile which puts a large, single process pressure on the VM. I consider this to be a more representative test of how the VM deals with pressure. The kernel compile is an ok test of basic VM handling, but from what I've been hearing on linux-kernel and linux-mm is that the VM goes to crap when you have a mix of stuff running, and one (or more) processes starts up or grows much larger and starts impacting the system performance. I'm also not knocking your contributions to this discussion, so stop being so touchy. I was trying to contribute and say (albeit poorly) that a *mix* of tests is needed to test the VM. More importantly, a *repeatable* set of tests is what is needed to test the VM and get consistent results from run to run, so you can see how your changes are impacting performance. The kernel compile doesn't really have any one process grow to a large fraction of memory, so dropping in a compile which *does* is a good thing. John John Stoffel - Senior Unix Systems Administrator - Lucent Technologies [EMAIL PROTECTED] - http://www.lucent.com - 978-952-7548 - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, John Stoffel wrote: Mike OK, riddle me this. If this test is a crummy test, just how is Mike it that I was able to warn Rik in advance that when 2.4.5 was Mike released, he should expect complaints? How did I _know_ that? Mike The answer is that I fiddle with Rik's code a lot, and I test Mike with this test because it tells me a lot. It may not tell you Mike anything, but it does me. I never said it was a crummy test, please do not read more into my words than was written. What I was trying to get across is that just one test (such as a compile of the kernel) isn't perfect at showing where the problems are with the VM sub-system. Hmm... Tobias Could you please explain what is good about this test? I Tobias understand that it will stress the VM, but will it do so in a Tobias realistic and relevant way? I agree, this isn't really a good test case. I'd rather see what happens when you fire up a gimp session to edit an image which is *almost* the size of RAM, or even just 50% the size of ram. Then how does that affect your other processes that are running at the same time? ...but anyway, yes it just one test from any number of possibles. Jonathan Morton has been using another large compile to also test the sub-system, and it includes a compile which puts a large, single process pressure on the VM. I consider this to be a more representative test of how the VM deals with pressure. What does 'more representative' mean given that the VM must react to every situation it runs into? The kernel compile is an ok test of basic VM handling, but from what Now we're communicating. I never said it was more than that ;-) I've been hearing on linux-kernel and linux-mm is that the VM goes to crap when you have a mix of stuff running, and one (or more) processes starts up or grows much larger and starts impacting the system performance. I'm also not knocking your contributions to this discussion, so stop being so touchy. I was trying to contribute and say (albeit poorly) that a *mix* of tests is needed to test the VM. Yes, more people need to test. I don't need to do all of those other tests (no have right toys), more people need to do repeatable tests. More importantly, a *repeatable* set of tests is what is needed to test the VM and get consistent results from run to run, so you can see how your changes are impacting performance. The kernel compile doesn't really have any one process grow to a large fraction of memory, so dropping in a compile which *does* is a good thing. I know I'm only watching basic functionality. I'm watching basic functionality with one very consistant test run very consistantly. -Mike - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, Mike Galbraith wrote: On Fri, 8 Jun 2001, Tobias Ringstrom wrote: On Fri, 8 Jun 2001, Mike Galbraith wrote: I gave this a shot at my favorite vm beater test (make -j30 bzImage) while testing some other stuff today. Could you please explain what is good about this test? I understand that it will stress the VM, but will it do so in a realistic and relevant way? Can you explain what is bad about this test? ;) It spins the same VM wheels I think a load of ~30 is quit uncommon, and therefor it is unclear to me that it would be a test that would be repesentative of most normal loads. as any other load does. What's the difference if I have a bunch of httpd allocating or a bunch of cc1/as/ld? This load has a modest cachable data set and is compute bound.. and above all gives very repeatable results. Not a big difference. The difference I was thinking abount is the difference between spawning lots of processes allocating, using and freeing lots of memory, compared to a case where you have a few processes touching a lot of already allocated pages in some pattern. I was wondering whether optimizing for your case would be good or bad for the other case. I know, I know, I should do more testing myself. And I should probably not ask you, since you really really like your test, and you will probably just say yes... ;-) At home, I'm running a couple of computers. One of them is a slow computer running Linux, serving mail, NFS, SMB, etc. I'm usually logged in on a couple of virtual consoles. On this machine, I do not mind if all shells, daemons and other idle processes are beeing swapped out in favor of disk cache for the NFS and SMB serving. In fact, that is a very good thing, and I want it that way. Another maching is my desktop machine. When using this maching, I really hate when my emacsen, browsers, xterms, etc are swapped out just to give me some stupid disk cache for my xmms or compilations. I do not care if a kernel compile is a little slower as long as my applications are snappy. How could Linux predict this? It is a matter of taste, IMHO. I use it to watch reaction to surge. I watch for the vm to build to a solid maximum throughput without thrashing. That's the portion of VM that I'm interested in, so that's what I test. Besides :) I simply don't have the hardware to try to simulate hairy chested server loads. There are lots of folks with hairy chested boxes.. they should test that stuff. Agreed. More testing is needed. Now if we would have those knobs and wheels to turn, we could perhaps also tune our systems to behave as we like them, and submit that as well. Right now you need to be a kernel hacker, and see through all the magic with shm, mmap, a bunch of caches, page lists, etc. I'd give a lot for a nice picture (or state diagram) showing the lifetime of a page, but I have not found such a picture anywhere. Besides, the VM seems to change every new release anyway. I've been repeating ~this test since 2.0 times, and have noticed a 1:1 relationship. When I notice that my box is ~happy doing this load test, I also notice very few VM gripes hitting the list. Ok, but as you say, we need more tests. Isn't the interesting case when you have a number of processes using lots of memory, but only a part of all that memory is beeing actively used, and that memory fits in RAM. In that case, the VM should make sure that the not used memory is swapped out. In RAM you should have the used memory, but also disk cache if there is any RAM left. Does the current VM handle this case fine yet? IMHO, this is the case most people care about. It is definately the case I care about, at least. :-) The interesting case is _every_ case. Try seeing my particular test as a simulation of a small classroom box with 30 students compiling their assignments and it'll suddenly become quite realistic. You'll notice by the numbers I post that I was very careful to not overload the box in a rediculous manner when selecting the total size of the job.. it's just a heavily loaded box. This test does not overload my IO resources, so it tests the VM's ability to choose and move the right stuff at the right time to get the job done with a minimum of additional overhead. I did not understand those numbers when I saw them the first time. Now, I must say that your test does not look as silly as it did before. The current VM handles things generally well imho, but has problems regulating itself under load. My test load hits the VM right in it's weakest point (not _that_ weak, but..) by starting at zero and building rapidly to max.. and keeping it _right there_. I'm not saying that it's a completely uninteresting case when your active memory is bigger than you RAM of course, but perhaps there should be other algorithms handling that case, such as putting some of the swapping processes to sleep for some time, especially if you
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, John Stoffel wrote: Mike OK, riddle me this. If this test is a crummy test, just how is Mike it that I was able to warn Rik in advance that when 2.4.5 was Mike released, he should expect complaints? How did I _know_ that? Mike The answer is that I fiddle with Rik's code a lot, and I test Mike with this test because it tells me a lot. It may not tell you Mike anything, but it does me. I never said it was a crummy test, please do not read more into my words than was written. What I was trying to get across is that just one test (such as a compile of the kernel) isn't perfect at showing where the problems are with the VM sub-system. Jonathan Morton has been using another large compile to also test the sub-system, and it includes a compile which puts a large, single process pressure on the VM. I consider this to be a more representative test of how the VM deals with pressure. The kernel compile is an ok test of basic VM handling, but from what I've been hearing on linux-kernel and linux-mm is that the VM goes to crap when you have a mix of stuff running, and one (or more) processes starts up or grows much larger and starts impacting the system performance. I'm also not knocking your contributions to this discussion, so stop being so touchy. I was trying to contribute and say (albeit poorly) that a *mix* of tests is needed to test the VM. More importantly, a *repeatable* set of tests is what is needed to test the VM and get consistent results from run to run, so you can see how your changes are impacting performance. The kernel compile doesn't really have any one process grow to a large fraction of memory, so dropping in a compile which *does* is a good thing. I agree with you. Mike, I'm sure you have noticed that stock kernel gives much better results than mine or Jonathan's patch. Now the stock kernel gives us crappy interactivity compared to my patch. (Note: my patch still does not gives me the interactivity I want under high VM loads, but I hope to get there soon). BTW, we are talking with the OSDL (http://www.osdlab.org) guys about a possibility to setup a test system which would run a different variety of benchmarks to give us results of different kinds of workloads. If that ever happens, we'll probably get rid of most of this testing problems. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
Marcelo Now the stock kernel gives us crappy interactivity compared Marcelo to my patch. (Note: my patch still does not gives me the Marcelo interactivity I want under high VM loads, but I hope to get Marcelo there soon). This raises the important question, how can we objectively measure interactive response in the kernel and relate it to the user's perceived interactive response? If we could come up with some sort of testing system that would show us this, it would help alot, since we could just have people run tests in a more automatic and repeatable manner. And I think it would also help us automatically tune the Kernel, since it would have a knowledge of it's own performance. There is the problem in terms of some people want pure interactive performance, while others are looking for throughput over all else, but those are both extremes of the spectrum. Though I suspect raw throughput is the less wanted (in terms of numbers of systems) than keeping interactive response good during VM pressure. I have zero knowledge of how we could do this, but giving the kernel some counters, even if only for use during debugging runs, which would give us some objective feedback on performance would be a big win. Having people just send in reports of I ran X,Y,Z and it was slow doesn't help us, since it's so hard to re-create their environment so you can run tests against it. Anyway, enjoy the weekend all. John John Stoffel - Senior Unix Systems Administrator - Lucent Technologies [EMAIL PROTECTED] - http://www.lucent.com - 978-952-7548 - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, John Stoffel wrote: Marcelo Now the stock kernel gives us crappy interactivity compared Marcelo to my patch. (Note: my patch still does not gives me the Marcelo interactivity I want under high VM loads, but I hope to get Marcelo there soon). This raises the important question, how can we objectively measure interactive response in the kernel and relate it to the user's perceived interactive response? If we could come up with some sort of testing system that would show us this, it would help alot, since we could just have people run tests in a more automatic and repeatable manner. And I think it would also help us automatically tune the Kernel, since it would have a knowledge of it's own performance. There is the problem in terms of some people want pure interactive performance, while others are looking for throughput over all else, but those are both extremes of the spectrum. Though I suspect raw throughput is the less wanted (in terms of numbers of systems) than keeping interactive response good during VM pressure. And this raises a very very important point: raw throughtput wins enterprise-like benchmarks, and the enterprise people are the ones who pay most of hackers here. (including me and Rik) We have to be careful about that. I have zero knowledge of how we could do this, but giving the kernel some counters, even if only for use during debugging runs, which would give us some objective feedback on performance would be a big win. Having people just send in reports of I ran X,Y,Z and it was slow doesn't help us, since it's so hard to re-create their environment so you can run tests against it. Lets wait for some test system to be set up (eg the OSDL thing). Once thats done, I'm sure we will find out some way of doing it. Well, good weekend for you too. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
[ Re-entering discussion after too long a day and a long sleep... ] There is the problem in terms of some people want pure interactive performance, while others are looking for throughput over all else, but those are both extremes of the spectrum. Though I suspect raw throughput is the less wanted (in terms of numbers of systems) than keeping interactive response good during VM pressure. And this raises a very very important point: raw throughtput wins enterprise-like benchmarks, and the enterprise people are the ones who pay most of hackers here. (including me and Rik) Very true. As well as the fact that interactivity is much harder to measure. The question is, what is interactivity (from the kernel's perspective)? It usually means small(ish) processes with intermittent working-set and CPU requirements. These types of process can safely be swapped out when not immediately in use, but the kernel has to be able to page them in quite quickly when needed. Doing that under heavy load is very non-trivial. It can also mean multimedia applications with a continuous (maybe small) working set, a continuous but not 100% CPU usage, and the special property that the user WILL notice if this process gets swapped out even briefly. mpg123 and XMMS fall into this category, and I sometimes tried running these alongside my compilation tests to see how they fared. I think I had it going fairly well towards the end, with mpg123 stuttering relatively rarely and briefly while VM load was high. On the subject of Mike Galbraith's kernel compilation test, how much physical RAM does he have for his machine, what type of CPU is it, and what (approximate) type of device does he use for swap? I'll see if I can partially duplicate his results at this end. So far all my tests have been done with a fast CPU - perhaps I should try the P166/MMX or even try loading linux-pmac onto my 8100. -- from: Jonathan Chromatix Morton mail: [EMAIL PROTECTED] (not for attachments) The key to knowledge is not to rely on people to teach you it. GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*) - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
Jonathan Morton wrote: [ Re-entering discussion after too long a day and a long sleep... ] There is the problem in terms of some people want pure interactive performance, while others are looking for throughput over all else, but those are both extremes of the spectrum. Though I suspect raw throughput is the less wanted (in terms of numbers of systems) than keeping interactive response good during VM pressure. And this raises a very very important point: raw throughtput wins enterprise-like benchmarks, and the enterprise people are the ones who pay most of hackers here. (including me and Rik) Very true. As well as the fact that interactivity is much harder to measure. The question is, what is interactivity (from the kernel's perspective)? It usually means small(ish) processes with intermittent working-set and CPU requirements. These types of process can safely be swapped out when not immediately in use, but the kernel has to be able to page them in quite quickly when needed. Doing that under heavy load is very non-trivial. For the low-latency stuff, latency can be defined as the worst-case time to schedule a userspace process in response to an interrupt. That metric is also appropriate in this case, (latency equals interactivity), although here you don't need to be so fanatical about the *worst case*. A few scheduling blips here are less fatal. I have tools to measure latency (aka interactivity). At http://www.uow.edu.au/~andrewm/linux/schedlat.html#downloads there is a kernel patch called `rtc-debug' which causes the PC RTC to generate a stream of interrupts. A user-space task called `amlat' responds to those interrupts and reads the RTC device. The patched RTC driver can then measure the elapsed time between the interrupt and the read from userspace. Voila: latency. When you close the RTC device (by killing amlat), the RTC driver will print out a histogram of the latencies. `amlat' at present gives itself SCHED_RR policy and runs under mlockall() - for your testing you'll need to delete those lines. So. Simple apply rtc-debug, run `amlat' and kill it when you've finished the workload. The challenge will be to relate the latency histogram to human-perceived interactivity. I'm not sure of the best way of doing that. Perhaps monitor the 90th percentile, and aim to keep it below 100 milliseconds. Also, `amlat' should do a bit of disk I/O as well. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, Mike Galbraith wrote: On Fri, 8 Jun 2001, John Stoffel wrote: I agree, this isn't really a good test case. I'd rather see what happens when you fire up a gimp session to edit an image which is *almost* the size of RAM, or even just 50% the size of ram. OK, riddle me this. If this test is a crummy test, just how is it Personally, I'd like to see BOTH of these tests, and many many more. Preferably, handed to the VM hackers in various colourful graphs that allow even severely undercaffeinated hackers to see how things changed for the good or the bad between kernel revisions. cheers, Rik -- Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose... http://www.surriel.com/ http://distro.conectiva.com/ Send all your spam to [EMAIL PROTECTED] (spam digging piggy) - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Sat, 9 Jun 2001, Jonathan Morton wrote: On the subject of Mike Galbraith's kernel compilation test, how much physical RAM does he have for his machine, what type of CPU is it, and what (approximate) type of device does he use for swap? I'll see if I can partially duplicate his results at this end. So far all my tests have been done with a fast CPU - perhaps I should try the P166/MMX or even try loading linux-pmac onto my 8100. It's a PIII/500 with one ide disk. -Mike - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On the subject of Mike Galbraith's kernel compilation test, how much physical RAM does he have for his machine, what type of CPU is it, and what (approximate) type of device does he use for swap? I'll see if I can partially duplicate his results at this end. So far all my tests have been done with a fast CPU - perhaps I should try the P166/MMX or even try loading linux-pmac onto my 8100. It's a PIII/500 with one ide disk. ...with how much RAM? That's the important bit. -- from: Jonathan Chromatix Morton mail: [EMAIL PROTECTED] (not for attachments) The key to knowledge is not to rely on people to teach you it. GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*) - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, Tobias Ringstrom wrote: On Fri, 8 Jun 2001, Mike Galbraith wrote: On Fri, 8 Jun 2001, Tobias Ringstrom wrote: On Fri, 8 Jun 2001, Mike Galbraith wrote: I gave this a shot at my favorite vm beater test (make -j30 bzImage) while testing some other stuff today. Could you please explain what is good about this test? I understand that it will stress the VM, but will it do so in a realistic and relevant way? Can you explain what is bad about this test? ;) It spins the same VM wheels I think a load of ~30 is quit uncommon, and therefor it is unclear to me that it would be a test that would be repesentative of most normal loads. It's not supposed to be repesentative. It's supposed to take the box rapidly (but not instantly) from idle through lo-medium-high and maintain solid throughput. as any other load does. What's the difference if I have a bunch of httpd allocating or a bunch of cc1/as/ld? This load has a modest cachable data set and is compute bound.. and above all gives very repeatable results. Not a big difference. The difference I was thinking abount is the difference between spawning lots of processes allocating, using and freeing lots of memory, compared to a case where you have a few processes touching a lot of already allocated pages in some pattern. I was wondering whether optimizing for your case would be good or bad for the other case. I know, I know, I should do more testing myself. And I should probably not ask you, since you really really like your test, and you will probably just say yes... ;-) It's not a matter of optimizing for my case.. that would be horrible. It's a matter of is the vm capable of rapid and correct responses. At home, I'm running a couple of computers. One of them is a slow computer running Linux, serving mail, NFS, SMB, etc. I'm usually logged in on a couple of virtual consoles. On this machine, I do not mind if all shells, daemons and other idle processes are beeing swapped out in favor of disk cache for the NFS and SMB serving. In fact, that is a very good thing, and I want it that way. Another maching is my desktop machine. When using this maching, I really hate when my emacsen, browsers, xterms, etc are swapped out just to give me some stupid disk cache for my xmms or compilations. I do not care if a kernel compile is a little slower as long as my applications are snappy. How could Linux predict this? It is a matter of taste, IMHO. I have no idea. It would be _wonderful_ if it could detect interactive tasks and give them preferencial treatment. I use it to watch reaction to surge. I watch for the vm to build to a solid maximum throughput without thrashing. That's the portion of VM that I'm interested in, so that's what I test. Besides :) I simply don't have the hardware to try to simulate hairy chested server loads. There are lots of folks with hairy chested boxes.. they should test that stuff. Agreed. More testing is needed. Now if we would have those knobs and wheels to turn, we could perhaps also tune our systems to behave as we like them, and submit that as well. Right now you need to be a kernel hacker, and see through all the magic with shm, mmap, a bunch of caches, page lists, etc. I'd give a lot for a nice picture (or state diagram) showing the lifetime of a page, but I have not found such a picture anywhere. Besides, the VM seems to change every new release anyway. I've been repeating ~this test since 2.0 times, and have noticed a 1:1 relationship. When I notice that my box is ~happy doing this load test, I also notice very few VM gripes hitting the list. Ok, but as you say, we need more tests. Isn't the interesting case when you have a number of processes using lots of memory, but only a part of all that memory is beeing actively used, and that memory fits in RAM. In that case, the VM should make sure that the not used memory is swapped out. In RAM you should have the used memory, but also disk cache if there is any RAM left. Does the current VM handle this case fine yet? IMHO, this is the case most people care about. It is definately the case I care about, at least. :-) The interesting case is _every_ case. Try seeing my particular test as a simulation of a small classroom box with 30 students compiling their assignments and it'll suddenly become quite realistic. You'll notice by the numbers I post that I was very careful to not overload the box in a rediculous manner when selecting the total size of the job.. it's just a heavily loaded box. This test does not overload my IO resources, so it tests the VM's ability to choose and move the right stuff at the right time to get the job done with a minimum of additional overhead. I did not understand those numbers when I saw them the first time. Now, I must say that your test does not look as silly as it did before. [snip..
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Fri, 8 Jun 2001, Marcelo Tosatti wrote: On Fri, 8 Jun 2001, John Stoffel wrote: More importantly, a *repeatable* set of tests is what is needed to test the VM and get consistent results from run to run, so you can see how your changes are impacting performance. The kernel compile doesn't really have any one process grow to a large fraction of memory, so dropping in a compile which *does* is a good thing. I agree with you. Mike, I'm sure you have noticed that stock kernel gives much better results than mine or Jonathan's patch. I noticed that Jonathan brought back waiting.. that (among others) made me very interested. Now the stock kernel gives us crappy interactivity compared to my patch. (Note: my patch still does not gives me the interactivity I want under high VM loads, but I hope to get there soon). (And that's why) Among other things (yes, I do love throughput) I've poked at the interactivity problem. I can't improve it anymore without doing some strategic waiting :( I used to be able to help it a little by doing a careful roll-up in scrub size as load builds.. trying to smooth the transition from latency oriented to hammer down throughput. BTW, we are talking with the OSDL (http://www.osdlab.org) guys about a possibility to setup a test system which would run a different variety of benchmarks to give us results of different kinds of workloads. If that ever happens, we'll probably get rid of most of this testing problems. Excellent! -Mike - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
On Sat, 9 Jun 2001, Jonathan Morton wrote: On the subject of Mike Galbraith's kernel compilation test, how much physical RAM does he have for his machine, what type of CPU is it, and what (approximate) type of device does he use for swap? I'll see if I can partially duplicate his results at this end. So far all my tests have been done with a fast CPU - perhaps I should try the P166/MMX or even try loading linux-pmac onto my 8100. It's a PIII/500 with one ide disk. ...with how much RAM? That's the important bit. Duh! :) I'm a dipstick. 128mb. -Mike - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
At 12:29 am +0100 8/6/2001, Shane Nay wrote: >(VM report at Marcelo Tosatti's request. He has mentioned that rather than >complaining about the VM that people mention what there experiences were. I >have tried to do so in the way that he asked.) >> By performance you mean interactivity or throughput? > >Interactivity. I don't have any throughput needs to speak of. > >I just ran a barage of tests on my machine, and the smallest it would ever >make the cache was 16M, it would prefer to kill processes rather than make >the cache smaller than that. http://www.chromatix.uklinux.net/linux-patches/vm-update-2.patch Try this. I can't guarantee it's SMP-safe yet (I'm leaving the gurus to that, but they haven't told me about any errors in the past hour so I'm assuming they aren't going to find anything glaringly wrong...), but you might like to see if your performance improves with it. It also fixes the OOM-killer bug, which you refer to above. Some measurements, from my own box (1GHz Athlon, 256Mb RAM): For the following benchmarks, physical memory availability was reduced according to the parameter in the left column. The benchmark is the wall-clock time taken to compile MySQL. mem=2.4.5 earlier tweaks now 48M 8m30s 6m30s 5m58s 32M unknown 2h15m 12m34s The following was performed with all 256Mb RAM available. This is compilation of MySQL using make -j 15. kernel: 2.4.5 now time: 6m30s 6m15s peak swap: 190M70M For the following test, the 256Mb swap partition on my IDE drive was disabled and replaced with a 1Gb swapfile on my Ultra160 SCSI drive. This is compilation of MySQL using make -j 20. kernel: 2.4.5 now time: 7m20s 6m30s peak swap: 370M254M Draw your own conclusions. :) -- from: Jonathan "Chromatix" Morton mail: [EMAIL PROTECTED] (not for attachments) The key to knowledge is not to rely on people to teach you it. GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*) - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: VM Report was:Re: Break 2.4 VM in five easy steps
At 12:29 am +0100 8/6/2001, Shane Nay wrote: (VM report at Marcelo Tosatti's request. He has mentioned that rather than complaining about the VM that people mention what there experiences were. I have tried to do so in the way that he asked.) By performance you mean interactivity or throughput? Interactivity. I don't have any throughput needs to speak of. I just ran a barage of tests on my machine, and the smallest it would ever make the cache was 16M, it would prefer to kill processes rather than make the cache smaller than that. http://www.chromatix.uklinux.net/linux-patches/vm-update-2.patch Try this. I can't guarantee it's SMP-safe yet (I'm leaving the gurus to that, but they haven't told me about any errors in the past hour so I'm assuming they aren't going to find anything glaringly wrong...), but you might like to see if your performance improves with it. It also fixes the OOM-killer bug, which you refer to above. Some measurements, from my own box (1GHz Athlon, 256Mb RAM): For the following benchmarks, physical memory availability was reduced according to the parameter in the left column. The benchmark is the wall-clock time taken to compile MySQL. mem=2.4.5 earlier tweaks now 48M 8m30s 6m30s 5m58s 32M unknown 2h15m 12m34s The following was performed with all 256Mb RAM available. This is compilation of MySQL using make -j 15. kernel: 2.4.5 now time: 6m30s 6m15s peak swap: 190M70M For the following test, the 256Mb swap partition on my IDE drive was disabled and replaced with a 1Gb swapfile on my Ultra160 SCSI drive. This is compilation of MySQL using make -j 20. kernel: 2.4.5 now time: 7m20s 6m30s peak swap: 370M254M Draw your own conclusions. :) -- from: Jonathan Chromatix Morton mail: [EMAIL PROTECTED] (not for attachments) The key to knowledge is not to rely on people to teach you it. GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*) - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
