Re: Subtle MM bug

2001-04-17 Thread H. Peter Anvin
Followup to: <[EMAIL PROTECTED]> By author:Rik van Riel <[EMAIL PROTECTED]> In newsgroup: linux.dev.kernel > > Suppose you have 8 high-priority tasks waiting on kswapd > and one lower-priority (but still higher than kswapd) > process running and preventing kswapd from doing its work. > Oh ..

Re: Subtle MM bug

2001-01-25 Thread Daniel Phillips
Rik van Riel wrote: > > On Tue, 9 Jan 2001, Daniel Phillips wrote: > > Linus Torvalds wrote: > > > (This is why I worked so hard at getting the PageDirty semantics right in > > > the last two months or so - and why I released 2.4.0 when I did. Getting > > > PageDirty right was the big step to mak

Re: Subtle MM bug

2001-01-18 Thread Rik van Riel
On 17 Jan 2001, Linus Torvalds wrote: > Rik van Riel <[EMAIL PROTECTED]> wrote: > >On Wed, 10 Jan 2001, Linus Torvalds wrote: > > > >> I looked at it a year or two ago myself, and came to the > >> conclusion that I don't want to blow up our page table size by a > >> factor of three or more, so I'

Re: Subtle MM bug

2001-01-17 Thread Rik van Riel
On 17 Jan 2001, Zlatko Calusic wrote: > Rik van Riel <[EMAIL PROTECTED]> writes: > > > > Second test: kernel compile make -j32 (empirically this puts the > > > VM under load, but not excessively!) > > > > > > 2.2.17 -> make -j32 392.49s user 47.87s system 168% cpu 4:21.13 total > > > 2.4.0 -> ma

Re: Subtle MM bug

2001-01-17 Thread Linus Torvalds
In article <[EMAIL PROTECTED]>, Rik van Riel <[EMAIL PROTECTED]> wrote: >On Wed, 10 Jan 2001, Linus Torvalds wrote: > >> I looked at it a year or two ago myself, and came to the >> conclusion that I don't want to blow up our page table size by a >> factor of three or more, so I'm not personally i

Re: Subtle MM bug

2001-01-17 Thread Rik van Riel
On Tue, 9 Jan 2001, Daniel Phillips wrote: > Linus Torvalds wrote: > > (This is why I worked so hard at getting the PageDirty semantics right in > > the last two months or so - and why I released 2.4.0 when I did. Getting > > PageDirty right was the big step to make all of the VM stuff possible in

Re: Subtle MM bug

2001-01-17 Thread Rik van Riel
On Wed, 10 Jan 2001, Linus Torvalds wrote: > I looked at it a year or two ago myself, and came to the > conclusion that I don't want to blow up our page table size by a > factor of three or more, so I'm not personally interested any > more. Maybe somebody else comes up with a better way to do it,

Re: Subtle MM bug

2001-01-17 Thread Rik van Riel
On Wed, 10 Jan 2001, Andrea Arcangeli wrote: > On Wed, Jan 10, 2001 at 10:46:07AM -0700, Eric W. Biederman wrote: > > My impression with the MM stuff is that everyone except linux is > > trying hard to clone BSD instead of thinking through the issues > > ourselves. > > I wasn't even thinking abou

Re: Subtle MM bug

2001-01-17 Thread Zlatko Calusic
Rik van Riel <[EMAIL PROTECTED]> writes: > > Second test: kernel compile make -j32 (empirically this puts the > > VM under load, but not excessively!) > > > > 2.2.17 -> make -j32 392.49s user 47.87s system 168% cpu 4:21.13 total > > 2.4.0 -> make -j32 389.59s user 31.29s system 182% cpu 3:50.2

Re: Subtle MM bug

2001-01-16 Thread Rik van Riel
On 9 Jan 2001, Zlatko Calusic wrote: > Rik van Riel <[EMAIL PROTECTED]> writes: > > > Now if 2.4 has worse _performance_ than 2.2 due to one > > reason or another, that I'd like to hear about ;) > > > > Oh, well, it seems that I was wrong. :) > > First test: hogmem 180 5 = allocate 180MB and dirty

Re: Subtle MM bug

2001-01-14 Thread Eric W. Biederman
Ralf Baechle <[EMAIL PROTECTED]> writes: > On Fri, Jan 12, 2001 at 09:11:43PM +, Russell King wrote: > > > Eric W. Biederman writes: > > > Hmm. I would think that increasing the logical page size in the kernel > > > would be the trivial way to handle virtual aliases. (i.e.) with a large >

Re: Subtle MM bug

2001-01-14 Thread Ralf Baechle
On Fri, Jan 12, 2001 at 09:11:43PM +, Russell King wrote: > Eric W. Biederman writes: > > Hmm. I would think that increasing the logical page size in the kernel > > would be the trivial way to handle virtual aliases. (i.e.) with a large > > enough page size you can't actually have a virtual

Re: Subtle MM bug

2001-01-14 Thread Ralf Baechle
On Fri, Jan 12, 2001 at 09:10:54AM -0700, Eric W. Biederman wrote: > > Having a reverse mappings is the least sucky way to handle virtual aliases > > of certain types of MIPS caches. > > Hmm. I would think that increasing the logical page size in the kernel would > be the trivial way to handle

Re: Subtle MM bug

2001-01-12 Thread Russell King
Eric W. Biederman writes: > Hmm. I would think that increasing the logical page size in the kernel > would be the trivial way to handle virtual aliases. (i.e.) with a large > enough page size you can't actually have a virtual alias. There are types of caches out there that no matter how large t

Re: Subtle MM bug

2001-01-12 Thread Eric W. Biederman
Ralf Baechle <[EMAIL PROTECTED]> writes: > On Thu, Jan 11, 2001 at 12:56:57AM +0100, David Weinehall wrote: > > > > The MMU on these systems is a CAM, and the mmu table is thus backwards to > > > convention. (It also means you can notionally map two physical addresses to > > > one virtual but th

Re: Subtle MM bug

2001-01-12 Thread Ralf Baechle
On Thu, Jan 11, 2001 at 12:56:57AM +0100, David Weinehall wrote: > > The MMU on these systems is a CAM, and the mmu table is thus backwards to > > convention. (It also means you can notionally map two physical addresses to > > one virtual but thats undefined in the implementation ;)) > > Are the

Re: Subtle MM bug

2001-01-11 Thread Alexander Viro
On Thu, 11 Jan 2001, Stephen C. Tweedie wrote: > > And how is that different from the current situation? > > It's not, which is the point I was making: COW doesn't actually solve > the pthreads problem. Far better to do it in user space. Oh, certainly. We need COW for completely unrelated re

Re: Subtle MM bug

2001-01-11 Thread Stephen C. Tweedie
Hi, On Thu, Jan 11, 2001 at 02:03:48PM -0500, Alexander Viro wrote: > On Thu, 11 Jan 2001, Stephen C. Tweedie wrote: > > > On Thu, Jan 11, 2001 at 02:12:05PM +0100, Trond Myklebust wrote: > > > > > > What's wrong with copy-on-write style semantics? IOW, anyone who > > > wants to change the cre

Re: Subtle MM bug

2001-01-11 Thread Albert D. Cahalan
Stephen C. Tweedie writes: > On Thu, Jan 11, 2001 at 11:50:21AM -0500, Albert D. Cahalan wrote: >> Stephen C. Tweedie writes: >>> But is it really worth the pain? I'd hate to have to audit the >>> entire VFS to make sure that it works if another thread changes our >>> credentials in the middle o

Re: Subtle MM bug

2001-01-11 Thread Alexander Viro
On Thu, 11 Jan 2001, Stephen C. Tweedie wrote: > Hi, > > On Thu, Jan 11, 2001 at 02:12:05PM +0100, Trond Myklebust wrote: > > > > What's wrong with copy-on-write style semantics? IOW, anyone who > > wants to change the credentials needs to make a private copy of the > > existing structure fi

Re: Subtle MM bug

2001-01-11 Thread Alexander Viro
On Thu, 11 Jan 2001, Stephen C. Tweedie wrote: > Hi, > > On Wed, Jan 10, 2001 at 12:11:16PM -0800, Linus Torvalds wrote: > > > > That said, we can easily support the notion of CLONE_CRED if we absolutely > > have to (and sane people just shouldn't use it), so if somebody wants to > > work on

Re: Subtle MM bug

2001-01-11 Thread Stephen C. Tweedie
Hi, On Thu, Jan 11, 2001 at 11:50:21AM -0500, Albert D. Cahalan wrote: > Stephen C. Tweedie writes: > > > > But is it really worth the pain? I'd hate to have to audit the > > entire VFS to make sure that it works if another thread changes our > > credentials in the middle of a syscall, so we eit

Re: Subtle MM bug

2001-01-11 Thread Albert D. Cahalan
Stephen C. Tweedie writes: > On Wed, Jan 10, 2001 at 12:11:16PM -0800, Linus Torvalds wrote: >> That said, we can easily support the notion of CLONE_CRED if >> we absolutely have to (and sane people just shouldn't use it), >> so if somebody wants to work on this for 2.5.x... > > But is it really

Re: Subtle MM bug

2001-01-11 Thread Jim Gettys
EMAIL PROTECTED]>, > Rik van Riel <[EMAIL PROTECTED]>, [EMAIL PROTECTED] > Subject: Re: Subtle MM bug > - > On Wed, 10 Jan 2001, David Woodhouse wrote: > > > > > [EMAIL PROTECTED] said: > > > The no-swap behaviour shoul dactually be pretty much

Re: Subtle MM bug

2001-01-11 Thread Stephen C. Tweedie
Hi, On Thu, Jan 11, 2001 at 02:12:05PM +0100, Trond Myklebust wrote: > > What's wrong with copy-on-write style semantics? IOW, anyone who > wants to change the credentials needs to make a private copy of the > existing structure first. Because COW only solves the problem if each task is only c

Re: Subtle MM bug

2001-01-11 Thread Trond Myklebust
> " " == Stephen C Tweedie <[EMAIL PROTECTED]> writes: > Hi, On Wed, Jan 10, 2001 at 12:11:16PM -0800, Linus Torvalds > wrote: >> >> That said, we can easily support the notion of CLONE_CRED if we >> absolutely have to (and sane people just shouldn't use it), so >> i

Re: Subtle MM bug

2001-01-11 Thread Andi Kleen
On Thu, Jan 11, 2001 at 12:56:04PM +, Stephen C. Tweedie wrote: > Hi, > > On Wed, Jan 10, 2001 at 12:11:16PM -0800, Linus Torvalds wrote: > > > > That said, we can easily support the notion of CLONE_CRED if we absolutely > > have to (and sane people just shouldn't use it), so if somebody wan

Re: Subtle MM bug

2001-01-11 Thread Stephen C. Tweedie
Hi, On Wed, Jan 10, 2001 at 12:11:16PM -0800, Linus Torvalds wrote: > > That said, we can easily support the notion of CLONE_CRED if we absolutely > have to (and sane people just shouldn't use it), so if somebody wants to > work on this for 2.5.x... But is it really worth the pain? I'd hate to

Re: Subtle MM bug

2001-01-11 Thread Trond Myklebust
> " " == Alan Cox <[EMAIL PROTECTED]> writes: >> As the thread started it's not only only needed for pthreads, >> but also for NFS and setuid (actually NFS already implements it >> privately), and probably other network file systems too. So >> it's far from being only a "bad

[2.4.0 pre-PATCH] 830MB barrier (was: Subtle MM bug)

2001-01-10 Thread Wayne Whitney
On Tue, 9 Jan 2001, Szabolcs Szakacsits wrote: > 3) ask kernel developers to get rid of this "brk hits the fixed start > address of mmapped areas" or the other way around complaints "mmapped > area should start at lower address" limitation. E.g. Solaris does > growing up heap, growing down mmap a

Re: Subtle MM bug

2001-01-10 Thread Alan Cox
> > The MMU on these systems is a CAM, and the mmu table is thus backwards to > > convention. (It also means you can notionally map two physical addresses to > > one virtual but thats undefined in the implementation ;)) > > Are there any other (not yet supported) platforms with similar (or other

Re: Subtle MM bug (really 830MB barrier question)

2001-01-10 Thread Wayne Whitney
On Tue, 9 Jan 2001, Szabolcs Szakacsits wrote: > 3) ask kernel developers to get rid of this "brk hits the fixed start > address of mmapped areas" or the other way around complaints "mmapped > area should start at lower address" limitation. E.g. Solaris does > growing up heap, growing down mmap a

Re: Subtle MM bug

2001-01-10 Thread David Weinehall
On Wed, Jan 10, 2001 at 07:36:43PM +, Alan Cox wrote: > > I looked at it a year or two ago myself, and came to the conclusion that I > > don't want to blow up our page table size by a factor of three or more, so > > I'm not personally interested any more. Maybe somebody else comes up with > >

Re: Subtle MM bug

2001-01-10 Thread Linus Torvalds
On Wed, 10 Jan 2001, Alan Cox wrote: > > That is extremely undesirable behaviour. setuid() changes for pthreads crud > should be done by the library emulation layer. Many people have very real > and very good reasons for running multiple parallel ids. Just try writing > a threaded ftp daemon (n

Re: Subtle MM bug

2001-01-10 Thread Chris Wing
Alan: > I've seen exactly nil cases where there are any security holes in apps caused > by that pthreads api non adherance. I don't know of any exploitable bugs that were found in it, but the identd server included in Red Hat 6.1 (pidentd 3.0.10) unintentionally ran as root instead of nobody be

Re: Subtle MM bug

2001-01-10 Thread Alan Cox
> As the thread started it's not only only needed for pthreads, but also for NFS > and setuid (actually NFS already implements it privately), and probably other network > file systems too. So it's far from being only a "bad standard corner case". I wonder how Linux 2.2 worked, that doesnt have

Re: Subtle MM bug

2001-01-10 Thread Andi Kleen
On Wed, Jan 10, 2001 at 07:48:04PM +, Alan Cox wrote: > > As the thread started it's not only only needed for pthreads, but also for NFS > > and setuid (actually NFS already implements it privately), and probably other >network > > file systems too. So it's far from being only a "bad standar

Re: Subtle MM bug

2001-01-10 Thread Andi Kleen
On Wed, Jan 10, 2001 at 07:40:49PM +, Alan Cox wrote: > > Of course not by default, it would be a new clone flag (with default to on in > > linuxthreads though, to not cause security holes in ported programs like today) > > I've seen exactly nil cases where there are any security holes in ap

Re: Subtle MM bug

2001-01-10 Thread Alan Cox
> Of course not by default, it would be a new clone flag (with default to on in > linuxthreads though, to not cause security holes in ported programs like today) I've seen exactly nil cases where there are any security holes in apps caused by that pthreads api non adherance. There are also far t

Re: Subtle MM bug

2001-01-10 Thread Alan Cox
> I looked at it a year or two ago myself, and came to the conclusion that I > don't want to blow up our page table size by a factor of three or more, so > I'm not personally interested any more. Maybe somebody else comes up with > a better way to do it, or with a really compelling reason to. The

Re: Subtle MM bug

2001-01-10 Thread Andi Kleen
On Wed, Jan 10, 2001 at 07:31:52PM +, Alan Cox wrote: > > struct ucred is also needed to get LinuxThreads POSIX compliant (sharing > > credentials between threads, but still keeping system calls atomic in > > relation to credential changes) > > That is extremely undesirable behaviour. setuid

Re: Subtle MM bug

2001-01-10 Thread Alan Cox
> struct ucred is also needed to get LinuxThreads POSIX compliant (sharing > credentials between threads, but still keeping system calls atomic in > relation to credential changes) That is extremely undesirable behaviour. setuid() changes for pthreads crud should be done by the library emulation

Re: Subtle MM bug

2001-01-10 Thread David S. Miller
Date:Wed, 10 Jan 2001 11:03:21 -0800 (PST) From: Linus Torvalds <[EMAIL PROTECTED]> "Feel free to try" is definitely the open source motto. I basically came to the conclusion that it sucks when I gave it a go. In my scheme I tried to save space by using very small descriptors t

Re: Subtle MM bug

2001-01-10 Thread Linus Torvalds
On 10 Jan 2001, Eric W. Biederman wrote: > Andrea Arcangeli <[EMAIL PROTECTED]> writes: > > > > Once I or other developer finishes with the reverse lookup from page to > > pte-chain (an implementation from DaveM just exists) we'll be able to put them > > in a separate lru, but it's certainly n

Re: Subtle MM bug

2001-01-10 Thread Andrea Arcangeli
On Wed, Jan 10, 2001 at 10:46:07AM -0700, Eric W. Biederman wrote: > Why do we even want to do reverse page tables? > It seems everyone is assuming this is a good thing and except for being I'm not assuming it's a good thing, but I believe it's something to try. > My impression with the MM stuff

Re: Subtle MM bug

2001-01-10 Thread Eric W. Biederman
Andrea Arcangeli <[EMAIL PROTECTED]> writes: > On Wed, Jan 10, 2001 at 11:46:03AM +, David Woodhouse wrote: > > So the VM code spends a fair amount of time scanning lists of pages which > > it really can't do anything about? > > Yes. > > > Would it be possible to put such pages on differen

Re: Subtle MM bug

2001-01-10 Thread Andi Kleen
On Tue, Jan 09, 2001 at 08:29:02PM +0100, Trond Myklebust wrote: > Al has mentioned that he wants us to move towards a *BSD-like system > of credentials (i.e. struct ucred) that could be used here, but that's > in the far future. In the meantime, we cache RPC credentials in the > struct file... s

Re: Subtle MM bug

2001-01-10 Thread Linus Torvalds
On Wed, 10 Jan 2001, David Woodhouse wrote: > > [EMAIL PROTECTED] said: > > The no-swap behaviour shoul dactually be pretty much identical, > > simply because both 2.2 and 2.4 will do the same thing: just skip > > dirty pages in the page tables because they cannot do anything about > > them.

Re: Subtle MM bug

2001-01-10 Thread Andrea Arcangeli
On Wed, Jan 10, 2001 at 11:46:03AM +, David Woodhouse wrote: > So the VM code spends a fair amount of time scanning lists of pages which > it really can't do anything about? Yes. > Would it be possible to put such pages on different list, so that the VM Currently to unmap the other pages w

Re: Subtle MM bug

2001-01-10 Thread David Woodhouse
[EMAIL PROTECTED] said: > The no-swap behaviour shoul dactually be pretty much identical, > simply because both 2.2 and 2.4 will do the same thing: just skip > dirty pages in the page tables because they cannot do anything about > them. So the VM code spends a fair amount of time scanning list

Re: Subtle MM bug

2001-01-09 Thread Linus Torvalds
On Wed, 10 Jan 2001, David Woodhouse wrote: > > How does this affect embedded systems with no swap space at all? The no-swap behaviour shoul dactually be pretty much identical, simply because both 2.2 and 2.4 will do the same thing: just skip dirty pages in the page tables because they cannot

Re: Subtle MM bug

2001-01-09 Thread Andrea Arcangeli
On Wed, Jan 10, 2001 at 01:45:47AM +, David Woodhouse wrote: > How does this affect embedded systems with no swap space at all? If there's no swap the swap-cache dirty-sticky issue can't arise. Andrea - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a

Re: Subtle MM bug

2001-01-09 Thread David Woodhouse
On Tue, 9 Jan 2001, Linus Torvalds wrote: > And this _is_ a downside, there's no question about it. There's the worry > about the potential loss of locality, but there's also the fact that you > effectively need a bigger swap partition with 2.4.x - never mind that > large portions of the allocati

Re: Subtle MM bug (really 830MB barrier question)

2001-01-09 Thread Wayne Whitney
On Tue, 9 Jan 2001, Szabolcs Szakacsits wrote: > Wayne, the patch below should fix your barrier problem [1 GB physical > memory configuration]. First, I just wanted to thank you and everyone else (Linus, Andrea, Dan Maas, Rik and others) who has responded to my emails. You guys are wonderful!

Re: Subtle MM bug

2001-01-09 Thread Zlatko Calusic
Simon Kirby <[EMAIL PROTECTED]> writes: > On Tue, Jan 09, 2001 at 10:47:57AM -0800, Linus Torvalds wrote: > > > And this _is_ a downside, there's no question about it. There's the worry > > about the potential loss of locality, but there's also the fact that you > > effectively need a bigger swa

Re: Subtle MM bug

2001-01-09 Thread Linus Torvalds
On Tue, 9 Jan 2001, Simon Kirby wrote: > > On Tue, Jan 09, 2001 at 10:47:57AM -0800, Linus Torvalds wrote: > > > And this _is_ a downside, there's no question about it. There's the worry > > about the potential loss of locality, but there's also the fact that you > > effectively need a bigger s

Re: Subtle MM bug

2001-01-09 Thread Simon Kirby
On Tue, Jan 09, 2001 at 10:47:57AM -0800, Linus Torvalds wrote: > And this _is_ a downside, there's no question about it. There's the worry > about the potential loss of locality, but there's also the fact that you > effectively need a bigger swap partition with 2.4.x - never mind that > large po

Re: Subtle MM bug (really 830MB barrier question)

2001-01-09 Thread Szabolcs Szakacsits
On Tue, 9 Jan 2001, Dan Maas wrote: > OK it's fairly obvious what's happening here. Your program is using > its own allocator, which relies solely on brk() to obtain more > memory. [... good explanation here ...] > Here's your short answer: ask the authors of your program to either > 1) replace

Re: Subtle MM bug

2001-01-09 Thread Linus Torvalds
In article <[EMAIL PROTECTED]>, Daniel Phillips <[EMAIL PROTECTED]> wrote: >Linus Torvalds wrote: >> (This is why I worked so hard at getting the PageDirty semantics right in >> the last two months or so - and why I released 2.4.0 when I did. Getting >> PageDirty right was the big step to make al

Re: Subtle MM bug

2001-01-09 Thread Trond Myklebust
> " " == Daniel Phillips <[EMAIL PROTECTED]> writes: > Linus Torvalds wrote: >> (This is why I worked so hard at getting the PageDirty >> semantics right in the last two months or so - and why I >> released 2.4.0 when I did. Getting PageDirty right was the big >> step to

Re: Subtle MM bug

2001-01-09 Thread Daniel Phillips
Linus Torvalds wrote: > (This is why I worked so hard at getting the PageDirty semantics right in > the last two months or so - and why I released 2.4.0 when I did. Getting > PageDirty right was the big step to make all of the VM stuff possible in > the first place. Even if it probably looked a bi

Re: Subtle MM bug

2001-01-09 Thread Linus Torvalds
On 9 Jan 2001, Zlatko Calusic wrote: > > But in the end I'm not sure. I made two simple tests and haven't found > any problems with 2.4.0 mm logic (opposed to 2.2.17). In fact, the new > kernel was faster in the more interesting (make -j32) test. I personally think 2.4.x is going to be as fast

Re: Subtle MM bug - 2 errors in MM code in pre-2.4.1?

2001-01-09 Thread Benjamin Redelings I
Hi - Based on my quick reading of this patch: + +empty: + spin_lock(&mmlist_lock); + return 0; The above should actually be spin_UNlock? Also the test for !inactive_shortage() seems to be inverted? + /* If refill_inactive_scan failed, try to page stuff out.. *

Re: Subtle MM bug

2001-01-09 Thread Eric W. Biederman
Linus Torvalds <[EMAIL PROTECTED]> writes: > On 8 Jan 2001, Eric W. Biederman wrote: > > > Zlatko Calusic <[EMAIL PROTECTED]> writes:> > > > > > > Yes, but a lot more data on the swap also means degraded performance, > > > because the disk head has to seek around in the much bigger area. Are >

Re: Subtle MM bug

2001-01-09 Thread Zlatko Calusic
Linus Torvalds <[EMAIL PROTECTED]> writes: > On 8 Jan 2001, Eric W. Biederman wrote: > > > Zlatko Calusic <[EMAIL PROTECTED]> writes:> > > > > > > Yes, but a lot more data on the swap also means degraded performance, > > > because the disk head has to seek around in the much bigger area. Are >

Re: Subtle MM bug (really 830MB barrier question)

2001-01-09 Thread Dan Maas
> 08048000-08b5c000 r-xp 03:05 1130923 /tmp/newmagma/magma.exe.dyn > 08b5c000-08cc9000 rw-p 00b13000 03:05 1130923 /tmp/newmagma/magma.exe.dyn > 08cc9000-0bd0 rwxp 00:00 0 > Now, subsequent to each memory allocation, only the second number in the > third line changes. It be

Re: Subtle MM bug

2001-01-08 Thread Linus Torvalds
On 8 Jan 2001, Eric W. Biederman wrote: > Zlatko Calusic <[EMAIL PROTECTED]> writes:> > > > > Yes, but a lot more data on the swap also means degraded performance, > > because the disk head has to seek around in the much bigger area. Are > > you sure this is all OK? > > I don't think we have

Re: Subtle MM bug

2001-01-08 Thread Eric W. Biederman
Zlatko Calusic <[EMAIL PROTECTED]> writes: > > Yes, but a lot more data on the swap also means degraded performance, > because the disk head has to seek around in the much bigger area. Are > you sure this is all OK? I don't think we have more data on the swap, just more data has an allocated ho

Re: Subtle MM bug

2001-01-08 Thread Linus Torvalds
On 9 Jan 2001, Zlatko Calusic wrote: > > Yes, but a lot more data on the swap also means degraded performance, > because the disk head has to seek around in the much bigger area. Are > you sure this is all OK? Yes and no. I'm not _sure_, obviously. However, one thing I _am_ sure of is that t

Re: Subtle MM bug (really 830MB barrier question)

2001-01-08 Thread Wayne Whitney
On Mon, Jan 08, 2001 at 03:22:44PM -0800, Wayne Whitney wrote: > I guess I conclude that either (1) MAGMA does not use libc's malloc > (checking on this, I doubt it) I'm still a bit unclear on this one. I now have two executables, magma.exe and magma.exe.dyn (ignore the .exe). magma.exe is st

Re: Subtle MM bug

2001-01-08 Thread Zlatko Calusic
Linus Torvalds <[EMAIL PROTECTED]> writes: > On Mon, 8 Jan 2001, Rik van Riel wrote: > > > On Sun, 7 Jan 2001, Wayne Whitney wrote: > > > > > Well, here is a workload that performs worse on 2.4.0 than on 2.2.19pre, > > > > > The typical machine is a dual Intel box with 512MB RAM and 512MB swap

Re: Subtle MM bug

2001-01-08 Thread Zlatko Calusic
Rik van Riel <[EMAIL PROTECTED]> writes: > Now if 2.4 has worse _performance_ than 2.2 due to one > reason or another, that I'd like to hear about ;) > Oh, well, it seems that I was wrong. :) First test: hogmem 180 5 = allocate 180MB and dirty it 5 times (on a 192MB machine) kernel | swap us

Re: Subtle MM bug

2001-01-08 Thread Linus Torvalds
In article <[EMAIL PROTECTED]>, Andrea Arcangeli <[EMAIL PROTECTED]> wrote: >On Mon, Jan 08, 2001 at 03:22:44PM -0800, Wayne Whitney wrote: >> I guess I conclude that either (1) MAGMA does not use libc's malloc >> (checking on this, I doubt it) or (2) glibc-2.1.92 knows of these >> variables but

Re: Subtle MM bug

2001-01-08 Thread Andrea Arcangeli
On Mon, Jan 08, 2001 at 03:22:44PM -0800, Wayne Whitney wrote: > I guess I conclude that either (1) MAGMA does not use libc's malloc > (checking on this, I doubt it) or (2) glibc-2.1.92 knows of these > variables but has not yet implemented the tuning (I'll try glibc-2.2) or > (3) this is not the

Re: Subtle MM bug

2001-01-08 Thread Wayne Whitney
On Mon, 8 Jan 2001, Wayne Whitney wrote: > On Mon, 8 Jan 2001, Szabolcs Szakacsits wrote: > > > AFAIK newer glibc = CVS glibc but the malloc() tune parameters work > > via environment variables for the current stable ones as well, > > I'll arrange a binary linked against glibc2.2, and then your s

Re: Subtle MM bug

2001-01-08 Thread Andrea Arcangeli
On Mon, Jan 08, 2001 at 02:00:19PM -0800, Wayne Whitney wrote: > I'd ask if this jives with your theory: if I configure the linux kernel > to be able to use 2GB of RAM, then the 870MB limit becomes much lower, to > 230MB. It's because the virtual address space for userspace tasks gets reduced fr

Re: Subtle MM bug

2001-01-08 Thread Wayne Whitney
On Mon, 8 Jan 2001, Szabolcs Szakacsits wrote: > AFAIK newer glibc = CVS glibc but the malloc() tune parameters work > via environment variables for the current stable ones as well, e.g. to > overcome the above "out of memory" one could do, > > % export MALLOC_MMAP_MAX_=100 > % export MALLOC_

Re: Subtle MM bug

2001-01-08 Thread Wayne Whitney
On Mon, 8 Jan 2001, Szabolcs Szakacsits wrote: > AFAIK newer glibc = CVS glibc but the malloc() tune parameters work > via environment variables for the current stable ones as well, Hmm, this must have been introduced in libc6? Unfortunately, I don't have the source code to MAGMA, and the binar

Re: Subtle MM bug

2001-01-08 Thread Wayne Whitney
On Mon, 8 Jan 2001, Rik van Riel wrote: > How does 2.4 perform when you add an extra GB of swap ? OK, some more data: First, I tried booting 2.4.0 with "nosmp" to see if the behavior I observe is SMP related. It isn't, there was no difference under 2.4.0 between 512MB/512MB/1CPU and 512MB/512M

Re: Subtle MM bug

2001-01-08 Thread Szabolcs Szakacsits
Andi Kleen <[EMAIL PROTECTED]> wrote: > On Sun, Jan 07, 2001 at 09:29:29PM -0800, Wayne Whitney wrote: > > package called MAGMA; at times this requires very large matrices. The > > RSS can get up to 870MB; for some reason a MAGMA process under linux > > thinks it has run out of memory at 870MB, r

Re: Subtle MM bug

2001-01-08 Thread Linus Torvalds
On Mon, 8 Jan 2001, Rik van Riel wrote: > > You are right in that we need to refill the inactive list > before calling page_launder(), but we'll also need a few > other modifications: NONE of your three additions do _anything_ to help us at all if we don't even see the dirty bit because the pa

Re: Subtle MM bug

2001-01-08 Thread Linus Torvalds
On Mon, 8 Jan 2001, Rik van Riel wrote: > On Sun, 7 Jan 2001, Wayne Whitney wrote: > > > Well, here is a workload that performs worse on 2.4.0 than on 2.2.19pre, > > > The typical machine is a dual Intel box with 512MB RAM and 512MB swap. > > How does 2.4 perform when you add an extra GB of

Re: Subtle MM bug

2001-01-08 Thread Rik van Riel
On 7 Jan 2001, Linus Torvalds wrote: > That doesn't resolve the "2.4.x behaves badly" thing, though. > > I've seen that one myself, and it seems to be simply due to the > fact that we're usually so good at gettign memory from > page_launder() that we never bother to try to swap stuff out. > And

Re: Subtle MM bug

2001-01-08 Thread Rik van Riel
On Sun, 7 Jan 2001, Wayne Whitney wrote: > Well, here is a workload that performs worse on 2.4.0 than on 2.2.19pre, > The typical machine is a dual Intel box with 512MB RAM and 512MB swap. How does 2.4 perform when you add an extra GB of swap ? 2.4 keeps dirty pages in the swap cache, so you w

Re: Subtle MM bug

2001-01-07 Thread Linus Torvalds
In article <[EMAIL PROTECTED]>, Andi Kleen <[EMAIL PROTECTED]> wrote: >On Sun, Jan 07, 2001 at 09:29:29PM -0800, Wayne Whitney wrote: >> The application is some mathematics computations (modular symbols) using a >> package called MAGMA; at times this requires very large matrices. The >> RSS can

Re: Subtle MM bug

2001-01-07 Thread Andi Kleen
On Sun, Jan 07, 2001 at 09:29:29PM -0800, Wayne Whitney wrote: > The application is some mathematics computations (modular symbols) using a > package called MAGMA; at times this requires very large matrices. The > RSS can get up to 870MB; for some reason a MAGMA process under linux > thinks it h

Re: Subtle MM bug

2001-01-07 Thread Wayne Whitney
On Sunday, January 7, 20001, Rik van Riel <[EMAIL PROTECTED]> wrote: > Now if 2.4 has worse _performance_ than 2.2 due to one reason or > another, that I'd like to hear about ;) Well, here is a workload that performs worse on 2.4.0 than on 2.2.19pre, and as it is the usual workload on my little

Re: Subtle MM bug

2001-01-07 Thread Zlatko Calusic
Rik van Riel <[EMAIL PROTECTED]> writes: > On 7 Jan 2001, Zlatko Calusic wrote: > > > Things go berzerk if you have one big process whose working set > > is around your physical memory size. > > "go berzerk" in what way? Does the system cause lots of extra > swap IO and does it make the system

Re: Subtle MM bug

2001-01-07 Thread Rik van Riel
On 7 Jan 2001, Zlatko Calusic wrote: > Things go berzerk if you have one big process whose working set > is around your physical memory size. "go berzerk" in what way? Does the system cause lots of extra swap IO and does it make the system thrash where 2.2 didn't even touch the disk ? > Final

Subtle MM bug

2001-01-07 Thread Zlatko Calusic
I'm trying to get more familiar with the MM code in 2.4.0, as can be seen from lots of questions I have on the subject. I discovered nasty mm behaviour under even moderate load (2.2 didn't have troubles). Things go berzerk if you have one big process whose working set is around your physical memo