Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-29 Thread Rik van Riel
On Fri, 29 Sep 2000, Andrea Arcangeli wrote: > On Fri, Sep 29, 2000 at 11:39:18AM -0300, Rik van Riel wrote: > > OK, good to see that we agree on the fact that we > > should age and swapout all pages equally agressively. > > Actually I think we should start looking at the mapped stuff > _only_

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-29 Thread Andrea Arcangeli
On Fri, Sep 29, 2000 at 11:39:18AM -0300, Rik van Riel wrote: > OK, good to see that we agree on the fact that we > should age and swapout all pages equally agressively. Actually I think we should start looking at the mapped stuff _only_ when the I/O cache aging is relevant. If the I/O cache

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-29 Thread Rik van Riel
On Thu, 28 Sep 2000, Andrea Arcangeli wrote: > On Thu, Sep 28, 2000 at 08:16:32AM -0300, Rik van Riel wrote: > > Andrea, I have the strong impression that your idea of > > memory balancing is based on the idea that the OS should > > out-smart the application instead of looking at the usage > >

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-29 Thread Rik van Riel
On Thu, 28 Sep 2000, Andrea Arcangeli wrote: On Thu, Sep 28, 2000 at 08:16:32AM -0300, Rik van Riel wrote: Andrea, I have the strong impression that your idea of memory balancing is based on the idea that the OS should out-smart the application instead of looking at the usage pattern of

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-29 Thread Andrea Arcangeli
On Fri, Sep 29, 2000 at 11:39:18AM -0300, Rik van Riel wrote: OK, good to see that we agree on the fact that we should age and swapout all pages equally agressively. Actually I think we should start looking at the mapped stuff _only_ when the I/O cache aging is relevant. If the I/O cache aging

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-29 Thread Rik van Riel
On Fri, 29 Sep 2000, Andrea Arcangeli wrote: On Fri, Sep 29, 2000 at 11:39:18AM -0300, Rik van Riel wrote: OK, good to see that we agree on the fact that we should age and swapout all pages equally agressively. Actually I think we should start looking at the mapped stuff _only_ when the

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-28 Thread Juan J. Quintela
> "ingo" == Ingo Molnar <[EMAIL PROTECTED]> writes: Hi ingo> 2) introducing sys_flush(), which flushes pages from the pagecache. It is not supposed that mincore can do that (yes, just now it is not implemented, but the interface is there to do that)? Just curious. -- In theory, practice

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-28 Thread Andrea Arcangeli
On Thu, Sep 28, 2000 at 05:13:59PM +0200, Ingo Molnar wrote: > Can anyone see any problems with the concept of this approach? This can be It works only on top of a filesystem while all the checkpointing clever stuff is done internally by the DB (infact it _needs_ O_SYNC when it works on the fs).

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-28 Thread Ingo Molnar
On Thu, 28 Sep 2000, Andrea Arcangeli wrote: > The DBMS uses shared SCSI disks across multiple hosts on the same SCSI > bus and synchronize the distributed cache via TCP. Tell me how to do > that with the OS cache and mmap. this could be supported by: 1) mlock()-ing the whole mapping. 2)

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-28 Thread Andrea Arcangeli
On Thu, Sep 28, 2000 at 01:31:40PM +0200, Ingo Molnar wrote: > if the shm contains raw I/O data, then thats flawed application design - > an mmap()-ed file should be used instead. Shm is equivalent to shared The DBMS uses shared SCSI disks across multiple hosts on the same SCSI bus and

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-28 Thread Andrea Arcangeli
On Thu, Sep 28, 2000 at 08:16:32AM -0300, Rik van Riel wrote: > Andrea, I have the strong impression that your idea of > memory balancing is based on the idea that the OS should > out-smart the application instead of looking at the usage > pattern of the pages in memory. Not sure what you mean

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-28 Thread Andrea Arcangeli
On Thu, Sep 28, 2000 at 07:08:51AM -0300, Rik van Riel wrote: > taking care of this itself. But this is not something the OS > should prescribe to the application. Agreed. > (unless the SHM users tell you that this is the normal way > they use SHM ... but as Christoph just told us, it isn't)

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-28 Thread Ingo Molnar
On Thu, 28 Sep 2000, Rik van Riel wrote: > The OS has no business knowing what's inside that SHM page. exactly. > IF the shm contains I/O cache, maybe you're right. However, > until you know that this is the case, optimising for that > situation just doesn't make any sense. if the shm

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-28 Thread Rik van Riel
On Thu, 28 Sep 2000, Rik van Riel wrote: > On Wed, 27 Sep 2000, Andrea Arcangeli wrote: > > But again: if the shm contains I/O cache it should be released > > and not swapped out. Swapping out shmfs that contains I/O cache > > would be exactly like swapping out page-cache. > > The OS has no

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-28 Thread Rik van Riel
On Wed, 27 Sep 2000, Andrea Arcangeli wrote: > On Wed, Sep 27, 2000 at 10:11:43AM +0200, Christoph Rohland wrote: > > I just checked one oracle system and it did not lock the memory. And I > > If that memory is used for I/O cache then such memory should > released when the system runs into swap

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-28 Thread Rik van Riel
On Wed, 27 Sep 2000, Andrea Arcangeli wrote: On Wed, Sep 27, 2000 at 10:11:43AM +0200, Christoph Rohland wrote: I just checked one oracle system and it did not lock the memory. And I If that memory is used for I/O cache then such memory should released when the system runs into swap

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-28 Thread Rik van Riel
On Thu, 28 Sep 2000, Rik van Riel wrote: On Wed, 27 Sep 2000, Andrea Arcangeli wrote: But again: if the shm contains I/O cache it should be released and not swapped out. Swapping out shmfs that contains I/O cache would be exactly like swapping out page-cache. The OS has no business

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-28 Thread Ingo Molnar
On Thu, 28 Sep 2000, Rik van Riel wrote: The OS has no business knowing what's inside that SHM page. exactly. IF the shm contains I/O cache, maybe you're right. However, until you know that this is the case, optimising for that situation just doesn't make any sense. if the shm contains

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-28 Thread Andrea Arcangeli
On Thu, Sep 28, 2000 at 08:16:32AM -0300, Rik van Riel wrote: Andrea, I have the strong impression that your idea of memory balancing is based on the idea that the OS should out-smart the application instead of looking at the usage pattern of the pages in memory. Not sure what you mean with

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-28 Thread Andrea Arcangeli
On Thu, Sep 28, 2000 at 01:31:40PM +0200, Ingo Molnar wrote: if the shm contains raw I/O data, then thats flawed application design - an mmap()-ed file should be used instead. Shm is equivalent to shared The DBMS uses shared SCSI disks across multiple hosts on the same SCSI bus and synchronize

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-28 Thread Andrea Arcangeli
On Thu, Sep 28, 2000 at 05:13:59PM +0200, Ingo Molnar wrote: Can anyone see any problems with the concept of this approach? This can be It works only on top of a filesystem while all the checkpointing clever stuff is done internally by the DB (infact it _needs_ O_SYNC when it works on the fs).

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-28 Thread Juan J. Quintela
"ingo" == Ingo Molnar [EMAIL PROTECTED] writes: Hi ingo 2) introducing sys_flush(), which flushes pages from the pagecache. It is not supposed that mincore can do that (yes, just now it is not implemented, but the interface is there to do that)? Just curious. -- In theory, practice and

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-27 Thread Andrea Arcangeli
On Wed, Sep 27, 2000 at 12:25:44PM -0600, Erik Andersen wrote: > Or sysinfo(2). Same thing... sysinfo structure doesn't export the number of active pages in the system. Andrea - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-27 Thread Erik Andersen
On Wed Sep 27, 2000 at 07:42:00PM +0200, Andrea Arcangeli wrote: > > You should of course poll the /proc/meminfo. (/proc/meminfo works in O(1) in > 2.4.x so it's just the overhead of a read syscall) Or sysinfo(2). Same thing... -Erik -- Erik B. Andersen email: [EMAIL PROTECTED] --This

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-27 Thread Andrea Arcangeli
On Wed, Sep 27, 2000 at 06:56:42PM +0200, Christoph Rohland wrote: > Yes, but how does the application detect that it should free the mem? The trivial way is not to detect it and to allow the user to select how much memory it will use as cache and to take it locked and then don't care (he will

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-27 Thread Christoph Rohland
Andrea Arcangeli <[EMAIL PROTECTED]> writes: > On Wed, Sep 27, 2000 at 10:11:43AM +0200, Christoph Rohland wrote: > > I just checked one oracle system and it did not lock the memory. And I > > If that memory is used for I/O cache then such memory should > released when the system runs into swap

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-27 Thread Andrea Arcangeli
On Wed, Sep 27, 2000 at 10:11:43AM +0200, Christoph Rohland wrote: > I just checked one oracle system and it did not lock the memory. And I If that memory is used for I/O cache then such memory should released when the system runs into swap instead of swapping it out too (otherwise it's not

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-27 Thread Christoph Rohland
Ingo Molnar <[EMAIL PROTECTED]> writes: > On 27 Sep 2000, Christoph Rohland wrote: > > > Nobody should rely on shm swapping for productive use. But you have > > changing/increasing loads on application servers and out of a sudden > > you run oom. In this case the system should behave and it is

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-27 Thread Ingo Molnar
On 27 Sep 2000, Christoph Rohland wrote: > Nobody should rely on shm swapping for productive use. But you have > changing/increasing loads on application servers and out of a sudden > you run oom. In this case the system should behave and it is _very_ > good to have a smooth behaviour. it

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-27 Thread Christoph Rohland
Andrea Arcangeli <[EMAIL PROTECTED]> writes: > Said that I heard of real world programs that have a .text larger than 2G =:-O > I know Oracle (and most other DB) are very shm intensive. However > the fact you say the shm is not locked in memory is really a news to > me. I really remembered

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-27 Thread Christoph Rohland
Andrea Arcangeli [EMAIL PROTECTED] writes: Said that I heard of real world programs that have a .text larger than 2G =:-O I know Oracle (and most other DB) are very shm intensive. However the fact you say the shm is not locked in memory is really a news to me. I really remembered that the

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-27 Thread Ingo Molnar
On 27 Sep 2000, Christoph Rohland wrote: Nobody should rely on shm swapping for productive use. But you have changing/increasing loads on application servers and out of a sudden you run oom. In this case the system should behave and it is _very_ good to have a smooth behaviour. it might

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-27 Thread Christoph Rohland
Ingo Molnar [EMAIL PROTECTED] writes: On 27 Sep 2000, Christoph Rohland wrote: Nobody should rely on shm swapping for productive use. But you have changing/increasing loads on application servers and out of a sudden you run oom. In this case the system should behave and it is _very_

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-27 Thread Andrea Arcangeli
On Wed, Sep 27, 2000 at 10:11:43AM +0200, Christoph Rohland wrote: I just checked one oracle system and it did not lock the memory. And I If that memory is used for I/O cache then such memory should released when the system runs into swap instead of swapping it out too (otherwise it's not cache

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-27 Thread Christoph Rohland
Andrea Arcangeli [EMAIL PROTECTED] writes: On Wed, Sep 27, 2000 at 10:11:43AM +0200, Christoph Rohland wrote: I just checked one oracle system and it did not lock the memory. And I If that memory is used for I/O cache then such memory should released when the system runs into swap instead

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-27 Thread Andrea Arcangeli
On Wed, Sep 27, 2000 at 06:56:42PM +0200, Christoph Rohland wrote: Yes, but how does the application detect that it should free the mem? The trivial way is not to detect it and to allow the user to select how much memory it will use as cache and to take it locked and then don't care (he will

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-27 Thread Erik Andersen
On Wed Sep 27, 2000 at 07:42:00PM +0200, Andrea Arcangeli wrote: You should of course poll the /proc/meminfo. (/proc/meminfo works in O(1) in 2.4.x so it's just the overhead of a read syscall) Or sysinfo(2). Same thing... -Erik -- Erik B. Andersen email: [EMAIL PROTECTED] --This

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-27 Thread Andrea Arcangeli
On Wed, Sep 27, 2000 at 12:25:44PM -0600, Erik Andersen wrote: Or sysinfo(2). Same thing... sysinfo structure doesn't export the number of active pages in the system. Andrea - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED]

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-26 Thread Christoph Rohland
Andrea Arcangeli <[EMAIL PROTECTED]> writes: > Could you tell me what's wrong in having an app with a 1.5G mapped executable > (or a tiny executable but with a 1.5G shared/private file mapping if you > prefer), O.K. that sound more reasonable. I was reading image as program text... and a 1.5GB

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-26 Thread Andrea Arcangeli
On Tue, Sep 26, 2000 at 06:20:47PM +0200, Christoph Rohland wrote: > O.K. that sound more reasonable. I was reading image as program > text... and a 1.5GB program text is a something I never have seen (and > hopefully will never see :-) :) >From the shrink_mmap complexity of the algorithm point

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-26 Thread Andrea Arcangeli
On Tue, Sep 26, 2000 at 08:54:23AM +0200, Christoph Rohland wrote: > "Stephen C. Tweedie" <[EMAIL PROTECTED]> writes: > > > Hi, > > > > On Mon, Sep 25, 2000 at 09:32:42PM +0200, Andrea Arcangeli wrote: > > > > > Having shrink_mmap that browse the mapped page cache is useless > > > as having

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-26 Thread Andrea Arcangeli
On Tue, Sep 26, 2000 at 08:54:23AM +0200, Christoph Rohland wrote: "Stephen C. Tweedie" [EMAIL PROTECTED] writes: Hi, On Mon, Sep 25, 2000 at 09:32:42PM +0200, Andrea Arcangeli wrote: Having shrink_mmap that browse the mapped page cache is useless as having shrink_mmap browsing

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-26 Thread Andrea Arcangeli
On Tue, Sep 26, 2000 at 06:20:47PM +0200, Christoph Rohland wrote: O.K. that sound more reasonable. I was reading image as program text... and a 1.5GB program text is a something I never have seen (and hopefully will never see :-) :) From the shrink_mmap complexity of the algorithm point of

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-26 Thread Christoph Rohland
Andrea Arcangeli [EMAIL PROTECTED] writes: Could you tell me what's wrong in having an app with a 1.5G mapped executable (or a tiny executable but with a 1.5G shared/private file mapping if you prefer), O.K. that sound more reasonable. I was reading image as program text... and a 1.5GB

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Linus Torvalds
On Tue, 26 Sep 2000, Andrea Arcangeli wrote: > > The machine will run low on memory as soon as I read 200mbyte from disk. So? Yes, at that point we'll do the LRU dance. Then we won't be low on memory any more, and we won't do the LRU dance any more. What's the magic in zoneinfo that makes

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Andrea Arcangeli
On Mon, Sep 25, 2000 at 03:30:10PM -0700, Linus Torvalds wrote: > On Tue, 26 Sep 2000, Andrea Arcangeli wrote: > > > > I'm talking about the fact that if you have a file mmapped in 1.5G of RAM > > test9 will waste time rolling between LRUs 384000 pages, while classzone > > won't ever see 1 of

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Andrea Arcangeli
On Tue, Sep 26, 2000 at 12:30:28AM +0200, Juan J. Quintela wrote: > Which is completely wrong if the program uses _any not completely_ > unusual locality of reference. Think twice about that, it is more > probable that you need more that 300MB of filesystem cache that you > have an aplication

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Rik van Riel
On Tue, 26 Sep 2000, Andrea Arcangeli wrote: > On Mon, Sep 25, 2000 at 08:54:57PM +0100, Stephen C. Tweedie wrote: > > basically the whole of memory is data cache, some of which is mapped > > and some of which is not? > > As as said in the last email aging on the cache is supposed to that. > >

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Andrea Arcangeli
On Mon, Sep 25, 2000 at 07:26:56PM -0300, Rik van Riel wrote: > IMHO this is a minor issue because: I don't think it's a minor issue. If you don't have reschedule point in your equivalent of shrink_mmap and this 1.5G will happen to be consecutive in the lru order (quite probably if it's been

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Andrea Arcangeli
On Mon, Sep 25, 2000 at 08:54:57PM +0100, Stephen C. Tweedie wrote: > OK, and here's another simple real life example. A 2GB RAM machine > running something like Oracle with a hundred client processes all > shm-mapping the same shared memory segment. Oracle takes the SHM locked, and it will

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Linus Torvalds
On Tue, 26 Sep 2000, Andrea Arcangeli wrote: > > I'm talking about the fact that if you have a file mmapped in 1.5G of RAM > test9 will waste time rolling between LRUs 384000 pages, while classzone > won't ever see 1 of those pages until you run low on fs cache. What drugs are you on? Nobody

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Juan J. Quintela
> "andrea" == Andrea Arcangeli <[EMAIL PROTECTED]> writes: Hi andrea> I'm talking about the fact that if you have a file mmapped in 1.5G of RAM andrea> test9 will waste time rolling between LRUs 384000 pages, while classzone andrea> won't ever see 1 of those pages until you run low on fs

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Rik van Riel
On Tue, 26 Sep 2000, Andrea Arcangeli wrote: > On Mon, Sep 25, 2000 at 04:26:17PM -0300, Rik van Riel wrote: > > > > It doesn't --- that is part of the design. The vm scanner propagates > > > > > > And that's the inferior part of the design IMHO. > > > > Indeed, but physical page based aging

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Andrea Arcangeli
On Mon, Sep 25, 2000 at 04:26:17PM -0300, Rik van Riel wrote: > > > It doesn't --- that is part of the design. The vm scanner propagates > > > > And that's the inferior part of the design IMHO. > > Indeed, but physical page based aging is a definate > 2.5 thing ... ;( I'm talking about the

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Stephen C. Tweedie
Hi, On Mon, Sep 25, 2000 at 09:32:42PM +0200, Andrea Arcangeli wrote: > Having shrink_mmap that browse the mapped page cache is useless > as having shrink_mmap browsing kernel memory and anonymous pages > as it does in 2.2.x as far I can tell. It's an algorithm > complexity problem and it will

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Rik van Riel
On Mon, 25 Sep 2000, Andrea Arcangeli wrote: > On Mon, Sep 25, 2000 at 07:06:57PM +0100, Stephen C. Tweedie wrote: > > Good. One of the problems we always had in the past, though, was that > > getting the relative aging of cache vs. vmas was easy if you had a > > small set of test loads, but it

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Stephen C. Tweedie
Hi, On Mon, Sep 25, 2000 at 07:03:47PM +0200, Andrea Arcangeli wrote: > > > This really seems to be the biggest difference between the two > > approaches right now. The FreeBSD folks believe fervently that one of > > [ aging cache and mapped pages in the same cycle ] > > Right. > > And since

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Andrea Arcangeli
On Mon, Sep 25, 2000 at 07:21:48PM +0200, bert hubert wrote: > Ok, sorry. Kernel development is proceding at a furious pace and I sometimes > lose track. No problem :). > I seem to remember that people were impressed by classzone, but that the > implementation was very non-trivial and hard to

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread bert hubert
> We're talking about shrink_[id]cache_memory change. That have _nothing_ to do > with the VM changes that happened anywhere between test8 and test9-pre6. > > You were talking about a different thing. Ok, sorry. Kernel development is proceding at a furious pace and I sometimes lose track. > I

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Andrea Arcangeli
On Mon, Sep 25, 2000 at 05:24:42PM +0100, Stephen C. Tweedie wrote: > Your other recent complaint, that newly-swapped pages end up on the > wrong end of the LRU lists and can't be reclaimed without cycling the > rest of the pages in shrink_mmap, is also cured in Rik's code, by > placing pages

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Stephen C. Tweedie
Hi, On Mon, Sep 25, 2000 at 01:41:37AM +0200, Andrea Arcangeli wrote: > > Since you're talking about this I'll soon (as soon as I'll finish some other > thing that is just work in progress) release a classzone against latest's > 2.4.x. My approch is _quite_ different from the curren VM. Current

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Andrea Arcangeli
On Mon, Sep 25, 2000 at 06:20:40PM +0200, Ingo Molnar wrote: > i only suggested this as a debugging helper, instead of the suggested I don't think removing the superlock from all fs is good thing at this stage (I agree with SCT doing it only for ext2 [that's what we mostly care about] would be

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Ingo Molnar
On Mon, 25 Sep 2000, Alexander Viro wrote: > > i'd suggest to simply BUG() in schedule() if the superblock lock is held > > not directly by lock_super. Holding the superblock lock is IMO quite rude > > anyway (for performance and latency) - is there any place where we hold it > > for a long

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Alexander Viro
On Mon, 25 Sep 2000, Ingo Molnar wrote: > > On Mon, 25 Sep 2000, Stephen C. Tweedie wrote: > > > Sorry, but in this case you have got a lot more variables than you > > seem to think. The obvious lock is the ext2 superblock lock, but > > there are side cases with quota and O_SYNC which are

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Ingo Molnar
On Mon, 25 Sep 2000, Stephen C. Tweedie wrote: > Sorry, but in this case you have got a lot more variables than you > seem to think. The obvious lock is the ext2 superblock lock, but > there are side cases with quota and O_SYNC which are much less > commonly triggered. That's not even

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Stephen C. Tweedie
Hi, On Mon, Sep 25, 2000 at 12:36:50AM +0200, bert hubert wrote: > On Mon, Sep 25, 2000 at 12:13:42AM +0200, Andrea Arcangeli wrote: > > On Sun, Sep 24, 2000 at 10:43:03PM +0100, Stephen C. Tweedie wrote: > > > any form of serialisation on the quota file). This feels like rather > > > a lot of

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Stephen C. Tweedie
Hi, On Mon, Sep 25, 2000 at 12:36:50AM +0200, bert hubert wrote: On Mon, Sep 25, 2000 at 12:13:42AM +0200, Andrea Arcangeli wrote: On Sun, Sep 24, 2000 at 10:43:03PM +0100, Stephen C. Tweedie wrote: any form of serialisation on the quota file). This feels like rather a lot of new and

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Ingo Molnar
On Mon, 25 Sep 2000, Stephen C. Tweedie wrote: Sorry, but in this case you have got a lot more variables than you seem to think. The obvious lock is the ext2 superblock lock, but there are side cases with quota and O_SYNC which are much less commonly triggered. That's not even starting to

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Alexander Viro
On Mon, 25 Sep 2000, Ingo Molnar wrote: On Mon, 25 Sep 2000, Stephen C. Tweedie wrote: Sorry, but in this case you have got a lot more variables than you seem to think. The obvious lock is the ext2 superblock lock, but there are side cases with quota and O_SYNC which are much less

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Ingo Molnar
On Mon, 25 Sep 2000, Alexander Viro wrote: i'd suggest to simply BUG() in schedule() if the superblock lock is held not directly by lock_super. Holding the superblock lock is IMO quite rude anyway (for performance and latency) - is there any place where we hold it for a long time and

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Andrea Arcangeli
On Mon, Sep 25, 2000 at 06:20:40PM +0200, Ingo Molnar wrote: i only suggested this as a debugging helper, instead of the suggested I don't think removing the superlock from all fs is good thing at this stage (I agree with SCT doing it only for ext2 [that's what we mostly care about] would be

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Andrea Arcangeli
On Mon, Sep 25, 2000 at 05:24:42PM +0100, Stephen C. Tweedie wrote: Your other recent complaint, that newly-swapped pages end up on the wrong end of the LRU lists and can't be reclaimed without cycling the rest of the pages in shrink_mmap, is also cured in Rik's code, by placing pages which

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread bert hubert
We're talking about shrink_[id]cache_memory change. That have _nothing_ to do with the VM changes that happened anywhere between test8 and test9-pre6. You were talking about a different thing. Ok, sorry. Kernel development is proceding at a furious pace and I sometimes lose track. I

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Andrea Arcangeli
On Mon, Sep 25, 2000 at 07:21:48PM +0200, bert hubert wrote: Ok, sorry. Kernel development is proceding at a furious pace and I sometimes lose track. No problem :). I seem to remember that people were impressed by classzone, but that the implementation was very non-trivial and hard to

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Stephen C. Tweedie
Hi, On Mon, Sep 25, 2000 at 07:03:47PM +0200, Andrea Arcangeli wrote: This really seems to be the biggest difference between the two approaches right now. The FreeBSD folks believe fervently that one of [ aging cache and mapped pages in the same cycle ] Right. And since you move

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Stephen C. Tweedie
Hi, On Mon, Sep 25, 2000 at 09:32:42PM +0200, Andrea Arcangeli wrote: Having shrink_mmap that browse the mapped page cache is useless as having shrink_mmap browsing kernel memory and anonymous pages as it does in 2.2.x as far I can tell. It's an algorithm complexity problem and it will

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Andrea Arcangeli
On Mon, Sep 25, 2000 at 04:26:17PM -0300, Rik van Riel wrote: It doesn't --- that is part of the design. The vm scanner propagates And that's the inferior part of the design IMHO. Indeed, but physical page based aging is a definate 2.5 thing ... ;( I'm talking about the fact that if

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Rik van Riel
On Tue, 26 Sep 2000, Andrea Arcangeli wrote: On Mon, Sep 25, 2000 at 04:26:17PM -0300, Rik van Riel wrote: It doesn't --- that is part of the design. The vm scanner propagates And that's the inferior part of the design IMHO. Indeed, but physical page based aging is a definate

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Linus Torvalds
On Tue, 26 Sep 2000, Andrea Arcangeli wrote: I'm talking about the fact that if you have a file mmapped in 1.5G of RAM test9 will waste time rolling between LRUs 384000 pages, while classzone won't ever see 1 of those pages until you run low on fs cache. What drugs are you on? Nobody

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Juan J. Quintela
"andrea" == Andrea Arcangeli [EMAIL PROTECTED] writes: Hi andrea I'm talking about the fact that if you have a file mmapped in 1.5G of RAM andrea test9 will waste time rolling between LRUs 384000 pages, while classzone andrea won't ever see 1 of those pages until you run low on fs cache.

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Andrea Arcangeli
On Mon, Sep 25, 2000 at 08:54:57PM +0100, Stephen C. Tweedie wrote: OK, and here's another simple real life example. A 2GB RAM machine running something like Oracle with a hundred client processes all shm-mapping the same shared memory segment. Oracle takes the SHM locked, and it will never

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Andrea Arcangeli
On Mon, Sep 25, 2000 at 07:26:56PM -0300, Rik van Riel wrote: IMHO this is a minor issue because: I don't think it's a minor issue. If you don't have reschedule point in your equivalent of shrink_mmap and this 1.5G will happen to be consecutive in the lru order (quite probably if it's been

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Rik van Riel
On Tue, 26 Sep 2000, Andrea Arcangeli wrote: On Mon, Sep 25, 2000 at 08:54:57PM +0100, Stephen C. Tweedie wrote: basically the whole of memory is data cache, some of which is mapped and some of which is not? As as said in the last email aging on the cache is supposed to that. Wasting

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Andrea Arcangeli
On Tue, Sep 26, 2000 at 12:30:28AM +0200, Juan J. Quintela wrote: Which is completely wrong if the program uses _any not completely_ unusual locality of reference. Think twice about that, it is more probable that you need more that 300MB of filesystem cache that you have an aplication that

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Andrea Arcangeli
On Mon, Sep 25, 2000 at 03:30:10PM -0700, Linus Torvalds wrote: On Tue, 26 Sep 2000, Andrea Arcangeli wrote: I'm talking about the fact that if you have a file mmapped in 1.5G of RAM test9 will waste time rolling between LRUs 384000 pages, while classzone won't ever see 1 of those pages

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-25 Thread Linus Torvalds
On Tue, 26 Sep 2000, Andrea Arcangeli wrote: The machine will run low on memory as soon as I read 200mbyte from disk. So? Yes, at that point we'll do the LRU dance. Then we won't be low on memory any more, and we won't do the LRU dance any more. What's the magic in zoneinfo that makes it

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-24 Thread Miles Lane
bert hubert wrote: > On Mon, Sep 25, 2000 at 12:13:42AM +0200, Andrea Arcangeli wrote: > >> On Sun, Sep 24, 2000 at 10:43:03PM +0100, Stephen C. Tweedie wrote: >> >>> any form of serialisation on the quota file). This feels like rather >>> a lot of new and interesting deadlocks to be

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-24 Thread Andrea Arcangeli
On Mon, Sep 25, 2000 at 12:36:50AM +0200, bert hubert wrote: > True. But they also appear to be found and solved at an impressive rate. We're talking about shrink_[id]cache_memory change. That have _nothing_ to do with the VM changes that happened anywhere between test8 and test9-pre6. You were

[patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-24 Thread bert hubert
On Mon, Sep 25, 2000 at 12:13:42AM +0200, Andrea Arcangeli wrote: > On Sun, Sep 24, 2000 at 10:43:03PM +0100, Stephen C. Tweedie wrote: > > any form of serialisation on the quota file). This feels like rather > > a lot of new and interesting deadlocks to be introducing so late in > > 2.4. :-)

[patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-24 Thread bert hubert
On Mon, Sep 25, 2000 at 12:13:42AM +0200, Andrea Arcangeli wrote: On Sun, Sep 24, 2000 at 10:43:03PM +0100, Stephen C. Tweedie wrote: any form of serialisation on the quota file). This feels like rather a lot of new and interesting deadlocks to be introducing so late in 2.4. :-) True.

Re: [patch] vmfixes-2.4.0-test9-B2 - fixing deadlocks

2000-09-24 Thread Andrea Arcangeli
On Mon, Sep 25, 2000 at 12:36:50AM +0200, bert hubert wrote: True. But they also appear to be found and solved at an impressive rate. We're talking about shrink_[id]cache_memory change. That have _nothing_ to do with the VM changes that happened anywhere between test8 and test9-pre6. You were