On Fri, 29 Sep 2000, Andrea Arcangeli wrote:
> On Fri, Sep 29, 2000 at 11:39:18AM -0300, Rik van Riel wrote:
> > OK, good to see that we agree on the fact that we
> > should age and swapout all pages equally agressively.
>
> Actually I think we should start looking at the mapped stuff
> _only_ wh
On Fri, Sep 29, 2000 at 11:39:18AM -0300, Rik van Riel wrote:
> OK, good to see that we agree on the fact that we
> should age and swapout all pages equally agressively.
Actually I think we should start looking at the mapped stuff _only_ when the
I/O cache aging is relevant. If the I/O cache agin
On Thu, 28 Sep 2000, Andrea Arcangeli wrote:
> On Thu, Sep 28, 2000 at 08:16:32AM -0300, Rik van Riel wrote:
> > Andrea, I have the strong impression that your idea of
> > memory balancing is based on the idea that the OS should
> > out-smart the application instead of looking at the usage
> > pat
> "ingo" == Ingo Molnar <[EMAIL PROTECTED]> writes:
Hi
ingo> 2) introducing sys_flush(), which flushes pages from the pagecache.
It is not supposed that mincore can do that (yes, just now it is not
implemented, but the interface is there to do that)?
Just curious.
--
In theory, practice
On Thu, Sep 28, 2000 at 05:13:59PM +0200, Ingo Molnar wrote:
> Can anyone see any problems with the concept of this approach? This can be
It works only on top of a filesystem while all the checkpointing clever stuff
is done internally by the DB (infact it _needs_ O_SYNC when it works on the
fs).
On Thu, 28 Sep 2000, Andrea Arcangeli wrote:
> The DBMS uses shared SCSI disks across multiple hosts on the same SCSI
> bus and synchronize the distributed cache via TCP. Tell me how to do
> that with the OS cache and mmap.
this could be supported by:
1) mlock()-ing the whole mapping.
2) intr
On Thu, Sep 28, 2000 at 01:31:40PM +0200, Ingo Molnar wrote:
> if the shm contains raw I/O data, then thats flawed application design -
> an mmap()-ed file should be used instead. Shm is equivalent to shared
The DBMS uses shared SCSI disks across multiple hosts on the same SCSI bus
and synchroniz
On Thu, Sep 28, 2000 at 08:16:32AM -0300, Rik van Riel wrote:
> Andrea, I have the strong impression that your idea of
> memory balancing is based on the idea that the OS should
> out-smart the application instead of looking at the usage
> pattern of the pages in memory.
Not sure what you mean wi
On Thu, Sep 28, 2000 at 07:08:51AM -0300, Rik van Riel wrote:
> taking care of this itself. But this is not something the OS
> should prescribe to the application.
Agreed.
> (unless the SHM users tell you that this is the normal way
> they use SHM ... but as Christoph just told us, it isn't)
sh
On Thu, 28 Sep 2000, Rik van Riel wrote:
> The OS has no business knowing what's inside that SHM page.
exactly.
> IF the shm contains I/O cache, maybe you're right. However,
> until you know that this is the case, optimising for that
> situation just doesn't make any sense.
if the shm contain
On Thu, 28 Sep 2000, Rik van Riel wrote:
> On Wed, 27 Sep 2000, Andrea Arcangeli wrote:
> > But again: if the shm contains I/O cache it should be released
> > and not swapped out. Swapping out shmfs that contains I/O cache
> > would be exactly like swapping out page-cache.
>
> The OS has no bus
On Wed, 27 Sep 2000, Andrea Arcangeli wrote:
> On Wed, Sep 27, 2000 at 10:11:43AM +0200, Christoph Rohland wrote:
> > I just checked one oracle system and it did not lock the memory. And I
>
> If that memory is used for I/O cache then such memory should
> released when the system runs into swap i
On Wed, Sep 27, 2000 at 12:25:44PM -0600, Erik Andersen wrote:
> Or sysinfo(2). Same thing...
sysinfo structure doesn't export the number of active pages in the system.
Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
On Wed Sep 27, 2000 at 07:42:00PM +0200, Andrea Arcangeli wrote:
>
> You should of course poll the /proc/meminfo. (/proc/meminfo works in O(1) in
> 2.4.x so it's just the overhead of a read syscall)
Or sysinfo(2). Same thing...
-Erik
--
Erik B. Andersen email: [EMAIL PROTECTED]
--This mes
On Wed, Sep 27, 2000 at 06:56:42PM +0200, Christoph Rohland wrote:
> Yes, but how does the application detect that it should free the mem?
The trivial way is not to detect it and to allow the user to select how much
memory it will use as cache and to take it locked and then don't care (he will
ha
Andrea Arcangeli <[EMAIL PROTECTED]> writes:
> On Wed, Sep 27, 2000 at 10:11:43AM +0200, Christoph Rohland wrote:
> > I just checked one oracle system and it did not lock the memory. And I
>
> If that memory is used for I/O cache then such memory should
> released when the system runs into swap
On Wed, Sep 27, 2000 at 10:11:43AM +0200, Christoph Rohland wrote:
> I just checked one oracle system and it did not lock the memory. And I
If that memory is used for I/O cache then such memory should released when the
system runs into swap instead of swapping it out too (otherwise it's not cache
Ingo Molnar <[EMAIL PROTECTED]> writes:
> On 27 Sep 2000, Christoph Rohland wrote:
>
> > Nobody should rely on shm swapping for productive use. But you have
> > changing/increasing loads on application servers and out of a sudden
> > you run oom. In this case the system should behave and it is _
On 27 Sep 2000, Christoph Rohland wrote:
> Nobody should rely on shm swapping for productive use. But you have
> changing/increasing loads on application servers and out of a sudden
> you run oom. In this case the system should behave and it is _very_
> good to have a smooth behaviour.
it might
Andrea Arcangeli <[EMAIL PROTECTED]> writes:
> Said that I heard of real world programs that have a .text larger than 2G
=:-O
> I know Oracle (and most other DB) are very shm intensive. However
> the fact you say the shm is not locked in memory is really a news to
> me. I really remembered tha
Andrea Arcangeli <[EMAIL PROTECTED]> writes:
> Could you tell me what's wrong in having an app with a 1.5G mapped executable
> (or a tiny executable but with a 1.5G shared/private file mapping if you
> prefer),
O.K. that sound more reasonable. I was reading image as program
text... and a 1.5GB p
On Tue, Sep 26, 2000 at 06:20:47PM +0200, Christoph Rohland wrote:
> O.K. that sound more reasonable. I was reading image as program
> text... and a 1.5GB program text is a something I never have seen (and
> hopefully will never see :-)
:)
>From the shrink_mmap complexity of the algorithm point
On Tue, Sep 26, 2000 at 08:54:23AM +0200, Christoph Rohland wrote:
> "Stephen C. Tweedie" <[EMAIL PROTECTED]> writes:
>
> > Hi,
> >
> > On Mon, Sep 25, 2000 at 09:32:42PM +0200, Andrea Arcangeli wrote:
> >
> > > Having shrink_mmap that browse the mapped page cache is useless
> > > as having shr
"Stephen C. Tweedie" <[EMAIL PROTECTED]> writes:
> Hi,
>
> On Mon, Sep 25, 2000 at 09:32:42PM +0200, Andrea Arcangeli wrote:
>
> > Having shrink_mmap that browse the mapped page cache is useless
> > as having shrink_mmap browsing kernel memory and anonymous pages
> > as it does in 2.2.x as far
On Mon, Sep 25, 2000 at 04:18:13PM -0700, Linus Torvalds wrote:
>
>
> On Tue, 26 Sep 2000, Andrea Arcangeli wrote:
> >
> > The machine will run low on memory as soon as I read 200mbyte from disk.
>
> So?
>
> Yes, at that point we'll do the LRU dance. Then we won't be low on memory
> any more
On Tue, 26 Sep 2000, Andrea Arcangeli wrote:
>
> The machine will run low on memory as soon as I read 200mbyte from disk.
So?
Yes, at that point we'll do the LRU dance. Then we won't be low on memory
any more, and we won't do the LRU dance any more. What's the magic in
zoneinfo that makes it
On Mon, Sep 25, 2000 at 03:30:10PM -0700, Linus Torvalds wrote:
> On Tue, 26 Sep 2000, Andrea Arcangeli wrote:
> >
> > I'm talking about the fact that if you have a file mmapped in 1.5G of RAM
> > test9 will waste time rolling between LRUs 384000 pages, while classzone
> > won't ever see 1 of tho
On Tue, Sep 26, 2000 at 12:30:28AM +0200, Juan J. Quintela wrote:
> Which is completely wrong if the program uses _any not completely_
> unusual locality of reference. Think twice about that, it is more
> probable that you need more that 300MB of filesystem cache that you
> have an aplication tha
On Tue, 26 Sep 2000, Andrea Arcangeli wrote:
> On Mon, Sep 25, 2000 at 08:54:57PM +0100, Stephen C. Tweedie wrote:
> > basically the whole of memory is data cache, some of which is mapped
> > and some of which is not?
>
> As as said in the last email aging on the cache is supposed to that.
>
>
On Mon, Sep 25, 2000 at 07:26:56PM -0300, Rik van Riel wrote:
> IMHO this is a minor issue because:
I don't think it's a minor issue.
If you don't have reschedule point in your equivalent of shrink_mmap and this
1.5G will happen to be consecutive in the lru order (quite probably if it's
been pag
On Mon, Sep 25, 2000 at 08:54:57PM +0100, Stephen C. Tweedie wrote:
> OK, and here's another simple real life example. A 2GB RAM machine
> running something like Oracle with a hundred client processes all
> shm-mapping the same shared memory segment.
Oracle takes the SHM locked, and it will neve
On Tue, 26 Sep 2000, Andrea Arcangeli wrote:
>
> I'm talking about the fact that if you have a file mmapped in 1.5G of RAM
> test9 will waste time rolling between LRUs 384000 pages, while classzone
> won't ever see 1 of those pages until you run low on fs cache.
What drugs are you on? Nobody l
> "andrea" == Andrea Arcangeli <[EMAIL PROTECTED]> writes:
Hi
andrea> I'm talking about the fact that if you have a file mmapped in 1.5G of RAM
andrea> test9 will waste time rolling between LRUs 384000 pages, while classzone
andrea> won't ever see 1 of those pages until you run low on fs cac
On Tue, 26 Sep 2000, Andrea Arcangeli wrote:
> On Mon, Sep 25, 2000 at 04:26:17PM -0300, Rik van Riel wrote:
> > > > It doesn't --- that is part of the design. The vm scanner propagates
> > >
> > > And that's the inferior part of the design IMHO.
> >
> > Indeed, but physical page based aging is
On Mon, Sep 25, 2000 at 04:26:17PM -0300, Rik van Riel wrote:
> > > It doesn't --- that is part of the design. The vm scanner propagates
> >
> > And that's the inferior part of the design IMHO.
>
> Indeed, but physical page based aging is a definate
> 2.5 thing ... ;(
I'm talking about the fac
Hi,
On Mon, Sep 25, 2000 at 09:32:42PM +0200, Andrea Arcangeli wrote:
> Having shrink_mmap that browse the mapped page cache is useless
> as having shrink_mmap browsing kernel memory and anonymous pages
> as it does in 2.2.x as far I can tell. It's an algorithm
> complexity problem and it will w
On Mon, 25 Sep 2000, Andrea Arcangeli wrote:
> On Mon, Sep 25, 2000 at 07:06:57PM +0100, Stephen C. Tweedie wrote:
> > Good. One of the problems we always had in the past, though, was that
> > getting the relative aging of cache vs. vmas was easy if you had a
> > small set of test loads, but it w
On Mon, Sep 25, 2000 at 07:06:57PM +0100, Stephen C. Tweedie wrote:
> Good. One of the problems we always had in the past, though, was that
> getting the relative aging of cache vs. vmas was easy if you had a
> small set of test loads, but it was really, really hard to find a
> balance that didn'
Hi,
On Mon, Sep 25, 2000 at 07:03:47PM +0200, Andrea Arcangeli wrote:
>
> > This really seems to be the biggest difference between the two
> > approaches right now. The FreeBSD folks believe fervently that one of
> > [ aging cache and mapped pages in the same cycle ]
>
> Right.
>
> And since
On Mon, Sep 25, 2000 at 07:21:48PM +0200, bert hubert wrote:
> Ok, sorry. Kernel development is proceding at a furious pace and I sometimes
> lose track.
No problem :).
> I seem to remember that people were impressed by classzone, but that the
> implementation was very non-trivial and hard to g
> We're talking about shrink_[id]cache_memory change. That have _nothing_ to do
> with the VM changes that happened anywhere between test8 and test9-pre6.
>
> You were talking about a different thing.
Ok, sorry. Kernel development is proceding at a furious pace and I sometimes
lose track.
> I
On Mon, Sep 25, 2000 at 05:24:42PM +0100, Stephen C. Tweedie wrote:
> Your other recent complaint, that newly-swapped pages end up on the
> wrong end of the LRU lists and can't be reclaimed without cycling the
> rest of the pages in shrink_mmap, is also cured in Rik's code, by
> placing pages whic
Hi,
On Mon, Sep 25, 2000 at 01:41:37AM +0200, Andrea Arcangeli wrote:
>
> Since you're talking about this I'll soon (as soon as I'll finish some other
> thing that is just work in progress) release a classzone against latest's
> 2.4.x. My approch is _quite_ different from the curren VM. Current
On Mon, Sep 25, 2000 at 06:20:40PM +0200, Ingo Molnar wrote:
> i only suggested this as a debugging helper, instead of the suggested
I don't think removing the superlock from all fs is good thing at this stage (I
agree with SCT doing it only for ext2 [that's what we mostly care about] would
be po
On Mon, 25 Sep 2000, Alexander Viro wrote:
> > i'd suggest to simply BUG() in schedule() if the superblock lock is held
> > not directly by lock_super. Holding the superblock lock is IMO quite rude
> > anyway (for performance and latency) - is there any place where we hold it
> > for a long time
On Mon, 25 Sep 2000, Ingo Molnar wrote:
>
> On Mon, 25 Sep 2000, Stephen C. Tweedie wrote:
>
> > Sorry, but in this case you have got a lot more variables than you
> > seem to think. The obvious lock is the ext2 superblock lock, but
> > there are side cases with quota and O_SYNC which are mu
On Mon, 25 Sep 2000, Stephen C. Tweedie wrote:
> Sorry, but in this case you have got a lot more variables than you
> seem to think. The obvious lock is the ext2 superblock lock, but
> there are side cases with quota and O_SYNC which are much less
> commonly triggered. That's not even starting
Hi,
On Mon, Sep 25, 2000 at 12:36:50AM +0200, bert hubert wrote:
> On Mon, Sep 25, 2000 at 12:13:42AM +0200, Andrea Arcangeli wrote:
> > On Sun, Sep 24, 2000 at 10:43:03PM +0100, Stephen C. Tweedie wrote:
> > > any form of serialisation on the quota file). This feels like rather
> > > a lot of n
bert hubert wrote:
> On Mon, Sep 25, 2000 at 12:13:42AM +0200, Andrea Arcangeli wrote:
>
>> On Sun, Sep 24, 2000 at 10:43:03PM +0100, Stephen C. Tweedie wrote:
>>
>>> any form of serialisation on the quota file). This feels like rather
>>> a lot of new and interesting deadlocks to be introduci
On Mon, Sep 25, 2000 at 12:36:50AM +0200, bert hubert wrote:
> True. But they also appear to be found and solved at an impressive rate.
We're talking about shrink_[id]cache_memory change. That have _nothing_ to do
with the VM changes that happened anywhere between test8 and test9-pre6.
You were
On Mon, Sep 25, 2000 at 12:13:42AM +0200, Andrea Arcangeli wrote:
> On Sun, Sep 24, 2000 at 10:43:03PM +0100, Stephen C. Tweedie wrote:
> > any form of serialisation on the quota file). This feels like rather
> > a lot of new and interesting deadlocks to be introducing so late in
> > 2.4. :-)
T
51 matches
Mail list logo
