Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-18 Thread Waiman Long
On 07/14/2018 02:34 PM, Al Viro wrote: > On Sat, Jul 14, 2018 at 11:00:32AM -0700, Linus Torvalds wrote: >> On Sat, Jul 14, 2018 at 10:35 AM Pavel Machek wrote: >>> Could we allocate -ve entries from separate slab? >> No, because negative dentrires don't stay negative. >> >> Every single positive

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-18 Thread Waiman Long
On 07/14/2018 02:34 PM, Al Viro wrote: > On Sat, Jul 14, 2018 at 11:00:32AM -0700, Linus Torvalds wrote: >> On Sat, Jul 14, 2018 at 10:35 AM Pavel Machek wrote: >>> Could we allocate -ve entries from separate slab? >> No, because negative dentrires don't stay negative. >> >> Every single positive

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-14 Thread Linus Torvalds
On Sat, Jul 14, 2018 at 11:36 AM Al Viro wrote: > > OK, this > /* > * No ordinary (disk based) filesystem counts links as inodes; > * but each new link needs a new dentry, pinning lowmem, and > * tmpfs dentries cannot be pruned until they are unlinked. >

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-14 Thread Linus Torvalds
On Sat, Jul 14, 2018 at 11:36 AM Al Viro wrote: > > OK, this > /* > * No ordinary (disk based) filesystem counts links as inodes; > * but each new link needs a new dentry, pinning lowmem, and > * tmpfs dentries cannot be pruned until they are unlinked. >

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-14 Thread Al Viro
On Sat, Jul 14, 2018 at 07:34:45PM +0100, Al Viro wrote: > On Sat, Jul 14, 2018 at 11:00:32AM -0700, Linus Torvalds wrote: > > On Sat, Jul 14, 2018 at 10:35 AM Pavel Machek wrote: > > > > > > Could we allocate -ve entries from separate slab? > > > > No, because negative dentrires don't stay

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-14 Thread Al Viro
On Sat, Jul 14, 2018 at 07:34:45PM +0100, Al Viro wrote: > On Sat, Jul 14, 2018 at 11:00:32AM -0700, Linus Torvalds wrote: > > On Sat, Jul 14, 2018 at 10:35 AM Pavel Machek wrote: > > > > > > Could we allocate -ve entries from separate slab? > > > > No, because negative dentrires don't stay

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-14 Thread Al Viro
On Sat, Jul 14, 2018 at 11:00:32AM -0700, Linus Torvalds wrote: > On Sat, Jul 14, 2018 at 10:35 AM Pavel Machek wrote: > > > > Could we allocate -ve entries from separate slab? > > No, because negative dentrires don't stay negative. > > Every single positive dentry starts out as a negative

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-14 Thread Al Viro
On Sat, Jul 14, 2018 at 11:00:32AM -0700, Linus Torvalds wrote: > On Sat, Jul 14, 2018 at 10:35 AM Pavel Machek wrote: > > > > Could we allocate -ve entries from separate slab? > > No, because negative dentrires don't stay negative. > > Every single positive dentry starts out as a negative

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-14 Thread Linus Torvalds
On Sat, Jul 14, 2018 at 10:35 AM Pavel Machek wrote: > > Could we allocate -ve entries from separate slab? No, because negative dentrires don't stay negative. Every single positive dentry starts out as a negative dentry that is passed in to "lookup()" to maybe be made positive. And most of the

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-14 Thread Linus Torvalds
On Sat, Jul 14, 2018 at 10:35 AM Pavel Machek wrote: > > Could we allocate -ve entries from separate slab? No, because negative dentrires don't stay negative. Every single positive dentry starts out as a negative dentry that is passed in to "lookup()" to maybe be made positive. And most of the

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-14 Thread Pavel Machek
> > Yes, "should be". I could understand that the presence of huge > > nunmbers of -ve dentries could result in undesirable reclaim of > > pagecache, etc. Triggering oom-killings is very bad, and presumably > > has the same cause. > > > > Before we go and add a large amount of code to do the

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-14 Thread Pavel Machek
> > Yes, "should be". I could understand that the presence of huge > > nunmbers of -ve dentries could result in undesirable reclaim of > > pagecache, etc. Triggering oom-killings is very bad, and presumably > > has the same cause. > > > > Before we go and add a large amount of code to do the

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-03 Thread Vlastimil Babka
On 07/03/2018 03:11 AM, Waiman Long wrote: > On 07/03/2018 05:18 AM, Andrew Morton wrote: >> On Mon, 2 Jul 2018 12:34:00 -0700 Linus Torvalds >> wrote: >> >>> On Sun, Jul 1, 2018 at 10:52 PM Waiman Long wrote: A rogue application can potentially create a large number of negative

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-03 Thread Vlastimil Babka
On 07/03/2018 03:11 AM, Waiman Long wrote: > On 07/03/2018 05:18 AM, Andrew Morton wrote: >> On Mon, 2 Jul 2018 12:34:00 -0700 Linus Torvalds >> wrote: >> >>> On Sun, Jul 1, 2018 at 10:52 PM Waiman Long wrote: A rogue application can potentially create a large number of negative

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-03 Thread Jan Kara
On Mon 02-07-18 16:19:25, Andrew Morton wrote: > On Mon, 02 Jul 2018 15:34:40 -0700 James Bottomley > wrote: > > > On Mon, 2018-07-02 at 14:18 -0700, Andrew Morton wrote: > > > On Mon, 2 Jul 2018 12:34:00 -0700 Linus Torvalds > > dation.org> wrote: > > > > > > > On Sun, Jul 1, 2018 at 10:52

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-03 Thread Jan Kara
On Mon 02-07-18 16:19:25, Andrew Morton wrote: > On Mon, 02 Jul 2018 15:34:40 -0700 James Bottomley > wrote: > > > On Mon, 2018-07-02 at 14:18 -0700, Andrew Morton wrote: > > > On Mon, 2 Jul 2018 12:34:00 -0700 Linus Torvalds > > dation.org> wrote: > > > > > > > On Sun, Jul 1, 2018 at 10:52

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Waiman Long
On 07/03/2018 07:19 AM, Andrew Morton wrote: > On Mon, 02 Jul 2018 15:34:40 -0700 James Bottomley > wrote: > >> On Mon, 2018-07-02 at 14:18 -0700, Andrew Morton wrote: >>> On Mon, 2 Jul 2018 12:34:00 -0700 Linus Torvalds >> dation.org> wrote: >>> On Sun, Jul 1, 2018 at 10:52 PM Waiman Long

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Waiman Long
On 07/03/2018 07:19 AM, Andrew Morton wrote: > On Mon, 02 Jul 2018 15:34:40 -0700 James Bottomley > wrote: > >> On Mon, 2018-07-02 at 14:18 -0700, Andrew Morton wrote: >>> On Mon, 2 Jul 2018 12:34:00 -0700 Linus Torvalds >> dation.org> wrote: >>> On Sun, Jul 1, 2018 at 10:52 PM Waiman Long

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Waiman Long
On 07/03/2018 05:18 AM, Andrew Morton wrote: > On Mon, 2 Jul 2018 12:34:00 -0700 Linus Torvalds > wrote: > >> On Sun, Jul 1, 2018 at 10:52 PM Waiman Long wrote: >>> A rogue application can potentially create a large number of negative >>> dentries in the system consuming most of the memory

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Waiman Long
On 07/03/2018 05:18 AM, Andrew Morton wrote: > On Mon, 2 Jul 2018 12:34:00 -0700 Linus Torvalds > wrote: > >> On Sun, Jul 1, 2018 at 10:52 PM Waiman Long wrote: >>> A rogue application can potentially create a large number of negative >>> dentries in the system consuming most of the memory

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Waiman Long
On 07/03/2018 03:34 AM, Linus Torvalds wrote: > On Sun, Jul 1, 2018 at 10:52 PM Waiman Long wrote: >> A rogue application can potentially create a large number of negative >> dentries in the system consuming most of the memory available if it >> is not under the direct control of a memory

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Waiman Long
On 07/03/2018 03:34 AM, Linus Torvalds wrote: > On Sun, Jul 1, 2018 at 10:52 PM Waiman Long wrote: >> A rogue application can potentially create a large number of negative >> dentries in the system consuming most of the memory available if it >> is not under the direct control of a memory

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Linus Torvalds
On Mon, Jul 2, 2018 at 4:19 PM Andrew Morton wrote: > > Before we go and add a large amount of code to do the shrinker's job > for it, we should get a full understanding of what's going wrong. Is > it because the dentry_lru had a mixture of +ve and -ve dentries? > Should we have a separate LRU

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Linus Torvalds
On Mon, Jul 2, 2018 at 4:19 PM Andrew Morton wrote: > > Before we go and add a large amount of code to do the shrinker's job > for it, we should get a full understanding of what's going wrong. Is > it because the dentry_lru had a mixture of +ve and -ve dentries? > Should we have a separate LRU

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Andrew Morton
On Mon, 02 Jul 2018 15:34:40 -0700 James Bottomley wrote: > On Mon, 2018-07-02 at 14:18 -0700, Andrew Morton wrote: > > On Mon, 2 Jul 2018 12:34:00 -0700 Linus Torvalds > dation.org> wrote: > > > > > On Sun, Jul 1, 2018 at 10:52 PM Waiman Long > > > wrote: > > > > > > > > A rogue

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Andrew Morton
On Mon, 02 Jul 2018 15:34:40 -0700 James Bottomley wrote: > On Mon, 2018-07-02 at 14:18 -0700, Andrew Morton wrote: > > On Mon, 2 Jul 2018 12:34:00 -0700 Linus Torvalds > dation.org> wrote: > > > > > On Sun, Jul 1, 2018 at 10:52 PM Waiman Long > > > wrote: > > > > > > > > A rogue

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Linus Torvalds
On Mon, Jul 2, 2018 at 3:54 PM Linus Torvalds wrote: > > Lookie here, for example: > > [torvalds@i7 linux]$ strace -e trace=%file -c git status So in the name of honestly, that's slightly misleading. "git" will happily thread the actual index file up-to-date testing. And that's hidden in the

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Linus Torvalds
On Mon, Jul 2, 2018 at 3:54 PM Linus Torvalds wrote: > > Lookie here, for example: > > [torvalds@i7 linux]$ strace -e trace=%file -c git status So in the name of honestly, that's slightly misleading. "git" will happily thread the actual index file up-to-date testing. And that's hidden in the

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Linus Torvalds
On Mon, Jul 2, 2018 at 3:34 PM James Bottomley wrote: > > There are still a lot of applications that keep looking up non-existent > files, so I think it's still beneficial to keep them. Apparently > apache still looks for a .htaccess file in every directory it > traverses, for instance. .. or

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Linus Torvalds
On Mon, Jul 2, 2018 at 3:34 PM James Bottomley wrote: > > There are still a lot of applications that keep looking up non-existent > files, so I think it's still beneficial to keep them. Apparently > apache still looks for a .htaccess file in every directory it > traverses, for instance. .. or

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread James Bottomley
On Mon, 2018-07-02 at 14:18 -0700, Andrew Morton wrote: > On Mon, 2 Jul 2018 12:34:00 -0700 Linus Torvalds dation.org> wrote: > > > On Sun, Jul 1, 2018 at 10:52 PM Waiman Long > > wrote: > > > > > > A rogue application can potentially create a large number of > > > negative > > > dentries in

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread James Bottomley
On Mon, 2018-07-02 at 14:18 -0700, Andrew Morton wrote: > On Mon, 2 Jul 2018 12:34:00 -0700 Linus Torvalds dation.org> wrote: > > > On Sun, Jul 1, 2018 at 10:52 PM Waiman Long > > wrote: > > > > > > A rogue application can potentially create a large number of > > > negative > > > dentries in

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Linus Torvalds
On Mon, Jul 2, 2018 at 3:21 PM Matthew Wilcox wrote: > > On Mon, Jul 02, 2018 at 02:18:11PM -0700, Andrew Morton wrote: > > > > Dumb question: do we know that negative dentries are actually > > worthwhile? Has anyone checked in the past couple of decades? Perhaps > > our lookups are so whizzy

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Linus Torvalds
On Mon, Jul 2, 2018 at 3:21 PM Matthew Wilcox wrote: > > On Mon, Jul 02, 2018 at 02:18:11PM -0700, Andrew Morton wrote: > > > > Dumb question: do we know that negative dentries are actually > > worthwhile? Has anyone checked in the past couple of decades? Perhaps > > our lookups are so whizzy

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Matthew Wilcox
On Mon, Jul 02, 2018 at 02:18:11PM -0700, Andrew Morton wrote: > In the [5/6] changelog it is mentioned that a large number of -ve > dentries can lead to oom-killings. This sounds bad - -ve dentries > should be trivially reclaimable and we shouldn't be oom-killing in such > a situation. > > Dumb

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Matthew Wilcox
On Mon, Jul 02, 2018 at 02:18:11PM -0700, Andrew Morton wrote: > In the [5/6] changelog it is mentioned that a large number of -ve > dentries can lead to oom-killings. This sounds bad - -ve dentries > should be trivially reclaimable and we shouldn't be oom-killing in such > a situation. > > Dumb

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Andrew Morton
On Mon, 2 Jul 2018 12:34:00 -0700 Linus Torvalds wrote: > On Sun, Jul 1, 2018 at 10:52 PM Waiman Long wrote: > > > > A rogue application can potentially create a large number of negative > > dentries in the system consuming most of the memory available if it > > is not under the direct control

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Andrew Morton
On Mon, 2 Jul 2018 12:34:00 -0700 Linus Torvalds wrote: > On Sun, Jul 1, 2018 at 10:52 PM Waiman Long wrote: > > > > A rogue application can potentially create a large number of negative > > dentries in the system consuming most of the memory available if it > > is not under the direct control

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Linus Torvalds
On Sun, Jul 1, 2018 at 10:52 PM Waiman Long wrote: > > A rogue application can potentially create a large number of negative > dentries in the system consuming most of the memory available if it > is not under the direct control of a memory controller that enforce > kernel memory limit. I

Re: [PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-02 Thread Linus Torvalds
On Sun, Jul 1, 2018 at 10:52 PM Waiman Long wrote: > > A rogue application can potentially create a large number of negative > dentries in the system consuming most of the memory available if it > is not under the direct control of a memory controller that enforce > kernel memory limit. I

[PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-01 Thread Waiman Long
v4->v5: - Backed to the latest 4.18 kernel and modify the code accordingly. Patch 1 "Relocate dentry_kill() after lock_parent()" is now no longer necessary. - Make tracking and limiting of negative dentries an user configurable option (CONFIG_DCACHE_TRACK_NEG_ENTRY) so that users

[PATCH v5 0/6] fs/dcache: Track & limit # of negative dentries

2018-07-01 Thread Waiman Long
v4->v5: - Backed to the latest 4.18 kernel and modify the code accordingly. Patch 1 "Relocate dentry_kill() after lock_parent()" is now no longer necessary. - Make tracking and limiting of negative dentries an user configurable option (CONFIG_DCACHE_TRACK_NEG_ENTRY) so that users