Re: Unexpected out of memory kills when running parallel find instances over millions of files

2023-10-21 Thread Johnny Billquist
I think I reported this something like 20 years ago, but noone really seemed to care. I noticed it pretty much right away after NetBSD switched to the unified memory thing, where all free memory usually was grabbed as disk cache. It was not fun on VAX, but at the time it seem other platforms

Re: Unexpected out of memory kills when running parallel find instances over millions of files

2023-10-21 Thread Manuel Bouyer
On Fri, Oct 20, 2023 at 10:26:05PM +0200, Reinoud Zandijk wrote: > Hi, > > On Thu, Oct 19, 2023 at 11:20:02AM +0200, Mateusz Guzik wrote: > > Running 20 find(1) instances, where each has a "private" tree with > > million of files runs into trouble with the kernel killing them (and > > others): >

Re: Unexpected out of memory kills when running parallel find instances over millions of files

2023-10-21 Thread Reinoud Zandijk
Hi, On Thu, Oct 19, 2023 at 11:20:02AM +0200, Mateusz Guzik wrote: > Running 20 find(1) instances, where each has a "private" tree with > million of files runs into trouble with the kernel killing them (and > others): > [ 785.194378] UVM: pid 1998.1998 (find), uid 0 killed: out of swap > [

Re: Unexpected out of memory kills when running parallel find instances over millions of files

2023-10-19 Thread Michael van Elst
mjgu...@gmail.com (Mateusz Guzik) writes: >> While vnodes would be recyclable, they hardly get recycled unless >> an filesystem object is deleted or the filesystem is unmounted. >They get recycled all the time by vdrain thread if numvnodes goes above >desiredvnodes, like it does in this test.

Re: Unexpected out of memory kills when running parallel find instances over millions of files

2023-10-19 Thread Mateusz Guzik
On Thu, Oct 19, 2023 at 10:49:37AM +, Michael van Elst wrote: > mjgu...@gmail.com (Mateusz Guzik) writes: > > >Running 20 find(1) instances, where each has a "private" tree with > >million of files runs into trouble with the kernel killing them (and > >others): > >[ 785.194378] UVM: pid

Re: Unexpected out of memory kills when running parallel find instances over millions of files

2023-10-19 Thread Michael van Elst
mjgu...@gmail.com (Mateusz Guzik) writes: >Running 20 find(1) instances, where each has a "private" tree with >million of files runs into trouble with the kernel killing them (and >others): >[ 785.194378] UVM: pid 1998.1998 (find), uid 0 killed: out of swap >This should not be happening --

Unexpected out of memory kills when running parallel find instances over millions of files

2023-10-19 Thread Mateusz Guzik
Running 20 find(1) instances, where each has a "private" tree with million of files runs into trouble with the kernel killing them (and others): [ 785.194378] UVM: pid 1998.1998 (find), uid 0 killed: out of swap [ 785.194378] UVM: pid 2010.2010 (find), uid 0 killed: out of swap [ 785.224675]