On 27/07/21 12:19 am, Paul Ripke wrote:
On Mon, Jul 26, 2021 at 05:53:19PM +1200, Lloyd Parkes wrote:
That's 12GB of RAM in use and 86MB of RAM free. Sounds pretty awful to me.
Sounds normal to me - I don't expect to see any free RAM unless I've just
- exited a large process
- deleted a large file with large cache footprint
- released a large chunk of RAM by other means (mmap, madvise, semctl, etc).
I haven't run NetBSD on a desktop for a while now, but I still think
12GB is a lot of memory in use. Maybe I'll get a new MacBook when they
start shipping 32GB Apple CPU ones and then put NetBSD on my current
MacBook.
A big chunk of it is in file cache, which is unsurprising when reading
thru a 400GiB file...
Page activity lasts 20s and at 30MB/s that means you should have 600MB
of file data active. Add 50% for inactive pages and that's still only
900MB. I'm willing to bet money that zstd only reads each block of data
once (sequentially in fact) and so it doesn't need any file data cache
at all. File metadata is a different matter, but that probably stays
active and there won't be much of it.
I suspect that your vm.filemax is set to more memory than you have
available for the file cache and once that happens anonymous pages start
to get swapped out. My experience is that while anonymous pages sound
unimportant, they are in fact the most important pages to keep in RAM.
Thinking about it, they are the irreplaceable bits of all our running
software.
Try setting vm.filemin=5 and vm.filemax=10. Really. I did it when
processing vast amounts of files in CVS and it worked for me.
Out of curiosity, what are you doing with zstd. You mentioned backups.
Is this dump or restore? dump implements its own file cache, which won;t
help with the memory burden.
"top -ores" will tell you what programs are using the most anonymous
pages, which might help identify where all this memory pressure is
coming from.
Cheers,
Lloyd
