On Mon, Mar 22, 2021 at 6:52 AM Eric Ernst <[email protected]> wrote: > > Hey ya’ll, > > One challenge I’ve been looking at is how to setup an appropriate memory > cgroup limit for workloads that are leveraging virtiofs (ie, running pods > with Kata Containers). I noticed that memory usage of the daemon itself can > grow considerably depending on the workload; though much more than I’d expect. > > I’m running workload that simply runs a build on kernel sources with -j3. In > doing this, the source of the linux kernel are shared via virtiofs (no DAX), > so as the build goes on, there are a lot of files opened, closed, as well as > created. The rss memory of virtiofsd grows into several hundreds of MBs. > > When taking a look, I’m suspecting that virtiofsd is carrying out the opens, > but never actually closing fds. In the guest, I’m seeing fd’s on the order of > 10-40 for all the container processes as it runs, whereas I see the number of > fds for virtiofsd continually increasing, reaching over 80,000 fds. I’m > guessing this isn’t expected?
The reason could be that guest is keeping a ref on the inodes (dcache->dentry->inode) and current implementation of server keeps an O_PATH fd open for each inode referenced by the client. One way to avoid this is to use the "cache=none" option, which forces the client to drop dentries immediately from the cache if not in use. This is not desirable if cache is actually in use. The memory use of the server should still be limited by the memory use of the guest: if there's memory pressure in the guest kernel, then it will clean out caches, which results in the memory use decreasing in the server as well. If the server memory use looks unbounded, that might be indicative of too much memory used for dcache in the guest (cat /proc/slabinfo | grep ^dentry). Can you verify? Thanks, Miklos _______________________________________________ Virtio-fs mailing list [email protected] https://listman.redhat.com/mailman/listinfo/virtio-fs
