On Tue, Apr 10, 2018 at 6:32 AM Wido den Hollander <w...@42on.com> wrote:
> Hi,
>
> There have been numerous threads about this in the past, but I wanted to
> bring this up again in a new situation.
>
> Running with Luminous v12.2.4 I'm seeing some odd Memory and CPU usage
> when using the ceph-fuse client to mount a multi-MDS CephFS filesystem.
>
> health: HEALTH_OK
>
> services:
> mon: 3 daemons, quorum luvil,sanomat,tide
> mgr: luvil(active), standbys: tide, sanomat
> mds: svw-2/2/2 up {0=luvil=up:active,1=tide=up:active}, 1 up:standby
> osd: 112 osds: 111 up, 111 in
>
> data:
> pools: 2 pools, 4352 pgs
> objects: 85549k objects, 4415 GB
> usage: 50348 GB used, 772 TB / 821 TB avail
> pgs: 4352 active+clean
>
> After running a rsync with millions of files (and some directories
> having 1M files) a ceph-fuse process was using 44GB RSS and using
> between 100% and 200% CPU usage.
>
> Looking at this FUSE client through the admin socket the objecter was
> one of my first suspects, but it claimed to only use ~300M of data in
> it's case spread out over tens of thousands of files.
>
> After unmounting and mounting again the Memory usage was gone and we
> tried the rsync again, but it wasn't reproducible.
>
> The CPU usage however is, a "simple" rsync would cause ceph-fuse to use
> up to 100% CPU.
>
> Switching to the kernel client (4.16 kernel) seems to solve this, but
> the reason for using ceph-fuse in this would be the lack of a recent
> kernel in Debian 9 in this case and the easiness to upgrade the FUSE
> client.
>
> I've tried to disable all logging inside the FUSE client, but that
> didn't help.
>
> When checking on the FUSE client's socket I saw that rename() operations
> were hanging and that's something which rsync does a lot.
>
> At the same time I saw a getfattr() being done to the same inode by the
> FUSE client, but to a different MDS:
>
> rename(): mds rank 0
> getfattr: mds rank 1
>
> Although the kernel client seems to perform better it has the same
> behavior when looking at the mdsc file in /sys
>
> 216729 mds0 create (unsafe)
> #100021abbd9/.ddd.010236269.mpeg21.a0065.folia.xml.gz.AuxBQj
> (reddata2/.ddd.010236269.mpeg21.a0065.folia.xml.gz.AuxBQj)
>
> 216731 mds1 rename
> #100021abbd9/ddd.010236269.mpeg21.a0065.folia.xml.gz
> (reddata2/ddd.010236269.mpeg21.a0065.folia.xml.gz)
> #100021abbd9/.ddd.010236269.mpeg21.a0065.folia.xml.gz.AuxBQj
> (reddata2/.ddd.010236269.mpeg21.a0065.folia.xml.gz.AuxBQj)
>
> So this is rsync talking to two MDS, one for a create and one for a rename.
>
> Is this normal? Is this expected behavior?
>
If the directory got large enough to be sharded across MDSes, yes, it's
expected behavior. There are filesystems that attempt to recognize rsync
and change their normal behavior specifically to deal with this case, but
CephFS isn't one of them (yet, anyway).
Not sure about the specifics of the client memory or CPU usage; I think
you'd have to profile. rsync is a pretty pessimal CephFS workload though
and I think I've heard about this before...
-Greg
>
> To me it seems like that possibly the Subtree Partitioning might be
> interfering here, but it wanted to double check.
>
> Apart from that the CPU and Memory usage of the FUSE client seems very
> high and that might be related to this.
>
> Any ideas?
>
> Thanks,
>
> Wido
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
