Hi folks, I'm struggling to find a solution to missing files on FUSE mounts. Which files are missing is different on different clients. I can stat or ls the missing files directly when called by filename but listing directories won't show them.
So far I've: - verified heal info shows no files in need of healing and no split brain condition - verified the same number of clients are connected to each brick - verified the file counts on the bricks match - upgraded Gluster server and clients from 3.x to 6.x and 7.x - run a stat on all files - run a heal full - rebooted / remounted FUSE clients File count from running a 'find' command on FUSE mounts on the bricks themselves. These counts should all be the same: *38823 *fuse-odroid1-share2 *38823 *fuse-odroid2-share2 *60962 *fuse-odroid3-share2 *7202 *fuse-odroid4-share2 ...and a FUSE mount on a seperate server: *38823 *fuse-phn2dsm-share2 File count from running a 'find' command on brick directories themselves:: *43382 *brick-odroid1-share2 *43382 *brick-odroid2-share2 *43382 *brick-arbiter-odroid3-share2 *23075 *brick-odroid3-share2 *23075 *brick-odroid4-share2 *23075 *brick-arbiter-odroid2-share2 Here's some info about the setup: *# gluster --version | head -1; cat /etc/lsb-release; uname -r* glusterfs 7.8 DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS" 4.14.157-171 *# gluster volume info* Volume Name: gvol0 Type: Distributed-Replicate Volume ID: 57e3a085-5fb7-417d-a71a-fed5cd0ae2d9 Status: Started Snapshot Count: 0 Number of Bricks: 2 x (2 + 1) = 6 Transport-type: tcp Bricks: Brick1: odroid1:/srv/gfs-brick/gvol0 Brick2: odroid2:/srv/gfs-brick/gvol0 Brick3: odroid3:/srv/gfs-brick/gvol0-arbiter2 (arbiter) Brick4: odroid3:/srv/gfs-brick/gvol0_2 Brick5: odroid4:/srv/gfs-brick/gvol0 Brick6: odroid2:/srv/gfs-brick/gvol0-arbiter2 (arbiter) Options Reconfigured: cluster.self-heal-daemon: enable performance.readdir-ahead: yes performance.cache-invalidation: on performance.stat-prefetch: on performance.quick-read: on cluster.shd-max-threads: 4 performance.parallel-readdir: on cluster.server-quorum-type: server server.event-threads: 4 client.event-threads: 4 performance.nl-cache-timeout: 600 performance.nl-cache: on network.inode-lru-limit: 200000 performance.md-cache-timeout: 600 performance.cache-samba-metadata: on features.cache-invalidation-timeout: 600 features.cache-invalidation: on storage.fips-mode-rchecksum: on performance.client-io-threads: off nfs.disable: on transport.address-family: inet features.bitrot: on features.scrub: Active features.scrub-throttle: lazy features.scrub-freq: daily cluster.min-free-disk: 10% *# gluster volume status gvol0 detail* Status of volume: gvol0 ------------------------------------------------------------------------------ Brick : Brick odroid1:/srv/gfs-brick/gvol0 TCP Port : 49152 RDMA Port : 0 Online : Y Pid : 702 File System : xfs Device : /dev/sda Mount Options : rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquota Inode Size : 512 Disk Space Free : 983.4GB Total Disk Space : 5.5TB Inode Count : 586052224 Free Inodes : 585835873 ------------------------------------------------------------------------------ Brick : Brick odroid2:/srv/gfs-brick/gvol0 TCP Port : 49152 RDMA Port : 0 Online : Y Pid : 30206 File System : xfs Device : /dev/sda Mount Options : rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquota Inode Size : 512 Disk Space Free : 983.3GB Total Disk Space : 5.5TB Inode Count : 586052224 Free Inodes : 585711242 ------------------------------------------------------------------------------ Brick : Brick odroid3:/srv/gfs-brick/gvol0-arbiter2 TCP Port : 49152 RDMA Port : 0 Online : Y Pid : 32449 File System : xfs Device : /dev/sda Mount Options : rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquota Inode Size : 512 Disk Space Free : 1.4TB Total Disk Space : 2.7TB Inode Count : 293026624 Free Inodes : 292378835 ------------------------------------------------------------------------------ Brick : Brick odroid3:/srv/gfs-brick/gvol0_2 TCP Port : 49153 RDMA Port : 0 Online : Y Pid : 32474 File System : xfs Device : /dev/sda Mount Options : rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquota Inode Size : 512 Disk Space Free : 1.4TB Total Disk Space : 2.7TB Inode Count : 293026624 Free Inodes : 292378835 ------------------------------------------------------------------------------ Brick : Brick odroid4:/srv/gfs-brick/gvol0 TCP Port : 49152 RDMA Port : 0 Online : Y Pid : 23138 File System : xfs Device : /dev/sda Mount Options : rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquota Inode Size : 512 Disk Space Free : 1.4TB Total Disk Space : 2.7TB Inode Count : 293026624 Free Inodes : 292891910 ------------------------------------------------------------------------------ Brick : Brick odroid2:/srv/gfs-brick/gvol0-arbiter2 TCP Port : 49153 RDMA Port : 0 Online : Y Pid : 30231 File System : xfs Device : /dev/sda Mount Options : rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquota Inode Size : 512 Disk Space Free : 983.3GB Total Disk Space : 5.5TB Inode Count : 586052224 Free Inodes : 585711242
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users