Hi,
We are facing a weird scenario where on the clients the glusterfs
volume mounts successfully but the size on the client is less than
the total volume size. A few bricks are missing and hence the size as seen
on the client is incorrect and we face issues of missing files. This
happens randomly and we are unable to reproduce it. The gluster volume
status shows all bricks up and running.
We are running distributed glusterfs version 9.3 with 13 nodes having 50
bricks divided amongst them. We are mounting it on the client with
the below entry in fstab
"10.132.2.101:/datavol /data glusterfs ro,_netdev 0 0" where 10.132.2.101
is one of the nodes and datavol is the volume name.
Any help would be appreciated. In the client glusterfs log we see the below
prints in case these help to figure out the issue.
[2025-01-28 11:00:11.404092 +0000] W [MSGID: 109222]
> [dht-selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory
> selfheal failed [{path=/}, {subvol-down=1}, {Not-fixing}, {gfid=}]
> [2025-01-28 11:00:11.405029 +0000] W [dict.c:1532:dict_get_with_ref]
> (-->/usr/lib64/glusterfs/9.3/xlator/cluster/distribute.so(+0x556ea)
> [0x7f63cdf176ea] -->/lib64/libglusterfs.so.0(dict_get_uint32+0x3f) [0x7f63
> dd12ffaf] -->/lib64/libglusterfs.so.0(dict_get_with_ref+0x8e)
> [0x7f63dd12f29e] ) 0-dict: dict OR key (trusted.glusterfs.dht.commithash)
> is NULL [Invalid argument]
> [2025-01-28 11:00:11.406595 +0000] I [MSGID: 109061]
> [dht-layout.c:641:dht_layout_normalize] 0-datavol-dht: Found anomalies
> [{path=/code}, {gfid=949ca970-0b14-4fed-af73-d9ea216905f1}, {holes=1},
> {overlaps=0}]
> [2025-01-28 11:00:11.406731 +0000] W [MSGID: 109222]
> [dht-selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory
> selfheal failed [{path=/code}, {subvol-down=1}, {Not-fixing},
> {gfid=949ca970-0b14-4fed-a
> f73-d9ea216905f1}]
> [2025-01-28 11:00:11.408356 +0000] W [dict.c:1532:dict_get_with_ref]
> (-->/usr/lib64/glusterfs/9.3/xlator/cluster/distribute.so(+0x556ea)
> [0x7f63cdf176ea] -->/lib64/libglusterfs.so.0(dict_get_uint32+0x3f) [0x7f63
> dd12ffaf] -->/lib64/libglusterfs.so.0(dict_get_with_ref+0x8e)
> [0x7f63dd12f29e] ) 0-dict: dict OR key (trusted.glusterfs.dht.commithash)
> is NULL [Invalid argument]
> [2025-01-28 11:00:11.409546 +0000] I [MSGID: 109061]
> [dht-layout.c:641:dht_layout_normalize] 0-datavol-dht: Found anomalies
> [{path=/code/summary}, {gfid=c96f0e78-dd0e-4c61-89a3-f94ca1b502c4},
> {holes=1}, {overlap
> s=0}]
> [2025-01-28 11:00:11.409605 +0000] W [MSGID: 109222]
> [dht-selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory
> selfheal failed [{path=/code/summary}, {subvol-down=1}, {Not-fixing},
> {gfid=c96f0e78-dd0
> e-4c61-89a3-f94ca1b502c4}]
> [2025-01-28 11:00:28.910500 +0000] W [MSGID: 109222]
> [dht-selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory
> selfheal failed [{path=/}, {subvol-down=1}, {Not-fixing}, {gfid=}]
> [2025-01-28 11:00:28.911125 +0000] I [MSGID: 109061]
> [dht-layout.c:641:dht_layout_normalize] 0-datavol-dht: Found anomalies
> [{path=/}, {gfid=00000000-0000-0000-0000-000000000001}, {holes=1},
> {overlaps=0}]
> [2025-01-28 11:00:28.911167 +0000] W [MSGID: 109222]
> [dht-selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory
> selfheal failed [{path=/}, {subvol-down=1}, {Not-fixing}, {gfid=}]
> [2025-01-28 11:00:40.359587 +0000] W [MSGID: 109222]
> [dht-selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory
> selfheal failed [{path=/}, {subvol-down=1}, {Not-fixing}, {gfid=}]
> [2025-01-28 11:00:40.375728 +0000] W [MSGID: 109011]
> [dht-layout.c:148:dht_layout_search] 0-datavol-dht: Failed to get hashed
> subvolume [{hash-value=0x5877673a}]
> [2025-01-28 11:00:40.377646 +0000] W [MSGID: 109011]
> [dht-layout.c:148:dht_layout_search] 0-datavol-dht: Failed to get hashed
> subvolume [{hash-value=0x5877673a}]
> [2025-01-28 11:00:40.428513 +0000] W [MSGID: 109011]
> [dht-layout.c:148:dht_layout_search] 0-datavol-dht: Failed to get hashed
> subvolume [{hash-value=0x599f6a37}]
--
Regards,
Shreyansh Shah
AlphaGrep* Securities Pvt. Ltd. *
-------
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel
