Re: [Gluster-users] How to identify a files shard?
On 25/04/2016 2:36 PM, Krutika Dhananjay wrote: Lindsay, Could you send logs from your setup: bricks, shds and client logs. Also send in the gfids of these 8 shards. -Krutika Will do, take me a bit to get it together. thanks, -- Lindsay Mathieson ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to identify a files shard?
Lindsay, Could you send logs from your setup: bricks, shds and client logs. Also send in the gfids of these 8 shards. -Krutika On Mon, Apr 25, 2016 at 3:27 AM, Paul Cuzner wrote: > > > Just wondering how shards can silently be different across bricks in a > replica? Lindsay caught this issue due to her due diligence taking on 'new' > tech - and resolved the inconsistency, but tbh this shouldn't be an admin's > job :( > > > > On Sun, Apr 24, 2016 at 7:06 PM, Krutika Dhananjay > wrote: > >> OK. Under normal circumstances it should have been possible to heal a >> single file by issuing a lookup on it (==> stat on the file from the >> mountpoint). But with shards this won't work. We take care not to expose >> /.shard on the mountpoint, and as a result any attempt to issue lookup on a >> shard from the mountpoint will be met with an 'operation not permitted' >> error. >> >> -Krutika >> >> On Sun, Apr 24, 2016 at 11:42 AM, Lindsay Mathieson < >> lindsay.mathie...@gmail.com> wrote: >> >>> On 24/04/2016 2:56 PM, Krutika Dhananjay wrote: >>> Nope, it's not necessary for them to all have the xattr. >>> >>> Thats good :) >>> >>> Do you see anything at least in .glusterfs/indices/dirty on all bricks? >>> >>> I checked, dirty dir empty on all bricks >>> >>> I used diff3 to compare the checksums of the shards and it revealed that >>> seven of the shards were the same on two bricks (vna & vng) and one of the >>> shards was the same on two other bricks (vna & vnb). Fortunately none were >>> different on all 3 bricks :) >>> >>> Using the checksum as a quorum I deleted all the singleton shards (7 on >>> vnb, 1 on vng), touched the file owner and issule a "heal full". All 8 >>> shards were restored with matching checksums for the other two bricks. A >>> rechack of the entire set of shards for the vm showed all 3 copies as >>> identical and the VM itself is functioning normally. >>> >>> Its one way to manually heal up shard mismatches which gluster hasn't >>> detected, if somewhat tedious. Its a method which lends itself to >>> automation though. >>> >>> >>> Cheers, >>> >>> >>> -- >>> Lindsay Mathieson >>> >>> >> >> ___ >> Gluster-users mailing list >> Gluster-users@gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> > > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to identify a files shard?
Just wondering how shards can silently be different across bricks in a replica? Lindsay caught this issue due to her due diligence taking on 'new' tech - and resolved the inconsistency, but tbh this shouldn't be an admin's job :( On Sun, Apr 24, 2016 at 7:06 PM, Krutika Dhananjay wrote: > OK. Under normal circumstances it should have been possible to heal a > single file by issuing a lookup on it (==> stat on the file from the > mountpoint). But with shards this won't work. We take care not to expose > /.shard on the mountpoint, and as a result any attempt to issue lookup on a > shard from the mountpoint will be met with an 'operation not permitted' > error. > > -Krutika > > On Sun, Apr 24, 2016 at 11:42 AM, Lindsay Mathieson < > lindsay.mathie...@gmail.com> wrote: > >> On 24/04/2016 2:56 PM, Krutika Dhananjay wrote: >> >>> Nope, it's not necessary for them to all have the xattr. >>> >> >> Thats good :) >> >> >>> Do you see anything at least in .glusterfs/indices/dirty on all bricks? >>> >> >> I checked, dirty dir empty on all bricks >> >> I used diff3 to compare the checksums of the shards and it revealed that >> seven of the shards were the same on two bricks (vna & vng) and one of the >> shards was the same on two other bricks (vna & vnb). Fortunately none were >> different on all 3 bricks :) >> >> Using the checksum as a quorum I deleted all the singleton shards (7 on >> vnb, 1 on vng), touched the file owner and issule a "heal full". All 8 >> shards were restored with matching checksums for the other two bricks. A >> rechack of the entire set of shards for the vm showed all 3 copies as >> identical and the VM itself is functioning normally. >> >> Its one way to manually heal up shard mismatches which gluster hasn't >> detected, if somewhat tedious. Its a method which lends itself to >> automation though. >> >> >> Cheers, >> >> >> -- >> Lindsay Mathieson >> >> > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to identify a files shard?
OK. Under normal circumstances it should have been possible to heal a single file by issuing a lookup on it (==> stat on the file from the mountpoint). But with shards this won't work. We take care not to expose /.shard on the mountpoint, and as a result any attempt to issue lookup on a shard from the mountpoint will be met with an 'operation not permitted' error. -Krutika On Sun, Apr 24, 2016 at 11:42 AM, Lindsay Mathieson < lindsay.mathie...@gmail.com> wrote: > On 24/04/2016 2:56 PM, Krutika Dhananjay wrote: > >> Nope, it's not necessary for them to all have the xattr. >> > > Thats good :) > > >> Do you see anything at least in .glusterfs/indices/dirty on all bricks? >> > > I checked, dirty dir empty on all bricks > > I used diff3 to compare the checksums of the shards and it revealed that > seven of the shards were the same on two bricks (vna & vng) and one of the > shards was the same on two other bricks (vna & vnb). Fortunately none were > different on all 3 bricks :) > > Using the checksum as a quorum I deleted all the singleton shards (7 on > vnb, 1 on vng), touched the file owner and issule a "heal full". All 8 > shards were restored with matching checksums for the other two bricks. A > rechack of the entire set of shards for the vm showed all 3 copies as > identical and the VM itself is functioning normally. > > Its one way to manually heal up shard mismatches which gluster hasn't > detected, if somewhat tedious. Its a method which lends itself to > automation though. > > > Cheers, > > > -- > Lindsay Mathieson > > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to identify a files shard?
On 24/04/2016 2:56 PM, Krutika Dhananjay wrote: Nope, it's not necessary for them to all have the xattr. Thats good :) Do you see anything at least in .glusterfs/indices/dirty on all bricks? I checked, dirty dir empty on all bricks I used diff3 to compare the checksums of the shards and it revealed that seven of the shards were the same on two bricks (vna & vng) and one of the shards was the same on two other bricks (vna & vnb). Fortunately none were different on all 3 bricks :) Using the checksum as a quorum I deleted all the singleton shards (7 on vnb, 1 on vng), touched the file owner and issule a "heal full". All 8 shards were restored with matching checksums for the other two bricks. A rechack of the entire set of shards for the vm showed all 3 copies as identical and the VM itself is functioning normally. Its one way to manually heal up shard mismatches which gluster hasn't detected, if somewhat tedious. Its a method which lends itself to automation though. Cheers, -- Lindsay Mathieson ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to identify a files shard?
Nope, it's not necessary for them to all have the xattr. Do you see anything at least in .glusterfs/indices/dirty on all bricks? -Krutika On Sun, Apr 24, 2016 at 4:17 AM, Lindsay Mathieson < lindsay.mathie...@gmail.com> wrote: > On 24/04/2016 1:24 AM, Krutika Dhananjay wrote: > >> Each shard is also associated with a gfid. >> >> So do you see the gfids of these 8 shards in the >> .glusterfs/indices/xattrop directory on any of the bricks? >> > > > There were no files at all in the xattrop dir. > > Also I did a spot check of the extended attr of the shards in question and > they did not look right (as I understand them). Shouldn't > trusted.afr.datastore4-client-0/1/2 be set for all shards (excluding their > brick index) > > ssh root@vng getfattr -d -m . -e hex > /tank/vmdata/datastore4/.shard/744c5059-303d-4e82-b5be-0a5f53b1aeff.1172 > # file: > tank/vmdata/datastore4/.shard/744c5059-303d-4e82-b5be-0a5f53b1aeff.1172 > trusted.afr.datastore4-client-2=0x > trusted.afr.dirty=0x > trusted.bit-rot.version=0x040057163218cfbf > trusted.gfid=0xac4dc6375d6a4b0c90763ffb5314a5a3 > > getfattr: Removing leading '/' from absolute path names > root@vna:~# ssh root@vna getfattr -d -m . -e hex > /tank/vmdata/datastore4/.shard/744c5059-303d-4e82-b5be-0a5f53b1aeff.1172 > # file: > tank/vmdata/datastore4/.shard/744c5059-303d-4e82-b5be-0a5f53b1aeff.1172 > trusted.afr.dirty=0x > trusted.bit-rot.version=0x0500571626b8000316f0 > trusted.gfid=0xac4dc6375d6a4b0c90763ffb5314a5a3 > > getfattr: Removing leading '/' from absolute path names > root@vna:~# ssh root@vnb getfattr -d -m . -e hex > /tank/vmdata/datastore4/.shard/744c5059-303d-4e82-b5be-0a5f53b1aeff.1172 > # file: > tank/vmdata/datastore4/.shard/744c5059-303d-4e82-b5be-0a5f53b1aeff.1172 > trusted.afr.datastore4-client-2=0x > trusted.afr.dirty=0x > trusted.bit-rot.version=0x040057160faa000e0632 > trusted.gfid=0xac4dc6375d6a4b0c90763ffb5314a5a3 > > -- > Lindsay Mathieson > > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to identify a files shard?
On 24/04/2016 1:24 AM, Krutika Dhananjay wrote: Each shard is also associated with a gfid. So do you see the gfids of these 8 shards in the .glusterfs/indices/xattrop directory on any of the bricks? There were no files at all in the xattrop dir. Also I did a spot check of the extended attr of the shards in question and they did not look right (as I understand them). Shouldn't trusted.afr.datastore4-client-0/1/2 be set for all shards (excluding their brick index) ssh root@vng getfattr -d -m . -e hex /tank/vmdata/datastore4/.shard/744c5059-303d-4e82-b5be-0a5f53b1aeff.1172 # file: tank/vmdata/datastore4/.shard/744c5059-303d-4e82-b5be-0a5f53b1aeff.1172 trusted.afr.datastore4-client-2=0x trusted.afr.dirty=0x trusted.bit-rot.version=0x040057163218cfbf trusted.gfid=0xac4dc6375d6a4b0c90763ffb5314a5a3 getfattr: Removing leading '/' from absolute path names root@vna:~# ssh root@vna getfattr -d -m . -e hex /tank/vmdata/datastore4/.shard/744c5059-303d-4e82-b5be-0a5f53b1aeff.1172 # file: tank/vmdata/datastore4/.shard/744c5059-303d-4e82-b5be-0a5f53b1aeff.1172 trusted.afr.dirty=0x trusted.bit-rot.version=0x0500571626b8000316f0 trusted.gfid=0xac4dc6375d6a4b0c90763ffb5314a5a3 getfattr: Removing leading '/' from absolute path names root@vna:~# ssh root@vnb getfattr -d -m . -e hex /tank/vmdata/datastore4/.shard/744c5059-303d-4e82-b5be-0a5f53b1aeff.1172 # file: tank/vmdata/datastore4/.shard/744c5059-303d-4e82-b5be-0a5f53b1aeff.1172 trusted.afr.datastore4-client-2=0x trusted.afr.dirty=0x trusted.bit-rot.version=0x040057160faa000e0632 trusted.gfid=0xac4dc6375d6a4b0c90763ffb5314a5a3 -- Lindsay Mathieson ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to identify a files shard?
On 24/04/2016 1:24 AM, Krutika Dhananjay wrote: Each shard is also associated with a gfid. So do you see the gfids of these 8 shards in the .glusterfs/indices/xattrop directory on any of the bricks? The .glusterfs/indices/xattrop directory is empty on all three bricks - you mean a file listing? -- Lindsay Mathieson ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to identify a files shard?
Each shard is also associated with a gfid. So do you see the gfids of these 8 shards in the .glusterfs/indices/xattrop directory on any of the bricks? -Krutika On Sat, Apr 23, 2016 at 8:51 PM, Lindsay Mathieson < lindsay.mathie...@gmail.com> wrote: > On 23/04/2016 11:14 PM, Krutika Dhananjay wrote: > >> You'll see at least 2 paths listed for this inode number (because they're >> hardlinks of each other). >> One of them is the .glusterfs/... path which is internally maintained by >> gluster. >> >> The other path (or paths if the application itself created hard links to >> this file) is the one belonging to the main file that this shard belongs to. >> > > > Thanks Krutika, very useful. > > As part of testing today I stopped the volume and generated a md5sum for > every shard on every brick. 20047 shards per brick, it took a while :) > > I then compared them and found there were 8 shards different and applying > your algorithm found they were all from one file: > > /images/307/vm-307-disk-1.qcow2 > > Is there anyway to launch a heal against just one file? > > -- > Lindsay Mathieson > > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to identify a files shard?
On 23/04/2016 11:14 PM, Krutika Dhananjay wrote: You'll see at least 2 paths listed for this inode number (because they're hardlinks of each other). One of them is the .glusterfs/... path which is internally maintained by gluster. The other path (or paths if the application itself created hard links to this file) is the one belonging to the main file that this shard belongs to. Thanks Krutika, very useful. As part of testing today I stopped the volume and generated a md5sum for every shard on every brick. 20047 shards per brick, it took a while :) I then compared them and found there were 8 shards different and applying your algorithm found they were all from one file: /images/307/vm-307-disk-1.qcow2 Is there anyway to launch a heal against just one file? -- Lindsay Mathieson ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to identify a files shard?
On Sat, Apr 23, 2016 at 2:44 PM, Lindsay Mathieson < lindsay.mathie...@gmail.com> wrote: > Give a file on a volume how do I identify it shards? it looks to be like a > uuid for the file with a .xxx extension > > > 1) Get the trusted.gfid extended attribute of the main file from the brick. # getfattr -d -m . -e hex For example: # pwd /bricks/3 # getfattr -d -m . -e hex file # file: file security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a6574635f72756e74696d655f743a733000 trusted.afr.dirty=0x trusted.bit-rot.version=0x0200571a0eae00016560 trusted.gfid=0xb577234394614334a397ecf8ee4ff75c 2) Convert the gfid into formatted string: 0xb577234394614334a397ecf8ee4ff75c ===> b5772343-9461-4334-a397-ecf8ee4ff75c Alternatively from a fuse mount you could execute: # getfattr -n glusterfs.gfid.string file # file: file glusterfs.gfid.string="b5772343-9461-4334-a397-ecf8ee4ff75c" > And how about the reverse? give a shard such as > "007c8fcb-49ba-4e7e-b744-4e3768ac6bf6.1004" how would I indetify and locate > the file it belongs to? > In the example you gave: 007c8fcb-49ba-4e7e-b744-4e3768ac6bf6 is the gfid of the original file. 1) On any of the replicas, do: # stat $BRICK_ROOT/.glusterfs/00/7c/007c8fcb-49ba-4e7e-b744-4e3768ac6bf6 In the above example, '00' represents the first two characters of the gfid. '7c' represents the next two characters. 2) Get the inode number from the stat output above. 3) On the brick root, execute: # find . -inum You'll see at least 2 paths listed for this inode number (because they're hardlinks of each other). One of them is the .glusterfs/... path which is internally maintained by gluster. The other path (or paths if the application itself created hard links to this file) is the one belonging to the main file that this shard belongs to. Thanks for asking. This is worth documenting. Will put up a .md soon. -Krutika > > -- > Lindsay Mathieson > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
