Hi Hari, Hi Sanoj, thank you very much for your patience and your support! The problem has been solved following your instructions :-)
N.B.: in order to reduce the running time, I executed the “du” command as follows: for i in {1..12} do du /gluster/mnt$i/brick/CSP/ans004/ftp done and not on each brick at "/gluster/mnt$i/brick" tree level. I hope it was a correct idea :-) Thank you again for helping me to solve this issue. Have a good day. Mauro > Il giorno 11 lug 2018, alle ore 09:16, Hari Gowtham <hgowt...@redhat.com> ha > scritto: > > Hi, > > There was a accounting issue in your setup. > The directory ans004/ftp/CMCC-CM2-VHR4-CTR/atm/hist and > ans004/ftp/CMCC-CM2-VHR4 > had wrong size value on them. > > To fix it, you will have to set dirty xattr (an internal gluster > xattr) on these directories > which will mark it for calculating the values again for the directory. > And then do a du on the mount after setting the xattrs. This will do a > stat that will > calculate and update the right values. > > To set dirty xattr: > setfattr -n trusted.glusterfs.quota.dirty -v 0x3100 <path to the directory> > This has to be done for both the directories one after the other on each > brick. > Once done for all the bricks issue the du command. > > Thanks to Sanoj for the guidance > On Tue, Jul 10, 2018 at 6:37 PM Mauro Tridici <mauro.trid...@cmcc.it> wrote: >> >> >> Hi Hari, >> >> sorry for the late. >> Yes, the gluster volume is a single volume that is spread between all the 3 >> node and has 36 bricks >> >> In attachment you can find a tar.gz file containing: >> >> - gluster volume status command output; >> - gluster volume info command output; >> - the output of the following script execution (it generated 3 files per >> server: s01.log, s02.log, s03.log). >> >> This is the “check.sh” script that has been executed on each server (servers >> are s01, s02, s03). >> >> #!/bin/bash >> >> #set -xv >> >> host=$(hostname) >> >> for i in {1..12} >> do >> ./quota_fsck_new-6.py --full-logs --sub-dir CSP/ans004 /gluster/mnt$i/brick >> >> $host.log >> done >> >> Many thanks, >> Mauro >> >> >> Il giorno 10 lug 2018, alle ore 12:12, Hari Gowtham <hgowt...@redhat.com> ha >> scritto: >> >> Hi Mauro, >> >> Can you send the gluster v status command output? >> >> Is it a single volume that is spread between all the 3 node and has 36 >> bricks? >> If yes, you will have to run on all the bricks. >> >> In the command use sub-dir option if you are running only for the >> directory where limit is set. else if you are >> running on the brick mount path you can remove it. >> >> The full-log will consume a lot of space as its going to record the >> xattrs for each entry inside the path we are >> running it. This data is needed to cross check and verify quota's >> marker functionality. >> >> To reduce resource consumption you can run it on one replica set alone >> (if its replicate volume) >> But its better if you can run it on all the brick if possible and if >> the size consumed is fine with you. >> >> Make sure you run it with the script link provided above by Sanoj. (patch >> set 6) >> On Tue, Jul 10, 2018 at 2:54 PM Mauro Tridici <mauro.trid...@cmcc.it> wrote: >> >> >> >> Hi Hari, >> >> thank you very much for your answer. >> I will try to use the script mentioned above pointing to each backend bricks. >> >> So, if I understand, since I have a gluster cluster composed by 3 nodes >> (with 12 bricks on each node), I have to execute the script 36 times. Right? >> >> You can find below the “df” command output executed on a cluster node: >> >> /dev/mapper/cl_s01-gluster 100G 33M 100G 1% /gluster >> /dev/mapper/gluster_vgd-gluster_lvd 9,0T 5,6T 3,5T 62% /gluster/mnt2 >> /dev/mapper/gluster_vge-gluster_lve 9,0T 5,7T 3,4T 63% /gluster/mnt3 >> /dev/mapper/gluster_vgj-gluster_lvj 9,0T 5,7T 3,4T 63% /gluster/mnt8 >> /dev/mapper/gluster_vgc-gluster_lvc 9,0T 5,6T 3,5T 62% /gluster/mnt1 >> /dev/mapper/gluster_vgl-gluster_lvl 9,0T 5,8T 3,3T 65% /gluster/mnt10 >> /dev/mapper/gluster_vgh-gluster_lvh 9,0T 5,7T 3,4T 64% /gluster/mnt6 >> /dev/mapper/gluster_vgf-gluster_lvf 9,0T 5,7T 3,4T 63% /gluster/mnt4 >> /dev/mapper/gluster_vgm-gluster_lvm 9,0T 5,4T 3,7T 60% /gluster/mnt11 >> /dev/mapper/gluster_vgn-gluster_lvn 9,0T 5,4T 3,7T 60% /gluster/mnt12 >> /dev/mapper/gluster_vgg-gluster_lvg 9,0T 5,7T 3,4T 64% /gluster/mnt5 >> /dev/mapper/gluster_vgi-gluster_lvi 9,0T 5,7T 3,4T 63% /gluster/mnt7 >> /dev/mapper/gluster_vgk-gluster_lvk 9,0T 5,8T 3,3T 65% /gluster/mnt9 >> >> I will execute the following command and I will put here the output. >> >> ./quota_fsck_new.py --full-logs --sub-dir /gluster/mnt{1..12} >> >> Thank you again for your support. >> Regards, >> Mauro >> >> Il giorno 10 lug 2018, alle ore 11:02, Hari Gowtham <hgowt...@redhat.com> ha >> scritto: >> >> Hi, >> >> There is no explicit command to backup all the quota limits as per my >> understanding. need to look further about this. >> But you can do the following to backup and set it. >> Gluster volume quota volname list which will print all the quota >> limits on that particular volume. >> You will have to make a note of the directories with their respective limit >> set. >> Once noted down, you can disable quota on the volume and then enable it. >> Once enabled, you will have to set each limit explicitly on the volume. >> >> Before doing this we suggest you can to try running the script >> mentioned above with the backend brick path instead of the mount path. >> you need to run this on the machines where the backend bricks are >> located and not on the mount. >> On Mon, Jul 9, 2018 at 9:01 PM Mauro Tridici <mauro.trid...@cmcc.it> wrote: >> >> >> Hi Sanoj, >> >> could you provide me the command that I need in order to backup all quota >> limits? >> If there is no solution for this kind of problem, I would like to try to >> follow your “backup” suggestion. >> >> Do you think that I should contact gluster developers too? >> >> Thank you very much. >> Regards, >> Mauro >> >> >> Il giorno 05 lug 2018, alle ore 09:56, Mauro Tridici <mauro.trid...@cmcc.it> >> ha scritto: >> >> Hi Sanoj, >> >> unfortunately the output of the command execution was not helpful. >> >> [root@s01 ~]# find /tier2/CSP/ans004 | xargs getfattr -d -m. -e hex >> [root@s01 ~]# >> >> Do you have some other idea in order to detect the cause of the issue? >> >> Thank you again, >> Mauro >> >> >> Il giorno 05 lug 2018, alle ore 09:08, Sanoj Unnikrishnan >> <sunni...@redhat.com> ha scritto: >> >> Hi Mauro, >> >> A script issue did not capture all necessary xattr. >> Could you provide the xattrs with.. >> find /tier2/CSP/ans004 | xargs getfattr -d -m. -e hex >> >> Meanwhile, If you are being impacted, you could do the following >> back up quota limits >> disable quota >> enable quota >> freshly set the limits. >> >> Please capture the xattr values first, so that we can get to know what went >> wrong. >> Regards, >> Sanoj >> >> >> On Tue, Jul 3, 2018 at 4:09 PM, Mauro Tridici <mauro.trid...@cmcc.it> wrote: >> >> >> Dear Sanoj, >> >> thank you very much for your support. >> I just downloaded and executed the script you suggested. >> >> This is the full command I executed: >> >> ./quota_fsck_new.py --full-logs --sub-dir /tier2/CSP/ans004/ /gluster >> >> In attachment, you can find the logs generated by the script. >> What can I do now? >> >> Thank you very much for your patience. >> Mauro >> >> >> >> >> Il giorno 03 lug 2018, alle ore 11:34, Sanoj Unnikrishnan >> <sunni...@redhat.com> ha scritto: >> >> Hi Mauro, >> >> This may be an issue with update of backend xattrs. >> To RCA further and provide resolution could you provide me with the logs by >> running the following fsck script. >> https://review.gluster.org/#/c/19179/6/extras/quota/quota_fsck.py >> >> Try running the script and revert with the logs generated. >> >> Thanks, >> Sanoj >> >> >> On Mon, Jul 2, 2018 at 2:21 PM, Mauro Tridici <mauro.trid...@cmcc.it> wrote: >> >> >> Dear Users, >> >> I just noticed that, after some data deletions executed inside >> "/tier2/CSP/ans004” folder, the amount of used disk reported by quota >> command doesn’t reflect the value indicated by du command. >> Surfing on the web, it seems that it is a bug of previous versions of >> Gluster FS and it was already fixed. >> In my case, the problem seems unfortunately still here. >> >> How can I solve this issue? Is it possible to do it without starting a >> downtime period? >> >> Thank you very much in advance, >> Mauro >> >> [root@s01 ~]# glusterfs -V >> glusterfs 3.10.5 >> Repository revision: git://git.gluster.org/glusterfs.git >> Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/> >> GlusterFS comes with ABSOLUTELY NO WARRANTY. >> It is licensed to you under your choice of the GNU Lesser >> General Public License, version 3 or any later version (LGPLv3 >> or later), or the GNU General Public License, version 2 (GPLv2), >> in all cases as published by the Free Software Foundation. >> >> [root@s01 ~]# gluster volume quota tier2 list /CSP/ans004 >> Path Hard-limit Soft-limit Used >> Available Soft-limit exceeded? Hard-limit exceeded? >> ------------------------------------------------------------------------------------------------------------------------------- >> /CSP/ans004 1.0TB 99%(1013.8GB) 3.9TB >> 0Bytes Yes Yes >> >> [root@s01 ~]# du -hs /tier2/CSP/ans004/ >> 295G /tier2/CSP/ans004/ >> >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users@gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users >> >> >> >> >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users@gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users >> >> >> >> >> -- >> Regards, >> Hari Gowtham. >> >> >> >> >> >> -- >> Regards, >> Hari Gowtham. >> >> > > > -- > Regards, > Hari Gowtham. ------------------------- Mauro Tridici Fondazione CMCC CMCC Supercomputing Center presso Complesso Ecotekne - Università del Salento - Strada Prov.le Lecce - Monteroni sn 73100 Lecce IT http://www.cmcc.it mobile: (+39) 327 5630841 email: mauro.trid...@cmcc.it
_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users