Hi Thomas, I think one way to get reliable quota values might be to reduce a couple of tunables:
- osc.*.max_dirty_mb : In this case you reduce the amount of non-committed data on the client cache and thus the potential for quota inconsistencies. I've been recently having quota issues on a filesystem with many OSTs set to 128MB as max_dirty_mb. That gives potential for a lot of dirty data depending on the number of clients you have, hence quota issues or better to say, false positives. That’s actually just my understanding of quota vs cache. - mds01 ~]# lctl get_param qmt.*.*.soft_least_qunit: tunable to reduce the qunit size between soft and hard quota thus allowing a fine tuning of quota allowance when quota is on this "danger zone". That should help to have more reliable values. That's at least on Lustre 2.10. I had the same problems you have w.r.t. to qunit. The documentation seems outdated or not accurate for, at least, Lustre 2.10. Regards, Diego On 10.07.19, 13:15, "lustre-discuss on behalf of Thomas Roth" <lustre-discuss-boun...@lists.lustre.org on behalf of t.r...@gsi.de> wrote: Yes, I have seen this pattern before. My guess: - cetafs-OST0017 - 21 are at the limit, thus the * there, thus the overall * The manual warns you that somebody might write more than his allotted amount because the quota is distributed over the OSTs, this might work the other way around, too. - 8k or 20k is far too low. However, I have no idea how to influence these values. Regards Thomas On 09/07/2019 08.46, Alfonso Pardo wrote: > Hi, > > If I set quota to "-b 0 -B 0" and inmedeately set guota to 20G again, I get > same result, quota exceded. > When I run "lfs quota -v", this is the output: > > ---------------------------------------------------------------------------- > ---------------------------------------------------------------------------- > --------------------- > Disk quotas for group XXXXXXX (gid 694): > Filesystem used quota limit grace files quota limit > grace > /mnt/data 2.307G* 20G 20G 6d23h59m58s 39997 100000 100000 > - > cetafs-MDT0000_UUID > 25.29M - 0k - 39997 - 65536 > - > cetafs-OST0014_UUID > 451.1M - 452.1M - - - - > - > cetafs-OST0015_UUID > 409.7M - 410.7M - - - - > - > cetafs-OST0016_UUID > 429.4M - 430.4M - - - - > - > cetafs-OST0017_UUID > 1.022G* - 1.022G - - - - > - > cetafs-OST0018_UUID > 8k* - 8k - - - - > - > cetafs-OST0019_UUID > 20k* - 20k - - - - > - > cetafs-OST001a_UUID > 8k* - 8k - - - - > - > cetafs-OST001b_UUID > 24k* - 24k - - - - > - > quotactl ost28 failed. > quotactl ost29 failed. > quotactl ost30 failed. > quotactl ost31 failed. > cetafs-OST0020_UUID > 116k* - 116k - - - - > - > cetafs-OST0021_UUID > 136k* - 136k - - - - > - > Total allocated inode limit: 65536, total allocated block limit: 2.285G > Some errors happened when getting quota info. Some devices may be not > working or deactivated. The data in "[]" is inaccurate. > ---------------------------------------------------------------------------- > ---------------------------------------------------------------------------- > --------------------- > > > As you can see, I have some OST deactivated, because I will remove them. > > I have set quotas without "quota" (soft with -b) only setting "limit" (hard > with -B), and it works fine, no quota exceded, but if I set "quota" (-b) the > error appears. > > > > -----Mensaje original----- > De: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org] En > nombre de Thomas Roth > Enviado el: lunes, 8 de julio de 2019 15:14 > Para: lustre-discuss@lists.lustre.org > Asunto: Re: [lustre-discuss] RV: Lustre quota issues > > Perhaps the same issue that we see from time to time. > > What happens if you remove the quota alltogether (-b 0 -B 0) and set them to > 20G immedeately afterwards? > That made Lustre reconsider and repent in our case. > > Still, I suspect it is connected to the parts of the quota-total attributed > to each OST: > Try 'lfs quota -v' to see these for each OST. > > The manual talks of the 'qunit size', but is not entirely clear whether that > is a tunable, a value derived from Lustre/OST size, a fixed value, and how > to dead with it. > > > Regards > Thomas > > > > On 08/07/2019 12.42, Alfonso Pardo wrote: >> Hi, >> >> >> >> We have an issue with Lustre quotas and groups. We assign storage >> quotas to users groups. In general it works fine, but sometimes with >> some groups Lustre says that the quota limit has been passed. But it >> not true, if you run "lfs quota -g XXXX /data" we get a storage used below > the quota limit: >> >> >> >> Disk quotas for group XXXXXX (gid 694): >> >> Filesystem used quota limit grace files quota limit grace >> >> /mnt/data 2.148G* 20G 20G 2d1h31m9s 32733 100000 100000 > - >> >> >> >> >> >> As you can see in this example the group has 2.148Gb used, and the >> quotas is established to 20Gb, but Lustre say that the quota has been >> passed (*) and a grace period has started. >> >> >> >> If I upgrade the quota to 50G, for example, the quota indicator (*) >> and no quota errors is vanished. >> >> >> >> >> >> Any idea or suggestion? >> >> >> >> >> >> Thanks in advance >> >> >> >> Alfonso Pardo Díaz >> CETA·Ciemat >> Departamento de Tecnología >> Conventual de San Francisco >> 10200 Trujillo (Cáceres) >> Tel: 927 659 317 | Ext: 214 >> alfonso.pa...@ciemat.es <mailto:alfonso.pa...@ciemat.es> >> >> <http://www.ceta-ciemat.es/> >> >> >> >> >> >> _______________________________________________ >> lustre-discuss mailing list >> lustre-discuss@lists.lustre.org >> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >> > > -- > -------------------------------------------------------------------- > Thomas Roth > Department: Informationstechnologie > Location: SB3 2.291 > Phone: +49-6159-71 1453 Fax: +49-6159-71 2986 > > > GSI Helmholtzzentrum für Schwerionenforschung GmbH > Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de > > Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528 > Managing Directors / Geschäftsführung: > Professor Dr. Paolo Giubellino, Ursula Weyrich, Jörg Blaurock > Chairman of the Supervisory Board / Vorsitzender des GSI-Aufsichtsrats: > State Secretary / Staatssekretär Dr. Georg Schütte > _______________________________________________ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > -- -------------------------------------------------------------------- Thomas Roth Department: Informationstechnologie Location: SB3 2.291 Phone: +49-6159-71 1453 Fax: +49-6159-71 2986 GSI Helmholtzzentrum für Schwerionenforschung GmbH Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528 Managing Directors / Geschäftsführung: Professor Dr. Paolo Giubellino, Ursula Weyrich, Jörg Blaurock Chairman of the Supervisory Board / Vorsitzender des GSI-Aufsichtsrats: State Secretary / Staatssekretär Dr. Georg Schütte _______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org _______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org