Re: disk quotas bug fix [was: quotas grace period none right away]
On Mon, Dec 01, 2014 at 09:52:18PM -0600, Boris Goldberg wrote: Hello misc, I've reported a detailed bug two months ago. The short story - grace period end time isn't being reset if the over_soft_quota stage is reached by chown command. I've confirmed it on i386 5.0 through current (as of month ago) and on amd64 5.4. Developers seemed to don't have time for it, so I've asked our consultant, Ed Bartosh bart...@gmail.com (not subscribed to the list), to look into this. It seems like he has fixed it. Here is the patch for 5.4 (tested on i386 only yet): Cool, but your mailer mangled the diff. Here it is for current. BTW, let's take this to tech@, that's a better place. So anybody replying, please rm misc@. -Otto Index: ufs_vnops.c === RCS file: /cvs/src/sys/ufs/ufs/ufs_vnops.c,v retrieving revision 1.116 diff -u -p -r1.116 ufs_vnops.c --- ufs_vnops.c 3 Nov 2014 21:28:35 - 1.116 +++ ufs_vnops.c 2 Dec 2014 08:10:42 - @@ -498,6 +498,7 @@ ufs_chown(struct vnode *vp, uid_t uid, g int error = 0; daddr_t change; enum ufs_quota_flags quota_flags = 0; + struct ucred *newcr; if (uid == (uid_t)VNOVAL) uid = DIP(ip, uid); @@ -534,16 +535,23 @@ ufs_chown(struct vnode *vp, uid_t uid, g if ((error = getinoquota(ip)) != 0) goto error; - if ((error = ufs_quota_alloc_blocks2(ip, change, cred, -quota_flags)) != 0) + newcr = crget(); + newcr-cr_uid = uid; + newcr-cr_gid = gid; + + if ((error = ufs_quota_alloc_blocks2(ip, change, newcr, + quota_flags)) != 0) { + crfree(newcr); goto error; + } - if ((error = ufs_quota_alloc_inode2(ip, cred , -quota_flags)) != 0) { - (void)ufs_quota_free_blocks2(ip, change, cred, - quota_flags); + if ((error = ufs_quota_alloc_inode2(ip, newcr, quota_flags)) != 0) { + (void)ufs_quota_free_blocks2(ip, change, newcr, quota_flags); + crfree(newcr); goto error; } + + crfree(newcr); if (getinoquota(ip)) panic(chown: lost quota);
disk quotas bug fix [was: quotas grace period none right away]
Hello misc, I've reported a detailed bug two months ago. The short story - grace period end time isn't being reset if the over_soft_quota stage is reached by chown command. I've confirmed it on i386 5.0 through current (as of month ago) and on amd64 5.4. Developers seemed to don't have time for it, so I've asked our consultant, Ed Bartosh bart...@gmail.com (not subscribed to the list), to look into this. It seems like he has fixed it. Here is the patch for 5.4 (tested on i386 only yet): Index: ufs_vnops.c === RCS file: /cvs/src/sys/ufs/ufs/ufs_vnops.c,v retrieving revision 1.107 diff -u -p -r1.107 ufs_vnops.c --- ufs_vnops.c 11 Jun 2013 16:42:19 - 1.107 +++ ufs_vnops.c 1 Dec 2014 21:54:44 - @@ -448,6 +448,8 @@ ufs_chown(struct vnode *vp, uid_t uid, g int error = 0; daddr_t change; enum ufs_quota_flags quota_flags = 0; + struct ucred *newcr; + if (uid == (uid_t)VNOVAL) uid = DIP(ip, uid); @@ -484,17 +486,26 @@ ufs_chown(struct vnode *vp, uid_t uid, g if ((error = getinoquota(ip)) != 0) goto error; - if ((error = ufs_quota_alloc_blocks2(ip, change, cred, -quota_flags)) != 0) + newcr = crget(); + newcr-cr_uid = uid; + newcr-cr_gid = gid; + + if ((error = ufs_quota_alloc_blocks2(ip, change, newcr, +quota_flags)) != 0) { + crfree(newcr); goto error; + } - if ((error = ufs_quota_alloc_inode2(ip, cred , + if ((error = ufs_quota_alloc_inode2(ip, newcr, quota_flags)) != 0) { - (void)ufs_quota_free_blocks2(ip, change, cred, + (void)ufs_quota_free_blocks2(ip, change, newcr, quota_flags); + crfree(newcr); goto error; } + crfree(newcr); + if (getinoquota(ip)) panic(chown: lost quota); Please advise if you see problems with that patch (besides the fact that it's for 5.4). -- Best regards, Boris mailto:bo...@twopoint.com
Re: quotas grace period none right away
Hello Otto, Monday, October 6, 2014, 10:42:32 AM, you wrote: OM Yeah. Have something similar in my tree. If -Wall is happy, so am I. OM Does it explain 5.4 problems though. OM I did not manage to reproduce those so far. It looks like the time_t patch is applicable to 5.5 (and later) only. Am I wrong? Is there going to be any (further) development about that bug in 5.4? -- Best regards, Borismailto:bo...@twopoint.com
Re: quotas grace period none right away
On Wed, 1 Oct 2014, Otto Moerbeek wrote: On Tue, Sep 30, 2014 at 07:31:20PM +0200, Otto Moerbeek wrote: On Tue, Sep 30, 2014 at 11:20:23AM -0500, Boris Goldberg wrote: Hello Otto, Wednesday, September 24, 2014, 2:36:58 PM, you wrote: OM Try to come up with a reproducable test case, include all relevant OM info and then we can investigate. I indeed see strange things on sparc64 more or less -current. Not exactly what you are seeing, but for starters, edquota -t is giving me what looks like unitialized mem. I hope to find some time to investigate further... -Otto There is indeed a bug in edquota -t in 5.5 and newer due to the time_t change, but that is unrelated to what you are seeing. Poked at this last night and came up with this, eliminating the bogus casting from time_t* to int*. Index: edquota.c === RCS file: /cvs/src/usr.sbin/edquota/edquota.c,v retrieving revision 1.53 diff -u -p -r1.53 edquota.c --- edquota.c 20 Jul 2014 01:38:40 - 1.53 +++ edquota.c 6 Oct 2014 15:34:24 - @@ -77,7 +77,7 @@ int readprivs(struct quotause *, int); intwritetimes(struct quotause *, int, int); intreadtimes(struct quotause *, int); char * cvtstoa(time_t); -intcvtatos(time_t, char *, time_t *); +intcvtatos(long long, char *, time_t *); void freeprivs(struct quotause *); intalldigits(char *s); inthasquota(struct fstab *, int, char **); @@ -569,7 +569,8 @@ readtimes(struct quotause *quplist, int FILE *fp; int cnt; char *cp; - time_t itime, btime, iseconds, bseconds; + long long itime, btime; + time_t iseconds, bseconds; char *fsp, bunits[10], iunits[10], line1[BUFSIZ]; lseek(infd, 0, SEEK_SET); @@ -594,8 +595,8 @@ readtimes(struct quotause *quplist, int return(0); } cnt = sscanf(cp, -block grace period: %d %9s file grace period: %d %9s, - (int *)btime, bunits, (int *)itime, iunits); +block grace period: %lld %9s file grace period: %lld %9s, + btime, bunits, itime, iunits); if (cnt != 4) { warnx(%s:%s: bad format, fsp, cp); return(0); @@ -639,19 +640,19 @@ cvtstoa(time_t time) if (time % (24 * 60 * 60) == 0) { time /= 24 * 60 * 60; - (void)snprintf(buf, sizeof buf, %d day%s, (int)time, + (void)snprintf(buf, sizeof buf, %lld day%s, (long long)time, time == 1 ? : s); } else if (time % (60 * 60) == 0) { time /= 60 * 60; - (void)snprintf(buf, sizeof buf, %d hour%s, (int)time, + (void)snprintf(buf, sizeof buf, %lld hour%s, (long long)time, time == 1 ? : s); } else if (time % 60 == 0) { time /= 60; - (void)snprintf(buf, sizeof buf, %d minute%s, (int)time, - time == 1 ? : s); + (void)snprintf(buf, sizeof buf, %lld minute%s, + (long long)time, time == 1 ? : s); } else - (void)snprintf(buf, sizeof buf, %d second%s, (int)time, - time == 1 ? : s); + (void)snprintf(buf, sizeof buf, %lld second%s, + (long long)time, time == 1 ? : s); return(buf); } @@ -659,7 +660,7 @@ cvtstoa(time_t time) * Convert ASCII input times to seconds. */ int -cvtatos(time_t time, char *units, time_t *seconds) +cvtatos(long long time, char *units, time_t *seconds) { if (bcmp(units, second, 6) == 0)
Re: quotas grace period none right away
Yeah. Have something similar in my tree. If -Wall is happy, so am I. Does it explain 5.4 problems though. I did not manage to reproduce those so far. -Otto Op 6 okt. 2014 om 17:38 heeft Philip Guenther guent...@gmail.com het volgende geschreven: On Wed, 1 Oct 2014, Otto Moerbeek wrote: On Tue, Sep 30, 2014 at 07:31:20PM +0200, Otto Moerbeek wrote: On Tue, Sep 30, 2014 at 11:20:23AM -0500, Boris Goldberg wrote: Hello Otto, Wednesday, September 24, 2014, 2:36:58 PM, you wrote: OM Try to come up with a reproducable test case, include all relevant OM info and then we can investigate. I indeed see strange things on sparc64 more or less -current. Not exactly what you are seeing, but for starters, edquota -t is giving me what looks like unitialized mem. I hope to find some time to investigate further... -Otto There is indeed a bug in edquota -t in 5.5 and newer due to the time_t change, but that is unrelated to what you are seeing. Poked at this last night and came up with this, eliminating the bogus casting from time_t* to int*. Index: edquota.c === RCS file: /cvs/src/usr.sbin/edquota/edquota.c,v retrieving revision 1.53 diff -u -p -r1.53 edquota.c --- edquota.c20 Jul 2014 01:38:40 -1.53 +++ edquota.c6 Oct 2014 15:34:24 - @@ -77,7 +77,7 @@ intreadprivs(struct quotause *, int); intwritetimes(struct quotause *, int, int); intreadtimes(struct quotause *, int); char *cvtstoa(time_t); -intcvtatos(time_t, char *, time_t *); +intcvtatos(long long, char *, time_t *); voidfreeprivs(struct quotause *); intalldigits(char *s); inthasquota(struct fstab *, int, char **); @@ -569,7 +569,8 @@ readtimes(struct quotause *quplist, int FILE *fp; int cnt; char *cp; -time_t itime, btime, iseconds, bseconds; +long long itime, btime; +time_t iseconds, bseconds; char *fsp, bunits[10], iunits[10], line1[BUFSIZ]; lseek(infd, 0, SEEK_SET); @@ -594,8 +595,8 @@ readtimes(struct quotause *quplist, int return(0); } cnt = sscanf(cp, - block grace period: %d %9s file grace period: %d %9s, -(int *)btime, bunits, (int *)itime, iunits); + block grace period: %lld %9s file grace period: %lld %9s, +btime, bunits, itime, iunits); if (cnt != 4) { warnx(%s:%s: bad format, fsp, cp); return(0); @@ -639,19 +640,19 @@ cvtstoa(time_t time) if (time % (24 * 60 * 60) == 0) { time /= 24 * 60 * 60; -(void)snprintf(buf, sizeof buf, %d day%s, (int)time, +(void)snprintf(buf, sizeof buf, %lld day%s, (long long)time, time == 1 ? : s); } else if (time % (60 * 60) == 0) { time /= 60 * 60; -(void)snprintf(buf, sizeof buf, %d hour%s, (int)time, +(void)snprintf(buf, sizeof buf, %lld hour%s, (long long)time, time == 1 ? : s); } else if (time % 60 == 0) { time /= 60; -(void)snprintf(buf, sizeof buf, %d minute%s, (int)time, -time == 1 ? : s); +(void)snprintf(buf, sizeof buf, %lld minute%s, +(long long)time, time == 1 ? : s); } else -(void)snprintf(buf, sizeof buf, %d second%s, (int)time, -time == 1 ? : s); +(void)snprintf(buf, sizeof buf, %lld second%s, +(long long)time, time == 1 ? : s); return(buf); } @@ -659,7 +660,7 @@ cvtstoa(time_t time) * Convert ASCII input times to seconds. */ int -cvtatos(time_t time, char *units, time_t *seconds) +cvtatos(long long time, char *units, time_t *seconds) { if (bcmp(units, second, 6) == 0)
Re: quotas grace period none right away
On Tue, Sep 30, 2014 at 07:31:20PM +0200, Otto Moerbeek wrote: On Tue, Sep 30, 2014 at 11:20:23AM -0500, Boris Goldberg wrote: Hello Otto, Wednesday, September 24, 2014, 2:36:58 PM, you wrote: OM Try to come up with a reproducable test case, include all relevant OM info and then we can investigate. I indeed see strange things on sparc64 more or less -current. Not exactly what you are seeing, but for starters, edquota -t is giving me what looks like unitialized mem. I hope to find some time to investigate further... -Otto There is indeed a bug in edquota -t in 5.5 and newer due to the time_t change, but that is unrelated to what you are seeing. I installed a 5.4 i386 machine to investigate further. Please tell me a few things: are you using softdep? uids in fstab? Please how me you full fstab line for the filesystem. Also, I like to see the output of both edquota -t and $ hexdump -C ../user.quota | head Note that changes made by edquota -t only end up in quota.user file after a a quoatcheck run. To be sure, reboot first. BTW please fix you email. Both you mx time out. -Otto
Re: quotas grace period none right away
Hello Otto, Wednesday, September 24, 2014, 2:36:58 PM, you wrote: OM Try to come up with a reproducable test case, include all relevant OM info and then we can investigate. Here is what I could reproduce: root@mail1 ~ # quota test_spam Disk quotas for user test_spam (uid 1003): Filesystem KBytesquota limit gracefiles quota limit grace /var/mail 28 10 100 81 10 root@mail1 ~ # dd if=/dev/random of=w00 bs=1M count=150 150+0 records in 150+0 records out 157286400 bytes transferred in 2.679 secs (58707553 bytes/sec) root@mail1 ~ # mv w00 ~test_spam/ root@mail1 ~ # chown test_spam /var/mail/test_spam/w00 root@mail1 ~ # quota test_spam Disk quotas for user test_spam (uid 1003): Filesystem KBytesquota limit gracefiles quota limit grace /var/mail 153660* 10 100 18:10 91 10 root@mail1 ~ # edquota -t Time units may be: days, hours, minutes, or seconds Grace period before enforcing soft limits for users: /var/mail: block grace period: 30 days, file grace period: 30 days root@mail1 ~ # date Mon Sep 29 14:12:42 CDT 2014 root@mail1 ~ # rm /var/mail/test_spam/w00 root@mail1 ~ # quota test_spam Disk quotas for user test_spam (uid 1003): Filesystem KBytesquota limit gracefiles quota limit grace /var/mail 28 10 100 81 10 root@mail1 ~ # date Mon Sep 29 18:47:44 CDT 2014 root@mail1 ~ # quota test_spam Disk quotas for user test_spam (uid 1003): Filesystem KBytesquota limit gracefiles quota limit grace /var/mail 28 10 100 81 10 root@mail1 ~ # dd if=/dev/random of=~test_spam/w00 bs=1M count=150 150+0 records in 150+0 records out 157286400 bytes transferred in 2.059 secs (76367302 bytes/sec) root@mail1 ~ # chown test_spam /var/mail/test_spam/w00 root@mail1 ~ # quota test_spam Disk quotas for user test_spam (uid 1003): Filesystem KBytesquota limit gracefiles quota limit grace /var/mail 153660* 10 100 13:31 91 10 root@mail1 ~ # rm /var/mail/test_spam/w00 root@mail1 ~ # quota test_spam Disk quotas for user test_spam (uid 1003): Filesystem KBytesquota limit gracefiles quota limit grace /var/mail 28 10 100 81 10 root@mail1 ~ # date Tue Sep 30 08:38:03 CDT 2014 root@mail1 ~ # quota test_spam Disk quotas for user test_spam (uid 1003): Filesystem KBytesquota limit gracefiles quota limit grace /var/mail 28 10 100 81 10 root@mail1 ~ # dd if=/dev/random of=~test_spam/w00 bs=1M count=150 150+0 records in 150+0 records out 157286400 bytes transferred in 2.074 secs (75822855 bytes/sec) root@mail1 ~ # chown test_spam /var/mail/test_spam/w00 root@mail1 ~ # quota test_spam Disk quotas for user test_spam (uid 1003): Filesystem KBytesquota limit gracefiles quota limit grace /var/mail 153660* 10 100none 91 10 root@mail1 ~ # rm /var/mail/test_spam/w00 root@mail1 ~ # quota test_spam Disk quotas for user test_spam (uid 1003): Filesystem KBytesquota limit gracefiles quota limit grace /var/mail 28 10 100 81 10 root@mail1 ~ # dmesg | head OpenBSD 5.4-stable (GENERIC.MP) #3: Wed Apr 2 16:44:04 CDT 2014 r...@build32.twopoint.com:/usr/src/sys/arch/i386/compile/GENERIC.MP cpu0: Intel(R) Xeon(R) CPU 3060 @ 2.40GHz (GenuineIntel 686-class) 2.41 GHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,NXE,LONG,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,LAHF,PERF real mem = 3621744640 (3453MB) avail mem = 3551121408 (3386MB) mainbus0 at root bios0 at mainbus0: AT/286+ BIOS, date 12/31/99, BIOS32 rev. 0 @ 0xf, SMBIOS rev. 2.3 @ 0xee000 (47 entries) bios0: vendor HP version W04 date 04/06/2007 bios0: HP ProLiant DL320 G5 I've also started the test case on another computer (turned on user quotas and created a new user) - everything starts unfolding the same way: # quota test Disk quotas for user test (uid 1002): Filesystem KBytesquota limit gracefiles quota limit grace /wrk 4 10 100 11 10 # dd if=/dev/random of=/wrk/test/w00 bs=1M count=150 150+0 records in 150+0 records out 157286400 bytes transferred in 14.572 secs (10793030 bytes/sec) # chown test /wrk/test/w00 # quota test Disk quotas for user test (uid 1002): Filesystem KBytesquota limit gracefiles quota limit grace /wrk 153636* 10 100 7days 21 10 # rm /wrk/test/w00 # quota test Disk quotas for user test (uid 1002): Filesystem KBytesquota limit gracefiles quota limit grace /wrk 4 10
Re: quotas grace period none right away
On Tue, Sep 30, 2014 at 11:20:23AM -0500, Boris Goldberg wrote: Hello Otto, Wednesday, September 24, 2014, 2:36:58 PM, you wrote: OM Try to come up with a reproducable test case, include all relevant OM info and then we can investigate. I indeed see strange things on sparc64 more or less -current. Not exactly what you are seeing, but for starters, edquota -t is giving me what looks like unitialized mem. I hope to find some time to investigate further... -Otto
Re: quotas grace period none right away
On 2014-09-24 Wed 09:22 AM |, Boris Goldberg wrote: Does this mean you tried and found out (or knew) that disk quotas where not going to work for you? At the moment Boris, I'm not using quotas - but did a few years ago. I don't remember having any problems then. I guessed Dovecot would work for you by going around any possible issue due to filesystem delivery ... temporarily drop privileges to users.
Re: quotas grace period none right away
On 2014-09-23 Tue 10:06 AM |, Boris Goldberg wrote: This might have something to do with the fact that this is a mail server, and mail is being delivered by root (by procmail to maildirs if it makes a difference). How about Dovecot sieve (rules can be edited remotely in Thunderbird): Filesystem quota supports both local filesystems and rquota (NFS).: http://wiki2.dovecot.org/Quota/FS http://wiki2.dovecot.org/Quota ... temporarily drop privileges to users.: http://wiki2.dovecot.org/LMTP http://wiki2.dovecot.org/Pigeonhole/Sieve http://wiki2.dovecot.org/Pigeonhole/ManageSieve/ Almost a GUI for semi-competent users: https://addons.mozilla.org/EN-US/thunderbird/addon/sieve/ I've configured Dovecot to deliver to (NFS) /var/mail/ dirs, with per user sieve scripts in /var/mail/user/sieve/ too. Mutt works fine with /etc/mutt/Muttrc: set folder=$MAIL set postponed=$MAIL/.Drafts set record=$MAIL/.Sent set trash=$MAIL/.Trash mailboxes + `echo $MAIL/.* | tr -s ' ' '\n' | egrep -v 'Archive|Trash.' | tr -s '\n' ' '` There is no mail kept in $HOME This way, mail deleted or sent locally via Mutt, can be accessed remotely via IMAP - it is uniformly seemless. Also, /home /var/mail can be unmounted dumped with different level 0 frequency. Say annually for /home quarterly for /var/mail.
Re: quotas grace period none right away
Hello Otto, Tuesday, September 23, 2014, 10:54:56 AM, you wrote: OM Grace moves to none if you go above the hard limit. If mail delivery OM is done by root, quota's are not enforced, so you can go over the hard OM limit, nulling the grace period. OM This is a problem I solved a long time ago by using a patch the do OM local mail delivery as a specific user, but that diff was never OM committed. I've read your post about it, but in my case it doesn't go above hard limit, just slightly above the soft one (for a time definitely less than one day). The quota is 10/100, so there is (should be) a long way between them. Just noticed something else. Was playing creating and deleting big files yesterday - it was showing grace period 7 days (before I deleted the files). Created another one today - the grace period is 6 days. Does it suppose to have that long memory (over 12 hours)? Don't believe it was like that before (in 5.0). -- Best regards, Borismailto:bo...@twopoint.com
Re: quotas grace period none right away
Hello Craig, Wednesday, September 24, 2014, 3:56:35 AM, you wrote: CRS How about Dovecot sieve ... Does this mean you tried and found out (or knew) that disk quotas where not going to work for you? -- Best regards, Borismailto:bo...@twopoint.com
Re: quotas grace period none right away
On Wed, Sep 24, 2014 at 09:18:10AM -0500, Boris Goldberg wrote: Hello Otto, Tuesday, September 23, 2014, 10:54:56 AM, you wrote: OM Grace moves to none if you go above the hard limit. If mail delivery OM is done by root, quota's are not enforced, so you can go over the hard OM limit, nulling the grace period. OM This is a problem I solved a long time ago by using a patch the do OM local mail delivery as a specific user, but that diff was never OM committed. I've read your post about it, but in my case it doesn't go above hard limit, just slightly above the soft one (for a time definitely less than one day). The quota is 10/100, so there is (should be) a long way between them. Just noticed something else. Was playing creating and deleting big files yesterday - it was showing grace period 7 days (before I deleted the files). Created another one today - the grace period is 6 days. Does it suppose to have that long memory (over 12 hours)? Don't believe it was like that before (in 5.0). Try to come up with a reproducable test case, include all relevant info and then we can investigate. -Otto
quotas grace period none right away
Hello misc, I'm using i386 5.4-stable (GENERIC.MP) with user quotas (only) set on one FS (it's default FS, nothing special). The grace period is 7 days, edquota -t confirms it. It works fine if I create/chown files from shell, but changes to none right away with every day operations (twice already). The box have rebooted after quotas where set, so the quotacheck did run. This might have something to do with the fact that this is a mail server, and mail is being delivered by root (by procmail to maildirs if it makes a difference). I've found an old Otto's message about something remotely related. Does someone have deeper understanding of this situation or experienced something similar? -- Best regards, Boris mailto:bo...@twopoint.com
Re: quotas grace period none right away
On Tue, Sep 23, 2014 at 10:06:29AM -0500, Boris Goldberg wrote: Hello misc, I'm using i386 5.4-stable (GENERIC.MP) with user quotas (only) set on one FS (it's default FS, nothing special). The grace period is 7 days, edquota -t confirms it. It works fine if I create/chown files from shell, but changes to none right away with every day operations (twice already). The box have rebooted after quotas where set, so the quotacheck did run. This might have something to do with the fact that this is a mail server, and mail is being delivered by root (by procmail to maildirs if it makes a difference). I've found an old Otto's message about something remotely related. Does someone have deeper understanding of this situation or experienced something similar? -- Best regards, Boris mailto:bo...@twopoint.com Grace moves to none if you go above the hard limit. If mail delivery is done by root, quota's are not enforced, so you can go over the hard limit, nulling the grace period. This is a problem I solved a long time ago by using a patch the do local mail delivery as a specific user, but that diff was never committed. -Otto