MESSAGE quota resource implemention

[Julien Coloos]

Hi,

As discussed on IRC last week, I worked on implementing MESSAGE quota 
resource in cyrus (see RFC 2087; STORAGE resource already being handled).
I created a branch based on Greg's 'annotate' one on github, since his 
work on annotation storage management made mine a lot easier.


Details on the changes I made:

cyrus-imapd:
  - added MESSAGE quota resource management
- updated cunit test
- added 'quotawarnmsg' option which behaviour is similar to 
'quotawarnkb'; warning message shown when selecting folder in IMAP tells 
which resource limit was triggered
- added 'autocreatequotamsg' option, to set MESSAGE limit 
(unlimited by default) when user auto-creates its mailbox

  - xfer transfers all non-unlimited resources
  - added helper function to compute annotation storage usage for a 
given mailbox

  - changed the way quota entries are read/written
- resources presence in fetched entry is remembered
- when writing entry, only present resources are written
- when setting resources limits, entries which were not present 
upon fetching are now marked as present and their current usage is computed
  - quota utility lists and computes (-f) all resources associated to 
mailbox
The branch 'quotamessage/gnb/annotate' is available here: 
git://github.com/worldline-messaging/cyrus-imapd.git. It is based on 
Greg's 'annotate' branch on github.


cassandane:
  - added tests for MESSAGE resource
  - modified current tests to play a bit with subfolders and setting 
quota after adding messages/annotations
  - special test to check new resource usage is computed when necessary 
for pre-existing mailboxes (which only have usage for STORAGE resource)
The branch 'quotamessage/gnb/master' is available here: 
git://github.com/worldline-messaging/cassandane.git. It is based on 
Greg's 'master' branch on github.



Things that may be worth noting:
  - DUMP/UNDUMP currently does nothing special about MESSAGE or 
X-ANNOTATION-STORAGE quota resources

- should it be transferred ?
- without breaking backward compatibility, limits could only be 
transferred through a 'fake' file entry, as for annotations
  - quota usage is currently stored in a uquota_t variable, and delta 
is computed as quota_t; so theorically there could be overflow issues if 
quota usage to add/substract cannot be held in a quota_t; in practice it 
should be unlikely since that would mean a usage of over 2^63-1



Why the change concerning quota entries read/write ?
The thing is that I wanted to make version upgrading as painless as 
possible, both for users and in the code.
With previous code, when quota entry exists and is read, missing 
resources are initialised with default values (0 usage, unlimited). Thus 
only usage delta is tracked, and actual usage computing would only 
happen if quota entry was missing: this is not nice when upgrading, 
since lots of mailboxes are likely to have an entry with only the 
STORAGE resource present.
So actual usage has to be computed at some point for newly handled 
resources. The idea here is to compute it when setting the resource for 
the first time. To do that it was necessary to know when the resource 
was not previously present and keep it that way until its limit is 
finally set for the first time.
This scheme has another advantage: for platforms where only STORAGE 
quota is used, quota entries size remains as it is now. Only people 
using those new quota resources will have their quota entries grow to 
store the new data.


Comments are welcomed :)


I think that there are two things that may also be done concerning quota 
entries:
  - always recompute resource usage when changing its limit from 
unlimited to bounded
- currently it is only done once when setting the usage limit for 
the first time
- that way, it may not be necessary to track resource presence 
when reading/writing quota entries
- but maybe it could be too time consuming in some cases, since it 
seems to be possible to associate a quota resource to a whole domain 
(recomputing usage for all mailboxes would then take some time)

  - do not write resource in quota entry when its usage is unlimited
- except for the STORAGE resource, for backward compatibility
- would also help keeping quota entries size to the bare minimum

What do you think ?


PS: should I open a new bugzilla ticket for that ?

PPS:
cunit: on my computer cunit tests succeed except the 'foreach' one in 
the 'quota' suite which timeouts (it seems messing with 4096 quotalegacy 
is too much for my computer).
cassandane: a few quota tests I added fail due to some of the issues I 
reported (#5327 and #5329). And poor lemmings (well thought out name in 
this context :)) are dying hopelessly and endlessly in the 
Cassandane::Cyrus::Master test, preventing it to complete. Don't know if 
it is normal.

Re: MESSAGE quota resource implemention

[Bron Gondwana]

On Wed, Aug 31, 2011 at 05:50:36PM +0200, Julien Coloos wrote:
 Things that may be worth noting:
   - DUMP/UNDUMP currently does nothing special about MESSAGE or
 X-ANNOTATION-STORAGE quota resources
 - should it be transferred ?

I'd like to replace DUMP/UNDUMP with replication protocol
communications for XFER.

 - without breaking backward compatibility, limits could only be
 transferred through a 'fake' file entry, as for annotations

But for now, that's definitely the pragmatic way to go.

   - quota usage is currently stored in a uquota_t variable, and
 delta is computed as quota_t; so theorically there could be overflow
 issues if quota usage to add/substract cannot be held in a quota_t;
 in practice it should be unlikely since that would mean a usage of
 over 2^63-1

I propose we scrap uquota_t - it is un-necessary for the medium-term
future, now that we're requiring 64 bit types.

 I think that there are two things that may also be done concerning
 quota entries:
   - always recompute resource usage when changing its limit from
 unlimited to bounded

That's not too expensive really - we do it in the quota tool anyway.

The bigger issue is locking.  You need to be very careful of deadlock
situations if you do this.  We have that issue in the quota tool now.

The only _real_ way to do it reliably would be to go through and
remove all the quota roots from each mailbox, then go through again
one at a time and add them back, recording usage as you go.  Otherwise
there's no way of knowing if your end result is consistent.

 - that way, it may not be necessary to track resource presence
 when reading/writing quota entries
 - but maybe it could be too time consuming in some cases, since
 it seems to be possible to associate a quota resource to a whole
 domain (recomputing usage for all mailboxes would then take some
 time)

Yeah, now this is what you could call all sorts of bad names.  It's
really not sane to keep a quota counter at EVERY possible level in
the tree just incase someone wants to hang a limit there later.
You would get insane amounts of contention on the domain quota
lock for a busy domain, which would be un-necessary.

 PS: should I open a new bugzilla ticket for that ?

Why not - numbers are cheap.

 PPS:
 cunit: on my computer cunit tests succeed except the 'foreach' one
 in the 'quota' suite which timeouts (it seems messing with 4096
 quotalegacy is too much for my computer).

There's stuff you can do with fd limits.  Check out /etc/pam.d/* for
pam_limits.so, and then /etc/security/limits.conf.  I just had to
learn about that stuff today!

Bron.