Re: MESSAGE quota resource implemention

2011-09-18 Thread Greg Banks
On 17/09/11 01:37, Julien Coloos wrote: > As discussed here and on IRC, I rebased my commits with all the changes: > - the 'utility' methods have been promoted to 'command' ones, which > are now generic and may (options hash) concern cyrus binaries > - the 'start_command_bg' method is now repla

Re: MESSAGE quota resource implemention

2011-09-16 Thread Julien Coloos
As discussed here and on IRC, I rebased my commits with all the changes: - the 'utility' methods have been promoted to 'command' ones, which are now generic and may (options hash) concern cyrus binaries - the 'start_command_bg' method is now replaced by a 'background' option in 'run_command'

Re: MESSAGE quota resource implemention

2011-09-14 Thread Greg Banks
On 15/09/11 03:14, Julien Coloos wrote: > Le 12/09/2011 23:09, Bron Gondwana a écrit : >> >>> - for 'old' mailboxes (those created before the annotation storage >>> usage field in the index header), current annotations usage shall be >>> computed (and added to the quota entry) upon upgrading; thi

Re: MESSAGE quota resource implemention

2011-09-14 Thread Julien Coloos
Le 12/09/2011 23:09, Bron Gondwana a écrit : - for 'old' mailboxes (those created before the annotation storage usage field in the index header), current annotations usage shall be computed (and added to the quota entry) upon upgrading; this way users won't have to run 'quota -f' for all quot

Re: MESSAGE quota resource implemention

2011-09-12 Thread Bron Gondwana
On Mon, Sep 12, 2011 at 04:46:52PM +0200, Julien Coloos wrote: > Le 09/09/2011 14:18, Greg Banks a écrit : > >Ok, here you go. Not completely tested yet, so caveat emptor. > Remaining things concerning annotations: > - when deleting messages, annotations length is not substracted; > the solution m

Re: MESSAGE quota resource implemention

2011-09-12 Thread Julien Coloos
Le 09/09/2011 14:18, Greg Banks a écrit : Ok, here you go. Not completely tested yet, so caveat emptor. Had to change two things: - mailbox_quota_check now expects a quota diff array which is good :), but change is now really applied if all diffs are '> 0' (instead of '>= 0' previously); in

Re: MESSAGE quota resource implemention

2011-09-09 Thread Greg Banks
On 08/09/11 00:45, Julien Coloos wrote: > Le 06/09/2011 10:23, Greg Banks a écrit : >> > >> >> ... >> >> I'm still not convinced we'll need quota.sets[], but I'll play along. >> >> Thanks again for your work, and sorry that my annotate branch wasn't >> quite as stable a base as you first thought :)

Re: MESSAGE quota resource implemention

2011-09-07 Thread Greg Banks
Sent from my iPhone On 08/09/2011, at 0:45, Julien Coloos wrote: Le 06/09/2011 10:23, Greg Banks a écrit : a) my commit "Make quota -f less racy" is going to cause lots of clashes (sorry!) b) Bron and I both think that your commit "Compute each quota resource upon setting it for the

Re: MESSAGE quota resource implemention

2011-09-07 Thread Julien Coloos
Le 06/09/2011 10:23, Greg Banks a écrit : a) my commit "Make quota -f less racy" is going to cause lots of clashes (sorry!) b) Bron and I both think that your commit "Compute each quota resource upon setting it for the first time." is unnecessary, given that i) quota -f doesn't suck now, an

Re: MESSAGE quota resource implemention

2011-09-06 Thread Julien Coloos
Le 06/09/2011 19:48, Julien Coloos a écrit : Le 06/09/2011 10:23, Greg Banks a écrit : So I think we'll need another round, sorry :( Given that the annotations quota is broken and I'll be reimplementing it anyway, you may as well ignore QUOTA_ANNOTSTORAGE in all commits, and leave out the func

Re: MESSAGE quota resource implemention

2011-09-06 Thread Julien Coloos
Le 06/09/2011 10:23, Greg Banks a écrit : So I think we'll need another round, sorry :( Given that the annotations quota is broken and I'll be reimplementing it anyway, you may as well ignore QUOTA_ANNOTSTORAGE in all commits, and leave out the function annotatemore_computequota() for now. We'

Re: MESSAGE quota resource implemention

2011-09-06 Thread Greg Banks
On 06/09/11 02:55, Julien Coloos wrote: > Le 05/09/2011 06:12, Greg Banks a écrit : >> >> Julien, I think we agreed on everything else, right? I'm looking >> forward to your next iteration. > After picking your 'uquota_t removal' commit, I also removed it on my > end, and changed the code accordin

Re: MESSAGE quota resource implemention

2011-09-05 Thread Greg Banks
On 05/09/11 20:16, Greg Banks wrote: > On 02/09/11 20:03, Bron Gondwana wrote: >> On Fri, Sep 02, 2011 at 07:36:20PM +1000, Greg Banks wrote: >> >>> How's about this for a strategy? >>> >>> When a quota resource is first enabled, (i.e. the limit is changed from >>> UNLIMITED to some finite value),

Re: MESSAGE quota resource implemention

2011-09-05 Thread Bron Gondwana
On Mon, Sep 05, 2011 at 07:19:00PM +0200, Julien Coloos wrote: > Le 05/09/2011 12:06, Bron Gondwana a écrit : > >On Mon, Sep 05, 2011 at 02:32:40PM +1000, Greg Banks wrote: > > > >> - add a 32b mailbox index header entry to track the storage in bytes > >>used by all annotations for the mailbox its

Re: MESSAGE quota resource implemention

2011-09-05 Thread Julien Coloos
Le 05/09/2011 12:06, Bron Gondwana a écrit : On Mon, Sep 05, 2011 at 02:32:40PM +1000, Greg Banks wrote: - add a 32b mailbox index header entry to track the storage in bytes used by all annotations for the mailbox itself or for messages in the mailbox Why not 64b? Admittedly hanging gigabyt

Re: MESSAGE quota resource implemention

2011-09-05 Thread Julien Coloos
Le 05/09/2011 06:12, Greg Banks a écrit : Julien, I think we agreed on everything else, right? I'm looking forward to your next iteration. After picking your 'uquota_t removal' commit, I also removed it on my end, and changed the code according to our previous discussions. Adding an helper fu

Re: MESSAGE quota resource implemention

2011-09-05 Thread Greg Banks
On 05/09/11 20:06, Bron Gondwana wrote: > On Mon, Sep 05, 2011 at 02:32:40PM +1000, Greg Banks wrote: >> Ok, I'm now convinced my first attempt at annotation quotas sucked too >> hard, here's how I want to re-implement it. Let me know what you think. > >> - add a 32b mailbox index header entry t

Re: MESSAGE quota resource implemention

2011-09-05 Thread Greg Banks
On 02/09/11 20:03, Bron Gondwana wrote: > On Fri, Sep 02, 2011 at 07:36:20PM +1000, Greg Banks wrote: > >> How's about this for a strategy? >> >> When a quota resource is first enabled, (i.e. the limit is changed from >> UNLIMITED to some finite value), the usage is stored as some special >> value

Re: MESSAGE quota resource implemention

2011-09-05 Thread Bron Gondwana
On Mon, Sep 05, 2011 at 02:32:40PM +1000, Greg Banks wrote: > Ok, I'm now convinced my first attempt at annotation quotas sucked too > hard, here's how I want to re-implement it. Let me know what you think. > - add a 32b mailbox index header entry to track the storage in bytes > used by all anno

Re: MESSAGE quota resource implemention

2011-09-04 Thread Greg Banks
On 01/09/11 22:22, Bron Gondwana wrote: > On Thu, Sep 01, 2011 at 01:27:00PM +0200, Julien Coloos wrote: >> Le 01/09/2011 03:03, Greg Banks a écrit : > >>> I'm thinking that there's now three places in the code which take >>> a mailbox* and fill out an array of quota diffs, interpreting the >>> co

Re: MESSAGE quota resource implemention

2011-09-04 Thread Greg Banks
On 01/09/11 21:27, Julien Coloos wrote: >> >> [...] Perhaps >> we should a) document it clearly and b) detect the situation and put >> an obvious message saying something like "you may need to run quota -f >> ..." where the human making the change will see it. Also, such a >> message might be us

Re: MESSAGE quota resource implemention

2011-09-04 Thread Greg Banks
On 01/09/11 03:21, Bron Gondwana wrote: > On Wed, Aug 31, 2011 at 05:50:36PM +0200, Julien Coloos wrote: >> Things that may be worth noting: >> - DUMP/UNDUMP currently does nothing special about MESSAGE or >> X-ANNOTATION-STORAGE quota resources >> -> should it be transferred ? > > I'd like

Re: MESSAGE quota resource implemention

2011-09-02 Thread Bron Gondwana
On Fri, Sep 02, 2011 at 07:36:20PM +1000, Greg Banks wrote: > If the software was robust, underflow would not happen and we would not > need to test for it and handle it. Thus the log messages are not > operational messages intended for the sysadmin, but warnings about > internal Cyrus problems in

Re: MESSAGE quota resource implemention

2011-09-02 Thread Greg Banks
On 01/09/11 22:22, Bron Gondwana wrote: > On Thu, Sep 01, 2011 at 01:27:00PM +0200, Julien Coloos wrote: >> Le 01/09/2011 03:03, Greg Banks a écrit : > > But more generally, the "update the quotaroot" is atomic-safe, > because the mailbox doesn't add/remove things from the quotaroot > racily - bu

Re: MESSAGE quota resource implemention

2011-09-01 Thread Bron Gondwana
On Fri, Sep 02, 2011 at 09:37:24AM +1000, Rob Mueller wrote: > > >Actually, really I'd like to create a new UNIQUEID - and store > >all the files in paths based on uniqueid rather than on folder > >name. This would not only mean renames could be fast and > >atomic, but that delayed delete would b

Re: MESSAGE quota resource implemention

2011-09-01 Thread Rob Mueller
Actually, really I'd like to create a new UNIQUEID - and store all the files in paths based on uniqueid rather than on folder name. This would not only mean renames could be fast and atomic, but that delayed delete would be fast. The downside is a more opaque filesystem layout. Oh, another up

Re: MESSAGE quota resource implemention

2011-09-01 Thread Julien Coloos
Le 01/09/2011 14:22, Bron Gondwana a écrit : A provisioning system could run quota -f itself after making the change, of course. Sure. But since the quota is being changed, clients would wonder why they need to call another quota utility to finish the job. Plus I wanted to have something simila

Re: MESSAGE quota resource implemention

2011-09-01 Thread Bron Gondwana
On Thu, Sep 01, 2011 at 02:50:30PM +0200, Michael Menge wrote: > Quoting Bron Gondwana : > > >Actually, really I'd like to create a new UNIQUEID - and store > >all the files in paths based on uniqueid rather than on folder > >name. This would not only mean renames could be fast and > >atomic, but

Re: MESSAGE quota resource implemention

2011-09-01 Thread Michael Menge
Quoting Bron Gondwana : Actually, really I'd like to create a new UNIQUEID - and store all the files in paths based on uniqueid rather than on folder name. This would not only mean renames could be fast and atomic, but that delayed delete would be fast. The downside is a more opaque filesystem

Re: MESSAGE quota resource implemention

2011-09-01 Thread Bron Gondwana
On Thu, Sep 01, 2011 at 01:27:00PM +0200, Julien Coloos wrote: > Le 01/09/2011 03:03, Greg Banks a écrit : > >This one's tough, I wasn't sure what to do. However, I'm happy to > >leave it to the administrator to have to manually run quota -f > >(maybe twice!) if they set a quota on a resource that

Re: MESSAGE quota resource implemention

2011-09-01 Thread Julien Coloos
Hi, thanks for your comments and reviewing the code Le 01/09/2011 03:03, Greg Banks a écrit : I think that there are two things that may also be done concerning quota entries: - always recompute resource usage when changing its limit from unlimited to bounded -> currently it is only done

Re: MESSAGE quota resource implemention

2011-09-01 Thread Greg Banks
Julien Coloos wrote: Hi, As discussed on IRC last week, I worked on implementing MESSAGE quota resource in cyrus (see RFC 2087; STORAGE resource already being handled). I created a branch based on Greg's 'annotate' one on github, since his work on annotation storage management made mine a lot

Re: MESSAGE quota resource implemention

2011-08-31 Thread Greg Banks
Julien Coloos wrote: Hi, As discussed on IRC last week, I worked on implementing MESSAGE quota resource in cyrus (see RFC 2087; STORAGE resource already being handled). I created a branch based on Greg's 'annotate' one on github, since his work on annotation storage management made mine a lot

Re: MESSAGE quota resource implemention

2011-08-31 Thread Bron Gondwana
On Wed, Aug 31, 2011 at 05:50:36PM +0200, Julien Coloos wrote: > Things that may be worth noting: > - DUMP/UNDUMP currently does nothing special about MESSAGE or > X-ANNOTATION-STORAGE quota resources > -> should it be transferred ? I'd like to replace DUMP/UNDUMP with replication protocol c

MESSAGE quota resource implemention

2011-08-31 Thread Julien Coloos
Hi, As discussed on IRC last week, I worked on implementing MESSAGE quota resource in cyrus (see RFC 2087; STORAGE resource already being handled). I created a branch based on Greg's 'annotate' one on github, since his work on annotation storage management made mine a lot easier. Details on t