Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2006-01-13 Thread Paul J Stevens
Aaron Stone wrote: > On Thu, 2006-01-12 at 15:39 +0100, Paul J Stevens wrote: > >>Simon Gray wrote: >> >> >>>http://www.danga.com/memcached/ >> >>Which is very cool stuff. Definitely a planned thing. >> > > > It is, Sean Chittenden of pgmemcache has been pushing it for over a > year. When we'r

RE: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2006-01-12 Thread Aaron Stone
On Thu, 2006-01-12 at 12:21 +, Simon Gray wrote: > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Aaron Stone > > >> Nope, you can't make that assumption. The reason is that IMAP > >> supports multiple simultaneous writers. If the mailbox changes out > >> from underneath yo

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2006-01-12 Thread Aaron Stone
On Thu, 2006-01-12 at 15:39 +0100, Paul J Stevens wrote: > Simon Gray wrote: > > > http://www.danga.com/memcached/ > > Which is very cool stuff. Definitely a planned thing. > It is, Sean Chittenden of pgmemcache has been pushing it for over a year. When we're ready for multimaster replication,

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2006-01-12 Thread Paul J Stevens
Simon Gray wrote: > http://www.danga.com/memcached/ Which is very cool stuff. Definitely a planned thing. -- Paul Stevens mailto:[EMAIL PROTECTED] NET FACILITIES GROUP PGP

RE: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2006-01-12 Thread Simon Gray
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Aaron Stone >> Nope, you can't make that assumption. The reason is that IMAP >> supports multiple simultaneous writers. If the mailbox changes out >> from underneath you then you need to realize that and recompute >> things. Right

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2006-01-10 Thread Paul J Stevens
Kevin Brown wrote: > By the way, I'm a little underwhelmed with the bug tracking system, > based on what I've seen of it. It should automatically listen to this > mailing list and whenever it sees a reply to a bug it should add the > contents of the reply to the bug. It's annoying to have to de

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2006-01-10 Thread Aaron Stone
On Mon, 2006-01-09 at 15:13 -0800, Kevin Brown wrote: > Paul J Stevens wrote: > > In this particular case: > > - the acl calls shouldn't be triggered every time when the acl for a > > particular mailbox has already been established. In general, the acl > > calls for a mailbox are too expensive alre

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2006-01-10 Thread Kevin Brown
Paul J Stevens wrote: > I've been rather busy keeping my business afloat doing paid-for production > work > lately. No problem at all. > The problem by now is well understood; repeating the same command > sequentially for many messages in the same mailbox needs to be > optimized using server-sid

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2006-01-08 Thread Paul J Stevens
Kevin Brown wrote: > Paul J Stevens wrote: > >> >>Kevin Brown wrote: >> >> From what I can tell, just doing a STORE of attributes is enough. I >>>can recreate the problem and email you the logs if you're interested. >> >>I was more like thinking about some sort of lazy caching where _ic_store

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2006-01-07 Thread Kevin Brown
Paul J Stevens wrote: > > > Kevin Brown wrote: > > >>From what I can tell, just doing a STORE of attributes is enough. I > > can recreate the problem and email you the logs if you're interested. > > I was more like thinking about some sort of lazy caching where _ic_store > does not call db_get

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-16 Thread Kevin Brown
Geo Carncross wrote: > On Thu, 2005-12-15 at 14:16 -0800, Kevin Brown wrote: > > Paul J Stevens wrote: > > > And I suspect 5.0 will be become a requirement for 2.3+ so we can > > > start working with views, triggers, etc... > > > > Hmm...that would imply ditching SQLite, wouldn't it? > > > > Not

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-16 Thread Geo Carncross
On Thu, 2005-12-15 at 14:16 -0800, Kevin Brown wrote: > Paul J Stevens wrote: > > And I suspect 5.0 will be become a requirement for 2.3+ so we can > > start working with views, triggers, etc... > > Hmm...that would imply ditching SQLite, wouldn't it? > > Not that I have a problem with that, mind

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-16 Thread Geo Carncross
On Thu, 2005-12-15 at 14:24 -0800, Kevin Brown wrote: > Geo Carncross wrote: > > I know we talked about this earlier, but I think for 2.2 we need to make > > it possible to have multiple queries- we talked about it per database > > layer, but why not a global conformance level setting? > > > > DB_

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-15 Thread Kevin Brown
I wrote: > I'll create a patch using it instead of what I proposed earlier. And it's attached. Dirt simple, really. Note that the patch is relative to the mailfilters patch that Paul supplies in his 2.0.7+20050927-1 version of the DBMail Debian package. But if I'm not mistaken, that should just

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-15 Thread Kevin Brown
Geo Carncross wrote: > I know we talked about this earlier, but I think for 2.2 we need to make > it possible to have multiple queries- we talked about it per database > layer, but why not a global conformance level setting? > > DB_CAN_VIEW > DB_CAN_TRIGGER > DB_CAN_SUBSELECT > DB_CAN_FOREIGNKEY >

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-15 Thread Kevin Brown
Paul J Stevens wrote: > And I suspect 5.0 will be become a requirement for 2.3+ so we can > start working with views, triggers, etc... Hmm...that would imply ditching SQLite, wouldn't it? Not that I have a problem with that, mind you. -- Kevin Brown [E

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-15 Thread Kevin Brown
Paul J Stevens wrote: > I was more like thinking about some sort of lazy caching where _ic_store > does not call db_getmailbox if the previous imap command was also > _ic_store. > > So yes: please do send me a command sequence I can work with. OK, I'm attaching the log generated by DBMail (trace_

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-15 Thread Kevin Brown
Geo Carncross wrote: > Thanks for clarifying this- I don't do much with MySQL- anything myself > really- because of bad experiences in the 3.3 days... Same for me, actually. I got seriously turned off to MySQL when I hit the "global table lock on insert" performance-killer. Once I started lookin

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-15 Thread Geo Carncross
On Thu, 2005-12-15 at 10:05 -0800, Kevin Brown wrote: > Geo Carncross wrote: > > I mean, dbmail knows what queries it uses, and I don't suggest for a > > moment that dbmail should attempt to find optimal queries for suboptimal > > (read: old) databases, but if dbmail 2.0 works with 4.0 then 2.2 sho

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-15 Thread Kevin Brown
Geo Carncross wrote: > I mean, dbmail knows what queries it uses, and I don't suggest for a > moment that dbmail should attempt to find optimal queries for suboptimal > (read: old) databases, but if dbmail 2.0 works with 4.0 then 2.2 should > as well, otherwise we'll be making upgrades hell for use

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-15 Thread Kevin Brown
Paul J Stevens wrote: > Geo Carncross wrote: > > I wrote this query back in <[EMAIL PROTECTED]> > > > > I also suggested the alternative: > > > > select count(*), count(case when seen_flag > 0 Then 1 else null end), > > count(case when recent_flag > 0 then 1 else null end) FROM > > dbmail_message

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-15 Thread Geo Carncross
On Thu, 2005-12-15 at 09:46 +0100, Paul J Stevens wrote: > Kevin Brown wrote: > > Aaron Stone wrote: > > > >>Something without a subquery is needed, as the 2.0 series must continue > >>to work with MySQL 4.0.x. > > > > > > We can't just force people to upgrade to 4.1.x? It'd be good for > > the

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-15 Thread Paul J Stevens
Geo Carncross wrote: > I wrote this query back in <[EMAIL PROTECTED]> > > I also suggested the alternative: > > select count(*), count(case when seen_flag > 0 Then 1 else null end), > count(case when recent_flag > 0 then 1 else null end) FROM > dbmail_messages WHERE status < '2' and mailbox_idnr=

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-15 Thread Paul J Stevens
Kevin Brown wrote: > Aaron Stone wrote: > >>Something without a subquery is needed, as the 2.0 series must continue >>to work with MySQL 4.0.x. > > > We can't just force people to upgrade to 4.1.x? It'd be good for > them. Forcing them to upgrade to PostgreSQL would be even better. > Hey, I ca

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-15 Thread Aaron Stone
On Wed, 2005-12-14 at 09:46 -0800, Kevin Brown wrote: > Aaron Stone wrote: > > Something without a subquery is needed, as the 2.0 series must continue > > to work with MySQL 4.0.x. > > We can't just force people to upgrade to 4.1.x? It'd be good for > them. Forcing them to upgrade to PostgreSQL

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-14 Thread Geo Carncross
I wrote this query back in <[EMAIL PROTECTED]> I also suggested the alternative: select count(*), count(case when seen_flag > 0 Then 1 else null end), count(case when recent_flag > 0 then 1 else null end) FROM dbmail_messages WHERE status < '2' and mailbox_idnr='1'; in <[EMAIL PROTECTED]> but I

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-14 Thread Abdullah Ibn Hamad Al-Marri
Kevin Brown <[EMAIL PROTECTED]> wrote: Aaron Stone wrote: > Something without a subquery is needed, as the 2.0 series must continue > to work with MySQL 4.0.x. We can't just force people to upgrade to 4.1.x? It'd be good for them. Forcing them to upgrade to PostgreSQL would be even better. Hey, I

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-14 Thread Kevin Brown
Aaron Stone wrote: > Something without a subquery is needed, as the 2.0 series must continue > to work with MySQL 4.0.x. We can't just force people to upgrade to 4.1.x? It'd be good for them. Forcing them to upgrade to PostgreSQL would be even better. Hey, I can dream, can't I? :-) Anyway, th

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-14 Thread Aaron Stone
On Tue, 2005-12-13 at 23:35 -0800, Kevin Brown wrote: > Paul J Stevens wrote: > > Just to be clear; If you do have an elegant, readable and > > maintainable solution I'm all for it. [snip] > The above works in MySQL versions 4.1 and greater (might work with > earlier versions, but I don't know and

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-14 Thread Paul J Stevens
Kevin Brown wrote: >>From what I can tell, just doing a STORE of attributes is enough. I > can recreate the problem and email you the logs if you're interested. I was more like thinking about some sort of lazy caching where _ic_store does not call db_getmailbox if the previous imap command was

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-14 Thread Paul J Stevens
Kevin Brown wrote: > I'll create a patch against 2.0.7 for this if you guys are interested. Please do. It won't work for mysql-4.0 because of the sub-select. But I can include it in the debian packages as a separate patch, and integrate in into trunk. -- _

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-14 Thread Kevin Brown
Paul J Stevens wrote: > Just to be clear; If you do have an elegant, readable and > maintainable solution I'm all for it. Hmm...well, the following query might not qualify but it certainly does the trick and removes the need for an internal array. We still have to do a sum of all the results but

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-14 Thread Kevin Brown
I wrote: > The latest problem I've had with 2.1.x-trunk is related to > dbmail-smtp. I use the mailbox2dbmail contrib script to populate > dbmail from my primary mail store and in the 2.1.x series the mail > server returns nothing when you attempt to retrieve a message. The > headers come back fi

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-14 Thread Kevin Brown
Paul J Stevens wrote: > > Paul seems to be more familiar with this area of code, so I'd prefer to > > leave it to him to integrate your work on this. > > I my view the problem is not that the current query is slow. The > core problem is that it's called too often. Yeah, that too. :-) I figured i

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-13 Thread Paul J Stevens
Aaron Stone wrote: > On Tue, 2005-12-13 at 13:09 -0800, Kevin Brown wrote: > >>I wrote: >> >>>In experimenting with DBMail 2.0.7 (even with the changes I submitted >>>regarding the headername query, I can't get 2.1-trunk to run well >>>enough to work properly with both Mutt and Thunderbird), I dis

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-13 Thread Aaron Stone
On Tue, 2005-12-13 at 13:09 -0800, Kevin Brown wrote: > I wrote: > > > > In experimenting with DBMail 2.0.7 (even with the changes I submitted > > regarding the headername query, I can't get 2.1-trunk to run well > > enough to work properly with both Mutt and Thunderbird), I discovered > > (as per

Re: [Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-13 Thread Kevin Brown
I wrote: > > In experimenting with DBMail 2.0.7 (even with the changes I submitted > regarding the headername query, I can't get 2.1-trunk to run well > enough to work properly with both Mutt and Thunderbird), I discovered > (as perhaps some of you have) that storing changes to the attributes > is

[Dbmail-dev] STORE is SLOOOOOW on folder with many, many messages

2005-12-12 Thread Kevin Brown
In experimenting with DBMail 2.0.7 (even with the changes I submitted regarding the headername query, I can't get 2.1-trunk to run well enough to work properly with both Mutt and Thunderbird), I discovered (as perhaps some of you have) that storing changes to the attributes is very slow when the f