Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-10 Thread Paul J Stevens
Aaron Stone wrote: On Wed, Mar 9, 2005, Paul J Stevens <[EMAIL PROTECTED]> said: I've also committed this to the 2.0 svn branch. About time we cut a 2.0.4rc1 I suppose. Aaron? We can defer bug #161 to 2.0.5 or even 2.1, but any guesstimates on #164, perhaps? Ok, thanks for reminding me.

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-10 Thread Aaron Stone
On Thu, Mar 10, 2005, Paul J Stevens <[EMAIL PROTECTED]> said: >> Ok, thanks for reminding me. Let's kick #161 because if we haven't started >> looking into what crashes the mime parser, now ain't the time. > > Well, the crash was triggered by malformed messages with crafted headers > (read: emb

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-10 Thread Paul J Stevens
Aaron Stone wrote: On Wed, Mar 9, 2005, Paul J Stevens <[EMAIL PROTECTED]> said: I've also committed this to the 2.0 svn branch. About time we cut a 2.0.4rc1 I suppose. Aaron? We can defer bug #161 to 2.0.5 or even 2.1, but any guesstimates on #164, perhaps? Ok, thanks for reminding me.

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-10 Thread Aaron Stone
On Wed, Mar 9, 2005, Paul J Stevens <[EMAIL PROTECTED]> said: > I've also committed this to the 2.0 svn branch. About time we cut a > 2.0.4rc1 I suppose. Aaron? We can defer bug #161 to 2.0.5 or even 2.1, > but any guesstimates on #164, perhaps? Ok, thanks for reminding me. Let's kick #161 becau

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-10 Thread Geo Carncross
On Wed, 2005-03-09 at 23:53 +0100, Thomas Mueller wrote: > Geo Carncross wrote: > > On Wed, 2005-03-09 at 23:06 +0100, Thomas Mueller wrote: > >> Geo Carncross wrote: > >>> On Wed, 2005-03-09 at 19:10 +0100, Thomas Mueller wrote: > >>> > No index will ever solve the problem for PostgreSQL. CO

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Thomas Mueller
Geo Carncross wrote: > On Wed, 2005-03-09 at 23:06 +0100, Thomas Mueller wrote: >> Geo Carncross wrote: >>> On Wed, 2005-03-09 at 19:10 +0100, Thomas Mueller wrote: >>> No index will ever solve the problem for PostgreSQL. COUNT() can't use any index, because indices know nothing about >>

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Thomas Mueller
Hi Paul, > Fix implemented and imap testsuite validates just fine. Packages are > uploading now. Great thanks! > Please note that the naming of the 2.0 packages has changed: they are > now called dbmail2-pgsql. This is required because I'm aiming for > inclusion of dbmail2 in debian starting at

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Geo Carncross
On Wed, 2005-03-09 at 23:06 +0100, Thomas Mueller wrote: > Geo Carncross wrote: > > On Wed, 2005-03-09 at 19:10 +0100, Thomas Mueller wrote: > > > >>No index will ever solve the problem for PostgreSQL. COUNT() can't use > >>any index, because indices know nothing about transactions. > >>That is wh

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Thomas Mueller
Geo Carncross wrote: > On Wed, 2005-03-09 at 19:10 +0100, Thomas Mueller wrote: > >>No index will ever solve the problem for PostgreSQL. COUNT() can't use >>any index, because indices know nothing about transactions. >>That is why every record that is counted using the index has to be >>checked to

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Paul J Stevens
Thomas, Fix implemented and imap testsuite validates just fine. Packages are uploading now. Please note that the naming of the 2.0 packages has changed: they are now called dbmail2-pgsql. This is required because I'm aiming for inclusion of dbmail2 in debian starting at 2.0.4, but will mainta

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Thomas Mueller
Paul J Stevens wrote: > Geo Carncross wrote: > >> 'q',COUNT(*) >> >> as the request for each. This was a minor note. The result would then be >> three separate rows, second column being the interesting part. > > Fixing the query thus, and adding the needed indexes result in *really* > fast return

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Geo Carncross
On Wed, 2005-03-09 at 19:10 +0100, Thomas Mueller wrote: > No index will ever solve the problem for PostgreSQL. COUNT() can't use > any index, because indices know nothing about transactions. > That is why every record that is counted using the index has to be > checked to see if it's involved in a

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Paul J Stevens
Geo Carncross wrote: 'q',COUNT(*) as the request for each. This was a minor note. The result would then be three separate rows, second column being the interesting part. Fixing the query thus, and adding the needed indexes result in *really* fast returns on mysql, comparable with the SUM() v

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Thomas Mueller
Paul J Stevens wrote: > Thomas Mueller wrote: > >> The main problem is in db.c, db_getmailbox(): >> SELECT COUNT(message_idnr), COUNT(message_idnr) - SUM(seen_flag), >> SUM(recent_flag) FROM dbmail_messages WHERE mailbox_idnr = '5' AND >> status < '2' >> That query takes up to 15 seconds (!!). >

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Geo Carncross
On Wed, 2005-03-09 at 15:42 +, Aaron Stone wrote: > Were those queries intended to produce the same results? NB: I didn't add > the indexes you suggest. You really should. No, I used UNION so that separate query plans would be generated. To make sure results are always given one should probab

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Aaron Stone
Were those queries intended to produce the same results? NB: I didn't add the indexes you suggest. mysql> SELECT COUNT(*) FROM dbmail_messages WHERE mailbox_idnr='5' AND -> (status='0' OR status='1') UNION -> SELECT COUNT(*) FROM dbmail_messages WHERE mailbox_idnr='5' AND -> (status='0

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Geo Carncross
On Wed, 2005-03-09 at 11:51 +0100, Thomas Mueller wrote: > Hi, > > I get more and more user complaints that my IMAP server is terrible > slow, opening a mailbox takes very long. > > I've analyzed that using PQA (http://pgfoundry.org/projects/pqa/, great > tool!), the output is attached if anyone

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Hans Kristian Rosbach
On Wed, 2005-03-09 at 13:03, Paul J Stevens wrote: > Hans Kristian Rosbach wrote: > > create index idx_test on dbmail_messages (mailbox_idnr, status, > > count(message_idnr)); > > This won't work, of course. > > sh-3.00$ psql dbmail > Welcome to psql 7.4.6, the PostgreSQL interactive terminal. >

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Paul J Stevens
Thomas Mueller wrote: Hi, I get more and more user complaints that my IMAP server is terrible slow, opening a mailbox takes very long. I've analyzed that using PQA (http://pgfoundry.org/projects/pqa/, great tool!), the output is attached if anyone is interested (includes all databases, not on

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Paul J Stevens
Hans Kristian Rosbach wrote: create index idx_test on dbmail_messages (mailbox_idnr, status, count(message_idnr)); This won't work, of course. sh-3.00$ psql dbmail Welcome to psql 7.4.6, the PostgreSQL interactive terminal. Type: \copyright for distribution terms \h for help with SQ

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Hans Kristian Rosbach
On Wed, 2005-03-09 at 11:51, Thomas Mueller wrote: > Hi, > > I get more and more user complaints that my IMAP server is terrible > slow, opening a mailbox takes very long. > > I've analyzed that using PQA (http://pgfoundry.org/projects/pqa/, great > tool!), the output is attached if anyone is int

[Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Thomas Mueller
Hi, I get more and more user complaints that my IMAP server is terrible slow, opening a mailbox takes very long. I've analyzed that using PQA (http://pgfoundry.org/projects/pqa/, great tool!), the output is attached if anyone is interested (includes all databases, not only dbmail). The main prob