Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Thomas Mueller
Geo Carncross wrote: > On Wed, 2005-03-09 at 23:06 +0100, Thomas Mueller wrote: >> Geo Carncross wrote: >>> On Wed, 2005-03-09 at 19:10 +0100, Thomas Mueller wrote: >>> No index will ever solve the problem for PostgreSQL. COUNT() can't use any index, because indices know nothing about >>

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Thomas Mueller
Hi Paul, > Fix implemented and imap testsuite validates just fine. Packages are > uploading now. Great thanks! > Please note that the naming of the 2.0 packages has changed: they are > now called dbmail2-pgsql. This is required because I'm aiming for > inclusion of dbmail2 in debian starting at

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Geo Carncross
On Wed, 2005-03-09 at 23:06 +0100, Thomas Mueller wrote: > Geo Carncross wrote: > > On Wed, 2005-03-09 at 19:10 +0100, Thomas Mueller wrote: > > > >>No index will ever solve the problem for PostgreSQL. COUNT() can't use > >>any index, because indices know nothing about transactions. > >>That is wh

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Thomas Mueller
Geo Carncross wrote: > On Wed, 2005-03-09 at 19:10 +0100, Thomas Mueller wrote: > >>No index will ever solve the problem for PostgreSQL. COUNT() can't use >>any index, because indices know nothing about transactions. >>That is why every record that is counted using the index has to be >>checked to

Re: [Dbmail-dev] where is development (CVS/SVN/etc)

2005-03-09 Thread Paul J Stevens
Geo Carncross wrote: http://dbmail.org/index.php?page=download says it's in CVS; I haven't seen a CVS update in a week. I see mention of things on the list before I see then in SVN. I know I've switched to SVN for both 2.0 and 2.1 (trunk). That page is outdated. For the record (and possibly

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Paul J Stevens
Thomas, Fix implemented and imap testsuite validates just fine. Packages are uploading now. Please note that the naming of the 2.0 packages has changed: they are now called dbmail2-pgsql. This is required because I'm aiming for inclusion of dbmail2 in debian starting at 2.0.4, but will mainta

[Dbmail-dev] where is development (CVS/SVN/etc)

2005-03-09 Thread Geo Carncross
http://dbmail.org/index.php?page=download says it's in CVS; I haven't seen a CVS update in a week. I see mention of things on the list before I see then in SVN. For the record (and possibly so the above page will get updated) WHERE exactly is dbmail development actually occurring now? -- Intern

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Thomas Mueller
Paul J Stevens wrote: > Geo Carncross wrote: > >> 'q',COUNT(*) >> >> as the request for each. This was a minor note. The result would then be >> three separate rows, second column being the interesting part. > > Fixing the query thus, and adding the needed indexes result in *really* > fast return

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Geo Carncross
On Wed, 2005-03-09 at 19:10 +0100, Thomas Mueller wrote: > No index will ever solve the problem for PostgreSQL. COUNT() can't use > any index, because indices know nothing about transactions. > That is why every record that is counted using the index has to be > checked to see if it's involved in a

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Paul J Stevens
Geo Carncross wrote: 'q',COUNT(*) as the request for each. This was a minor note. The result would then be three separate rows, second column being the interesting part. Fixing the query thus, and adding the needed indexes result in *really* fast returns on mysql, comparable with the SUM() v

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Thomas Mueller
Paul J Stevens wrote: > Thomas Mueller wrote: > >> The main problem is in db.c, db_getmailbox(): >> SELECT COUNT(message_idnr), COUNT(message_idnr) - SUM(seen_flag), >> SUM(recent_flag) FROM dbmail_messages WHERE mailbox_idnr = '5' AND >> status < '2' >> That query takes up to 15 seconds (!!). >

Re: [Dbmail-dev] Re: dbmail-pgsql horrible slow

2005-03-09 Thread Geo Carncross
On Wed, 2005-03-09 at 16:54 +0100, Tom Ivar Helbekkmo wrote: > Geo Carncross <[EMAIL PROTECTED]> writes: > > > A better query would be: > > > > SELECT COUNT(*) FROM dbmail_messages WHERE mailbox_idnr='5' AND > > (status='0' OR status='1') UNION > > SELECT COUNT(*) FROM dbmail_messages WHERE mailb

[Dbmail-dev] Re: dbmail-pgsql horrible slow

2005-03-09 Thread Tom Ivar Helbekkmo
Geo Carncross <[EMAIL PROTECTED]> writes: > An aggregate index wouldn't be meaningless with GROUP BY- a better > way to look at it [syntax] would be indexes on views. Indexes on materialized views could make sense, of course, but something like 'create index i on t (count(c))' does not. -tih --

Re: [Dbmail-dev] Re: dbmail-pgsql horrible slow

2005-03-09 Thread Geo Carncross
On Wed, 2005-03-09 at 16:43 +0100, Tom Ivar Helbekkmo wrote: > Paul J Stevens <[EMAIL PROTECTED]> writes: > > > ERROR: cannot use aggregate function in index expression > > > > A recent discussion shows fixing this is on the TODO list for postgresql. > > That sounded as if they're planning to al

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Geo Carncross
On Wed, 2005-03-09 at 15:42 +, Aaron Stone wrote: > Were those queries intended to produce the same results? NB: I didn't add > the indexes you suggest. You really should. No, I used UNION so that separate query plans would be generated. To make sure results are always given one should probab

[Dbmail-dev] Re: dbmail-pgsql horrible slow

2005-03-09 Thread Tom Ivar Helbekkmo
Geo Carncross <[EMAIL PROTECTED]> writes: > A better query would be: > > SELECT COUNT(*) FROM dbmail_messages WHERE mailbox_idnr='5' AND > (status='0' OR status='1') UNION > SELECT COUNT(*) FROM dbmail_messages WHERE mailbox_idnr='5' AND > (status='0' OR status='1') AND seen_flag=1 UNION > SELECT

[Dbmail-dev] Re: dbmail-pgsql horrible slow

2005-03-09 Thread Tom Ivar Helbekkmo
Paul J Stevens <[EMAIL PROTECTED]> writes: > ERROR: cannot use aggregate function in index expression > > A recent discussion shows fixing this is on the TODO list for postgresql. That sounded as if they're planning to allow indexing on aggregate functions, which is, of course, meaningless. Wha

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Aaron Stone
Were those queries intended to produce the same results? NB: I didn't add the indexes you suggest. mysql> SELECT COUNT(*) FROM dbmail_messages WHERE mailbox_idnr='5' AND -> (status='0' OR status='1') UNION -> SELECT COUNT(*) FROM dbmail_messages WHERE mailbox_idnr='5' AND -> (status='0

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Geo Carncross
On Wed, 2005-03-09 at 11:51 +0100, Thomas Mueller wrote: > Hi, > > I get more and more user complaints that my IMAP server is terrible > slow, opening a mailbox takes very long. > > I've analyzed that using PQA (http://pgfoundry.org/projects/pqa/, great > tool!), the output is attached if anyone

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Hans Kristian Rosbach
On Wed, 2005-03-09 at 13:03, Paul J Stevens wrote: > Hans Kristian Rosbach wrote: > > create index idx_test on dbmail_messages (mailbox_idnr, status, > > count(message_idnr)); > > This won't work, of course. > > sh-3.00$ psql dbmail > Welcome to psql 7.4.6, the PostgreSQL interactive terminal. >

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Paul J Stevens
Thomas Mueller wrote: Hi, I get more and more user complaints that my IMAP server is terrible slow, opening a mailbox takes very long. I've analyzed that using PQA (http://pgfoundry.org/projects/pqa/, great tool!), the output is attached if anyone is interested (includes all databases, not on

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Paul J Stevens
Hans Kristian Rosbach wrote: create index idx_test on dbmail_messages (mailbox_idnr, status, count(message_idnr)); This won't work, of course. sh-3.00$ psql dbmail Welcome to psql 7.4.6, the PostgreSQL interactive terminal. Type: \copyright for distribution terms \h for help with SQ

Re: [Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Hans Kristian Rosbach
On Wed, 2005-03-09 at 11:51, Thomas Mueller wrote: > Hi, > > I get more and more user complaints that my IMAP server is terrible > slow, opening a mailbox takes very long. > > I've analyzed that using PQA (http://pgfoundry.org/projects/pqa/, great > tool!), the output is attached if anyone is int

[Dbmail-dev] to sieve or not to sieve, and where is regexsort

2005-03-09 Thread Paul J Stevens
Aaron, I've been working on further reorganising the delivery chain. I'm currently assimilating the sort_and_deliver function into the dbmail-message framework. There seems to be a lot of dead wood in that code. I'm therefore sorely tempted to clean out the sieve code from svn-trunk. It's no

[Dbmail-dev] dbmail-pgsql horrible slow

2005-03-09 Thread Thomas Mueller
Hi, I get more and more user complaints that my IMAP server is terrible slow, opening a mailbox takes very long. I've analyzed that using PQA (http://pgfoundry.org/projects/pqa/, great tool!), the output is attached if anyone is interested (includes all databases, not only dbmail). The main prob