A NOTE has been added to this issue.
======================================================================
http://dbmail.org/mantis/view.php?id=924
======================================================================
Reported By: gerritcap
Assigned To:
======================================================================
Project: DBMail
Issue ID: 924
Category: Command-Line programs (dbmail-users, dbmail-util)
Reproducibility: always
Severity: minor
Priority: normal
Status: new
target:
======================================================================
Date Submitted: 04-Sep-11 22:53 CEST
Last Modified: 23-Dec-11 10:23 CET
======================================================================
Summary: dbmail-util -by doesn't fix un-cached physmessages
Description:
Every time I run dbmail-util -by it informs me that there are 3 un-cached
physmessages. It seems to fix it (3 dots are displayed) and finally reports
that the errors were fixed and suggest to rerun dbmail-util to validate if
they really are.
Unfortunately repeatedly I relaunched dbmail-util but it continues to give
the same information.
Note: I had this problem with postgresql 8.4.1/dbmail-2-2.11 and recently
upgraded to postgresql 9.0.4/dbmail-2.2.17. Both combinations had the same
problem.
I then tried to upgrade to dbmail-3.0.0 rc3:
1) upgraded the dbmail software
2) upgraded the database with the upgrade script to upgrade 2_2 to
3_0
3) tried twice the dbmail-util -by script: it twice reported all of my
15.000+ emails, it also showed a lot of dots while correcting (or trying to
correct)
4) ran dbmail-util -My twice to update all messages to new database
structure
5) ran dbmail-util -by twice again, and again twice the same 15.000+
records were reported and were part of a fix
Note that after upgrading to 3.0.0 rc3, using thunderbird to connect to
the dbmails imap server, every message had empty header records: no
subject, no sender info was shown, no date etc... and this after step 2, 3,
4 & 5 detailed above. Each test was performed after emptying the
thunderbird cache in the thunderbird profile directory on disk
======================================================================
----------------------------------------------------------------------
(0003371) mverbert (reporter) - 21-Dec-11 18:58
http://dbmail.org/mantis/view.php?id=924#c3371
----------------------------------------------------------------------
I tried the latest tree (3ef7b075f722387d917836eb93fd09745aa53fa1), and the
same problem can be reproduced on different environment:
- debian stable (6.0.3)
- postgresql server 9.1.1 (package version: 9.1.1-1~bpo60+1)
I used the following sequence:
- stop dbmail 2.2.17 servers (imap and sieve)
- run dbmail -a, nothing to fix so no action taken
- run the update script 2_2-3_0.pgsql as the same user used by dbmail
- install dbmail 3.0.0rc3 (in reality it matches the commit mentioned
above)
- run dbmail-util -My multiple times (I didn't use -m to do the > 40000
messages at once)
- run dbmail -b, it complains that physmessages are un-cached, so ran
"dbmail -by". Ran again "dbmail -b", same complain.
I repeated the last step 3 times, no real progress is ever made.
For completeness, I re-ran the full procedure (with, obviously, a database
restore in between), and it is 100% reproducible.
Is there any step in the upgrade procedure that I missed ?
for me, the issue is a show-stopper to go for dbmail 3 as no previous mail
is fully accessible...
----------------------------------------------------------------------
(0003372) paul (administrator) - 22-Dec-11 09:27
http://dbmail.org/mantis/view.php?id=924#c3372
----------------------------------------------------------------------
Without logs there's precious little I can do.
dbmail-util -yM will migrate all physmessage to 3.0 style storage
dbmail-util -ym will migrate 10000 physmessages
dbmail-util -ym 100 will migrate 100 physmessages
note however, that migrating physmessages is *not* required.
in all cases
dbmail-util -by
is required since the migration script drops and creates the headercache
tables that need to be re-filled.
So please include logs that show SQL (level 511)
----------------------------------------------------------------------
(0003373) mverbert (reporter) - 22-Dec-11 22:07
http://dbmail.org/mantis/view.php?id=924#c3373
----------------------------------------------------------------------
OK, I ran the experiment once more with the correct log level (511).
The log files are the following (in order of generation):
1) migration.log: output of "dbmail -yM"
2) migration_rest.log: output of "dbmail -yM -m 50000" ; most probably not
very interesting. At the end of the process "dbmail -M" shows no more
messages to migrate.
3) uncache.log: output of "dbmail -by"
4) after_uncache.log: output of "dbmail -b" showing that no progress was
made
I restore the db and tried just "dbmail -by" generating uncache2.log.
Needless to say that it didn't work, but that was just for the sake of
experimenting without migrating the physmessages.
The log files are well above the 150k limit of Mantis, so I put them
there:
http://moutarde.dyndns.org/dbmail/
Let me know if you need more information.
Thanks for your support and Merry Christmas !
----------------------------------------------------------------------
(0003374) paul (administrator) - 22-Dec-11 23:07
http://dbmail.org/mantis/view.php?id=924#c3374
----------------------------------------------------------------------
The uncache.log definitely doesn't look right. But I have never tested the
order in which you do stuff.
Normal steps to follow would be:
apply schema migration (required)
rebuild header cache (required)
migrate storage (optional)
So leave out the last one, and just run the first. Everything must work
then apart from imap-search on message bodies.
When it does, only then try the last step.
I'll try to reproduce your steps and see if fixing it isn't too difficult.
----------------------------------------------------------------------
(0003375) mverbert (reporter) - 23-Dec-11 07:34
http://dbmail.org/mantis/view.php?id=924#c3375
----------------------------------------------------------------------
It seems the last part of my previous message was not completely clear:
uncache2.log represents the "rebuild header cache" of the "normal steps",
given the database was fully restored from a backup (ie. including drop of
database) just before.
Thanks for your help.
----------------------------------------------------------------------
(0003376) paul (administrator) - 23-Dec-11 10:23
http://dbmail.org/mantis/view.php?id=924#c3376
----------------------------------------------------------------------
Did some tests. Had an epiphany. You guys are running pg9. You *must* set
in postgresql.conf:
bytea_output = 'escape'
or else dbmail won't be able to interpret the query results involving
bytea fields.
Issue History
Date Modified Username Field Change
======================================================================
04-Sep-11 22:53 gerritcap New Issue
21-Dec-11 18:58 mverbert Note Added: 0003371
22-Dec-11 09:27 paul Note Added: 0003372
22-Dec-11 22:07 mverbert Note Added: 0003373
22-Dec-11 23:07 paul Note Added: 0003374
23-Dec-11 07:34 mverbert Note Added: 0003375
23-Dec-11 10:23 paul Note Added: 0003376
======================================================================
_______________________________________________
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev
