A NOTE has been added to this issue. ====================================================================== http://www.dbmail.org/mantis/view.php?id=752 ====================================================================== Reported By: maximP Assigned To: paul ====================================================================== Project: DBMail Issue ID: 752 Category: General Reproducibility: always Severity: major Priority: normal Status: assigned target: ====================================================================== Date Submitted: 12-Jan-09 11:04 CET Last Modified: 16-Feb-09 12:32 CET ====================================================================== Summary: Dbmail changes header charset Description: Message header values for messages stored in mimeparts table may differ from those of messages injected to dbmail. For instance, Russian headers From: and To: in UTF-8 turn to iso-8859-5. It seems to be gmime feature (trying to select the appropriate charset for UTF-8 data), but it is completely wrong in this case, imho.
====================================================================== ---------------------------------------------------------------------- (0002733) paul (administrator) - 25-Jan-09 13:35 http://www.dbmail.org/mantis/view.php?id=752#c2733 ---------------------------------------------------------------------- Maxim, the mimeparts contains the messageparts as-is. The header caching tables contain strings converted into the charset specified in dbmail.conf. If you still feel there is a bug, please provide steps to reproduce. ---------------------------------------------------------------------- (0002736) maximP (reporter) - 26-Jan-09 09:23 http://www.dbmail.org/mantis/view.php?id=752#c2736 ---------------------------------------------------------------------- Steps to reproduce: 1. Take the attached file and look in it. Will see UTF-8 in From: and To: headers. 2. Run dbmail-deliver -d someuser <test.eml 3. See mimeparts table contents. Will see iso-8859-5 in From: and To: headers. ---------------------------------------------------------------------- (0002753) paul (administrator) - 03-Feb-09 09:21 http://www.dbmail.org/mantis/view.php?id=752#c2753 ---------------------------------------------------------------------- Please provide the relevant setting (encoding) from dbmail.conf. Also please provide a file_logging_levels=511 trace of the dbmail-deliver run. ---------------------------------------------------------------------- (0002755) maximP (reporter) - 03-Feb-09 10:13 http://www.dbmail.org/mantis/view.php?id=752#c2755 ---------------------------------------------------------------------- I use these parameters: encoding = utf8 default_msg_encoding = cp1251 BTW, the system is openSUSE 11.1 with gmime-2.2.23. The log file for 511 level (log.gz) is attached. ---------------------------------------------------------------------- (0002756) paul (administrator) - 03-Feb-09 11:44 http://www.dbmail.org/mantis/view.php?id=752#c2756 ---------------------------------------------------------------------- Please clarify step 3 of steps-to-reproduce. As I said: mimeparts contain unmodified parts of the message. How did you verify the modified encoding? I've inserted and retrieved your message on a pg8.3 installation. Looks good to me. Retrieval was tested with dbmail-export and with Thunderbird on dbmail-imapd. ---------------------------------------------------------------------- (0002757) maximP (reporter) - 03-Feb-09 12:50 http://www.dbmail.org/mantis/view.php?id=752#c2757 ---------------------------------------------------------------------- I tried both dbmail-export and pg_dump for mimeparts table. The resulting files DID contain From: =?iso-8859-5?<sometext> (and To: also) headers. openSUSE compiles gmime with --enable-rfc2047-workarounds. Can it be the source of this problem? ---------------------------------------------------------------------- (0002759) maximP (reporter) - 04-Feb-09 16:00 http://www.dbmail.org/mantis/view.php?id=752#c2759 ---------------------------------------------------------------------- The problem is connected with the following line in _map_headers function in dbmail-message.c: g_mime_header_set_raw (GMIME_MESSAGE(self->content)->mime_part->headers, NULL); As far as I understand, it asks gmime to use parsed headers instead of raw ones, so, after reconstruction the message, the header values can be changed. It can lead to treating the message as illegal during DKIM verification in mail client. Is it possible to avoid this situation? ---------------------------------------------------------------------- (0002761) paul (administrator) - 05-Feb-09 22:26 http://www.dbmail.org/mantis/view.php?id=752#c2761 ---------------------------------------------------------------------- I've reproduced this now. Will investigate and fix it asap. ---------------------------------------------------------------------- (0002762) paul (administrator) - 08-Feb-09 11:30 http://www.dbmail.org/mantis/view.php?id=752#c2762 ---------------------------------------------------------------------- On closes inspection: I am *not* able to reproduce this. I'll take a look at the changes in gmime since 2.2.11 (ubuntu/hardy). ---------------------------------------------------------------------- (0002763) paul (administrator) - 08-Feb-09 13:54 http://www.dbmail.org/mantis/view.php?id=752#c2763 ---------------------------------------------------------------------- Ok, now I've finally reproduced this using gmime-2.2.21 ---------------------------------------------------------------------- (0002764) jasb (reporter) - 08-Feb-09 13:57 http://www.dbmail.org/mantis/view.php?id=752#c2764 ---------------------------------------------------------------------- It may be useful for you to know, I also have this problem with gmime-2.2.22 from debian sid. (dbmail 2.3.5 +/-, git on the http://www.dbmail.org/mantis/view.php?id=755 fix) ---------------------------------------------------------------------- (0002765) paul (administrator) - 08-Feb-09 15:17 http://www.dbmail.org/mantis/view.php?id=752#c2765 ---------------------------------------------------------------------- I've committed a series of fixes for this problem. Dbmail now depends on gmime >= 2.2.22. Please let me know if these problems persist. maxim: your idea was spot on; removing the calls to gmime_header_set_raw and other work-arounds for bugs in earlier versions of gmime did the trick (hopefully). Thanks. ---------------------------------------------------------------------- (0002766) maximP (reporter) - 09-Feb-09 10:11 http://www.dbmail.org/mantis/view.php?id=752#c2766 ---------------------------------------------------------------------- The problem still exists. Now the change is caused by g_mime_message_set_header function. I added two debug lines which print the whole message in dbmail_message_set_header function: void dbmail_message_set_header(DbmailMessage *self, const char *header, const char *value) { TRACE(TRACE_CRIT, g_mime_message_to_string(self->content)); g_mime_message_set_header(GMIME_MESSAGE(self->content), header, value); TRACE(TRACE_CRIT, g_mime_message_to_string(self->content)); if (self->headers) _map_headers(self); } And got two different (in terms of From and To header charset) results in stderr. ---------------------------------------------------------------------- (0002785) maximP (reporter) - 16-Feb-09 12:32 http://www.dbmail.org/mantis/view.php?id=752#c2785 ---------------------------------------------------------------------- Finally, I've found the source of this problem. Being asked to write message headers to the stream, gmime iterates through the header list and calls "writer function" which can be set for every header name. Default writer function just writes the header intact, but for some header names complicated writer function are set (see g_mime_message_init function in gmime-message.c). For instance, one of them, write_addrspec function (which is used for From, To, and similar headers), parses header value, converts it to UTF-8, and then converts it back to the charset gmime thinks is best for this header (see internet-address.c and gmime-utils.c). The philosophical question: should dbmail treat the input message as something precious (and the only permitted action is inserting of additional message headers)? The alternative is to permit modifying the message form (for instance, header charset), but not the message value (meaning). The possible solutions: replace complicated writer functions by something trivial or do not use g_mime_message_set_header, g_mime_header_set_raw functions at all and write additional headers by hand (not using gmime). Issue History Date Modified Username Field Change ====================================================================== 12-Jan-09 11:04 maximP New Issue 25-Jan-09 13:35 paul Note Added: 0002733 25-Jan-09 13:35 paul Status new => feedback 26-Jan-09 09:23 maximP Note Added: 0002736 26-Jan-09 09:23 maximP File Added: test.eml 03-Feb-09 09:21 paul Note Added: 0002753 03-Feb-09 10:12 maximP Note Added: 0002755 03-Feb-09 10:12 maximP File Added: log.gz 03-Feb-09 10:13 maximP Note Edited: 0002755 03-Feb-09 10:37 jasb Issue Monitored: jasb 03-Feb-09 11:44 paul Note Added: 0002756 03-Feb-09 12:45 maximP Note Added: 0002757 03-Feb-09 12:50 maximP Note Edited: 0002757 04-Feb-09 16:00 maximP Note Added: 0002759 05-Feb-09 22:26 paul Note Added: 0002761 05-Feb-09 22:26 paul Assigned To => paul 05-Feb-09 22:26 paul Status feedback => confirmed 08-Feb-09 11:30 paul Note Added: 0002762 08-Feb-09 11:30 paul Status confirmed => acknowledged 08-Feb-09 13:54 paul Note Added: 0002763 08-Feb-09 13:54 paul Status acknowledged => confirmed 08-Feb-09 13:57 jasb Note Added: 0002764 08-Feb-09 15:17 paul Note Added: 0002765 08-Feb-09 15:17 paul Status confirmed => assigned 08-Feb-09 15:17 paul Resolution open => fixed 08-Feb-09 15:17 paul Fixed in Version => 2.3.6 09-Feb-09 10:11 maximP Note Added: 0002766 16-Feb-09 12:32 maximP Note Added: 0002785 ====================================================================== _______________________________________________ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
