A NOTE has been added to this issue. ====================================================================== http://www.dbmail.org/mantis/view.php?id=1020 ====================================================================== Reported By: igorbelykh86 Assigned To: ====================================================================== Project: DBMail Issue ID: 1020 Category: IMAP daemon Reproducibility: always Severity: minor Priority: normal Status: new target: ====================================================================== Date Submitted: 17-Aug-13 09:37 CEST Last Modified: 23-Sep-13 18:12 CEST ====================================================================== Summary: dbmail 3.1.2. Imap server returns wrong decoded header fields. Description: Hi
After upgrading the dbmail to 3.1.2 the IMAP server has began to return wrong decoded email headers. I've thought that it does it only for old emails which were been received before upgrading, but no. New mails have the problem too. This problem appears only when application try to get a header field. Example: CLIENT: UID FETCH 1017 (BODY.PEEK[HEADER.FIELDS (SUBJECT FROM CC TO)] INTERNALDATE BODY FLAGS) SERVER: * 127 FETCH (INTERNALDATE "16-Aug-2013 21:27:40 +0000" FLAGS (\Seen) UID 1017 BODY (("text" "plain" ("charset" "UTF-8") NIL NIL "base64" 60 1)("text" "html" ("charset" "UTF-8") NIL NIL "base64" 90 2) "alternative") BODY[HEADER.FIELDS (SUBJECT FROM CC TO)] {98} From: Igor Belykh <myem...@address.com> Subject: ????? ?????? To: Igor <anotherem...@address.com> ) A003 OK UID FETCH completed The subject is in russian. It is encoded correctly. If get full header of the message the server returns: .... Subject: =?UTF-8?B?0J3QvtCy0L7QtSDQv9C40YHRjNC80L4=?= .... what I can do to always get encoded header fields? Thanks. ====================================================================== ---------------------------------------------------------------------- (0003559) paul (administrator) - 17-Aug-13 10:33 http://www.dbmail.org/mantis/view.php?id=1020#c3559 ---------------------------------------------------------------------- Question is: what is in the database? Was it during insertion or during retrieval that something went wrong. The message_idnr is 1017 in your case. Try following: select m.message_idnr,n.headername,v.headervalue from dbmail_header h join dbmail_headername n on h.headername_id=n.id join dbmail_headervalue v on h.headervalue_id=v.id join dbmail_physmessage p on p.id=h.physmessage_id join dbmail_messages m on m.physmessage_id=p.id where m.message_idnr=1017; This should give you a dump of the cached headers. If your database is UTF-8 encoded, and your terminal is as well, it should give you nicely readable values. Of course, the order of the headers returned is also wrong, but that was fixed already and will be ok in 3.1.3. ---------------------------------------------------------------------- (0003560) igorbelykh86 (reporter) - 17-Aug-13 13:15 http://www.dbmail.org/mantis/view.php?id=1020#c3560 ---------------------------------------------------------------------- Hi Paul, All data is readable in the database. All tables is utf-8 encoded. And system (and terminal too) default encoding is en_US.UTF-8. Any idea? ---------------------------------------------------------------------- (0003561) igorbelykh86 (reporter) - 19-Aug-13 12:02 http://www.dbmail.org/mantis/view.php?id=1020#c3561 ---------------------------------------------------------------------- Hi Paul, I think that the problem is in receiving data from database? I've checked the mysql server and mysql client settings. Server and client default character set is utf8 too: [client] ...... default-character-set=utf8 ...... [mysql] ...... default-character-set=utf8 ...... [mysqld] ...... collation-server = utf8_unicode_ci init-connect='SET NAMES utf8' character-set-server = utf8 ...... What do you think what's the problem? Maybe some thing in dbmail settings or mysql or something else? ---------------------------------------------------------------------- (0003562) jasb (reporter) - 19-Aug-13 12:04 http://www.dbmail.org/mantis/view.php?id=1020#c3562 ---------------------------------------------------------------------- The [client] section of my.ini/cnf is for the mysql console client.. not 3rd party software. ---------------------------------------------------------------------- (0003563) igorbelykh86 (reporter) - 19-Aug-13 12:24 http://www.dbmail.org/mantis/view.php?id=1020#c3563 ---------------------------------------------------------------------- just tried to comment line "encoding = utf8". Now it works fine. CLIENT REQUEST: UID FETCH 1017 (BODY.PEEK[HEADER.FIELDS (SUBJECT FROM CC TO)] INTERNALDATE BODY FLAGS) SERVER RESPONSE: * 127 FETCH (INTERNALDATE "16-Aug-2013 21:27:40 +0000" FLAGS (\Seen) UID 1017 BODY (("text" "plain" ("charset" "UTF-8") NIL NIL "base64" 60 1)("text" "html" ("charset" "UTF-8") NIL NIL "base64" 90 2) "alternative") BODY[HEADER.FIELDS (SUBJECT FROM CC TO)] {119} From: Igor Belykh <igorbelyk...@gmail.com> Subject: =?iso-8859-5?b?vd7S3tUg39jh7Nze?= To: Igor <i...@zubybox.com> ) A003 OK UID FETCH completed How I understand these feader fields are received from source code of the message. ---------------------------------------------------------------------- (0003564) igorbelykh86 (reporter) - 19-Aug-13 12:29 http://www.dbmail.org/mantis/view.php?id=1020#c3564 ---------------------------------------------------------------------- no, in the source code the subject header field is different: Subject: =?UTF-8?B?0J3QvtCy0L7QtSDQv9C40YHRjNC80L4=?= So the imap server works fine if the line "encoding = utf8" is commented. ---------------------------------------------------------------------- (0003565) paul (administrator) - 19-Aug-13 12:54 http://www.dbmail.org/mantis/view.php?id=1020#c3565 ---------------------------------------------------------------------- the value of the 'encoding' parameter must match the collation of the database. Apparently your database is not in UTF8. try: mysql dbmail mysql> show variables like 'collation_%'; +----------------------+-------------------+ | Variable_name | Value | +----------------------+-------------------+ | collation_connection | utf8_general_ci | | collation_database | koi8r_general_ci | | collation_server | koi8r_general_ci | +----------------------+-------------------+ 3 rows in set (0.00 sec) if the collation of the database and connection are *not* the same, you need to set the encoding of the connection to match the collation of the database: mysql> SET NAMES koi8r; mysql> show variables like 'collation_%'; +----------------------+-------------------+ | Variable_name | Value | +----------------------+-------------------+ | collation_connection | koi8r_general_ci | | collation_database | koi8r_general_ci | | collation_server | koi8r_general_ci | +----------------------+-------------------+ 3 rows in set (0.00 sec) this means in this example the 'encoding' parameter should be 'koi8r'; Without the encoding paramater, it falls back to your system's default locale, I syspect. ---------------------------------------------------------------------- (0003566) igorbelykh86 (reporter) - 19-Aug-13 13:07 http://www.dbmail.org/mantis/view.php?id=1020#c3566 ---------------------------------------------------------------------- This is result of the query: mysql> show variables like 'collation_%'; +----------------------+-----------------+ | Variable_name | Value | +----------------------+-----------------+ | collation_connection | utf8_general_ci | | collation_database | utf8_general_ci | | collation_server | utf8_general_ci | +----------------------+-----------------+ 3 rows in set (0.00 sec) Thus the encoding parameter should be 'utf8'. Right? ---------------------------------------------------------------------- (0003580) igorbelykh86 (reporter) - 21-Sep-13 19:44 http://www.dbmail.org/mantis/view.php?id=1020#c3580 ---------------------------------------------------------------------- Hi Paul, I've found one issue with encoding in data base. There is `sortfield` cell in the `dbmail_headervalue` table. This cell has utf8_general_ci collation but in rows I see following: Привет из РоÑÑии (Hello from Russia) instead Привет из России (Hello from Russia) But the `headervalue` cell contains the right value in right encoding Regards, Igor ---------------------------------------------------------------------- (0003581) Lionel (reporter) - 22-Sep-13 18:07 http://www.dbmail.org/mantis/view.php?id=1020#c3581 ---------------------------------------------------------------------- Hi Paul, The problem is not only with russian characters but with any language that needs UTF-8 support. I've done tests with russian, hebrew, arabic and greek characters, they all have the same issue. I suspect the problem to be in the place where the sortfield is being created/processed when there is a new email because the sortfield is not inserted in the database correctly. The headervalue is inserted correctly, but not the sortfield. The way the sortfield appears in the database looks like a typical utf-8 meta tag missing in html. However, I do not know where in the src files this is handled. Your input on this in order to fix it is welcomed as I think this is a major issue for all dbmail users trying to run dbmail 3.x as with this issue dbmail is totally useless for anyone receiving or sending emails with characters other than english/dutch. Thanks. ---------------------------------------------------------------------- (0003582) paul (administrator) - 23-Sep-13 16:44 http://www.dbmail.org/mantis/view.php?id=1020#c3582 ---------------------------------------------------------------------- I wonder how does this affect the issue mentioned here, since the sortfield only affects 'sorting'? ---------------------------------------------------------------------- (0003583) paul (administrator) - 23-Sep-13 16:48 http://www.dbmail.org/mantis/view.php?id=1020#c3583 ---------------------------------------------------------------------- can you please provide a simple test message to verify? ---------------------------------------------------------------------- (0003584) Lionel (reporter) - 23-Sep-13 16:49 http://www.dbmail.org/mantis/view.php?id=1020#c3584 ---------------------------------------------------------------------- yes you are right. We (me & Igor) figured this out yesterday while looking at the source files. However, we did not find a fix for the issue regarding the headers and the subject that is totally unreadable (???? ???? ??????). Any idea what might be the issue? or where we should be looking at to try to figure it out? Thanks. ---------------------------------------------------------------------- (0003585) Lionel (reporter) - 23-Sep-13 16:52 http://www.dbmail.org/mantis/view.php?id=1020#c3585 ---------------------------------------------------------------------- just send an email and put in the subject the following: Привет из России (Hello from Russia) Same problem as well with hebrew: שלום בעברית (Hello in Hebrew) In arabic: مرحبا باللغة العربية (Hellow in Arabic) Thanks. ---------------------------------------------------------------------- (0003586) paul (administrator) - 23-Sep-13 16:57 http://www.dbmail.org/mantis/view.php?id=1020#c3586 ---------------------------------------------------------------------- ignore that last message ---------------------------------------------------------------------- (0003587) Lionel (reporter) - 23-Sep-13 16:58 http://www.dbmail.org/mantis/view.php?id=1020#c3587 ---------------------------------------------------------------------- If you want to try other languages like chinese or japanese for example, translate (Hellow in Chinese) with google translate. This issue did not exist with dbmail 2.x ---------------------------------------------------------------------- (0003589) paul (administrator) - 23-Sep-13 17:28 http://www.dbmail.org/mantis/view.php?id=1020#c3589 ---------------------------------------------------------------------- When I test above subjects with Thunderbird, everything is working just fine as far as I can tell. Are you using messages with raw 8bit headers? If so, that would violate RFC 2822 If you are *not* using 8bit headers please provide one or more test messages with encoded headers ---------------------------------------------------------------------- (0003591) Lionel (reporter) - 23-Sep-13 17:49 http://www.dbmail.org/mantis/view.php?id=1020#c3591 ---------------------------------------------------------------------- Thundrbird is the only email client that we tested that shows the subject properly in the message itself; not in the list of emails. I also made tests with Apple Mail and it does not show up properly. ---------------------------------------------------------------------- (0003592) Lionel (reporter) - 23-Sep-13 17:50 http://www.dbmail.org/mantis/view.php?id=1020#c3592 ---------------------------------------------------------------------- By the way; on my side, even when I send the emails with thunderbird the subject is unreadable in the emails list. All settings are the default one. ---------------------------------------------------------------------- (0003593) igorbelykh86 (reporter) - 23-Sep-13 18:12 http://www.dbmail.org/mantis/view.php?id=1020#c3593 ---------------------------------------------------------------------- Hi Paul Let me show you example of headers: Received: from mail-vb0-f49.google.com (mail-vb0-f49.google.com [209.85.212.49]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mail.zubybox.com (Postfix) with ESMTPS id 1D156781587 for <i...@zubybox.com>; Sun, 22 Sep 2013 12:58:31 -0500 (CDT) Received: by mail-vb0-f49.google.com with SMTP id w16so1572291vbb.22 for <i...@zubybox.com>; Sun, 22 Sep 2013 10:58:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=H8rshYhYYDgKX2iy+I9JDjA3nltOr4SWC6SGSMGbS58=; b=uG7fs/xEvi7mptyqURCvDefwIItiq+cIwgynj/8FHwcIJyThwH1N3z6oGiJ89KDK2n NDbrII/GkQuQ3radtjRX3R9+8D52f0+JN8kxmaLPlrYYzSz2vu0QE+1QNh5ec/sNuF3y TULQJpFm6hELvN63yyItAACH1zGSJvfIA4P9bI+SxHZjJMt30Xuh6mgcMNFlfyEdZ5WF vJbZ/I62+hfzTz5vmO3spsow+lNUclXmalUrMcFWb8GytMunVIVqQ9kuCkRNyQej/en0 7IIjkcHLrXH0wgXEIZrK+qmpvJczYR53TcyPn6QJkn00//FuJ5qor3bWUo+BepKYhef5 TV6Q== MIME-Version: 1.0 X-Received: by 10.221.51.206 with SMTP id vj14mr17765026vcb.17.1379872710605; Sun, 22 Sep 2013 10:58:30 -0700 (PDT) Received: by 10.52.191.40 with HTTP; Sun, 22 Sep 2013 10:58:30 -0700 (PDT) Date: Mon, 23 Sep 2013 00:58:30 +0700 Message-ID: <CA+jC7cWEg30O6UP4nKfy9BbbBBzH3_4w4g9=4oqyqfnpkmj...@mail.gmail.com> Subject: =?UTF-8?B?0KHQsNC80L7QtSDQvdC+0LLQvtC1INGB0L7QvtCx0YnQtdC90LjQtSDQvtGCINCY0LPQvg==?= =?UTF-8?B?0YDRjw==?= From: Igor Belykh <igorbelyk...@gmail.com> To: Igor <i...@zubybox.com> Return-Path: igorbelyk...@gmail.com Content-Type: multipart/alternative; boundary=001a11332252c042b904e6fca6f2 all fields is readable in database except sortfield Issue History Date Modified Username Field Change ====================================================================== 17-Aug-13 09:37 igorbelykh86 New Issue 17-Aug-13 10:33 paul Note Added: 0003559 17-Aug-13 13:15 igorbelykh86 Note Added: 0003560 19-Aug-13 12:02 igorbelykh86 Note Added: 0003561 19-Aug-13 12:04 jasb Note Added: 0003562 19-Aug-13 12:24 igorbelykh86 Note Added: 0003563 19-Aug-13 12:29 igorbelykh86 Note Added: 0003564 19-Aug-13 12:54 paul Note Added: 0003565 19-Aug-13 13:07 igorbelykh86 Note Added: 0003566 21-Sep-13 19:44 igorbelykh86 Note Added: 0003580 22-Sep-13 18:07 Lionel Note Added: 0003581 23-Sep-13 16:44 paul Note Added: 0003582 23-Sep-13 16:48 paul Note Added: 0003583 23-Sep-13 16:49 Lionel Note Added: 0003584 23-Sep-13 16:52 Lionel Note Added: 0003585 23-Sep-13 16:57 paul Note Added: 0003586 23-Sep-13 16:58 Lionel Note Added: 0003587 23-Sep-13 17:28 paul Note Added: 0003589 23-Sep-13 17:49 Lionel Note Added: 0003591 23-Sep-13 17:50 Lionel Note Added: 0003592 23-Sep-13 18:12 igorbelykh86 Note Added: 0003593 ====================================================================== _______________________________________________ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev