A NOTE has been added to this issue. ====================================================================== http://www.dbmail.org/mantis/view.php?id=265 ====================================================================== Reported By: idk Assigned To: paul ====================================================================== Project: DBMail Issue ID: 265 Category: IMAP daemon Reproducibility: always Severity: minor Priority: normal Status: feedback target: ====================================================================== Date Submitted: 31-Aug-05 15:16 CEST Last Modified: 03-Jul-07 23:01 CEST ====================================================================== Summary: Non US-ASCII character in mail header breaks fetching Description: When non-us-ascii (8bit) is present in header (eg To:, see additional information), IMAP server send this character unescaped, thus MSOE client logs [db] PARSE ERROR: hr=2148322516 and shows nothing. I know 8bit charecetr si wrong at this place, but IMAP server should handle this issue, IMHO. ====================================================================== Relationships ID Summary ---------------------------------------------------------------------- related to 0000426 invalid byte sequence for encoding &quo... related to 0000499 incorrect envelope related to 0000304 with ref: to ID 0000265 , bug was suspe... ======================================================================
---------------------------------------------------------------------- paul - 01-Oct-05 11:16 ---------------------------------------------------------------------- This is expected to be fixed in the upcoming 2.1.3 release, which will be the first release without *any* of the old homegrown mime-parser. I'm therefor suspending this bug until 2.1.3 is released. Once gmime is used for all mime and encoding issues this should be trivial to fix, if not fixed then already. ---------------------------------------------------------------------- paul - 30-Apr-06 10:11 ---------------------------------------------------------------------- Current svn-trunk is handling multibyte encodings in the headers just fine. I'm therefore closing this bug. ---------------------------------------------------------------------- idk - 06-Mar-07 22:14 ---------------------------------------------------------------------- Sorry, I had not check this issue till now. Aftter upgrade to 2.2.2 I checked some message (previously inserted into db by 2.0 version) with same result (as in additional information above described). The same message I inserted by 2.2.2 version (sendmail user <message.eml). The same result on new message copy. So parsing mail headers still wrong. My test message has ASCII >0x7F (here 0xE9) in "to" header (To: <"Odoln\xe9 palmy"@domain>), LMTP daemon receives this message from postfix in unchanged form (so with 0xE9 character), inserts it into db, then IMAP daemon returns on FETCH query this character, but IMAP should communicate in UTF7 encoding (0xE7 is not valid UTF7 character). SELECT hex( headervalue ) FROM `dbmail_headervalue` WHERE physmessage_id =148748 AND headername_id =8 22 4F 64 6F 6C 6E E9 20 70 61 6C 6D 79 22 40 6D 78 2E 6D 6F 72 65 2E 63 7A I changed this character in database manually (in dbmail_headervalue cache), but without effect, nor after daemon restart. IMAPD still returns 0xE9 ((NIL NIL "Odoln&http://www.dbmail.org/mantis/view.php?id=9618; palmy " "domain")). ---------------------------------------------------------------------- paul - 06-Mar-07 22:55 ---------------------------------------------------------------------- You are correct about 2.2.2, but please test the current active branch (dbmail_2_2_branch). There have been some relevant changes since 2.2.2 that have addressed (and hopefully resolve) all 8bit issues in the envelope response. I'm relating this issue to the ones I'm refering to. ---------------------------------------------------------------------- idk - 06-Mar-07 23:32 ---------------------------------------------------------------------- No, no, sorry. I tried https://svn.ic-s.nl/svn/dbmail/branches/dbmail_2_2_branch at revision 2451. Still the same. Nor old messages (inserted by 2.0.10 and by 2.2.2), nor newly inserted message (2.2 rev 2451). It is not to hard to check yourself. Procedure to reproduce I had reported almost two years before. Under linux with postfix 2.2.8 and DBMail via LMTP daemon (maybe various configurations) run sendmail userid <file_with_non_ASCII7_in_header.eml (I can attach sample), then fetch from IMAP daemon like UID FETCH 1:* (BODY.PEEK[HEADER.FIELDS (References X-Ref X-Priority X-MSMail-Priority X-MSOESRec Newsgroups)] ENVELOPE RFC822.SIZE UID FLAGS INTERNALDATE) via telnet (nc, or so of). You get non-UTF7 character in reply. Try it, please. ---------------------------------------------------------------------- paul - 07-Mar-07 09:50 ---------------------------------------------------------------------- Please provide steps to reproduce, because when I test this all is well. However, you do need to change dbmail.conf and add two new entries: encoding=utf8 default_msg_encoding=utf8 the first one needs to be the same as the encoding of the database connection. the second one is used to decode 8bit headers where the correct encoding can not be determined from the message itself. Both are needed for this to work properly. ---------------------------------------------------------------------- idk - 08-Mar-07 01:59 ---------------------------------------------------------------------- I set utf8 encoding, but before I had to switch to utf8 database too, only utf8_general_ci collation worked properly, I set to utf8 all tables, varchars and texts (all ones were set to latin1_swedish_ci). Newly inserted message (via sendmail -> lmtpd) with non ASCII7 character in header are processed nearly well (in OE message list is a "?" character at position of 0xE9, but in message view at 0xE9 is "correct" character (which I expected)). Old messages stills unreadable. Attached msgs.tgz contains two sample (spam) messages, msg.148748.eml is a message with 0xE9 in To field, msg.86479.eml contains some non ASCII7 characters in To field (many email addresses) and in Subject field too. Second one is not displayed by OE never, log says PARSE ERROR: hr=2148322516. I am unable to found the cause. Maybe relevant to http://www.dbmail.org/mantis/view.php?id=499? I tested it on 2452 revision. ---------------------------------------------------------------------- idk - 08-Mar-07 13:18 ---------------------------------------------------------------------- Attached message msg.86479.eml has strange To: field: To: AccurateAlerts5787 Content-Type: text/[EMAIL PROTECTED];, "charset=\"us-ascii\""@h1430.serverkompetenz.net, "MIME-Version:1.0"@h1430.serverkompetenz.net, "Content-Transfer-Encoding:8bit"@h1430.serverkompetenz.net, "Subject:We"@h1430.serverkompetenz.net, ... Is it possible of invalid parsing of this header? IMAP on FETCH ENVELOPE replies: * 82 FETCH (UID 534908 ENVELOPE ("Mon, 12 Dec 2005 06:14:06 +0100" "Danke f?r Deine Nachricht!" ((NIL NIL "Henry" "Krasemann.com")) ((NIL NIL "Henry" "Krasemann.com")) ((NIL NIL "Henry" "Krasemann.com")) ((NIL NIL "AccurateAlerts5787 Content-Type" NIL)(NIL NIL "text/plain" "h1430.serverkompetenz.net")(NIL NIL "charset=\ us-ascii\ " "h1430.serverkompetenz.net")(NIL NIL "MIME-Version:1.0 " "h1430.serverkompetenz.net")(NIL NIL "Content-Transfer-Encoding:8bit " "h1430.serverkompetenz.net")(NIL NIL "Subject:We " "h1430.serverkompetenz.net").... Email "Subject:We @h1430.serverkompetenz.net" dont look as a valid email address (colon? space?), maybe this is the reason for rejecting by OE client. Possibly all emails in a form "string"@domain causes this problem. It can look like no problem, because this message is buggy (sent by buggy spambot), but this message I got into INBOX, since the time OE singalize new messages, but I'm not able to fetch/read/delete. With this message SqurrelMail has the same problem (with message with only invalid UTF7 character one works well, messages with incorrect email address, if it is the reason, are not shown in list and are not manageable too). I can use telnet and delete message manually, but common users not. I have no idea to solve problem. Maybe ignore buggy mail addresses by server, maybe replace invalid characters by an underscore (not by a question mark)... I don't know. ---------------------------------------------------------------------- idk - 20-Apr-07 19:59 ---------------------------------------------------------------------- Till rev. 2481 LMTP daemon dead when header message contains an 8bit character. In my case I have a message with an 0xB9 character in Subject and From headers. Rev. 2481 dead after logging a line "INSERT INTO dbmail_subjectfield (physmessage_id, subjectfield) VALUES (289645,0xB9)". Next line is "register child". After replacing this character by the valid 7bit one, LMTPD lives till "handling a standard address [0xB9] [EMAIL PROTECTED]". In both casese postfix said "status=deferred (lost connection while sending end of data)". But this issue is solved by a rev. 2483+, however still not correctly. For the same message this will happens: - a From header for headervalue is encoded incorrectly as 0xCC 0x8C - a Subject header for headervalue is encoded correctly as 0xC4 0xA1 - a From header for fromfield is encoded correctly as 0xC4 0xA1 - a Subject header for subjectfield is encoded incorrectly as 0xCC 0x8C Tested with rev. 2520. ---------------------------------------------------------------------- paul - 03-Jul-07 23:01 ---------------------------------------------------------------------- is this still playing up in 2.2.5? Issue History Date Modified Username Field Change ====================================================================== 31-Aug-05 15:16 idk New Issue 01-Oct-05 11:16 paul Note Added: 0000937 01-Oct-05 11:16 paul Status new => acknowledged 01-Oct-05 11:16 paul Resolution open => suspended 01-Oct-05 11:16 paul Projection none => major rework 01-Oct-05 11:16 paul ETA none => < 1 month 30-Apr-06 09:27 aaron Relationship added related to 0000304 30-Apr-06 10:11 paul Note Added: 0001134 30-Apr-06 10:11 paul Assigned To => paul 30-Apr-06 10:11 paul Status acknowledged => resolved 30-Apr-06 10:11 paul Resolution suspended => fixed 30-Apr-06 10:11 paul Fixed in Version => SVN Trunk 06-Mar-07 22:14 idk Status resolved => feedback 06-Mar-07 22:14 idk Resolution fixed => reopened 06-Mar-07 22:14 idk Note Added: 0001874 06-Mar-07 22:55 paul Note Added: 0001875 06-Mar-07 22:55 paul Status feedback => closed 06-Mar-07 22:55 paul Resolution reopened => fixed 06-Mar-07 22:55 paul Fixed in Version SVN Trunk => 2.2.3 06-Mar-07 22:56 paul Relationship added related to 0000426 06-Mar-07 22:56 paul Relationship added related to 0000499 06-Mar-07 23:32 idk Status closed => feedback 06-Mar-07 23:32 idk Resolution fixed => reopened 06-Mar-07 23:32 idk Note Added: 0001876 07-Mar-07 09:50 paul Note Added: 0001877 08-Mar-07 01:59 idk Note Added: 0001885 08-Mar-07 02:00 idk File Added: msgs.tgz 08-Mar-07 13:16 idk Note Added: 0001887 08-Mar-07 13:17 idk Note Edited: 0001887 08-Mar-07 13:18 idk Note Edited: 0001887 20-Apr-07 19:59 idk Note Added: 0002078 03-Jul-07 23:01 paul Note Added: 0002278 ====================================================================== _______________________________________________ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
