The following issue has been RESOLVED.
======================================================================
http://www.dbmail.org/mantis/view.php?id=499
======================================================================
Reported By: AntonZ
Assigned To: paul
======================================================================
Project: DBMail
Issue ID: 499
Category: IMAP daemon
Reproducibility: always
Severity: feature
Priority: low
Status: resolved
target:
Resolution: fixed
Fixed in Version: 2.2.3
======================================================================
Date Submitted: 24-Jan-07 13:15 CET
Last Modified: 26-Feb-07 10:01 CET
======================================================================
Summary: incorrect envelope
Description:
locale and mysql charset is utf-8.
Message with not encoded(8 bit) subject have incorrect value in
dbmail_envelope.
dbmail_envelope.envelope value:
("Mon, 22 Jan 2007 19:54:04 +0400" {24}
Outlook express not show that messages. IMAP4.log:
IMAP: 19:44:48 [rx] * 1 FETCH (INTERNALDATE "22-Jan-2007 19:02:59 +0300"
RFC822.SIZE 20418 FLAGS (\Seen) UID 71569 ENVELOPE ("Mon, 22 Jan 2007
11:49:35 -0400" {24}
IMAP: 19:44:48 [rx] Buffer (literal) of length 24
IMAP: 19:44:48 [rx] erences X-Ref X-Priority X-MSMail-Priority X-MSOESRec
Newsgroups)] {44}
IMAP: 19:44:48 [rx] Buffer (literal) of length 44
IMAP: 19:44:48 [rx] )
IMAP: 19:44:48 [db] PARSE ERROR: hr=2148322516
IMAP: 19:44:48 [rx] * 2 FETCH (INTERNALDATE "22-Jan-2007 19:03:03 +0300"
RFC822.SIZE 20423 FLAGS (\Seen) UID 71570 ENVELOPE ("Mon, 22 Jan 2007
11:49:35 -0400" {24}
IMAP: 19:44:48 [rx] Buffer (literal) of length 24
IMAP: 19:44:48 [rx] erences X-Ref X-Priority X-MSMail-Priority X-MSOESRec
Newsgroups)] {44}
IMAP: 19:44:48 [rx] Buffer (literal) of length 44
IMAP: 19:44:48 [rx] )
IMAP: 19:44:48 [db] PARSE ERROR: hr=2148322516
IMAP: 19:44:48 [rx] * 3 FETCH (INTERNALDATE "22-Jan-2007 19:03:07 +0300"
RFC822.SIZE 20457 FLAGS (\Seen) UID 71571 ENVELOPE ("Mon, 22 Jan 2007
19:54:04 +0400" {24}
IMAP: 19:44:48 [rx] Buffer (literal) of length 24
IMAP: 19:44:48 [rx] erences X-Ref X-Priority X-MSMail-Priority X-MSOESRec
Newsgroups)] {44}
IMAP: 19:44:48 [rx] Buffer (literal) of length 44
IMAP: 19:44:48 [rx] )
IMAP: 19:44:48 [db] PARSE ERROR: hr=2148322516
IMAP: 19:44:48 [rx] * 4 FETCH (INTERNALDATE "22-Jan-2007 19:03:09 +0300"
RFC822.SIZE 20442 FLAGS (\Seen) UID 71572 ENVELOPE ("Mon, 22 Jan 2007
19:54:04 +0400" {24}
IMAP: 19:44:48 [rx] Buffer (literal) of length 24
IMAP: 19:44:48 [rx] erences X-Ref X-Priority X-MSMail-Priority X-MSOESRec
Newsgroups)] {44}
IMAP: 19:44:48 [rx] Buffer (literal) of length 44
IMAP: 19:44:48 [rx] )
IMAP: 19:44:48 [db] PARSE ERROR: hr=2148322516
value of subject headervalue for this messages is empty.
also empty fromname for not encoded headers From
======================================================================
Relationships ID Summary
----------------------------------------------------------------------
duplicate of 0000426 invalid byte sequence for encoding &quo...
======================================================================
----------------------------------------------------------------------
paul - 24-Jan-07 15:03
----------------------------------------------------------------------
Anton, please post trace_level=5 logs.
I don't see anything obviously wrong with the OE logs you posted.
8bit headers work for me.
----------------------------------------------------------------------
AntonZ - 24-Jan-07 15:41
----------------------------------------------------------------------
encoded headers work OK.
if subject have 8bit, dbmail generate and send incorrect:
Debug:[mailbox] dbmail-mailbox.c,dbmail_mailbox_get_set(+1233): [74813:*]
Debug:[misc] misc.c,g_tree_merge(+1219): a[0] [OR] b[1] -> a[1]
Debug:[sql] dbmysql.c,db_query(+286): query [SELECT seen_flag,
answered_flag, deleted_flag, flagged_flag, draft_flag, recent_flag,
DATE_FORMAT(internal_date, '%Y-%m-%d %T'), rfcsize, message_idnr FROM
dbmail_messages msg, dbmail_physmessage pm WHERE pm.id =
msg.physmessage_id AND message_idnr BETWEEN 73322 AND 73322 AND
mailbox_idnr = 1 AND status IN (0,1) ORDER BY message_idnr ASC]
Debug:[imapsession]
dbmail-imapsession.c,dbmail_imap_session_printf(+1292): RESPONSE: [* 6
FETCH (]
Debug:[imapsession]
dbmail-imapsession.c,dbmail_imap_session_printf(+1292): RESPONSE:
[INTERNALDATE "24-Jan-2007 10:42:34 +0300"]
Debug:[imapsession]
dbmail-imapsession.c,dbmail_imap_session_printf(+1292): RESPONSE: [ ]
Debug:[imapsession]
dbmail-imapsession.c,dbmail_imap_session_printf(+1292): RESPONSE:
[RFC822.SIZE 15709]
Debug:[imapsession]
dbmail-imapsession.c,dbmail_imap_session_printf(+1292): RESPONSE: [ ]
Debug:[imapsession]
dbmail-imapsession.c,dbmail_imap_session_printf(+1292): RESPONSE: [FLAGS
(\Seen)]
Debug:[imapsession]
dbmail-imapsession.c,dbmail_imap_session_printf(+1292): RESPONSE: [ ]
Debug:[imapsession]
dbmail-imapsession.c,dbmail_imap_session_printf(+1292): RESPONSE: [UID
73322]
Debug:[imapsession]
dbmail-imapsession.c,dbmail_imap_session_printf(+1292): RESPONSE: [ ]
Debug:[sql] dbmysql.c,db_query(+286): query [SELECT message_idnr,envelope
FROM dbmail_envelope e JOIN dbmail_messages m ON
m.physmessage_id=e.physmessage_id JOIN dbmail_mailboxes b ON
b.mailbox_idnr=m.mailbox_idnr WHERE m.mailbox_idnr = 1 AND message_idnr
BETWEEN 73322 AND 73322 ]
Debug:[imapsession]
dbmail-imapsession.c,dbmail_imap_session_printf(+1292): RESPONSE:
[ENVELOPE ("Wed, 24 Jan 2007 03:22:54 -0400" {38}
]
Debug:[imapsession] dbmail-imapsession.c,_imap_show_body_section(+1052):
itemtype [3] partspec []
Debug:[imapsession]
dbmail-imapsession.c,dbmail_imap_session_printf(+1292): RESPONSE: [ ]
Debug:[imapsession]
dbmail-imapsession.c,dbmail_imap_session_printf(+1292): RESPONSE: [BODY[]
Debug:[imapsession]
dbmail-imapsession.c,dbmail_imap_session_printf(+1292): RESPONSE:
[HEADER.FIELDS (References X-Ref X-Priority X-MSMail-Priority X-MSOESRec
Newsgroups)] ]
Debug:[sql] dbmysql.c,db_query(+286): query [SELECT
message_idnr,headername,headervalue FROM dbmail_headervalue v JOIN
dbmail_messages m ON v.physmessage_id=m.physmessage_id JOIN
dbmail_mailboxes b ON m.mailbox_idnr=b.mailbox_idnr JOIN dbmail_headername
n ON v.headername_id=n.id WHERE m.mailbox_idnr = 1 AND message_idnr BETWEEN
73322 AND 73322 AND lower(headername) IN
('references','x-ref','x-priority','x-msmail-priority','x-msoesrec','newsgroups')]
Debug:[imapsession]
dbmail-imapsession.c,dbmail_imap_session_printf(+1292): RESPONSE: [{44}
X-MSMail-Priority: Normal
X-Priority: 3
]
Debug:[imapsession]
dbmail-imapsession.c,dbmail_imap_session_printf(+1292): RESPONSE: [)
]
Debug:[imapsession]
dbmail-imapsession.c,dbmail_imap_session_printf(+1292): RESPONSE: [mb95 OK
UID FETCH completed
]
Info:[imap] imap4.c,IMAPClientHandler(+336): Finished command uid [0]
----------------------------------------------------------------------
paul - 24-Jan-07 15:52
----------------------------------------------------------------------
Anton,
I've done some minor changes in the envelope code. 8bit subjects are now
encoded in the envelope. Please tell me if it helps.
rev 2428
----------------------------------------------------------------------
paul - 24-Jan-07 15:55
----------------------------------------------------------------------
Anton,
sorry, I meant level=5 logs of message insertion.
----------------------------------------------------------------------
AntonZ - 24-Jan-07 17:08
----------------------------------------------------------------------
Ok. Found [1] missing envelope values.
Debug:[sql] dbmysql.c,db_query(+286): query [SELECT messageblk FROM
dbmail_messageblks WHERE physmessage_id = 33715 AND is_header = '1']
Debug:[message] dbmail-message.c,_set_content_from_stream(+383): parse
message
Debug:[misc] misc.c,imap_get_envelope(+2014): encoding 8bit subject [
----------------------------------------------------------------------
AntonZ - 24-Jan-07 17:19
----------------------------------------------------------------------
logs from "dbmail-utils -by" after delete from database corrupted records
gmime_utils_header_encode not encode subject
field has utf-8 charset and now in table dbmail_envelope 2 records (bug in
dbmail-utils???) with envelope=[
("Wed, 24 Jan 2007 03:22:54 -0400" {29}
acop
]
in dbmail_headervalue subject headervalue is empty
----------------------------------------------------------------------
paul - 24-Jan-07 18:41
----------------------------------------------------------------------
Anton,
UTF8 is *not* a charset. utf8 is an 8bit representation of Unicode. Dbmail
is not (yet) unicode clean.
Please attach the message involved (or a similar one) to this bug.
----------------------------------------------------------------------
AntonZ - 26-Jan-07 10:34
----------------------------------------------------------------------
g_mime_utils_header_encode_text always return not encode header.
mysql on insert truncate string to first unrecognized simbol(in utf8
representation).
I solve problem by convert envelope field from text to blob type.
Problem is keep in headervalue,subjectfield,fromfield,tofield,ccfield.
----------------------------------------------------------------------
paul - 30-Jan-07 20:29
----------------------------------------------------------------------
I'm setting this report to 'suspended - no change required'. Looks
suspiciously like a garbage-in garbage-out situation.
Injecting messages with 8bit headers of unknown (non-utf8) encoding - in
themselves a violation - into a utf8 table will result in unexpected
results. I don't see any solutions other than rejecting such messages
during insertion.
The proposed solution (setting some fields to BLOB, or BYTEA) is not valid
and not supported.
----------------------------------------------------------------------
AntonZ - 31-Jan-07 17:25
----------------------------------------------------------------------
patch description:
on save (cache,fields) convert wrong encoded string to database encoding.
source encoding getting from message header,default_msg_encoding (config
parameters) or locale.
convert wrong string to utf8 on envelope field generate.
8 bit chars replaced with "?" on error.
database encoding in dbmail.conf must be in iconv notation or known gmime.
----------------------------------------------------------------------
paul - 31-Jan-07 22:51
----------------------------------------------------------------------
AntonZ, this looks really good. Patch applied.
----------------------------------------------------------------------
paul - 26-Feb-07 10:01
----------------------------------------------------------------------
AntonZ, your patch resolves many utf8 related problems it seems.
People who run into this should make sure they setup the 'encoding' and
'default_msg_encoding' config items in the DBMAIL section of dbmail.conf
Issue History
Date Modified Username Field Change
======================================================================
24-Jan-07 13:15 AntonZ New Issue
24-Jan-07 15:03 paul Note Added: 0001782
24-Jan-07 15:41 AntonZ Note Added: 0001783
24-Jan-07 15:52 paul Note Added: 0001784
24-Jan-07 15:55 paul Note Added: 0001785
24-Jan-07 17:08 AntonZ Note Added: 0001786
24-Jan-07 17:09 AntonZ File Added: debug.dbmailutil
24-Jan-07 17:19 AntonZ Note Added: 0001787
24-Jan-07 18:41 paul Note Added: 0001791
24-Jan-07 18:54 AntonZ File Added: spam1.msg
26-Jan-07 10:34 AntonZ Note Added: 0001792
30-Jan-07 20:29 paul Note Added: 0001794
30-Jan-07 20:29 paul Priority normal => low
30-Jan-07 20:29 paul Severity major => feature
30-Jan-07 20:29 paul Status new => acknowledged
30-Jan-07 20:29 paul Resolution open => no change
required
30-Jan-07 20:29 paul Category General => IMAP daemon
30-Jan-07 21:26 paul Relationship added duplicate of 0000426
31-Jan-07 17:11 AntonZ File Added: cp-new.patch
31-Jan-07 17:25 AntonZ Note Added: 0001806
31-Jan-07 20:51 AntonZ File Added: cp-new-2.patch
31-Jan-07 22:51 paul Note Added: 0001807
26-Feb-07 10:01 paul Note Added: 0001842
26-Feb-07 10:01 paul Assigned To => paul
26-Feb-07 10:01 paul Status acknowledged => resolved
26-Feb-07 10:01 paul Resolution no change required =>
fixed
26-Feb-07 10:01 paul Fixed in Version => 2.2.3
======================================================================
_______________________________________________
Dbmail-dev mailing list
[email protected]
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
