Subject field corruption
I use Postfix 2.0.10 with TLS patch, cyrus-imapd 2.1.13, sasl 2.1.13 The problem is as follow: When a message with russian charset characters (KOI8-R, WIN1251) in Subject field arrives or is moved into the Sent Items folders by Outlook Express, each russian character is replaced by an X (ex: Re: mazda 626 ). I happens only when the Subject field is without charaterset specification. I dumped mailboxes with cyrdump, the Xes are in. What can be wrog? I will appreciate any help. Max. -- Real programmers never sleep. They enter brainsaver mode.
Re: Subject field corruption
On Thu, 5 Jun 2003, NOwlar wrote: I dumped mailboxes with cyrdump, the Xes are in. What can be wrog? I will appreciate any help. The sending client included raw 8-bit data in the headers, which is not allowed. Cyrus dealt with this problem by munging the header to be valid. You can set reject8bit: t in your imapd.conf to outright reject the messages. -Rob -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456 Research Systems Programmer * /usr/contributed Gatekeeper
Re: Subject field corruption
NOwlar wrote: I use Postfix 2.0.10 with TLS patch, cyrus-imapd 2.1.13, sasl 2.1.13 The problem is as follow: When a message with russian charset characters (KOI8-R, WIN1251) in Subject field arrives or is moved into the Sent Items folders by Outlook Express, each russian character is replaced by an X (ex: Re: mazda 626 ). I happens only when the Subject field is without charaterset specification. I dumped mailboxes with cyrdump, the Xes are in. What can be wrog? The problem is that 8-bit characters are not allowed in the header of an RFC822 message. There are extensions designed to allow 8-bit characters, by encoding the character set and a base64 or quoted-printable representation of them. The problem is that Cyrus cannot properly transform these messages to standards compliance since it cannot know what character set was used, only that it wasn't US-ASCII, so Cyrus chooses to either drop the message or replace all 8-bit characters with X depending on a configuration setting. There was some discussion of hacking Cyrus to allow an administrator-supplied default character set in the case of 8-bit data without a character set, but the consensus of the developers seemed to be that was more an MTA function than Cyrus, and nobody felt like doing a hack that would not be accepted into the official source. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Subject field corruption
NOwlar wrote: I use Postfix 2.0.10 with TLS patch, cyrus-imapd 2.1.13, sasl 2.1.13 The problem is as follow: When a message with russian charset characters (KOI8-R, WIN1251) in Subject field arrives or is moved into the Sent Items folders by Outlook Express, each russian character is replaced by an X (ex: Re: mazda 626 ). I happens only when the Subject field is without charaterset specification. I dumped mailboxes with cyrdump, the Xes are in. What can be wrog? Maybe postfix can auto-convert these messages for you ? In sendmail you could do something like: define(`confSEVEN_BIT_INPUT', `false')dnl define(`confEIGHT_BIT_HANDLING', `mime')dnl define(`confDEF_CHAR_SET', `iso-8859-15')dnl but such a solution can produce quite ugly results if the message beeing autoconverted does not apply to the default charset in use. I stopped using such features because squatter stopped working for certain mailboxes in which auto-converted messages resided, I think. I would suggest setting reject8bit in imapd.conf to yes so that these non-standard-compliant messages get rejected. It is more a mta issue somehow than cyrus! Your mta delivers non-standard-compliant messages to cyrus and leaves the decision what to do with them to cyrus instead of just rejecting everything non-standarad-compliant. For me, the sender should get notice about the wrong behaving client and is the one to fix the problem on his side. I mainly had problems with messages coming from some webmail clients and I stopped caring about anybody who knows about the existence of e.g. php's mail function but does not know how to send messages with it correctly! If you see X in your messages cyrus just corrected a message the sender's client was not able to specify a character-set for correctly! --Christian
Re: Subject field corruption
On Thu, Jun 05, 2003 at 08:57:38PM +0400, NOwlar wrote: I use Postfix 2.0.10 with TLS patch, cyrus-imapd 2.1.13, sasl 2.1.13 The problem is as follow: When a message with russian charset characters (KOI8-R, WIN1251) in Subject field arrives or is moved into the Sent Items folders by Outlook Express, each russian character is replaced by an X (ex: Re: mazda 626 ). I happens only when the Subject field is without charaterset specification. I dumped mailboxes with cyrdump, the Xes are in. What can be wrog? I will appreciate any help. Max. Cyrus kills the 8th bit character in the header, and replaces it with 'X'. There is a configuration option, that would discard unconditionally every message with the 8th bit turned on in the headers. Luckily you are *only* seeing the 'X' inserted instead. In Cyrus-2.0.16 one file needed to be patched. In the latest Cyrus-es you apparently need to patch 2 files. In my opinion, this breaks the Be liberal in what you accept, strict in what you send. Unfortunately (regardless of what the developers of Cyrus think) there are still software out there that generates email messages with 8th bit headers. -- [EMAIL PROTECTED]
Re: Subject field corruption
Einar Indridason wrote: On Thu, Jun 05, 2003 at 08:57:38PM +0400, NOwlar wrote: I use Postfix 2.0.10 with TLS patch, cyrus-imapd 2.1.13, sasl 2.1.13 The problem is as follow: When a message with russian charset characters (KOI8-R, WIN1251) in Subject field arrives or is moved into the Sent Items folders by Outlook Express, each russian character is replaced by an X (ex: Re: mazda 626 ). I happens only when the Subject field is without charaterset specification. I dumped mailboxes with cyrdump, the Xes are in. What can be wrog? I will appreciate any help. Max. Cyrus kills the 8th bit character in the header, and replaces it with 'X'. There is a configuration option, that would discard unconditionally every message with the 8th bit turned on in the headers. Luckily you are *only* seeing the 'X' inserted instead. In Cyrus-2.0.16 one file needed to be patched. In the latest Cyrus-es you apparently need to patch 2 files. In my opinion, this breaks the Be liberal in what you accept, strict in what you send. There is a difference between being liberal and being foolish. Unfortunately (regardless of what the developers of Cyrus think) there are still software out there that generates email messages with 8th bit headers. The users of such broken software should either pressure the author(s) to fix it or abandon it. Darwin's theory of evolution applies equally as well to software as it does to plants/animals. It has been mentioned MANY times that even if Cyrus was to accept 8-bit headers, there is no reasonable way to determine the character set. -- Kenneth Murchison Oceana Matrix Ltd. Software Engineer 21 Princeton Place 716-662-8973 x26 Orchard Park, NY 14127 --PGP Public Key--http://www.oceana.com/~ken/ksm.pgp
Re: Subject field corruption
On Fri, 6 Jun 2003, Einar Indridason wrote: In my opinion, this breaks the Be liberal in what you accept, strict in what you send. Actually, munging the data to Xs *is* being liberal in what is accepted and strict in what is sent. Since 8-bit characters in headers are nonsensical (there's no character set to go with them), changing them to Xs is the only way that cyrus can be strict in what it sends (IMAP is a 7-bit clean protocol, excepting the new BINARY extention which wouldn't allow 8-bit in this situation anyway). Having a default charset is a reasonable compromise, but I haven't seen anyone yet do the work (which would include properly QP-encoding the bogus headers). -Rob -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456 Research Systems Programmer * /usr/contributed Gatekeeper
