Re: AW: Re: AW: Re: Message contains NUL characters - howto dump?

[Joseph Brennan]

> (IMHO is the max. line lenght in emails 4000 characters.)


RFC 2821 sec 4.5.3.1 says the max length is 1000 characters including
the two CR LF characters.

However if the MTA fixes this, Cyrus won't see it.  Sendmail for example
breaks long lines at 997 characters and inserts ! CR LF.

Joseph Brennan
Lead Email Systems Engineer
Columbia University Information Technology


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: AW: Re: Message contains NUL characters - howto dump?

[Markus Rebensburg]

Klemens Puritscher schrieb:
> Phil Brutsche schrieb:
>   
>> The error message is being created by the LMTP service - NUL characters
>> aren't valid in ASCII messages. The email in question is being generated
>> incorrectly somewhere, somehow.
>> 
>
> thanks for your reply.
> I know that in the email must be a NUL character, but I cannot see this NUL 
> character during a tcpdump.
>
> Do you know, or someone else in this list, who can I safe find this NUL 
> character?
>
>   
Maybe it is a problem of lines ine the email which are longer than the
standard allows. Cyrus has a fixed buffer for each line in the email. If
the line is longer than this buffer lmtp inserts a terminating string
character (NUL) itself. This could be the reason you cannot see the
Character in the tcp stream of your mail.
We have the same problem with the NUL character here produced by replies
to emails with some HTML Attachments which have only one linebreak in
the whole file.

Regards,
 Markus
>> What you need to do is either have the MTA reject the message during the
>> DATA portion of the SMTP transaction, or have the MTA remove the NUL
>> characters before it passes the message on to the LMTP service.
>> 
>
> Yes, this will be the next step.
>
>   
>> Your email headers indicate you are using Postfix as your MTA, and I am
>> not familiar enough with that to tell you how to do what is necessary.
>> 
>
> Yes, that's right for outgoing emails.
> The MTA for incoming emails (mx host) is exim.
>
> regards,
> Klemens
>
>
>   
>> Klemens Puritscher wrote:
>> 
>>> Hello,
>>>
>>> I have a problem with one of our customers.
>>> When he forwards an email with the thunderbird email client (windows
>>> version), the lmtp-daemon on my cyrus-imapd (v2.3.13) rejects those
>>> emails with the error "554 5.6.0 Message contains NUL characters".
>>> ...ok, that's clear, there are "NUL" characters in the email.
>>>
>>> But I would show my customer, where the "NUL" character is.
>>>
>>> For tests, I generate a testmail, with "echo -e
>>> "From:\nTo:\nSubject:
>>> test\n\ntest\test\n.\n" > mail_with_NUL.txt
>>>
>>> Now I dump the lmtp-session on the cyrus-imapd host with:
>>> tcpdump -vv -XX -s 65535 -n -i eth1 "port lmtp
>>>
>>> and I see the "NUL" character:
>>> ...
>>> 0x0230:  7065 6564 2e61 740d 0a0d 0a74 6573 7400  peed.at
>>>   
>> test.
>> 
>>> 0x0240:  7465 7374 0d0a 2e0d 0a   test.
>>> ...
>>> 65 = e
>>> 73 = s
>>> 74 = t
>>> 00 = NUL
>>>
>>> ...ok, fine, I can find the "NUL" character.
>>>
>>> But when I dump the lmtp-session with the customer email (which get's
>>> the error "554 5.6.0 Message contains NUL characters"), I cannot find
>>> this "NUL" character.
>>>
>>> Can someone tell me, what I did wrong?
>>>
>>> Thanks in advance.
>>>
>>> Klemens
>>>
>>> 
>>> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
>>> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
>>> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>>>   
>> -- 
>>
>> Phil Brutsche
>> p...@optimumdata.com
>> 
>
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>
>   

begin:vcard
fn:Markus Rebensburg
n:Rebensburg;Markus
org;quoted-printable:Christian-Albrechts-Universit=C3=A4t zu Kiel;Rechenzentrum
adr:;;Ludewig-Meyn-Str. 4;Kiel;;24098;Germany
email;internet:rebensb...@rz.uni-kiel.de
tel;work:(+49)-431-880-3582
tel;fax:(+49)-431-880-1523 
x-mozilla-html:FALSE
url:http://www.uni-kiel.de/rz/
version:2.1
end:vcard


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

AW: Re: Message contains NUL characters - howto dump?

[Klemens Puritscher]

Phil Brutsche schrieb:
> The error message is being created by the LMTP service - NUL characters
> aren't valid in ASCII messages. The email in question is being generated
> incorrectly somewhere, somehow.

thanks for your reply.
I know that in the email must be a NUL character, but I cannot see this NUL 
character during a tcpdump.

Do you know, or someone else in this list, who can I safe find this NUL 
character?

> What you need to do is either have the MTA reject the message during the
> DATA portion of the SMTP transaction, or have the MTA remove the NUL
> characters before it passes the message on to the LMTP service.

Yes, this will be the next step.

> Your email headers indicate you are using Postfix as your MTA, and I am
> not familiar enough with that to tell you how to do what is necessary.

Yes, that's right for outgoing emails.
The MTA for incoming emails (mx host) is exim.

regards,
Klemens


> Klemens Puritscher wrote:
> > Hello,
> > 
> > I have a problem with one of our customers.
> > When he forwards an email with the thunderbird email client (windows
> > version), the lmtp-daemon on my cyrus-imapd (v2.3.13) rejects those
> > emails with the error "554 5.6.0 Message contains NUL characters".
> > ...ok, that's clear, there are "NUL" characters in the email.
> > 
> > But I would show my customer, where the "NUL" character is.
> > 
> > For tests, I generate a testmail, with "echo -e
> > "From:\nTo:\nSubject:
> > test\n\ntest\test\n.\n" > mail_with_NUL.txt
> > 
> > Now I dump the lmtp-session on the cyrus-imapd host with:
> > tcpdump -vv -XX -s 65535 -n -i eth1 "port lmtp
> > 
> > and I see the "NUL" character:
> > ...
> > 0x0230:  7065 6564 2e61 740d 0a0d 0a74 6573 7400  peed.at
> test.
> > 0x0240:  7465 7374 0d0a 2e0d 0a   test.
> > ...
> > 65 = e
> > 73 = s
> > 74 = t
> > 00 = NUL
> > 
> > ...ok, fine, I can find the "NUL" character.
> > 
> > But when I dump the lmtp-session with the customer email (which get's
> > the error "554 5.6.0 Message contains NUL characters"), I cannot find
> > this "NUL" character.
> > 
> > Can someone tell me, what I did wrong?
> > 
> > Thanks in advance.
> > 
> > Klemens
> > 
> > 
> > Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
> 
> 
> -- 
> 
> Phil Brutsche
> p...@optimumdata.com


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Patrick J. LoPresti]

For the record, this problem is not just caused by viruses and spam.

Where I work, some of our customers are in Japan and China.  I have
examples of messages sent via Outlook/Exchange in Japanese and Chinese
which include embedded NUL characters.  Rejecting these messages is
simply not an option from a business perspective.

If the default behavior of the major MTAs were to reject NUL
characters, all of the sending software would eventually get fixed.
But until that happens, I have to live with these messages.

So my only option is to to strip the NUL characters on the way in.  I
hate mangling message content as much as anyone, but I really have no
choice (other than to migrate off of Cyrus).  Since the messages are
not technically valid anyway, I feel less badly about mangling them.

Right now I use a very bad hack with an intermediate MTA to do this.
Having it as a configuration option in Cyrus would be VERY convenient.

Is such a change acceptable to the Cyrus developers?  If so, should I
(or someone) open a Bugzilla ticket to track this feature request?

 - Pat
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Greg A. Woods]

[ On Thursday, May 12, 2005 at 09:38:03 (+0200), John Fawcett wrote: ]
> Subject: Re: Message contains NUL characters ...
>
> I meant they are not readable in cyrus: at least not without my
> patch :-)

Well to be even more pedantic, they're not storable by Cyrus.

Were they to be delivered to some other system they may well be stored
and readable and valid.  :-)

> I think we all agree that nuls don't belong in email. There are various
> options for what to do about it. The current situation is the worst:
> MTA lets them through, cyrus rejects.
>
> The current behaviour of cyrus (reject) produces bounces that may go
> to the wrong people.

Neither of those are problems with Cyrus.  They are problems with broken
and/or mis-configured MTAs that some folks have unfortunately chosen to
use with Cyrus, perhaps through being mis-informed that the chosen MTA
should work well with Cyrus.  As you've discovered that is simply not
true of Postfix "out-of-the-box".


> Rejecting at the MTA is one option as is stripping in cyrus or the MTA.

Stripping "arbitrary" characters in either the MTA, or worse in Cyrus,
is, as I said, the moral equivalent of destructively stripping the 8'th
bit off all bytes in order to "convert" the message to ASCII.

I.e. it would be very wrong.

Conversion or rejection are the only valid choices.


> I've made proposed patches for all these options (configurable).

That configurability is the worst part -- software which is intended to
be _operated_ by non-expert users should not give them so much rope that
they can do very obviously wrong things, especially not in such a way
that the users they affect cannot know what is happening or why.

The right thing to do is to fix the (arguably broken) MTA so that it
always does the right thing and so that it can only do the right thing.

Postfix design, as in this example, was originally misguided into
believing that it would be a good thing to try to be a fully binary
transparent e-mail transport.  However experience has shown that this
could only work in an ideal world where every other e-mail tool involved
in sending, transmitting, and reading e-mail were also fully binary
transparent while at the same time being intelligent enough to know how
to automatically and properly classify the content it handles.  Witness
the evolution of a full MIME handling engine inside Postfix, as well as
recent options such as the "strict_7bit_headers" and
"strict_8bitmime_body" configuration flags.

For all intents and purposes a really simple and obvious fix to modern
Postfix would be to treat NUL as if it were an "8-bit" character
(i.e. so that it will be rejected or converted as necessary) and to make
sure that any Postfix install used with Cyrus has "strict_8bitmime"
turned on and that "disable_mime_output_conversion" is kept turned off
(and make sure that the Cyrus LMTP daemon does not advertise 8BITMIME
and that Postfix does right MIME conversions when sending to LMTP).

-- 
Greg A. Woods

H:+1 416 218-0098  W:+1 416 489-5852 x122  VE3TCP  RoboHack <[EMAIL PROTECTED]>
Planix, Inc. <[EMAIL PROTECTED]>  Secrets of the Weird <[EMAIL 
PROTECTED]>
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[John Fawcett]

Greg A. Woods wrote:
> [ On Thursday, May 12, 2005 at 00:30:46 (+0200), John Fawcett wrote: ]
> 
>>Subject: Re: Message contains NUL characters ...
>>
>>nothing actually breaks by removing the nuls. The messges weren't
>>readable before and may not be readable afterwards.
> 
> 
> You cannot know that they were not readable, and/or that they were not
> signed with the NULs in place.
I meant they are not readable in cyrus: at least not without my
patch :-)
> 
> As I said, finding a NUL embedded anywhere in the data is usually an
> indication of either a serious bug on the sending side, or at least an
> attempt to send opaque binary content of some type (which should have
> been encoded somewhow and thus also indicates either a serious bug on
> the sending side; or else it indicates malicious intent on the sending
> side).
> 
> The opaque binary data containing NULs could well be quite valid and
> useful and usable from both the sender's and recipient's point of view
> -- they may simply be expecting the e-mail transport to be binary
> transparent and are not being helped by proper mailer software that
> would have properly encoded their content for them.  Of course when the
> transport involves SMTP (and IMAP), it is not necessarily transparent.
I think we all agree that nuls don't belong in email. There are various
options for what to do about it. The current situation is the worst:
MTA lets them through, cyrus rejects.

> However if you silently and destructively remove the NULs from the
> stream without giving any indication to anyone of what you've done then
> you've definitely "broken" the message.  This is morally equivalent to
> blindly and destructively stripping off the 8th bit in order to
> "convert" (i.e. pretend to convert) a message to ASCII.
> 
> So, the only safe and sane thing to do when a NUL is found anywhere in
> SMTP data is to reject the whole message immediately with a fatal error
> response indicating the problem so that the sender is informed as
> quickly as possible that their content cannot be sent as-is.  (The
> sending MTA should have properly MIME-encoded the message if it didn't
> want to bounce it, or have it rejected.)
The current behaviour of cyrus (reject) produces bounces that may go
to the wrong people.
To be honest I throw away so many spam messages today without every
reading them that the idea of someone wanting to send nul characters
and needing to be advised as quickly as possible of the problem so
as to be able to resend them to me correctly is just not an issue.
I would apply the same logic as for the spam: if someone sends me
something and doesn't get a reply, it's up to them to contact me.
They cannot assume I read the message. It may have been filtered out
for whatever reason.
> 
> (Someone should make sure the Cyrus install guide strongly recommends to
> properly configure the MTA of the user's choice so that such it rejects
> invalid content, and to choose an MTA that can do so!)
> 
Rejecting at the MTA is one option as is stripping in cyrus or the MTA.
I've made proposed patches for all these options (configurable).
Then people can chose (if the patches are accepted of course),
otherwise they may have more limited options. Only one of
these is needed to solve my original problem. Given that I use postfix
my preference is also for the postfix enhancement.

John

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Greg A. Woods]

[ On Thursday, May 12, 2005 at 00:30:46 (+0200), John Fawcett wrote: ]
> Subject: Re: Message contains NUL characters ...
>
> nothing actually breaks by removing the nuls. The messges weren't
> readable before and may not be readable afterwards.

You cannot know that they were not readable, and/or that they were not
signed with the NULs in place.

As I said, finding a NUL embedded anywhere in the data is usually an
indication of either a serious bug on the sending side, or at least an
attempt to send opaque binary content of some type (which should have
been encoded somewhow and thus also indicates either a serious bug on
the sending side; or else it indicates malicious intent on the sending
side).

The opaque binary data containing NULs could well be quite valid and
useful and usable from both the sender's and recipient's point of view
-- they may simply be expecting the e-mail transport to be binary
transparent and are not being helped by proper mailer software that
would have properly encoded their content for them.  Of course when the
transport involves SMTP (and IMAP), it is not necessarily transparent.

However if you silently and destructively remove the NULs from the
stream without giving any indication to anyone of what you've done then
you've definitely "broken" the message.  This is morally equivalent to
blindly and destructively stripping off the 8th bit in order to
"convert" (i.e. pretend to convert) a message to ASCII.

So, the only safe and sane thing to do when a NUL is found anywhere in
SMTP data is to reject the whole message immediately with a fatal error
response indicating the problem so that the sender is informed as
quickly as possible that their content cannot be sent as-is.  (The
sending MTA should have properly MIME-encoded the message if it didn't
want to bounce it, or have it rejected.)

(Someone should make sure the Cyrus install guide strongly recommends to
properly configure the MTA of the user's choice so that such it rejects
invalid content, and to choose an MTA that can do so!)

-- 
Greg A. Woods

H:+1 416 218-0098  W:+1 416 489-5852 x122  VE3TCP  RoboHack <[EMAIL PROTECTED]>
Planix, Inc. <[EMAIL PROTECTED]>  Secrets of the Weird <[EMAIL 
PROTECTED]>
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[John Fawcett]

Greg A. Woods wrote:
> [ On Tuesday, May 10, 2005 at 22:33:43 (+0200), John Fawcett wrote: ]
> 
>>Subject: Re: Message contains NUL characters ...
>>
>>by the way, while looking into the code to do the patch, I saw that
>>cyrus already carries out some modifications to messages (example
>>removing bare \r and replacing bare \n with \r\n) so there is a
>>precedent.
> 
> 
> Note that EOL cleanups are a somewhat different issue -- if I'm not
> mistaken some encapsulation methods are already able to deal with EOL
> changes since this is a very common issue.  In any case one can conceive
> of arguments which show that these kinds of EOL cleanups are not
> destructive modifications of the message content (since EOL
> modifications may be necessary to even view the message on the
> recipient's system).
> 
> However finding a NUL embedded anywhere in the data is usually an
> indication of either a serious bug on the sending side, or at least an
> attempt to send opaque binary content of some type (which should have
> been encoded somewhow and thus also indicates either a serious bug on
> the sending side; or else it indicates malicious intent on the sending
> side).
> 
nothing actually breaks by removing the nuls. The messges weren't
readable before and may not be readable afterwards.
John
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Greg A. Woods]

[ On Tuesday, May 10, 2005 at 22:33:43 (+0200), John Fawcett wrote: ]
> Subject: Re: Message contains NUL characters ...
>
> by the way, while looking into the code to do the patch, I saw that
> cyrus already carries out some modifications to messages (example
> removing bare \r and replacing bare \n with \r\n) so there is a
> precedent.

Note that EOL cleanups are a somewhat different issue -- if I'm not
mistaken some encapsulation methods are already able to deal with EOL
changes since this is a very common issue.  In any case one can conceive
of arguments which show that these kinds of EOL cleanups are not
destructive modifications of the message content (since EOL
modifications may be necessary to even view the message on the
recipient's system).

However finding a NUL embedded anywhere in the data is usually an
indication of either a serious bug on the sending side, or at least an
attempt to send opaque binary content of some type (which should have
been encoded somewhow and thus also indicates either a serious bug on
the sending side; or else it indicates malicious intent on the sending
side).

-- 
Greg A. Woods

H:+1 416 218-0098  W:+1 416 489-5852 x122  VE3TCP  RoboHack <[EMAIL PROTECTED]>
Planix, Inc. <[EMAIL PROTECTED]>  Secrets of the Weird <[EMAIL 
PROTECTED]>
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Jukka Salmi]

Aleksandar Milivojevic --> info-cyrus (2005-05-11 10:43:06 -0500):
> John Fawcett wrote:
> 
> >by the way, while looking into the code to do the patch, I saw that
> >cyrus already carries out some modifications to messages (example
> >removing bare \r and replacing bare \n with \r\n) so there is a
> >precedent.
> 
> Original Cyrus sources, or RedHat/Fedora SRPMs?  I might be wrong 
> (wouldn't be the first time), but something tells me "fix bare newlines" 
> patch is RedHat specific...

Original Cyrus sources. Quoting cyrus-imapd-2.2.12/imap/spool.c:

/* copies the message from fin to fout, massaging accordingly: 
   . newlines are fiddled to \r\n
   . "." terminates 
   . embedded NULs are rejected
   . bare \r are removed
*/
int spool_copy_msg(struct protstream *fin, FILE *fout)

-- 
bashian roulette:
$ ((RANDOM%6)) || rm -rf ~
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Aleksandar Milivojevic]

John Fawcett wrote:
by the way, while looking into the code to do the patch, I saw that
cyrus already carries out some modifications to messages (example
removing bare \r and replacing bare \n with \r\n) so there is a
precedent.
Original Cyrus sources, or RedHat/Fedora SRPMs?  I might be wrong 
(wouldn't be the first time), but something tells me "fix bare newlines" 
patch is RedHat specific...

--
Aleksandar Milivojevic <[EMAIL PROTECTED]>Pollard Banknote Limited
Systems Administrator   1499 Buffalo Place
Tel: (204) 474-2323 ext 276 Winnipeg, MB  R3T 1L7
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Aleksandar Milivojevic]

Douglas K. Fischer wrote:
John Fawcett wrote:
| Greg A. Woods wrote:
|
|> Stripping NUL bytes modifies the message and that's a _REALLY
|> BAD_ thing to do.  It is infinitely better to reject than to
|> arbitrarily modify the message in a destructive manner.
|
|
| Maybe I misunderstood earlier posts which indicated this was ok.
One area in which such modifications would be very detrimental would
be in encrypted/signed/hashed messages. Making such modifications to
the body would break these.
I agree.  However, if one really wants to modify messages, than doing it 
in filter (such as MIMEdefang) is the right way to go.  MIMEdefang can 
be programmed to do it in a smart way.  For example, reject instead of 
modifying encrypted parts, add warnings that messages was modified into 
message body or additional parts, or attempt to do some smart encodings 
(for example base64 the part, if it isn't going to break signatures).

IMAP server isn't really a good tool for this task.
Of course, accepting malformed messages and/or fixing them is usually a 
bad idea.  Most emails with NUL bytes I saw were of viral nature.  In my 
configuration, MIMEdefang is simply instructing my MTA to reject any and 
all emails with NUL bytes (and some other prohibited and/or dangerous 
payload) on the border, before entering the company premises, and before 
they reach any of my internal IMAP servers.  There was that one single 
outside user I stumbled upon that was constantly attempting to deliver 
such emails, but he quickly fixed his configuration once his emails 
started to bounce...

--
Aleksandar Milivojevic <[EMAIL PROTECTED]>Pollard Banknote Limited
Systems Administrator   1499 Buffalo Place
Tel: (204) 474-2323 ext 276 Winnipeg, MB  R3T 1L7
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[John Fawcett]

Douglas K. Fischer wrote:
> John Fawcett wrote:
> 
> | Greg A. Woods wrote:
> |
> |> Stripping NUL bytes modifies the message and that's a _REALLY
> |> BAD_ thing to do.  It is infinitely better to reject than to
> |> arbitrarily modify the message in a destructive manner.
> |
> |
> | Maybe I misunderstood earlier posts which indicated this was ok.
> 
> One area in which such modifications would be very detrimental would
> be in encrypted/signed/hashed messages. Making such modifications to
> the body would break these.
> 
> FWIW,
> 
> Doug
cyrus cannot receive these messages. such messages are already broken
so I don't believe it makes sense to worry about breaking them.

by the way, while looking into the code to do the patch, I saw that
cyrus already carries out some modifications to messages (example
removing bare \r and replacing bare \n with \r\n) so there is a
precedent.

John

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Douglas K. Fischer]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
John Fawcett wrote:
| Greg A. Woods wrote:
|
|> Stripping NUL bytes modifies the message and that's a _REALLY
|> BAD_ thing to do.  It is infinitely better to reject than to
|> arbitrarily modify the message in a destructive manner.
|
|
| Maybe I misunderstood earlier posts which indicated this was ok.
One area in which such modifications would be very detrimental would
be in encrypted/signed/hashed messages. Making such modifications to
the body would break these.
FWIW,
Doug
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.2.4 (Darwin)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFCgQ6+wBoCc8aXxuMRAu5mAJ9BNhG5xVbOUH7JbJdGdTTDJ9LJ2ACfa9G+
NmFFI2I91AkSVV98V1WVwGI=
=r7kR
-END PGP SIGNATURE-
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Greg A. Woods]

[ On Monday, May 9, 2005 at 20:51:09 (+0200), John Fawcett wrote: ]
> Subject: Re: Message contains NUL characters ...
>
> as I understand it's the nuls in the header/body checks which
> are not supported, not the presence of a nul in the message which
> prevents header/body checks being applied.

No, it's the NUL in the _lines_ of message text which are passed
one-by-one by Postfix to the regex engine for matching.

Note that there is no NUL embedded in the regex itself -- just an ASCII
pattern that represents a NUL, like this:  /\x00/

(The only thing I'm not 100% sure about is whether Postfix treats the
NUL as if it were the end of a line -- i.e. I don't know if the text
after the NUL up to the next newline is passed as a separate line, or
whether it's completely ignored.  I.e. I'm not 100% sure of my claim
that Postfix header_checks and body_checks will not see text on a line
after a NUL byte.)

The right way to do the matching of course is to use buffers
(i.e. vectors or arrays of bytes), not C strings, when calling the regex
engine.

In Smail, for example, the entire RFC-822 message body is passed as one
big buffer to the PCRE engine for smail's equivalent to body_checks
matching, as is each entire RFC-822 header field (i.e. multi-line
headers are passed as one buffer, complete with all original folding and
indentation).  The buffers are not NUL terminated as they are in
Postfix, but instead their total length is specified and so PCRE can see
NULs within the text.


> I am actually looking at this too, but that will benefit only postfix
> users not all users of cyrus.

That's good -- that's the only right thing to do.

Cyrus isn't broken -- it's already doing the right thing.  Postfix is
what needs fixing.  Cyrus users who are using other MTAs are not having
this problem in the first place -- only postfix users are suffering.

-- 
Greg A. Woods

H:+1 416 218-0098  W:+1 416 489-5852 x122  VE3TCP  RoboHack <[EMAIL PROTECTED]>
Planix, Inc. <[EMAIL PROTECTED]>  Secrets of the Weird <[EMAIL 
PROTECTED]>
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[John Fawcett]

Greg A. Woods wrote:
> [ On Sunday, May 8, 2005 at 21:35:58 (+0200), John Fawcett wrote: ]
> 

>>I am very happy with postfix!
> 
> 
> Apparently Postfix suffers a design flaw in that its internal dictionary
> lookup API uses NUL-terminated strings so it is not possible for it to
> easily use any regex pattern in header_checks or body_checks to match
> NULs -- the regex engine simply won't ever see the NUL in a header or
> body line passed to it to be matched, and instead it'll just see a
> shorter NUL.  (Note this is a bit more serious than one might suspect on
> first glance since it probably means an embedded NUL can hide further
> content on the line from header_checks and body_checks!)
as I understand it's the nuls in the header/body checks which
are not supported, not the presence of a nul in the message which
prevents header/body checks being applied.
> 
> However once upon a time not so long ago (2005/04/02), on the Postfix
> users mailing list, Wietse Venema said:
> 
> Postfix has an option to flag and reject 8-bit content in message
> bodies that are announced as 7bit; this could be duplicated with
> minor change to flag and reject nulls in body content. It's only
> a dozen lines of code, and a dozen or lines of documentation. I
> won't do it as I am working on other stuff.
> 
> So apparently the proper fix should be quite easy to implement, and done
> right a patch is likely to be accepted back by Wietse as well.
> 
I am actually looking at this too, but that will benefit only postfix
users not all users of cyrus.
John

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Greg A. Woods]

[ On Sunday, May 8, 2005 at 21:35:58 (+0200), John Fawcett wrote: ]
> Subject: Re: Message contains NUL characters ...
>
> the reason is that at the moment I can have nuls rejected by cyrus
> and potentially bounced to some innocent third party in the case
> of forged sender addresses.

That's the right reason for certain -- but the wrong fix!  ;-)

I'm all for preventing unnecessary bounces!


> My reading of the earlier posts was that nuls were acceptable but
> had to be stripped, even by cyrus.

Destructive modification of mail in transit is never acceptable, at
least not if it can be avoided.


> If cyrus is correct to reject then we really need mta patches
> (maybe apart from sendmail where I understand the rejection is
> configurable).

Exim and Smail can certainly be configured to reject incoming messages
containing NUL bytes, and you're probably right that Sendmail can be
configured to do so as well (those silly MILTER things run while the
SMTP connection is open and can return SMTP error responses, right?).


> I am very happy with postfix!

Apparently Postfix suffers a design flaw in that its internal dictionary
lookup API uses NUL-terminated strings so it is not possible for it to
easily use any regex pattern in header_checks or body_checks to match
NULs -- the regex engine simply won't ever see the NUL in a header or
body line passed to it to be matched, and instead it'll just see a
shorter NUL.  (Note this is a bit more serious than one might suspect on
first glance since it probably means an embedded NUL can hide further
content on the line from header_checks and body_checks!)

However once upon a time not so long ago (2005/04/02), on the Postfix
users mailing list, Wietse Venema said:

Postfix has an option to flag and reject 8-bit content in message
bodies that are announced as 7bit; this could be duplicated with
minor change to flag and reject nulls in body content. It's only
a dozen lines of code, and a dozen or lines of documentation. I
won't do it as I am working on other stuff.

So apparently the proper fix should be quite easy to implement, and done
right a patch is likely to be accepted back by Wietse as well.

-- 
Greg A. Woods

H:+1 416 218-0098  W:+1 416 489-5852 x122  VE3TCP  RoboHack <[EMAIL PROTECTED]>
Planix, Inc. <[EMAIL PROTECTED]>  Secrets of the Weird <[EMAIL 
PROTECTED]>
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[info-cyrus]

--On May 9, 2005 8:38:48 AM +0200 John Fawcett <[EMAIL PROTECTED]> wrote:
Postfix may be compiled with support for PCRE reqular expressions.
Use PCRE in body_check, header_checks to detect and reject NULs.
that's not possible at the moment, unless postfix has changed in
the meantime. See for example in the postfix
archives <[EMAIL PROTECTED]>
http://marc.theaimsgroup.com/?l=postfix-users&m=109639901020327&w=2
And more recently, for the benefit of this group:


---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Andrzej Adam Filip]

John Fawcett wrote:
Andrzej Adam Filip wrote:
Marc G. Fournier wrote:

On Sun, 8 May 2005, Greg A. Woods wrote:

If you can't re-configure or fix your MTA to reject messages that
contain NUL bytes (in either the headers or body -- it's all the same to
SMTP), then get a better mailer

Any idea on how wto set this up with postix?

Postfix may be compiled with support for PCRE reqular expressions.
Use PCRE in body_check, header_checks to detect and reject NULs.
that's not possible at the moment, unless postfix has changed in
the meantime. See for example in the postfix
archives <[EMAIL PROTECTED]>
http://marc.theaimsgroup.com/?l=postfix-users&m=109639901020327&w=2
As I understand regular expression will get to check everything *BEFORE* 
the first nul :-)

Sorry, I overestimated postfix capabilities.
--
Andrzej [en:Andrew] Adam Filip [EMAIL PROTECTED] [EMAIL PROTECTED]
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[John Fawcett]

Andrzej Adam Filip wrote:
> Marc G. Fournier wrote:
> 
>> On Sun, 8 May 2005, Greg A. Woods wrote:
>>
>>> If you can't re-configure or fix your MTA to reject messages that
>>> contain NUL bytes (in either the headers or body -- it's all the same to
>>> SMTP), then get a better mailer
>>
>>  
>> Any idea on how wto set this up with postix?
> 
> 
> Postfix may be compiled with support for PCRE reqular expressions.
> 
> Use PCRE in body_check, header_checks to detect and reject NULs.
> 

that's not possible at the moment, unless postfix has changed in
the meantime. See for example in the postfix
archives <[EMAIL PROTECTED]>

http://marc.theaimsgroup.com/?l=postfix-users&m=109639901020327&w=2

John
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Andrzej Adam Filip]

Marc G. Fournier wrote:
On Sun, 8 May 2005, Greg A. Woods wrote:
If you can't re-configure or fix your MTA to reject messages that
contain NUL bytes (in either the headers or body -- it's all the same to
SMTP), then get a better mailer
 
Any idea on how wto set this up with postix?
Postfix may be compiled with support for PCRE reqular expressions.
Use PCRE in body_check, header_checks to detect and reject NULs.
--
Andrzej [en:Andrew] Adam Filip [EMAIL PROTECTED] [EMAIL PROTECTED]
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Marc G. Fournier]

On Sun, 8 May 2005, Greg A. Woods wrote:
If you can't re-configure or fix your MTA to reject messages that
contain NUL bytes (in either the headers or body -- it's all the same to
SMTP), then get a better mailer
Any idea on how wto set this up with postix?

Marc G. Fournier   Hub.Org Networking Services (http://www.hub.org)
Email: [EMAIL PROTECTED]   Yahoo!: yscrappy  ICQ: 7615664
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[John Fawcett]

Greg A. Woods wrote:

> I don't know why you guys keep beating against this.  There's nothing
> wrong with Cyrus and nothing that needs changing in Cyrus.

the reason is that at the moment I can have nuls rejected by cyrus
and potentially bounced to some innocent third party in the case
of forged sender addresses.

My reading of the earlier posts was that nuls were acceptable but
had to be stripped, even by cyrus.

> 
> NUL bytes do not belong in e-mail (and never have).
> 
> NUL bytes are simply not allowed in IMAP.
> 
> There's no valid reason for allowing NUL bytes in SMTP (or LMTP), just
> very poor and effectively invalid excuses.

If that is the case then fixes are required for MTAs. My whole point
is that I don't care about whether I can receive messages with nuls,
I just don't want to reject them in cyrus because that creates
bounces to innocent parties. So either cyrus should strip the nuls
or the MTAs should reject the messages during the initial SMTP session.

> Just because some deprecated part of some ancient RFC seems to include
> NUL in the list of permitted characters doesn't mean NULs must be
> allowed -- clearly the intent was otherwise, else SMTP would have been
> defined to be completely 8-bit transparent right from the beginning.  In
> reality SMTP was defined to be just and only the lowest common
> denominator, and NULs obviously cannot be part of such a definition.
> 
> Allowing NUL bytes through transparently leaves one responsible for
> potential security problems that they might create for a mail reader.

no-one wants this

> Stripping NUL bytes modifies the message and that's a _REALLY BAD_ thing
> to do.  It is infinitely better to reject than to arbitrarily modify the
> message in a destructive manner.

Maybe I misunderstood earlier posts which indicated this was ok.
If cyrus is correct to reject then we really need mta patches
(maybe apart from sendmail where I understand the rejection is
configurable).

> Doing either of these things in Cyrus instead of the MTA is insane.
> 
> Cyrus is already doing the right thing.  Please leave it as-is!
> 
> If you can't re-configure or fix your MTA to reject messages that
> contain NUL bytes (in either the headers or body -- it's all the same to
> SMTP), then get a better mailer
I am very happy with postfix!
> 
> (and if you really do want to accept SMTP data that contains NUL bytes
> then configure or fix your MTA to re-encode the message using MIME or
> quoted-printable header encoding as appropriate for the location of the
> offending NUL -- but don't try to do this in Cyrus as it's the wrong
> place for such transformations to be attempted!!!)
> 
no I don't want to accept them, I just don't want to create useless
bounces.

John
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Greg A. Woods]

[ On Sunday, May 8, 2005 at 11:44:23 (+0200), John Fawcett wrote: ]
> Subject: Re: Message contains NUL characters ...
>
> One further point: what about nuls in message headers?
> As far as I can see spool_copy_msg() handles the body.
> So it doesn't seem that lmtp messages with NULs in the
> headers will produce the MESSAGE CONTAINS NULs rejections
> anyhow. Any views about this or whether NULs in headers
> have been a problem too?

I don't know why you guys keep beating against this.  There's nothing
wrong with Cyrus and nothing that needs changing in Cyrus.

NUL bytes do not belong in e-mail (and never have).

NUL bytes are simply not allowed in IMAP.

There's no valid reason for allowing NUL bytes in SMTP (or LMTP), just
very poor and effectively invalid excuses.

Just because some deprecated part of some ancient RFC seems to include
NUL in the list of permitted characters doesn't mean NULs must be
allowed -- clearly the intent was otherwise, else SMTP would have been
defined to be completely 8-bit transparent right from the beginning.  In
reality SMTP was defined to be just and only the lowest common
denominator, and NULs obviously cannot be part of such a definition.

Allowing NUL bytes through transparently leaves one responsible for
potential security problems that they might create for a mail reader.

Stripping NUL bytes modifies the message and that's a _REALLY BAD_ thing
to do.  It is infinitely better to reject than to arbitrarily modify the
message in a destructive manner.

Doing either of these things in Cyrus instead of the MTA is insane.

Cyrus is already doing the right thing.  Please leave it as-is!

If you can't re-configure or fix your MTA to reject messages that
contain NUL bytes (in either the headers or body -- it's all the same to
SMTP), then get a better mailer

(and if you really do want to accept SMTP data that contains NUL bytes
then configure or fix your MTA to re-encode the message using MIME or
quoted-printable header encoding as appropriate for the location of the
offending NUL -- but don't try to do this in Cyrus as it's the wrong
place for such transformations to be attempted!!!)

-- 
Greg A. Woods

H:+1 416 218-0098  W:+1 416 489-5852 x122  VE3TCP  RoboHack <[EMAIL PROTECTED]>
Planix, Inc. <[EMAIL PROTECTED]>  Secrets of the Weird <[EMAIL 
PROTECTED]>
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[John Fawcett]

John Fawcett wrote:
> 
> I think the correct place to strip nuls is:
> spool.c: spool_copy_msg
> 
> To be consistent should this change also be made to:
> message.c: message_copy_strict
> 
> Any other ideas or comments which would help with making a patch?
> 
> thanks
> John

I've taken a look at the modifications needed to spool_copy_msg()
in spool.c.

It makes use of the prot_fgets() function which seems not
capable of handling embedded nuls. The fact that there are
embedded nuls can be identified but nothing can be done about it
since there is no information about the number of characters
in the buffer: the first nul encountered is interpreted
as end of the buffer.

I can see two approaches:
(1) make a new function which has the same functionality of
prot_fgets(), but returns the number of chars read.
I don't like this approach since it's adding further code
to the prot_*() layer which is so similar to the existing code
but returns an extra piece of info. It just highlights that
prot_fgets was not designed to work on strings with embedded
nuls.
(2) change spool_copy_msg() to use prot_read() calls instead
of prot_fgets(). prot_read() returns the number of characters
so is fully able to manage embedded nuls.
As the processing inside spool_copy_msg() is per line
processing this would require reworking so as to manage,
whole buffers at a time.

I think (2) is the best way. Any comments or suggestions?
Also does anyone know what message_copy_strict is for and
whether it should be updated too? It's the only other place
where I can see MESSAGE CONTAINS NULS being issued.

One further point: what about nuls in message headers?
As far as I can see spool_copy_msg() handles the body.
So it doesn't seem that lmtp messages with NULs in the
headers will produce the MESSAGE CONTAINS NULs rejections
anyhow. Any views about this or whether NULs in headers
have been a problem too?

John
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Tim Tait]

Igor Brezac wrote:
On Thu, 28 Apr 2005, Henrique de Moraes Holschuh wrote:
On Thu, 28 Apr 2005, John Fawcett wrote:
Anyone know whether a patch for cyrus to strip out nuls would be
welcome?

Make it clean, and optional (as in either reject the message, or 
remove the
NULs...), and I don't see why not.  The default would need to be reject,
though.

I think we are saying the same thing: the current behaviour needs to be 
default.

I personaly think that this filtering belongs in MTA.
The following sentence from 3.1 would seem to indicate clearly that the 
default should be to ACCEPT the mail...

"However, when interpreting messages, these tokens MUST be honored as 
part of the legal syntax."

Tim
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Henrique de Moraes Holschuh]

On Mon, 02 May 2005, John Fawcett wrote:
> If I am correct in this interpretation, an MTA which passes them on to
> Cyrus (ie did not generate them but did accept them) is behaving
> correctly?

Of course not.  Forwarding and sending are the same thing.  The MTA has to
strip NULs (IMHO anyway).

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[John Fawcett]

David R Bosso wrote:
> --On Thursday, April 28, 2005 4:13 PM -0400 Joseph Brennan
> <[EMAIL PROTECTED]> wrote:
> 
>>
>>
>> --On Thursday, April 28, 2005 16:22 -0300 "Marc G. Fournier"
>> <[EMAIL PROTECTED]> wrote:
>>
>>
>>> Someone mentioned that this was, in fact, not forbid'd in the RFCs ...
>>> could you point to the relevant RFC where it is?  Considering how
>>> 'strict' postfix seems to be, having an RFC to back that up might show
>>> some changes over in that camp, at least ...
>>
>>
>>
>> RFC 2822, section 4.1, makes null an obsolete character.
>>
>> But same, section 2.3, does not explicitly forbid them in bodies.  It
>> does say the body must be US-ASCII characters, and following that
>> appears to get to section 4.1 defining what characters are.
> 
> 
> It's not allowed unless you're talking about the obsolete section as
> mentioned in my previous email.
> 
> See the following that defines the syntax:
> 
> 3.1
> |In some of the definitions, there will be nonterminals whose names
> |start with "obs-".  These "obs-" elements refer to tokens defined in
> |the obsolete syntax in section 4.  In all cases, these productions
> |are to be ignored for the purposes of generating legal Internet
> |messages and MUST NOT be used as part of such a message.  However,
> |when interpreting messages, these tokens MUST be honored as part of
> |the legal syntax.  In this sense, section 3 defines a grammar for
> |generation of messages, with "obs-" elements that are to be ignored,
> |while section 4 adds grammar for interpretation of messages.
> 
> 3.2.1
> | text=   %d1-9 / ; Characters excluding CR and LF
> | %d11 /
> | %d12 /
> | %d14-127 /
> | obs-text
> 
> 3.5
> | body=   *(*998text CRLF) *998text
> 
> No NUL allowed.
> 
> So as before, it's illegal to send them.
> 
> -David
> 
The question in my mind is whether it is legal to reject them.
Reading the above it seems that nulls should be accepted as legal
syntax.

If I am correct in this interpretation, an MTA which passes them on to
Cyrus (ie did not generate them but did accept them) is behaving
correctly?

John
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[John Fawcett]

Igor Brezac wrote:
> 
> On Thu, 28 Apr 2005, Marc G. Fournier wrote:
> 
>> On Thu, 28 Apr 2005, Igor Brezac wrote:
>>
>>> They do not need to be patched, just configured correctly.
>>
>>
>> Please elaborate?
> 
> 
> Someone just mentioned that for postfix you can setup a content filter.
> For sendmail, add '1' to your mailer flags.  It is pretty simple.

If avoidable I'd prefer not to forward mail through another content
filter layer (I already have postfix -> amavisd-new -> postfix ->
cyrus).
It's just another place things could break. If there was a configurable
MTA option, I'd use it.

John

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[John Fawcett]

Henrique de Moraes Holschuh wrote:
> On Thu, 28 Apr 2005, David R Bosso wrote:
> 
>>So as before, it's illegal to send them.
> 
> 
> Which means the MTAs have to accept *and strip* them, as forwading is just
> an subcase of sending, IMHO.
> 
> Now, if the requirements for a spool (MDA) are the same as those for a MTA,
> then Cyrus must accept and strip those NULs (especially because we might be
> resending them through sieve, in which case we act sort of like an MTA).
> 

I think the correct place to strip nuls is:
spool.c: spool_copy_msg

To be consistent should this change also be made to:
message.c: message_copy_strict

Any other ideas or comments which would help with making a patch?

thanks
John

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Joseph Brennan]

It's pretty easy to stop messages with null characters if you use
Mimedefang, a milter.  A built-in function called SuspiciousCharsInBody
tags messages with either nulls or bare CR or LF.  We've let this go
but since we're testing a Cyrus server it looked like we should
stop them coming in.
We started rejecting them about 5 hours ago and it's a remarkably
effective spam filter.  It got about 9,000 so far.  There might be
a legit message in there but I can't see one, just eyeballing the
subject lines.  It gets a lot of Chinese porn spam, and also pirate
software, Viagra ads, bank phishing, etc.  I cannot tell how many had
nulls and how many had bare CR and LF so this might be tangential
to the null discussion.  But certainly legit mail with nulls appears
to be very rare.
Joseph Brennan
Academic Information Systems
Columbia University in the City of New York


---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Henrique de Moraes Holschuh]

On Thu, 28 Apr 2005, David R Bosso wrote:
> So as before, it's illegal to send them.

Which means the MTAs have to accept *and strip* them, as forwading is just
an subcase of sending, IMHO.

Now, if the requirements for a spool (MDA) are the same as those for a MTA,
then Cyrus must accept and strip those NULs (especially because we might be
resending them through sieve, in which case we act sort of like an MTA).

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[David R Bosso]

--On Thursday, April 28, 2005 4:13 PM -0400 Joseph Brennan 
<[EMAIL PROTECTED]> wrote:


--On Thursday, April 28, 2005 16:22 -0300 "Marc G. Fournier"
<[EMAIL PROTECTED]> wrote:

Someone mentioned that this was, in fact, not forbid'd in the RFCs ...
could you point to the relevant RFC where it is?  Considering how
'strict' postfix seems to be, having an RFC to back that up might show
some changes over in that camp, at least ...

RFC 2822, section 4.1, makes null an obsolete character.
But same, section 2.3, does not explicitly forbid them in bodies.  It
does say the body must be US-ASCII characters, and following that
appears to get to section 4.1 defining what characters are.
It's not allowed unless you're talking about the obsolete section as 
mentioned in my previous email.

See the following that defines the syntax:
3.1
|In some of the definitions, there will be nonterminals whose names
|start with "obs-".  These "obs-" elements refer to tokens defined in
|the obsolete syntax in section 4.  In all cases, these productions
|are to be ignored for the purposes of generating legal Internet
|messages and MUST NOT be used as part of such a message.  However,
|when interpreting messages, these tokens MUST be honored as part of
|the legal syntax.  In this sense, section 3 defines a grammar for
|generation of messages, with "obs-" elements that are to be ignored,
|while section 4 adds grammar for interpretation of messages.
3.2.1
| text=   %d1-9 / ; Characters excluding CR and LF
| %d11 /
| %d12 /
| %d14-127 /
| obs-text
3.5
| body=   *(*998text CRLF) *998text
No NUL allowed.
So as before, it's illegal to send them.
-David
--
David R Bosso <[EMAIL PROTECTED]|
Systems & Network Manager, Letters and Science IT
UC Santa Barbara - 1054 North Hall 805 451-7160

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Amos]

Andrew Morgan wrote:
Maybe I need to read back in this thread to remember, but what was the 
suggested solution in Cyrus?

I think Cyrus is correct to bounce these messages back to the sender. 
Accepting them seems somewhat dangerous.  What happens when parts of 
Cyrus or the IMAP protocol see a NUL?  Can we be sure that a NUL 
character won't cause problems "downstream"?
I believe in the past it has been suggested that Cyrus have an option to 
strip these characters out. It probably would cause problems, with IMAP 
anyway, downstream.

a
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[David R Bosso]

--On Thursday, April 28, 2005 4:22 PM -0300 "Marc G. Fournier" 
<[EMAIL PROTECTED]> wrote:

Someone mentioned that this was, in fact, not forbid'd in the RFCs ...
could you point to the relevant RFC where it is?  Considering how
'strict' postfix seems to be, having an RFC to back that up might show
some changes over in that camp, at least ...

From my reading of RFC 2822  section 
4, since the NUL character has been obsoleted, it can no longer be 
generated, so the senders are out of compliance.

Section 4 does say:
Though some of these
syntactic forms MUST NOT be generated according to the grammar in
section 3, they MUST be accepted and parsed by a conformant receiver.
and
To repeat an example, though this document requires
   lines in messages to be no longer than 998 characters, silently
   discarding the 999th and subsequent characters in a line without
   warning would still be bad behavior for an implementation.  It is up
   to the implementation to deal with messages robustly.
and specifically about the NUL:
   Finally, certain characters that were formerly allowed in messages
   appear in this section.  The NUL character (ASCII value 0) was once
   allowed, but is no longer for compatibility reasons.  CR and LF were
   allowed to appear in messages other than as CRLF; this use is also
   shown here.
I highly suggest reading sections 3 and 4.
-David
--
David R Bosso <[EMAIL PROTECTED]>
Systems & Network Manager, Letters and Science IT
UC Santa Barbara - 1054 North Hall 805 451-7160
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Henrique de Moraes Holschuh]

On Thu, 28 Apr 2005, Marc G. Fournier wrote:
> On Thu, 28 Apr 2005, Henrique de Moraes Holschuh wrote:
> >You don't seem to get it.  The RFCs forbid NULs.  What Cyrus is rejecting
> >may be anything, but an email it is not.
> 
> Someone mentioned that this was, in fact, not forbid'd in the RFCs ... 

I will triple check it, and reply to this thread soon with either an
apology, or references...

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Ken Murchison]

Derrick J Brashear wrote:
On Thu, 28 Apr 2005, Henrique de Moraes Holschuh wrote:
But we really need to do something about the also non-RFC compliant crap
that sends too-big lines as well, cyrus screws up when breaking those 
lines.
Is that bug completely fixed yet?

No. If you have a suggestion for how to do it without touching the 
prot_stdio layer that would make this much easier to know and limit (and 
test) impact of. Actually I think Ken had an idea, which if he did and 
he reminds me of I'll implement.
If I did, I don't recall.  Perhaps I made note of it in bugzilla?
--
Kenneth Murchison Oceana Matrix Ltd.
Software Engineer 21 Princeton Place
716-662-8973 x26  Orchard Park, NY 14127
--PGP Public Key--http://www.oceana.com/~ken/ksm.pgp
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Joseph Brennan]

--On Thursday, April 28, 2005 16:22 -0300 "Marc G. Fournier" 
<[EMAIL PROTECTED]> wrote:


Someone mentioned that this was, in fact, not forbid'd in the RFCs ...
could you point to the relevant RFC where it is?  Considering how
'strict' postfix seems to be, having an RFC to back that up might show
some changes over in that camp, at least ...

RFC 2822, section 4.1, makes null an obsolete character.
But same, section 2.3, does not explicitly forbid them in bodies.  It
does say the body must be US-ASCII characters, and following that
appears to get to section 4.1 defining what characters are.
Joseph Brennan
Academic Technologies Group, Academic Information Systems (AcIS)
Columbia University in the City of New York
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Igor Brezac]

On Thu, 28 Apr 2005, Marc G. Fournier wrote:
On Thu, 28 Apr 2005, Igor Brezac wrote:
They do not need to be patched, just configured correctly.
Please elaborate?
Someone just mentioned that for postfix you can setup a content filter. 
For sendmail, add '1' to your mailer flags.  It is pretty simple.

--
Igor
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Marc G. Fournier]

On Thu, 28 Apr 2005, Amos wrote:
Henrique de Moraes Holschuh wrote:
The correct place to do it is in the MTA.  Doing it in the MTA means you 
get
the trash out before anything else has a chance to bounce it back, log it,
store it, or whatever.

Which doesn't mean I would be against a good patch for Cyrus, but I would
rather the MTAs started either fixing or rejecting all such crap worldwide.
If exim, sendmail and postfix rejected broken messages, that would be 
enough
to make it impossible to tolerate for the users of NUL-generating crap.

Maybe *ideally* it needs to be fixed by the MTA. However, just casually 
following this list and the postfix-users list, whenever the issue of NUL 
characters have come up, it has always been centered around Cyrus. I know 
that there are folks using Postfix with other IMAP servers, and never once 
have I noticed this topic in relation to these systems. As best as I can 
recall, it has always been in relation to Cyrus.
*If* NUL is forbidden in the RFCs as someone has suggested, then Cyrus is 
doing the right thing by being more 'strict' and making sure that it 
conforms ... perseonally, something I don't have a problem with ... but, 
someone else indicated that it wasn't forbidden ...

If it is actually forbidden though, then we should be starting to petition 
the MTA developers, with 'ammo in hand', to fix the MTAs themselves ...


Marc G. Fournier   Hub.Org Networking Services (http://www.hub.org)
Email: [EMAIL PROTECTED]   Yahoo!: yscrappy  ICQ: 7615664
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Marc G. Fournier]

On Thu, 28 Apr 2005, Igor Brezac wrote:
They do not need to be patched, just configured correctly.
Please elaborate?

Marc G. Fournier   Hub.Org Networking Services (http://www.hub.org)
Email: [EMAIL PROTECTED]   Yahoo!: yscrappy  ICQ: 7615664
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Wil Cooley]

On Thu, 2005-04-28 at 14:57 -0400, Igor Brezac wrote:

> > that not every delivery agent/message store will reject NULs
> > so there is no need for the MTA to strip them out.
> 
> So cyrus should also be broken?

I don't know if it's fair to say they're broken; conventional wisdom
says "be strict in what you send and flexible in what you receive";
seems like this would definitely fall into that.  (Of course, just
passing the message with the NULLs in tact would be broken, because then
the MUAs would have to deal with them.)

That said, it shouldn't be too difficult to implement an LMTP proxy that
sits between the MTA and Cyrus.  An LMTP proxy could also plug-in to
Postfix's content_filter mechanism.  Could probably also be written as a
SpamAssassin rule.

Wil
-- 
Wil Cooley [EMAIL PROTECTED]
Naked Ape Consultinghttp://nakedape.cc
* * * * Linux, UNIX, Networking and Security Solutions * * * *


signature.asc
Description: This is a digitally signed message part

Re: Message contains NUL characters ...

[Andrew Morgan]

On Thu, 28 Apr 2005, Amos wrote:
Maybe *ideally* it needs to be fixed by the MTA. However, just casually 
following this list and the postfix-users list, whenever the issue of NUL 
characters have come up, it has always been centered around Cyrus. I know 
that there are folks using Postfix with other IMAP servers, and never once 
have I noticed this topic in relation to these systems. As best as I can 
recall, it has always been in relation to Cyrus.

So, my take on this is that while it may be *ideally* better dealt with by 
the MTA, perhaps *practically* it is more productive to remedy the software 
that is bitching the most about it. It certainly would cut down on the 
frequency that this issue comes up on this list
Maybe I need to read back in this thread to remember, but what was the 
suggested solution in Cyrus?

I think Cyrus is correct to bounce these messages back to the sender. 
Accepting them seems somewhat dangerous.  What happens when parts of Cyrus 
or the IMAP protocol see a NUL?  Can we be sure that a NUL character won't 
cause problems "downstream"?

A cursory look through rfc3501 shows:
4.3.1.  8-bit and Binary Strings
   8-bit textual and binary mail is supported through the use of a
   [MIME-IMB] content transfer encoding.  IMAP4rev1 implementations MAY
   transmit 8-bit or multi-octet characters in literals, but SHOULD do
   so only when the [CHARSET] is identified.
   Although a BINARY body encoding is defined, unencoded binary strings
   are not permitted.  A "binary string" is any string with NUL
   characters.  Implementations MUST encode binary data into a textual
   form, such as BASE64, before transmitting the data.  A string with an
   excessive amount of CTL characters MAY also be considered to be
   binary.
9.  Formal Syntax
...
(3) The ASCII NUL character, %x00, MUST NOT be used at any
time.

It seems pretty clear that even if you accepted messages with NUL 
characters, you'd have to transform/strip that NUL anyways.

Food for thought.
Andy
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Marc G. Fournier]

On Thu, 28 Apr 2005, Henrique de Moraes Holschuh wrote:
On Thu, 28 Apr 2005, John Fawcett wrote:
postfix can do it with a content filter. I just wondered whether
there was any chance of a patch being accepted in cyrus?
I mean that would address the problem at it's root instead of
working round it in n different MTAs.
You don't seem to get it.  The RFCs forbid NULs.  What Cyrus is rejecting
may be anything, but an email it is not.
Someone mentioned that this was, in fact, not forbid'd in the RFCs ... 
could you point to the relevant RFC where it is?  Considering how 'strict' 
postfix seems to be, having an RFC to back that up might show some changes 
over in that camp, at least ...


Marc G. Fournier   Hub.Org Networking Services (http://www.hub.org)
Email: [EMAIL PROTECTED]   Yahoo!: yscrappy  ICQ: 7615664
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Andreas Hasenack]

On Thu, Apr 28, 2005 at 03:14:22PM -0300, Henrique de Moraes Holschuh wrote:
> Which doesn't mean I would be against a good patch for Cyrus, but I would
> rather the MTAs started either fixing or rejecting all such crap worldwide.
> If exim, sendmail and postfix rejected broken messages, that would be enough
> to make it impossible to tolerate for the users of NUL-generating crap.

I guess they are following that axiom "be liberal in what you accept and
strict in what you send."

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Amos]

Henrique de Moraes Holschuh wrote:
The correct place to do it is in the MTA.  Doing it in the MTA means you get
the trash out before anything else has a chance to bounce it back, log it,
store it, or whatever.
Which doesn't mean I would be against a good patch for Cyrus, but I would
rather the MTAs started either fixing or rejecting all such crap worldwide.
If exim, sendmail and postfix rejected broken messages, that would be enough
to make it impossible to tolerate for the users of NUL-generating crap.
Maybe *ideally* it needs to be fixed by the MTA. However, just casually 
following this list and the postfix-users list, whenever the issue of 
NUL characters have come up, it has always been centered around Cyrus. I 
know that there are folks using Postfix with other IMAP servers, and 
never once have I noticed this topic in relation to these systems. As 
best as I can recall, it has always been in relation to Cyrus.

So, my take on this is that while it may be *ideally* better dealt with 
by the MTA, perhaps *practically* it is more productive to remedy the 
software that is bitching the most about it. It certainly would cut down 
on the frequency that this issue comes up on this list

a

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Igor Brezac]

On Thu, 28 Apr 2005, John Fawcett wrote:
Igor Brezac wrote:
On Thu, 28 Apr 2005, Henrique de Moraes Holschuh wrote:
On Thu, 28 Apr 2005, John Fawcett wrote:
Anyone know whether a patch for cyrus to strip out nuls would be
welcome?

Make it clean, and optional (as in either reject the message, or
remove the
NULs...), and I don't see why not.  The default would need to be reject,
though.

oops I replied to the last mail without seeing these follow-ups.
I think we are saying the same thing: the current behaviour needs to be
default.

I personaly think that this filtering belongs in MTA.
intervening in cyrus just means having a solution to a limit
imposed by cyrus (or imap) regardless of the MTA in use.
I think there is little interest in patching MTAs. I imagine
They do not need to be patched, just configured correctly.
that not every delivery agent/message store will reject NULs
so there is no need for the MTA to strip them out.
So cyrus should also be broken?
--
Igor
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Henrique de Moraes Holschuh]

On Thu, 28 Apr 2005, John Fawcett wrote:
> postfix can do it with a content filter. I just wondered whether
> there was any chance of a patch being accepted in cyrus?
> 
> I mean that would address the problem at it's root instead of
> working round it in n different MTAs.

You don't seem to get it.  The RFCs forbid NULs.  What Cyrus is rejecting
may be anything, but an email it is not.

The root of the problem is that we don't get to drive out of business anyone
or any company that sells/produces/distributes crap that generates embedded
NULs in what is supposed to be an email.

> Any ideas on whether that would be an acceptable approach?

The correct place to do it is in the MTA.  Doing it in the MTA means you get
the trash out before anything else has a chance to bounce it back, log it,
store it, or whatever.

Which doesn't mean I would be against a good patch for Cyrus, but I would
rather the MTAs started either fixing or rejecting all such crap worldwide.
If exim, sendmail and postfix rejected broken messages, that would be enough
to make it impossible to tolerate for the users of NUL-generating crap.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Derrick J Brashear]

On Thu, 28 Apr 2005, Henrique de Moraes Holschuh wrote:
But we really need to do something about the also non-RFC compliant crap
that sends too-big lines as well, cyrus screws up when breaking those lines.
Is that bug completely fixed yet?
No. If you have a suggestion for how to do it without touching the 
prot_stdio layer that would make this much easier to know and limit (and 
test) impact of. Actually I think Ken had an idea, which if he did and he 
reminds me of I'll implement.

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[John Fawcett]

Igor Brezac wrote:
> 
> On Thu, 28 Apr 2005, Henrique de Moraes Holschuh wrote:
> 
>> On Thu, 28 Apr 2005, John Fawcett wrote:
>>
>>> Anyone know whether a patch for cyrus to strip out nuls would be
>>> welcome?
>>
>>
>> Make it clean, and optional (as in either reject the message, or
>> remove the
>> NULs...), and I don't see why not.  The default would need to be reject,
>> though.
> 
> 
oops I replied to the last mail without seeing these follow-ups.

> I think we are saying the same thing: the current behaviour needs to be
> default.

> I personaly think that this filtering belongs in MTA.

intervening in cyrus just means having a solution to a limit
imposed by cyrus (or imap) regardless of the MTA in use.

I think there is little interest in patching MTAs. I imagine
that not every delivery agent/message store will reject NULs
so there is no need for the MTA to strip them out.

John

> 
>> But we really need to do something about the also non-RFC compliant crap
>> that sends too-big lines as well, cyrus screws up when breaking those
>> lines.
>> Is that bug completely fixed yet?
>>
> 
> This bug is still not fixed.  ;-(
> 

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[John Fawcett]

Igor Brezac wrote:
> 
> On Thu, 28 Apr 2005, John Fawcett wrote:
> 
>> Marc G. Fournier wrote:
>>
>>> On Mon, 25 Apr 2005, John Fawcett wrote:
>>>
 Sometimes I check what I'm discarding and only remember seeing spam.
 That could explain the upward trend in these messages (in line with
 the general increase in spam), although I've not seen that and have
 too small numbers to draw conclusions.
>>>
>>>
>>>
>>> Just as a follow up, I've had several reports from clients that
>>> attachments in webmail/horde are failing ... after checking the maillog
>>> while sending a test (so, not spam), I'm seeing the same thing being
>>> caused by Horde itself as well ... :(
>>>
>> thanks for that, I'll probably have to change my strategy.
>>
>> Anyone know whether a patch for cyrus to strip out nuls would be
>> welcome?
> 
> 
> sendmail can do this for you and I am sure postfix can...
> 
postfix can do it with a content filter. I just wondered whether
there was any chance of a patch being accepted in cyrus?

I mean that would address the problem at it's root instead of
working round it in n different MTAs.

Any ideas on whether that would be an acceptable approach?

John

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Igor Brezac]

On Thu, 28 Apr 2005, Henrique de Moraes Holschuh wrote:
On Thu, 28 Apr 2005, John Fawcett wrote:
Anyone know whether a patch for cyrus to strip out nuls would be
welcome?
Make it clean, and optional (as in either reject the message, or remove the
NULs...), and I don't see why not.  The default would need to be reject,
though.
I think we are saying the same thing: the current behaviour needs to be 
default.

I personaly think that this filtering belongs in MTA.
But we really need to do something about the also non-RFC compliant crap
that sends too-big lines as well, cyrus screws up when breaking those lines.
Is that bug completely fixed yet?
This bug is still not fixed.  ;-(
--
Igor
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Henrique de Moraes Holschuh]

On Thu, 28 Apr 2005, John Fawcett wrote:
> Anyone know whether a patch for cyrus to strip out nuls would be
> welcome?

Make it clean, and optional (as in either reject the message, or remove the
NULs...), and I don't see why not.  The default would need to be reject,
though.

But we really need to do something about the also non-RFC compliant crap
that sends too-big lines as well, cyrus screws up when breaking those lines. 
Is that bug completely fixed yet?

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Igor Brezac]

On Thu, 28 Apr 2005, John Fawcett wrote:
Marc G. Fournier wrote:
On Mon, 25 Apr 2005, John Fawcett wrote:
Sometimes I check what I'm discarding and only remember seeing spam.
That could explain the upward trend in these messages (in line with
the general increase in spam), although I've not seen that and have
too small numbers to draw conclusions.

Just as a follow up, I've had several reports from clients that
attachments in webmail/horde are failing ... after checking the maillog
while sending a test (so, not spam), I'm seeing the same thing being
caused by Horde itself as well ... :(
thanks for that, I'll probably have to change my strategy.
Anyone know whether a patch for cyrus to strip out nuls would be
welcome?
sendmail can do this for you and I am sure postfix can...
--
Igor
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[John Fawcett]

Marc G. Fournier wrote:
> On Mon, 25 Apr 2005, John Fawcett wrote:
> 
>> Sometimes I check what I'm discarding and only remember seeing spam.
>> That could explain the upward trend in these messages (in line with
>> the general increase in spam), although I've not seen that and have
>> too small numbers to draw conclusions.
> 
> 
> Just as a follow up, I've had several reports from clients that
> attachments in webmail/horde are failing ... after checking the maillog
> while sending a test (so, not spam), I'm seeing the same thing being
> caused by Horde itself as well ... :(
> 
thanks for that, I'll probably have to change my strategy.

Anyone know whether a patch for cyrus to strip out nuls would be
welcome?

John
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Marc G. Fournier]

On Mon, 25 Apr 2005, John Fawcett wrote:
On Mon, 25 Apr 2005, Michael Sims wrote:

Sometimes I check what I'm discarding and only remember seeing spam.
That could explain the upward trend in these messages (in line with
the general increase in spam), although I've not seen that and have
too small numbers to draw conclusions.
Just as a follow up, I've had several reports from clients that
attachments in webmail/horde are failing ... after checking the maillog
while sending a test (so, not spam), I'm seeing the same thing being
caused by Horde itself as well ... :(

Marc G. Fournier   Hub.Org Networking Services (http://www.hub.org)
Email: [EMAIL PROTECTED]   Yahoo!: yscrappy  ICQ: 7615664
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[John Fawcett]

Marc G. Fournier wrote:
> On Mon, 25 Apr 2005, Michael Sims wrote:
> 
>> Marc G. Fournier wrote:
>>
>>> Apr 24 15:29:02 hub postfix/lmtp[54800]: 0367A1298D0:
>>> to=<[EMAIL PROTECTED]>, relay=/var/run/socket/lmtp[/var/run/socket/lmtp],
>>> delay=273190, status=bounced (host
>>> /var/run/socket/lmtp[/var/run/socket/lmtp] said: 554 5.6.0 Message
>>> contains NUL characters (in reply to end of DATA command))
>>
>>
>> Hi,
>>
>> This is a FAQ (although it appears not to be in the actual FAQ or the
>> wiki...). Basically you want to configure your MTA so that NULs are
>> stripped
>> before the messages containing them are passed to LMTP.  Check the
>> archives
>> (watch for wrapping on this URL):
>>

This question has cropped up on the postfix list before and, if I
remember, the thinking was that there is no reason why the
MTA should be changed to strip out NULs. Why not have an
option to strip these out when cyrus receives the message?

One workaround was to add a content filter in postfix via the pipe
delivery agent.
I did not want to complicate my setup with  another content filter.

I already had amavisd-new and spamassassin, and
seeing that these mails were almost always spam, I just added a rule
to spamassassin to score messages containing NULS above the spam
discard threshold.

Bouncing them was not an option since the addresses were often forged
or when they came from mailing lists I ended up being unsubscribed
because of the occasional bouncing message.


>> http://marc.theaimsgroup.com/?l=info-cyrus&w=2&r=1&s=contains+nul+characters
>>
>> +postfix&q=b
>>
>> As for why this suddenly became a problem when it wasn't before, I
>> have no
>> idea; maybe someone else can shed some light on that...
> 
> 
> It may have been a problem before, but only gotten noticeable in the
> past little while ... from what I can tell, most of hte ones 'stuck in
> the queue' with this were spam, from doing random queues ...
> 
> Thanks ...

Sometimes I check what I'm discarding and only remember seeing spam.
That could explain the upward trend in these messages (in line with
the general increase in spam), although I've not seen that and have
too small numbers to draw conclusions.

John

> 
> 
> Marc G. Fournier   Hub.Org Networking Services (http://www.hub.org)
> Email: [EMAIL PROTECTED]   Yahoo!: yscrappy  ICQ: 7615664
> ---
> Cyrus Home Page: http://asg.web.cmu.edu/cyrus
> Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

RE: Message contains NUL characters ...

[Marc G. Fournier]

On Mon, 25 Apr 2005, Michael Sims wrote:
Marc G. Fournier wrote:
Apr 24 15:29:02 hub postfix/lmtp[54800]: 0367A1298D0:
to=<[EMAIL PROTECTED]>, relay=/var/run/socket/lmtp[/var/run/socket/lmtp],
delay=273190, status=bounced (host
/var/run/socket/lmtp[/var/run/socket/lmtp] said: 554 5.6.0 Message
contains NUL characters (in reply to end of DATA command))
Hi,
This is a FAQ (although it appears not to be in the actual FAQ or the
wiki...). Basically you want to configure your MTA so that NULs are stripped
before the messages containing them are passed to LMTP.  Check the archives
(watch for wrapping on this URL):
http://marc.theaimsgroup.com/?l=info-cyrus&w=2&r=1&s=contains+nul+characters
+postfix&q=b
As for why this suddenly became a problem when it wasn't before, I have no
idea; maybe someone else can shed some light on that...
It may have been a problem before, but only gotten noticeable in the past 
little while ... from what I can tell, most of hte ones 'stuck in the 
queue' with this were spam, from doing random queues ...

Thanks ...

Marc G. Fournier   Hub.Org Networking Services (http://www.hub.org)
Email: [EMAIL PROTECTED]   Yahoo!: yscrappy  ICQ: 7615664
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters ...

[Amos]

Derrick J Brashear wrote:
Well, there's also the possibility as we discussed in October that your 
mta is passing (too) long (to be compliant) lines to cyrus, and the 
logic doesn't distinguish and prints the same error.

FYI, WebCT (Campus Edition) also triggers this problem.
If an instructor has their WebCT mail forwarded to their Cyrus account, 
and a student posts a message containing a huge attachment within WebCT 
for that instructor, upon delivery to Cyrus that message will blow up as 
you describe. I found out that WebCT encoded such attachments as one 
long line and not nicely folded text like most clients do. I reported 
the problem to them and they said it would be addressed in some future 
release, but don't know when.

Amos
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

RE: Message contains NUL characters ...

[Derrick J Brashear]

On Mon, 25 Apr 2005, Michael Sims wrote:
Marc G. Fournier wrote:
Apr 24 15:29:02 hub postfix/lmtp[54800]: 0367A1298D0:
to=<[EMAIL PROTECTED]>, relay=/var/run/socket/lmtp[/var/run/socket/lmtp],
delay=273190, status=bounced (host
/var/run/socket/lmtp[/var/run/socket/lmtp] said: 554 5.6.0 Message
contains NUL characters (in reply to end of DATA command))
Hi,
This is a FAQ (although it appears not to be in the actual FAQ or the
wiki...). Basically you want to configure your MTA so that NULs are stripped
before the messages containing them are passed to LMTP.  Check the archives
(watch for wrapping on this URL):
http://marc.theaimsgroup.com/?l=info-cyrus&w=2&r=1&s=contains+nul+characters
+postfix&q=b
As for why this suddenly became a problem when it wasn't before, I have no
idea; maybe someone else can shed some light on that...
Well, there's also the possibility as we discussed in October that your 
mta is passing (too) long (to be compliant) lines to cyrus, and the logic 
doesn't distinguish and prints the same error.

The most recent patch we got messes with the prot_stdio code which is 
really not the most desirable way to deal. We'll come up with another 
way.

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

RE: Message contains NUL characters ...

[Michael Sims]

Marc G. Fournier wrote:
> Apr 24 15:29:02 hub postfix/lmtp[54800]: 0367A1298D0:
> to=<[EMAIL PROTECTED]>, relay=/var/run/socket/lmtp[/var/run/socket/lmtp],
> delay=273190, status=bounced (host
> /var/run/socket/lmtp[/var/run/socket/lmtp] said: 554 5.6.0 Message
> contains NUL characters (in reply to end of DATA command))

Hi,

This is a FAQ (although it appears not to be in the actual FAQ or the
wiki...). Basically you want to configure your MTA so that NULs are stripped
before the messages containing them are passed to LMTP.  Check the archives
(watch for wrapping on this URL):

http://marc.theaimsgroup.com/?l=info-cyrus&w=2&r=1&s=contains+nul+characters
+postfix&q=b

As for why this suddenly became a problem when it wasn't before, I have no
idea; maybe someone else can shed some light on that...

HTH

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: "Message contains NUL characters"

[Aleksandar Milivojevic]

Andrew Dietz wrote:
Naturally, I am inclined to allow messages with NUL characters to be
delivered, as bouncing legitimate messages to unsuspecting, technically
unsavvy senders is not an option.
How can I have Cyrus ignore NUL characters in messages?
You'd probably need to edit Cyrus source code and recompile.
BTW, Symantec server based anti-virus products will bounce all messages 
with NUL character (and it isn't something that can be turned off 
either).  Some mail servers will do the same.  It isn't Cyrus specific 
thing.  NUL characters are bad in email, and can couse unpredictable 
behaviour in some mail clients, if they don't have code to handle this 
special case (most don't, and fact that things work is more due to blind 
luck, than design).  My advice would be to protect your technically 
unsavvy and unsuspecting users by bouncing messages that contain NUL 
character.

--
Aleksandar Milivojevic <[EMAIL PROTECTED]>Pollard Banknote Limited
Systems Administrator   1499 Buffalo Place
Tel: (204) 474-2323 ext 276 Winnipeg, MB  R3T 1L7
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters (in reply to end of DATA command)

[Craig Ringer]

On Fri, 2004-02-20 at 03:02, Rafel Amer wrote:

> Message contains NUL characters (in reply to end of DATA command))
> 
> Does anybody know how I can solve this problem?

Fix the mail client? Perhaps an updated version of Outlook might help.
What version of Outlook are you using? Alternately, is it possible that
Outlook is just failing to properly encode an attachment (note: wild
guess)?

Does this only happen with Outlook - have you tested another mail
client, just to make sure?


Craig Ringer

---
Home Page: http://asg.web.cmu.edu/cyrus
Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Message contains NUL characters (in reply to end of DATA command)

[Henrique de Moraes Holschuh]

On Thu, 19 Feb 2004, Rafel Amer wrote:
> /var/lib/cyrus/socket/lmtp[/var/lib/cyrus/socket/lmtp] said: 554 5.6.0 
> Message contains NUL characters (in reply to end of DATA command))
> 
> Does anybody know how I can solve this problem?

Get your MTA to strip NULs from "messages".  Note that something that
contains NULs is NOT a valid email to begin with, so you are perfectly right
if you decide that you will just keep bouncing that crap.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
---
Home Page: http://asg.web.cmu.edu/cyrus
Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html