Re: APPEND vs RFC2822 vs STD0011

[John Alton Tamplin]

Edward Reid wrote:

Obviously there's a problem with the RFC in this case, in that it makes
a recommendation to the client but no recommendation or requirement for
the server.
But the RFC clearly says that the client is allowed to store a
non-RFC2822 message, if it has a valid reason. Nowhere do I see that it
says the client should deal with the server refusing to handle cases
which the RFC says the client is allowed to do.
 

Clearly a client is required to properly handle a NO or BAD response to 
an APPEND command.  Since (see below) a client is not permitted to send 
unencoded NUL characters, that would seem to be a violation of the IMAP 
protocol and therefore elicit either a NO (as the server can tell the 
error was in the message text) or a BAD (for a protocol error) tagged 
response.

Aside from the specific case of NUL characters, the client should be 
expected to properly handle a NO response if for whatever reason the 
server is unable to store the data.  The RFC specifies the behavior of 
both the client and the server, and saying the message SHOULD be in 
RFC2822 format means that the server can choose to relax that rule if it 
has a good reason just as well as it does for the client.  Certainly the 
server is not required to accept non-RFC2822 messages, so the client 
should be prepared to handle a refusal if it chooses to relax that 
requirement.

I tried to make it clear that I do not consider storing a non-RFC822
message to be a valid reason, in the RFC2119 sense, to violate the
"SHOULD". IMAP is designed for storing Internet email, and that
requires at minimum RFC822. (RFC733,  the RFC822 predecessor, is far to
old to consider here. RFC822 is over twenty years old; RFC2822 is only
two years old. We long ago reached that point where it's reasonable to
assume that all Internet email is RFC822-compliant, but we just are not
at the point where it's reasonable to assume that all Internet email is
RFC2822-compliant.)
 

RFC3501 specifically requires 2822 rather than 822, and even says that 
all references to 822 should be considered as 2822.  If a mail client 
claims to conform to RFC3501, then the mail messages it sends should 
conform to RFC2822 not RFC822.  If it wants to claim conformance only to 
an older IMAP RFC, that is fine.

Using null-terminated strings with data that might contain nulls is
problematic.
 

The data is not allowed to contain nulls -- from RFC3501, 4.3.1:

  Although a BINARY body encoding is defined, unencoded binary strings
  are not permitted.  A "binary string" is any string with NUL
  characters.  Implementations MUST encode binary data into a textual
  form, such as BASE64, before transmitting the data.  A string with an
  excessive amount of CTL characters MAY also be considered to be
  binary.
Note the use of MUST.  If Eudora or another mail client wants to send 
data containing NULs, it must encode it into another form before doing so.

You are basically saying that even if (emphasize "if") the code is
clearly wrong, that it won't be changed because it's too difficult to
write correct code? I don't follow this argument at all. This problem
with null-terminated strings has been widely known since long before
cyrus.
 

Yes, and since the IMAP spec specifically forbids the presence of 
unencode binary data (defined as strings containing the NUL character), 
it is perfectly reasonable to assume they don't exist.  Cyrus validates 
that the message does not violate the standard by including unencoded 
NULs, and rejects the message if it does.

Regarding bare newlines, RFC3501 2.2 states:

  All interactions transmitted by client and server are in the form of
  lines, that is, strings that end with a CRLF.  The protocol receiver
  of an IMAP4rev1 client or server is either reading a line, or is
  reading a sequence of octets with a known count followed by a line.
--

John A. Tamplin                               Unix System Administrator
Emory University, School of Public Health     +1 404/727-9931