Re: APPEND vs RFC2822 vs STD0011
At 12:28 PM -0400 7/9/03, Cyrus Daboo wrote: > On the NULL issue, IMAP does not allow bare NULLs in any data that >either > the server or client sends. If you check the formal syntax you will >see > that the 'literal' element used to send the message content in an >APPEND > explicitly excludes NULL as a valid character. Ah-ha. Excellent. Thanks, I can definitely run with that. > Bare CR or LF is another issue... That's the one that's left. BTW, in response to email off the list, I should make it clear that I'm only concerned with the body, not headers. At 01:35 PM -0400 7/9/03, John Alton Tamplin wrote: > Clearly a client is required to properly handle a NO or BAD response >to > an APPEND command. I agree. I have other issues with what Eudora considers "proper handling". They are saying that a dialog box and ceasing operation is proper handling, I say it's not. Them: "Where does it say that Eudora is an autoresponder?" Me: "Well, duh, in the User Manual." I did not mention this part because it's not an issue for this list. It's certainly going to be part of my next response to Qualcomm, but I wanted to gather information on the other issues as well. > The RFC specifies the behavior of > both the client and the server, and saying the message SHOULD be in > RFC2822 format means that the server can choose to relax that rule if >it > has a good reason just as well as it does for the client. What RFC3501 says is The APPEND command appends the literal argument as a new message to the end of the specified destination mailbox. This argument SHOULD be in the format of an [RFC-2822] message. Since it's the client that constructs the literal, it appears to me that the RFC is giving the client the choice. The language appears to specify only the client/server combination. Obviously it would be better if the language were clearer, but it's not. In other places, the RFC explicitly refers to the server, for example, the lines immediately after the above: 8-bit characters are permitted in the message. A server implementation that is unable to preserve 8-bit data properly MUST be able to reversibly convert 8-bit APPEND data to 7-bit using a [MIME-IMB] content transfer encoding. Note: There MAY be exceptions, e.g., draft messages, in which required [RFC-2822] header lines are omitted in the message literal argument to APPEND. The full implications of doing so MUST be understood and carefully weighed. I had not previously noted the implications of that last paragraph. It implies that even the headers are not necessarily required to be RFC2822 compliant. But it's even less clear about whether the client or the server makes the choice. But again, I'm not personally concerned about headers. The formal syntax is no help in this case, because it includes no restriction beyond "literal". > RFC3501 specifically requires 2822 rather than 822, and even says that > all references to 822 should be considered as 2822. If a mail client > claims to conform to RFC3501, then the mail messages it sends should > conform to RFC2822 not RFC822. If it wants to claim conformance only >to > an older IMAP RFC, that is fine. We are not talking about a case where RFC3501 mentions RFC822. We're talking about a case where RFC3501 mentions RFC2822 but does not make it a requirement, and is vague on exactly what is allowed and whether the client or the server decides what is allowed. > The data is not allowed to contain nulls -- from RFC3501, 4.3.1: Thanks. The formal syntax, as Cyrus Daboo mentioned, is clearer. But in any case, I now have more than adequate documentation about nulls to hit the Eudora people with. > Regarding bare newlines, RFC3501 2.2 states: > >All interactions transmitted by client and server are in the form >of >lines, that is, strings that end with a CRLF. The protocol >receiver >of an IMAP4rev1 client or server is either reading a line, or is >reading a sequence of octets with a known count followed by a line. This clearly does not mean that a literal cannot contain CR or LF -- in fact, in general the APPEND literal will always contain CRLFs which do not end the literal. The literal is prefix-coded with the octet count, and so the rule about lines ending in CRLF does not apply until the octet count is exhausted. The part of 2.2 that applies to this situation is the last line, "reading a sequence of octets with a known count". Mind, you, I'd prefer to find solid reasons to tell Eudora to clean up its act. I have that now with respect to nulls. Bare newlines still look ambiguous. Edward
Re: APPEND vs RFC2822 vs STD0011
At 01:30 PM -0400 7/7/03, John Alton Tamplin wrote: > That's what sieve is for -- do it in the server and you won't have to > rely on a particular client doing it for you. OTOH, if I do it in my client, then I don't have to rely on all the servers I have to deal with all running sieve. They don't, and so I can't count on it. I only mentioned the one server, but I'm not ready to count on it being the only server I ever deal with. I have a number of issues relevant to the way I set up my email, that are not relevant to the question at hand. > I disagree - it sounds like it would be defensible if Cyrus supported > storing such messages even though it is clearly recommended against by > the standard. If Eudora insists on storing such messages, it should >be > prepared to deal with a server that is unwilling to do so. Obviously there's a problem with the RFC in this case, in that it makes a recommendation to the client but no recommendation or requirement for the server. But the RFC clearly says that the client is allowed to store a non-RFC2822 message, if it has a valid reason. Nowhere do I see that it says the client should deal with the server refusing to handle cases which the RFC says the client is allowed to do. More importantly, I don't see that the Eudora people are going to think they should work around an IMAP server lacking a feature which the RFC says the client should be able to use, especially when they are claiming that other IMAP servers don't have this restriction. If I'm going to go back to Eudora and say they should change their code, I need a stronger analysis than just "I disagree" -- particularly when the RFC clearly says the client should be able to do this. > If you allow the IMAP server to store arbitrary data, it makes the >other > functions much more difficult I tried to make it clear that I do not consider storing a non-RFC822 message to be a valid reason, in the RFC2119 sense, to violate the "SHOULD". IMAP is designed for storing Internet email, and that requires at minimum RFC822. (RFC733, the RFC822 predecessor, is far to old to consider here. RFC822 is over twenty years old; RFC2822 is only two years old. We long ago reached that point where it's reasonable to assume that all Internet email is RFC822-compliant, but we just are not at the point where it's reasonable to assume that all Internet email is RFC2822-compliant.) > Allowing null characters in particular is problematic for any code >that > uses null-terminated strings for messages or parts of messages, and Using null-terminated strings with data that might contain nulls is problematic. > would require changing the code everywhere to use and pass the length >of > all the strings instead. You are basically saying that even if (emphasize "if") the code is clearly wrong, that it won't be changed because it's too difficult to write correct code? I don't follow this argument at all. This problem with null-terminated strings has been widely known since long before cyrus. > As far as STD0011 not being obsoleted, there are plenty of RFCs etc. > that are not obsoleted by something but are still not best current > practice. But it's very seldom that two years is considered an adequate transition. > Clearly if the RFC it has been based on has been obsoleted, > the STD should be updated as well. But rfc-editor.org says it hasn't been. I'd very much like to see more discussion on this, and not just a brush-off "that's not how we do it here". Edward
APPEND vs RFC2822 vs STD0011
The mail provider (MX) for my domain, fastmail.fm, runs cyrus. I use Eudora (for Mac, v5.2), mostly in POP mode, but I use some IMAP features too. In particular, some of my filters copy incoming (POP) messages to an IMAP mailbox at fastmail.fm. That's where the problems start. Some of these incoming messages contain NULs or bare CR or LF. Yes, the sender is broken as far as RFC2822 is concerned, but the messages get through anyway. And the messages are valid RFC822/STD0011 format. When Eudora tries to copy these (APPEND them) to the IMAP mailbox, cyrus (at fastmail.fm) returns an error. I could live with an occasional copy failure, but the worst part is that when Eudora gets the server error, it thinks it's a terrible problem and throws up a dialog box and ceases all processing. Since I (like many people) depend on Eudora cleaning up my mailbox and doing other things with incoming mail automatically when I'm not at my desk, this gets to be a serious problem. So I started reading RFC3501 to find the reason. I assumed that I'd find a good reason that I could quote to Eudora support, telling them why Eudora has to clean up the message before storing it in an IMAP mailbox. But I didn't find that. What I found -- under the APPEND command (section 6.3.11) -- is : The APPEND command appends the literal argument as a new message : to the end of the specified destination mailbox. This argument SHOULD : be in the format of an [RFC-2822] message. Note well: that's "SHOULD", not "MUST". This is important. RFC2119 gives the meaning of SHOULD: : This word [...] mean[s] that there may exist valid reasons in : particular circumstances to ignore a particular item [...] So based on my reading of the RFC, it's the client's choice: it should normally append RFC2822 messages, but if it has a valid reason, it's allowed to append something that's not RFC2822. Now, IMAP mailboxes are intended for email -- "Internet message format" or "Internet text messages" in the RFC language -- and so it would be hard to make a case for storing anything that's not such a message. But RFC822 messages are still rampant on the Internet. In fact, as I understand it, although RFC2822 has obsoleted RFC822, STD0011 (which is identical to RFC822) is still a standard and has not yet been superseded. And it certainly seems to me that making a copy of an existing message is a "valid reason" for copying it intact, without the modifications needed to force it to conform to the stricter format of RFC2822. Since RFC3501 leaves this decision up to the client, it follows that cyrus is broken when it refuses the message. If RFC3501 said "MUST", then I'd say it's Eudora's responsibility to fix the message before attempting an APPEND. But the RFC says "SHOULD". Is there any good argument for cyrus' action? If there is, I'd be happy to take it to Eudora and push them to fix Eudora. Eudora's not exactly known for its stellar IMAP support, and I'd like them to improve this. I've shoved the RFCs in their face plenty of times in the past. But in this case, my reading of the RFCs is that Eudora's APPEND action is defensible and cyrus' action is incorrect. I'm very interested in hearing the cases for both sides. Edward
