Re: cyrus-8bit-2.3.3.diff nonsense (Cyrus Patches used at FastMail.FM)
At Mon, 29 May 2006 23:27:10 +0200 (CEST), Simon Matter wrote: Stop, are you joking? Did you ever work in the corporate world? Did you ever have a boss in a large international company? Did you ever tell him that the server you installed _rejects_ mail? Indeed. Many times. In store-and-forward mail systems like SMTP on the Internet (as well the good old real-work physical postal system) the full and total responsibility for successfully, packaging, addressing, and delivering mail falls solely on the shoulders of the sender (and the sender's agents, e.g. the poor postal worker who has to read the sender's horrid handwriting). The only real difference between centralized systems such as the physical postal system and peer-to-peer systems such as SMTP on the internet is that in the former the post office returns your mail to you (if they can), while in the latter the recipient's mail server rejects it directly. I agree that this does give a different perception, but the end result is the same: Corrupted (or dangerous) messages are returned to the sender. If the boss is too stupid to understand such simple and common-place concepts then I really do not want to work for the idiot. Some people do like to wallow in their own garbage I suppose, but happily most people I've dealt with are quick to catch on to the idea of holding the sender responsible for correctly packaging and addressing their messages, and they all appreciate the reliability and robustness of something that is known to be working correctly just as it was designed to work. Indeed many sending sites who discover (through their bouncing mail) that their software is generating corrupt messages are quite thankful to learn of the problem, and are eager to fix it too. Maybe it's OK for internal-only systems which never see incoming messages from the real world and which have their own brain-dead but Ah, what you are talking about is the real wold? The real world is the one where languages and character encoding systems are _not_ universal, but in fact are rather quite diverse. In the real world mail systems must correctly identify and preserve character encoding if the recipient is to have any hope of being able to read the message on any different kind of system, or even different locale, from that it was sent from. The rules in protocol standards aren't there just because that's the way the author likes to do things. Sure sometimes it's just stupid encoding of various punctuation characters which us humans can usually read around OK; but often enough the result of allowing incorrectly coded crap through to an unsuspecting recipient is a screen full of unreadable garbage. Don't be too quick to dismiss the security risks either. Implementations of multi-byte encoding systems have proven to be rather fragile in some cases, and that's just the most obvious issue. If you start allowing raw data through then there are even better chances that sanity checks will be skipped and foreign data will be trusted by unsuspecting client software. -- Greg A. Woods H:+1 416 218-0098 W:+1 416 489-5852 x122 VE3TCP RoboHack [EMAIL PROTECTED] Planix, Inc. [EMAIL PROTECTED] Secrets of the Weird [EMAIL PROTECTED] Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
Robert Mueller wrote: accept the message, store it as is and ignore the stuff you don't understand when building indexes. And ignore it when using the indexes. Yes, but where's the patch to do that? I'll have a guess that something like this already happens. My guess is that the charset mapping tables ignore 8-bit data in a MIME header that should be encoded QP or BASE64 data. I have no proof of this, maybe Ken can comment? (I don't have time to look through the code at this point...) After a brief look, I believe you are correct in that 8-bit chars are ignored in the header. Rob, Bron, do you have a sample message with 8-bit chars in the header(s) that I could do some testing on? I suppose I could manufacture something, but I'd prefer to use something that came from the wild. -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus-8bit-2.3.3.diff nonsense (Cyrus Patches used at FastMail.FM)
At Sat, 27 May 2006 11:56:38 +1000, Robert Mueller wrote: Comments on the non-local ones: - cyrus-8bit-2.3.3.diff Was it not the position held by CMU in the last five years or so that one such patch would *have* do to it properly and re-encode the header to some default charset, in order not to corrupt the store with lack of charset information (for searches) ? Or am I getting completely confused, here? I think having this as an *option* is perfectly reasonable. There are already other imapd options like rfc2046_strict and rfc3028_strict that control some strictness of RFC behaviour. Just for the record I don't think that a run-time option should be considered in this case, especially not in the case of a site like an ISP which must deal with international mail. This isn't anything like those other two cases, and especially not the latter one. In this case it seems to me the options are: 1. Reject the message This is the only sane thing to do. It's actually not that hard at all to explain to people that their other ISP is offering sub-standard service by accepting provably corrupt messages and that by doing so they may even be making it easier for the bad guys to cause direct damage to the customer's own systems and data should they download such corrupt messages. It might be FUD-like, but it works. Maybe it's OK for internal-only systems which never see incoming messages from the real world and which have their own brain-dead but uniform way of dealing with non-standard encodings. If a site really wants to blindly ignore these issues then they can hack and patch their own custom code to allow it. I.e. if this feature really must be an option for those who can deal with the idiocity then it should be an _undocumented_ internal compile-time only option; and those who turn it on without even further patching should suffer from having their stored messages identified to their users as being potentially corrupt with some additional header text. - users hate loosing email just because it's not fully RFC compliant, especially when my other provider doesn't have a problem Also for the record: sites which do allow their users to get away with using broken mailers which generate corrupted messages should be warned that their users may find it increasingly difficult to deal with growing numbers of other sites which will not accept their user's corrupt messages. I.e. those lame sites may eventually have to deal with the issue anyway -- it's not going to go away on its own, especially not where the sites in question don't also control all the software that users might use on their own computers. -- Greg A. Woods H:+1 416 218-0098 W:+1 416 489-5852 x122 VE3TCP RoboHack [EMAIL PROTECTED] Planix, Inc. [EMAIL PROTECTED] Secrets of the Weird [EMAIL PROTECTED] Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus-8bit-2.3.3.diff nonsense (Cyrus Patches used at FastMail.FM)
At Sat, 27 May 2006 11:56:38 +1000, Robert Mueller wrote: Comments on the non-local ones: - cyrus-8bit-2.3.3.diff Was it not the position held by CMU in the last five years or so that one such patch would *have* do to it properly and re-encode the header to some default charset, in order not to corrupt the store with lack of charset information (for searches) ? Or am I getting completely confused, here? I think having this as an *option* is perfectly reasonable. There are already other imapd options like rfc2046_strict and rfc3028_strict that control some strictness of RFC behaviour. Just for the record I don't think that a run-time option should be considered in this case, especially not in the case of a site like an ISP which must deal with international mail. This isn't anything like those other two cases, and especially not the latter one. Cyrus is not only used with ISPs. It's also used in the corporate world where other rules exist. In this case it seems to me the options are: 1. Reject the message This is the only sane thing to do. It's actually not that hard at all Stop, are you joking? Did you ever work in the corporate world? Did you ever have a boss in a large international company? Did you ever tell him that the server you installed _rejects_ mail? to explain to people that their other ISP is offering sub-standard And if the other one is not an ISP but your important customer? service by accepting provably corrupt messages and that by doing so they may even be making it easier for the bad guys to cause direct damage to And if the bad guys are your customers who in the end pay you? the customer's own systems and data should they download such corrupt messages. It might be FUD-like, but it works. Maybe you want to fix the customers mail system for free, do you? You say it works, but hey, it doesn't work for all of us. Maybe it's OK for internal-only systems which never see incoming messages from the real world and which have their own brain-dead but Ah, what you are talking about is the real wold? No, it's just a part of a quite large world. It's all a question of your point of view. uniform way of dealing with non-standard encodings. If a site really wants to blindly ignore these issues then they can hack and patch their own custom code to allow it. Or they can enable an option in the config file. What's wrong with the option? I.e. if this feature really must be an option for those who can deal with the idiocity then it should be an _undocumented_ internal Undocumented feature is the M$ way, do you want that? No sorry, _undocumented_ is just stupid and evil. compile-time only option; and those who turn it on without even further patching should suffer from having their stored messages identified to their users as being potentially corrupt with some additional header text. Did you ever explain some thousand of normal users what email headers are? Believe me, it's not easy. However, your idea is not bad, do you have a patch? - users hate loosing email just because it's not fully RFC compliant, especially when my other provider doesn't have a problem Also for the record: sites which do allow their users to get away with using broken mailers which generate corrupted messages should be warned that their users may find it increasingly difficult to deal with growing numbers of other sites which will not accept their user's corrupt messages. I.e. those lame sites may eventually have to deal with the issue anyway -- it's not going to go away on its own, especially not where the sites in question don't also control all the software that users might use on their own computers. Unfortunately the problem is not going away anyway, with or without cyrus-8bit-2.3.3.diff patch. However, the patch is a good solution for some users of cyrus-imapd and I just don't understand why it's such an issue of discussion now. My congratulations to Ken, who has had a good feeling for the situation when he decided to accept the patch! Simon Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
accept the message, store it as is and ignore the stuff you don't understand when building indexes. And ignore it when using the indexes. Yes, but where's the patch to do that? I'll have a guess that something like this already happens. My guess is that the charset mapping tables ignore 8-bit data in a MIME header that should be encoded QP or BASE64 data. I have no proof of this, maybe Ken can comment? (I don't have time to look through the code at this point...) Rob Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
On Sat, 27 May 2006, Carlos Horowicz wrote: I was just asking to the list if Fastmail's cyrus-8bit-2.3.3.diff would let avoid rejections due to Message contains NUL chars No it won't, and you *really* don't want to store anything with embedded NULs, you have to at least remove them, as replacing them with X or anything else is probably going to cause some real weird output. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
On 05/26/06 20:37, Simon Matter wrote: Hij - cyrus-fromheader-2.3.3.diff Outlook sucks. I hate to have to commit yet another M$ workaround. How widespread are these non-RFC2822-compliant messages? I do not think it is just outlook. I've also seen problems with notes doing stuff like this. I get errors like this one in my logs: host /var/lib/imap/socket/lmtp[/var/lib/imap/socket/lmtp] said: 554 5.6.0 Message contains invalid header (in reply to end of DATA command) Does anyone know if there is a way to configure postfix or amavis to clean up emails with with errors like these? I'm not sure that is the same problem here but I have this in my /etc/postfix/header_checks: # Remove empty Message-ID headers from broken Notes implementation /^Message-ID:[[:space:]]*$/ IGNORE Thanks for the tip. I'll try this. Tarjei Simon Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
On Sat, 27 May 2006 02:06:20 -0300, Henrique de Moraes Holschuh [EMAIL PROTECTED] said: On Sat, 27 May 2006, Robert Mueller wrote: I think having this as an *option* is perfectly reasonable. There are So do I, Debian has this patch for three or four years now. It comes with a big warning to the likes of You are stupid to enable this and still wonder why multi-language search is now broken in your server. But this has never been the instance on this issue for upstream inclusion of this patch, and I was (and still am) quite surprised it was included without any further comment. Ph33r the power of my 'l33t bamboozling skills. 3. Accept the message and store as is, leave it up to the email client to handle - This seems to be the 90% good enough that works well. Most clients This breaks IMAP SEARCH, or so I have been told for four years, now. This is why I want a better explanation now for the inclusion of such a patch :-) Well, it may be worth adding more comments to the config option making it clear that it breaks IMAP SEARCH. I'd certainly support that. 4. Recode correctly in MIME encoding based on the message charset - Would be great but: Nah, just according to some site-wide config option from imap.conf would be more than good enough. Wow, I'm glad you know enough about everyone else's situtation to be able to make a blanket determination like that. We have users from all over the world, using: COUNT(DISTINCT DefCharSet) 59 59 different charactersets - all on the same set of servers. I don't know any characterset that would be sufficient to put in imapd.conf that would give the expected results for all of them. It would be lovely if that was the case, or everyone was using UTF-8 (except I'd likely be out of a job because things would actually just work[tm]) I agree that this option is truly a hack, and technically the email is non-RFC compliant so you can do whatever you want, but unfortunately out in This is not the issue. The issue is whether Cyrus is corrupting its innards or not by letting this thing into the spool. If everyone is applying the patch because the real world isn't such a pretty place as all that, then maybe it really does belong in upstream, especially if you get a choice at config time how much corruption you want. There's a reason I didn't run Courier's MTA back when I was running Courier's IMAP server at a previous job and it's because I lost too much mail that actual human beings were trying to send to other actual human beings. I'm a strong believer in both liberal in what you accept (unless you're in a position to enforce a contract over an interface, in which case you should be careful to enforce every clause) and ... how best to phrase this, it's a bit: * don't alter things you don't have to. * don't break things you don't need to, even if they're wrong. * don't throw away information - even if you don't understand it. In this case, I'm presuming the breaks multi-language search is an indexing issue - and an alternative would be to skip/replace/guess the character at indexing stage but leave the full message on disk untouched. Tough luck if you can't search it as expected - at least you haven't LOST information. That last point is particularly important. By rejecting the message out of hand, you are preserving your pristine innards but lose interoperability with reality. By silently replacing unknown 8bit data with an 'X' you are throwing away information and lying to whoever delivered it that you've faithfully saved/reproduced what you were told. The middle ground is to accept the message, store it as is and ignore the stuff you don't understand when building indexes. Erm.. but I rant. Sorry about that. Bron. -- Bron Gondwana [EMAIL PROTECTED] Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
On Sat, 27 May 2006, Bron Gondwana wrote: 4. Recode correctly in MIME encoding based on the message charset - Would be great but: Nah, just according to some site-wide config option from imap.conf would be more than good enough. Wow, I'm glad you know enough about everyone else's situtation to be able to make a blanket determination like that. We have users from all over the world, using: That's not it. I was just pointing out that the bar for a won't break anything patch is not as high. Well, for the old won't break anything definition on this matter, anyway. The best heuristics I can think of for this is a bit more complex, but it still is not perfect: you *are* always reduced to guessing if the broken PoS that generated the email gives you no charset information at all anywhere... 0. [config: optional] Do MUA imbecility charset fixup, checking at repairing mistakes like this table (each one configurabe to not be done): ISO-8859-1 [config: ISO-8859-15 instead] data encoded as UTF-8 Windows CP1252 data encoded as ISO-8859-1 or ISO-8859-15 (check both) Windows CP1252 data encoded as UTF-8 other rules people contribute as common MUA fuckups, I bet there are some common patters for CJK and Cirilic, at the very least). 1. Reject messages with invalid charset (note: data *missing* charset information is ignored for this step: this step is not to reject 8bit data in headers, just completely bogus data). This step is non-optional. 2. In non-strict-mode: Determine a set of all charsets declared for the headers, use frequency count as key for priority. Then, do the same for the message body, but at a lower pirority band. Then do the same for a admin configured set of charsets, at an yet lower priority band. Do the same for US-ASCII at the lowest possible priority. Try to encode all header data missing charset information using the priority-ordered set. If no valid encodings are possible, reject the message or do the X thing (according to config). As you see, it is far more complex, but also far more likely to do what you want. But I have never seen advocated in this list that such level of functionality should be required of a patch dealing with 8bit data in headers... BTW: suggestions to improve the above algo are welcome. Implementation suggestion: add filter plugins (for all ways a message can enter the spool), and the above as a filter. I know a lot of people would be happy to plug an AV as a filter too for IMAP APPEND... 59 different charactersets - all on the same set of servers. I don't know any characterset that would be sufficient to put in imapd.conf that would give the expected results for all of them. There is none. If everyone is applying the patch because the real world isn't such a pretty place as all that, then maybe it really does belong in upstream, Or a different version of the patch... but really, so far nobody who asked (or wrote) such a patch really cared for even the simplest recode to configured charset version. IMAP SEARCH is simply kicked to hell (or whatever, I am still wating someone to tell me it doesn't break ) :-) In this case, I'm presuming the breaks multi-language search is an indexing issue - and an alternative would be to skip/replace/guess the character at indexing stage but leave the full message on disk untouched. Yes, you can do that too. Flag all charset-less 8bit data as charset Unknown, never give it to any unaware UTF-8-processing function (security issue), and do binary matches against it. Sounds like quite a good compromise to me, actually. But it does cause reduced functionality when compared to the do your best do fix the message when it arrives in the spool. OTOH hand, it is probably a lot easier to implement... Tough luck if you can't search it as expected - at least you haven't LOST information. Well, basically that's what all of us have been doing when we have no choice, I suppose. That last point is particularly important. By rejecting the message out of hand, you are preserving your pristine innards but lose interoperability We *have* to preserve our innards to a degree where the code will never malfunction because we broke expectations. If the Cyrus code can, and does deal well with the unexpected 8bit data, then the condition of preserving the innards is *already met*. If it doesn't, we have to either fix it so that it does, or we should never accept such data. with reality. By silently replacing unknown 8bit data with an 'X' you are throwing away information and lying to whoever delivered it that you've faithfully saved/reproduced what you were told. The middle ground is to Well, they lied to you when they said they had an e-mail for you too, so don't give me that excuse :-) An e-mail is something that perfectly follows the proper RFCs, anything else is just __broken___ e-mail, and all bets are off. accept the message, store it as is and ignore the stuff you
Re: Cyrus Patches used at FastMail.FM
Henrique de Moraes Holschuh wrote: On Fri, 26 May 2006, Ken Murchison wrote: Comments on the non-local ones: - cyrus-8bit-2.3.3.diff As I said in my reply to the CVS commit (which I noticed before this mail), I am quite surprised by this patch being accepted. Was it not the position held by CMU in the last five years or so that one such patch would *have* do to it properly and re-encode the header to some default charset, in order not to corrupt the store with lack of charset information (for searches) ? I decided that I don't care to fight the fight any longer. If people want to continue to write/use crappy email clients that don't follow specs, that's fine by me. If admins want to let non-compliant email messages into there store, they are now free to do so by disabling munge8bit. Let it be known that I (CMU) am not endorsing crappy/misconfigured clients and/or non-compliant messages, nor am I endorsing disabling munge8bit. I will add text to imapd.conf(5) that states that disabling munge8bit will most likely break SEARCH. Henrique, do you recall if SEARCH dies, or just doesn't get any useful hits on non-tagged 8-bit headers? -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
On Sat, 27 May 2006, Ken Murchison wrote: Henrique, do you recall if SEARCH dies, or just doesn't get any useful hits on non-tagged 8-bit headers? I don't recall this either. But if you did accept the patch, I think it is time to go over IMAP SEARCH and make sure it doesn't do stupid things with non-tagged 8bit data... -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
Bron Gondwana wrote: I've finally split out all the patches that we use here into individual items. Comments on the non-local ones: - cyrus-8bit-2.3.3.diff Applied to CVS. - cyrus-anyone-2.3.3.diff If you want to complete the documentation, I'll consider applying the patch. - cyrus-commandtimer-2.3.3.diff This looks useful, and I'll probably commit it. - cyrus-defaultfix-2.3.3.diff Already fixed in CVS. - cyrus-fromheader-2.3.3.diff Outlook sucks. I hate to have to commit yet another M$ workaround. How widespread are these non-RFC2822-compliant messages? - cyrus-imapd-2.3.1-autoreply-0.1-0.diff The Sieve notify is undergoing a lot of changes, so I'll probably wait until the dust settles. - cyrus-isdigit-2.3.3.diff This seems like an OS problem rather than a Cyrus problem. Have you taken this up with the Debian folks? - cyrus-longwords-2.3.3.diff I have to see if there is a better way to handle long UID lists. - cyrus-plainsync-2.3.3.diff I just committed some code which does this in another way. I added a '-p' option to sync_server, pop3d, nntpd, that does the same thing as the 'imapd -p' option. It allows you to specify an external layer security strength factor, so setting it to =2 will allow PLAIN to be advertised w/o TLS. - cyrus-reconstruct-2.3.1.diff Already fixed in CVS. - cyrus-statuscache-2.3.3.diff This looks reasonable, but I want to give it some more thought. -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
Quoting Ken Murchison ([EMAIL PROTECTED]): - cyrus-fromheader-2.3.3.diff Outlook sucks. I hate to have to commit yet another M$ workaround. How widespread are these non-RFC2822-compliant messages? Its not an Outlook issue. Its other IMAP providers that deliver non-standard headers to IMAP clients. The problem arises when copying mail from foreign IMAP account to an account on a Cyrus server. John Capo Tuffmail.com Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
Hij - cyrus-fromheader-2.3.3.diff Outlook sucks. I hate to have to commit yet another M$ workaround. How widespread are these non-RFC2822-compliant messages? I do not think it is just outlook. I've also seen problems with notes doing stuff like this. I get errors like this one in my logs: host /var/lib/imap/socket/lmtp[/var/lib/imap/socket/lmtp] said: 554 5.6.0 Message contains invalid header (in reply to end of DATA command) Does anyone know if there is a way to configure postfix or amavis to clean up emails with with errors like these? kind regards, Tarjei - cyrus-imapd-2.3.1-autoreply-0.1-0.diff The Sieve notify is undergoing a lot of changes, so I'll probably wait until the dust settles. - cyrus-isdigit-2.3.3.diff This seems like an OS problem rather than a Cyrus problem. Have you taken this up with the Debian folks? - cyrus-longwords-2.3.3.diff I have to see if there is a better way to handle long UID lists. - cyrus-plainsync-2.3.3.diff I just committed some code which does this in another way. I added a '-p' option to sync_server, pop3d, nntpd, that does the same thing as the 'imapd -p' option. It allows you to specify an external layer security strength factor, so setting it to =2 will allow PLAIN to be advertised w/o TLS. - cyrus-reconstruct-2.3.1.diff Already fixed in CVS. - cyrus-statuscache-2.3.3.diff This looks reasonable, but I want to give it some more thought. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
On Fri, May 26, 2006 at 08:14:51PM +0200, Tarjei Huse wrote: Hij - cyrus-fromheader-2.3.3.diff Outlook sucks. I hate to have to commit yet another M$ workaround. How widespread are these non-RFC2822-compliant messages? I do not think it is just outlook. I've also seen problems with notes doing stuff like this. I get errors like this one in my logs: host /var/lib/imap/socket/lmtp[/var/lib/imap/socket/lmtp] said: 554 5.6.0 Message contains invalid header (in reply to end of DATA command) Does anyone know if there is a way to configure postfix or amavis to clean up emails with with errors like these? Maybe a postfix header regexp? There was a time I used sed, because I was already calling a shell script of mine to do other stuff. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
Hij - cyrus-fromheader-2.3.3.diff Outlook sucks. I hate to have to commit yet another M$ workaround. How widespread are these non-RFC2822-compliant messages? I do not think it is just outlook. I've also seen problems with notes doing stuff like this. I get errors like this one in my logs: host /var/lib/imap/socket/lmtp[/var/lib/imap/socket/lmtp] said: 554 5.6.0 Message contains invalid header (in reply to end of DATA command) Does anyone know if there is a way to configure postfix or amavis to clean up emails with with errors like these? I'm not sure that is the same problem here but I have this in my /etc/postfix/header_checks: # Remove empty Message-ID headers from broken Notes implementation /^Message-ID:[[:space:]]*$/ IGNORE Simon Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
Hi I get only a daily few of Message contains invalid header but a couple of thousand per day of Message contains NUL characters from lmtpd ... I've read through all threads both in postfix and cyrus lists about this, but didn't find any conclusion on how to avoid the bounces ... has the Fastmail.FM patch anything to do with the NUL chars in the body ? -Carlos Tarjei Huse wrote: Hij - cyrus-fromheader-2.3.3.diff Outlook sucks. I hate to have to commit yet another M$ workaround. How widespread are these non-RFC2822-compliant messages? I do not think it is just outlook. I've also seen problems with notes doing stuff like this. I get errors like this one in my logs: host /var/lib/imap/socket/lmtp[/var/lib/imap/socket/lmtp] said: 554 5.6.0 Message contains invalid header (in reply to end of DATA command) Does anyone know if there is a way to configure postfix or amavis to clean up emails with with errors like these? kind regards, Tarjei - cyrus-imapd-2.3.1-autoreply-0.1-0.diff The Sieve notify is undergoing a lot of changes, so I'll probably wait until the dust settles. - cyrus-isdigit-2.3.3.diff This seems like an OS problem rather than a Cyrus problem. Have you taken this up with the Debian folks? - cyrus-longwords-2.3.3.diff I have to see if there is a better way to handle long UID lists. - cyrus-plainsync-2.3.3.diff I just committed some code which does this in another way. I added a '-p' option to sync_server, pop3d, nntpd, that does the same thing as the 'imapd -p' option. It allows you to specify an external layer security strength factor, so setting it to =2 will allow PLAIN to be advertised w/o TLS. - cyrus-reconstruct-2.3.1.diff Already fixed in CVS. - cyrus-statuscache-2.3.3.diff This looks reasonable, but I want to give it some more thought. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
On Fri, 26 May 2006, Ken Murchison wrote: Comments on the non-local ones: - cyrus-8bit-2.3.3.diff As I said in my reply to the CVS commit (which I noticed before this mail), I am quite surprised by this patch being accepted. Was it not the position held by CMU in the last five years or so that one such patch would *have* do to it properly and re-encode the header to some default charset, in order not to corrupt the store with lack of charset information (for searches) ? Or am I getting completely confused, here? -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
On Fri, May 26, 2006 at 05:02:25PM -0300, Carlos Horowicz wrote: Hi I get only a daily few of Message contains invalid header but a couple of thousand per day of Message contains NUL characters from lmtpd ... What version of cyrus? There was a bug that caused the NUL error with long lines. This has already been fixed in later versions. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
Andreas Hasenack wrote: On Fri, May 26, 2006 at 05:02:25PM -0300, Carlos Horowicz wrote: Hi I get only a daily few of Message contains invalid header but a couple of thousand per day of Message contains NUL characters from lmtpd ... What version of cyrus? There was a bug that caused the NUL error with long lines. This has already been fixed in later versions. all of my servers complain : there are several 2.3.3 and several 2.2.12 ... Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
On Fri, May 26, 2006 at 05:35:30PM -0300, Carlos Horowicz wrote: Andreas Hasenack wrote: On Fri, May 26, 2006 at 05:02:25PM -0300, Carlos Horowicz wrote: Hi I get only a daily few of Message contains invalid header but a couple of thousand per day of Message contains NUL characters from lmtpd ... What version of cyrus? There was a bug that caused the NUL error with long lines. This has already been fixed in later versions. all of my servers complain : there are several 2.3.3 and several 2.2.12 ... Then it's probably for real: there are NUL characters in the messages. There is a postfix patch floating around that can get rid of these, and I think it's integrated in postfix's snapshot. Here is a version for 2.2.x: http://cvs.mandriva.com/cgi-bin/cvsweb.cgi/SPECS/postfix/patch_reject_strip-postfix-2.2.4.diff Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
Comments on the non-local ones: - cyrus-8bit-2.3.3.diff Was it not the position held by CMU in the last five years or so that one such patch would *have* do to it properly and re-encode the header to some default charset, in order not to corrupt the store with lack of charset information (for searches) ? Or am I getting completely confused, here? I think having this as an *option* is perfectly reasonable. There are already other imapd options like rfc2046_strict and rfc3028_strict that control some strictness of RFC behaviour. In this case it seems to me the options are: 1. Reject the message - users hate loosing email just because it's not fully RFC compliant, especially when my other provider doesn't have a problem 2. Replace with X/? - users hate mail being mangled and not being able to read it, especially when my other provider doesn't have a problem 3. Accept the message and store as is, leave it up to the email client to handle - This seems to be the 90% good enough that works well. Most clients well display 8-bit data from the message headers in the same charset as the message body, which generally works 4. Recode correctly in MIME encoding based on the message charset - Would be great but: a) Hard because what if it's a multipart/alternative message, and the alternative parts have different charsets? What do recode the header as? b) Some messages just don't have a charset at all (some Japanese clients seems particularly bad at this), what to do then? c) Some HTML messages only include the charset in the HTML as a meta header. Do you parse the HTML? I agree that this option is truly a hack, and technically the email is non-RFC compliant so you can do whatever you want, but unfortunately out in the real world I think a hack like this allows people to actually get their email and should be provided. I'm sure quite a few sites already apply this patch to their installations in some way (eg I think Simon's RPMs include it?) Rob Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
Andreas Hasenack wrote: On Fri, May 26, 2006 at 05:35:30PM -0300, Carlos Horowicz wrote: Andreas Hasenack wrote: On Fri, May 26, 2006 at 05:02:25PM -0300, Carlos Horowicz wrote: Hi I get only a daily few of Message contains invalid header but a couple of thousand per day of Message contains NUL characters from lmtpd ... What version of cyrus? There was a bug that caused the NUL error with long lines. This has already been fixed in later versions. all of my servers complain : there are several 2.3.3 and several 2.2.12 ... Then it's probably for real: there are NUL characters in the messages. There is a postfix patch floating around that can get rid of these, and I think it's integrated in postfix's snapshot. Here is a version for 2.2.x: http://cvs.mandriva.com/cgi-bin/cvsweb.cgi/SPECS/postfix/patch_reject_strip-postfix-2.2.4.diff Thanks but I was just asking to the list if Fastmail's cyrus-8bit-2.3.3.diff would let avoid rejections due to Message contains NUL chars Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
On Sat, 27 May 2006, Robert Mueller wrote: I think having this as an *option* is perfectly reasonable. There are So do I, Debian has this patch for three or four years now. It comes with a big warning to the likes of You are stupid to enable this and still wonder why multi-language search is now broken in your server. But this has never been the instance on this issue for upstream inclusion of this patch, and I was (and still am) quite surprised it was included without any further comment. 3. Accept the message and store as is, leave it up to the email client to handle - This seems to be the 90% good enough that works well. Most clients This breaks IMAP SEARCH, or so I have been told for four years, now. This is why I want a better explanation now for the inclusion of such a patch :-) 4. Recode correctly in MIME encoding based on the message charset - Would be great but: Nah, just according to some site-wide config option from imap.conf would be more than good enough. I agree that this option is truly a hack, and technically the email is non-RFC compliant so you can do whatever you want, but unfortunately out in This is not the issue. The issue is whether Cyrus is corrupting its innards or not by letting this thing into the spool. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
I've finally split out all the patches that we use here into individual items. I know some people were interested in the don't allow users to set the anyone ACL patch as well. http://cyrus.brong.fastmail.fm/ Ken - I'd love to work with you on getting as many as possible of these into upstream, at least the ones that are labelled as worth doing - some are so cheap-hack as to require a complete rewrite to make them worthwhile. Cool, some of the patches look really interesting and I'm considering to include one or the other into my rpm packages. For example the statuscache patch seems very nice. Just to be sure, are there any license restrictions on the patches? I'd love to see some of the patches integrated upstream, it makes my own work easier :) Simon Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
--On May 23, 2006 8:37:45 AM +0200 Simon Matter [EMAIL PROTECTED] wrote: Cool, some of the patches look really interesting and I'm considering to include one or the other into my rpm packages. For example the statuscache patch seems very nice. Just to be sure, are there any license restrictions on the patches? I'd love to see some of the patches integrated upstream, it makes my own work easier :) Statuscache also piqued my interestDoes it give any win for POP3 clients? We've a *HUGE* number of Outlook users that have the terribly wrong idea that they just MUST poll every minute. That along with the seen state stuff would be a good thing for us. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
Cool, some of the patches look really interesting and I'm considering to include one or the other into my rpm packages. For example the statuscache patch seems very nice. Just to be sure, are there any license restrictions on the patches? No, no license restrictions. We'd love these to get back into cyrus itself, so happy to contribute them back to whatever licence cyrus is under. The statuscache is actually very useful, more so than Bron's comment on the page immediately suggests. Originally I thought it was our webmail client that was doing most of the status calls, so I actually put caching into our web client. It turned out that in fact lots of email clients also blast the IMAP server with status calls, so putting it in the server itself has made a noticeable difference to load on our servers. The other one that I've just finished today is the fastindex one, which I'm also excited about. I discovered that on our brand new server with only a couple of hundred users on it, cmdtimer was still showing that some status calls were taking 2 seconds for large mailboxes (for status calls that weren't cached that is) on a virtually unloaded server (load 0.05). Checking it out, it became pretty obvious that the problem was the core status loop which calls index_insequence() on every message to see if it's seen/unseen. When the seen state sequence list becomes quite large, you're doing an awful lot of repeated scans of the seen list. With a bit of slightly messy hacking (static variables + extra parameter) instead of cleaner but more major code surgery, I've added a caching mode when scanning an assumed sequence list of messages into a sequential seen list. This has reduced the status call to basically instant even on 40,000+ message folders with lots of random seen/unseen messages. I've put the latest version of the patch up now: http://cyrus.brong.fastmail.fm/patches/cyrus-fastindex-2.3.3.diff I'd love to see some of the patches integrated upstream, it makes my own work easier :) So would we, the less we have to maintain, the better, and for large installations with many IMAP users, the performance impact of the status and fastindex patches should be helpful. Rob Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
Statuscache also piqued my interestDoes it give any win for POP3 clients? We've a *HUGE* number of Outlook users that have the terribly wrong idea that they just MUST poll every minute. That along with the seen state stuff would be a good thing for us. No, it's only for IMAP clients and speeds up the STATUS IMAP command, it does nothing for POP3. I'm surprised you're having issues with lots of POP polling, generally POP is a lot less resource intensive than IMAP I always believed... Rob Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
Bron Gondwana wrote: I've finally split out all the patches that we use here into individual items. I know some people were interested in the don't allow users to set the anyone ACL patch as well. http://cyrus.brong.fastmail.fm/ Ken - I'd love to work with you on getting as many as possible of these into upstream, at least the ones that are labelled as worth doing - some are so cheap-hack as to require a complete rewrite to make them worthwhile. I'm in the process of porting a couple of patches from UMich into 2.3 and then I'm going to make a release which includes a fix for an easily exploitable buffer overflow in pop3d. -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
Ken Murchison wrote: Bron Gondwana wrote: I've finally split out all the patches that we use here into individual items. I know some people were interested in the don't allow users to set the anyone ACL patch as well. http://cyrus.brong.fastmail.fm/ Ken - I'd love to work with you on getting as many as possible of these into upstream, at least the ones that are labelled as worth doing - some are so cheap-hack as to require a complete rewrite to make them worthwhile. I'm in the process of porting a couple of patches from UMich into 2.3 and then I'm going to make a release which includes a fix for an easily exploitable buffer overflow in pop3d. Clicked too fast. After I make the release, I'll look at your stuff. -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
I'm in the process of porting a couple of patches from UMich into 2.3 and then I'm going to make a release which includes a fix for an easily exploitable buffer overflow in pop3d. Ehm? Does this affect the 2.2.x branch as well? John -- John Madden Sr. UNIX Systems Engineer Ivy Tech Community College of Indiana [EMAIL PROTECTED] Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
John Madden wrote: I'm in the process of porting a couple of patches from UMich into 2.3 and then I'm going to make a release which includes a fix for an easily exploitable buffer overflow in pop3d. Ehm? Does this affect the 2.2.x branch as well? No. The exploit is in a feature added in 2.3. -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
Robert Mueller wrote: Cool, some of the patches look really interesting and I'm considering to include one or the other into my rpm packages. For example the statuscache patch seems very nice. Just to be sure, are there any license restrictions on the patches? No, no license restrictions. We'd love these to get back into cyrus itself, so happy to contribute them back to whatever licence cyrus is under. The statuscache is actually very useful, more so than Bron's comment on the page immediately suggests. Originally I thought it was our webmail client that was doing most of the status calls, so I actually put caching into our web client. It turned out that in fact lots of email clients also blast the IMAP server with status calls, so putting it in the server itself has made a noticeable difference to load on our servers. I know we discussed this in the past, but I can't seem to find the thread. What part of the existing STATUS code causes the bottleneck? Is it STATUS_RECENT and STATUS_UNSEEN? -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
--On May 23, 2006 10:34:20 PM +1000 Robert Mueller [EMAIL PROTECTED] wrote: Statuscache also piqued my interestDoes it give any win for POP3 clients? We've a *HUGE* number of Outlook users that have the terribly wrong idea that they just MUST poll every minute. That along with the seen state stuff would be a good thing for us. No, it's only for IMAP clients and speeds up the STATUS IMAP command, it does nothing for POP3. I'm surprised you're having issues with lots of POP polling, generally POP is a lot less resource intensive than IMAP I always believed... Well being as the POP3d runs the mailbox list every time a client connects. :) It certainly would also help because a lot of our accesses are via webmail as well. Rob -- Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds. -- Samuel Butler Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus Patches used at FastMail.FM
I know we discussed this in the past, but I can't seem to find the thread. What part of the existing STATUS code causes the bottleneck? Is it STATUS_RECENT and STATUS_UNSEEN? It's a combination of both. The main things are: 1. Status UNSEEN and RECENT both have to loop over the cyrus.index file. With users with many folders and large message counts that don't change that often, it doesn't stop their IMAP clients issuing a STATUS call on every folder every 5 minutes. On a busy server, even with lots of memory the system disk cache will be completely flushed over a 5 minute period, so these STATUS calls cause every cyrus.index file to be re-read into memory. Given most folders don't change that often, with the cache these STATUS calls are serviced without having to read the cyrus.index file at all. 2. Status UNSEEN was particularly slow with large seen sequences. The fastindex patch mostly fixes that is it's not as much of an issue any more. Rob Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus Patches used at FastMail.FM
I've finally split out all the patches that we use here into individual items. I know some people were interested in the don't allow users to set the anyone ACL patch as well. http://cyrus.brong.fastmail.fm/ Ken - I'd love to work with you on getting as many as possible of these into upstream, at least the ones that are labelled as worth doing - some are so cheap-hack as to require a complete rewrite to make them worthwhile. Bron. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
