Re: atom feeds for projects
Paul Querna wrote: robert burrell donkin wrote: On 7/2/06, Sam Ruby <[EMAIL PROTECTED]> wrote: robert burrell donkin wrote: the mailing list archives at apache run on mod_mbox which also supplies atom feeds for these lists. i've added the feed from general to the front page and think it'd be cool to add feeds to the pages in projects as well. since the focus of podlings should be recruiting developers (not users) i'm thinking of adding feeds to the dev lists. opinions? volunteers? Just be aware that the feeds produced are rarely well formed XML, mostly due to encoding issues. For example: http://tinyurl.com/h5f7t I tried to submit a patch based on my limited understanding of the code, and was told that my patch wasn't acceptable To be clear, AFAIK, there was never a patch for mod_mbox -- it was a Ruby file that only solved part of the problem. Again, AFAIK, no one ever wrote a patch in C for mod_mbox to attempt to resolve this issue. I offered. The response was, and I quote, "Erm, no". and that XML parsers that require well-formedness were broken anyway -- despite that being explicitly what the spec requires. Its unfortunate that the discussion degraded into that. I'd be willing to try again, but only if there was active interest in actually fixing the problem. Yes, there is active interest in making mod_box better. This thread was in October, and since then the feed has not improved. IMO we should fix the feed but i'm not involved with mod_mbox (or httpd). anyone who is want to jump in here? The primary bug is lack of encoding support. mod-mbox just doesn't even try to do it. Someone needs to write something that touches many parts of the code, using the apr_xlate API to convert the content to utf-8. (This would also help it validate as HTML). Once that is done, we do need to worry about out of range characters, some of which would be removed, others possibly HTML encoded. Inside the message, there may be a content-type header. Inside this header, there may be a charset parameter. This charset parameter may be quoted, or it may not. It may be correct, or it may not. It would be worthwhile to attempt to extract this, and to attempt to convert at least the body portion of the message to utf-8. But in any case, the results after the conversion need to be sanitized. The Ruby code that I offered to convert to C does exactly that - takes something that is allegedly utf-8 and corrects a number of common errors, and produces something that is guaranteed to be well formed. Of course, if you feed in absolute garbage, what you will get back is well formed line noise. As promised, here is a C version that does approximately the same thing: http://intertwingly.net/stories/2006/07/04/clean_utf8_for_xml.c This may be useful in display_atom_entry, and mbox_static_message, mbox_xml_message. It is safer than using as email messages (such as this one) may contain such strings. Also note that if the content_type of the original MIME message contains the string "html", you might want to adjust the type attribute on the atom:content element accordingly. But back to the original point: even if nobody puts in the effort to correctly interpret that message based on the specified charset, the addition of this code or something similar is (1) necessary anyway, (2) will make the result no worse than it currently is and has been for months, and (3) will make a marked improvement in that it will correct a number of common errors. Please feel free to treat the code mentioned above as being under the Apache Software License version 2.0. If you don't like my indentation or bracing style, by all means, adjust it to your tastes. Convert the malloc to use the appropriate apr call. Or if you prefer, throw it all away, and start over. I don't care, I just want to see the Atom feeds produced to be clean and valid. For future discussion of this please use [EMAIL PROTECTED] OK Thanks, -Paul - Sam Ruby
Re: atom feeds for projects
Garrett Rooney wrote: On 7/4/06, Sam Ruby <[EMAIL PROTECTED]> wrote: > To be clear, AFAIK, there was never a patch for mod_mbox -- it was a > Ruby file that only solved part of the problem. Again, AFAIK, no one > ever wrote a patch in C for mod_mbox to attempt to resolve this issue. I offered. The response was, and I quote, "Erm, no". The "Erm, no" was in response to the approach, not the offer to help, IIRC. If you're willing to fix the problem the right way, by adding real support for character sets to mod_mbox, I'm sure nobody would have a problem with that. You chose to snip the portion where I argue that the approach I outlined is necessary, at least as a fall-back/safety net. Care to explain why such a fall-back/safety net isn't necessary or appropriate? - Sam Ruby
Re: atom feeds for projects
Garrett Rooney wrote: On 7/4/06, Sam Ruby <[EMAIL PROTECTED]> wrote: Garrett Rooney wrote: > On 7/4/06, Sam Ruby <[EMAIL PROTECTED]> wrote: > >> > To be clear, AFAIK, there was never a patch for mod_mbox -- it was a >> > Ruby file that only solved part of the problem. Again, AFAIK, no one >> > ever wrote a patch in C for mod_mbox to attempt to resolve this issue. >> >> I offered. The response was, and I quote, "Erm, no". > > The "Erm, no" was in response to the approach, not the offer to help, IIRC. > > If you're willing to fix the problem the right way, by adding real > support for character sets to mod_mbox, I'm sure nobody would have a > problem with that. You chose to snip the portion where I argue that the approach I outlined is necessary, at least as a fall-back/safety net. Care to explain why such a fall-back/safety net isn't necessary or appropriate? No argument that it's necessary, but it seems kind of pointless to fix that part without fixing the underlying fact that mod_mbox is totally ignorant of character sets. You'll get perfectly "valid" junk in the vast majority of cases, that doesn't seem like a real step forward to me. "vast majority"? I beg to differ. In any case, the current code assumes that everything is valid utf-8. And that assumption does not seem to have any indication of changing since I posted my offer last October. For e-mail messages that are either correct UTF-8 or US-ASCII, the current code just works. That's a substantial portion of messages. With the code I posted, the majority of the messages which are iso-8859-1 will be converted to utf-8. Even if they don't contain the proper charset headers. And if they happen to be "salted" with win-1252 characters like "smart quotes", those will be corrected too. I wager that this covers a substantial portion of the non-spam messages in your in-box. And the impact is dramatic. IE7 won't display any feed that is not well formed. FireFox 2 will stop at the first error. Bloglines (I'm told, but haven't verified) will fall back to a rather sub-optimal RSS parser to handle broken Atom feeds - and the results aren't pretty. Suffice it to say (and I say this primarily for Paul's benefit) - I believe that either this code, or code that performs a similar function - will provide an immediate improvement to Bloglines users who subscribe to mod_mbox produced feeds. As to handing the charset correctly, this can proceed incrementally. Parsing the header isn't all that hard. Fixing the body given the charset should be only one call. Expanding this to the subject and from headers (presuming that they, too, are covered by the charset, I haven't checked what the specs and/or common practice indicates in this manner) can be done at leisure. I'm willing to help there too. But I have seen too many emails and too many tools that are broken when it comes to encoding to want to invest the time in learning how to build and deploy a test version of mod_mbox as long as the prevailing mood of the project can be summed up with "Erm, no". - Sam Ruby
mod_mbox and atom 1.0
I note that mod_mbox now produces Atom 1.0 feeds. Excellent! = = = There is a feedvalidator that can be used to identify areas of improvement. http://www.feedvalidator.org/ The highest priority is to make sure that the encoding is correct. As it currently stands, many of these feeds are not well formed XML, meaning that they will be rejected by conformant XML parsers. Fixing this will improve the usability of the HTML pages. An outline of what needs to be done can be found here: http://intertwingly.net/stories/2005/09/28/xchar.rb This is in Ruby. I can translate to C any portions you may have questions on. = = = There also is a minor issue regarding canonicalization. Also, email addresses should be split out from the name (I'll go fix the feedvalidator to issue warnings on this). Finally, it is clear that the authors of mod_mbox know a thing or two about CSS. Such techniques can also be applied to feeds. Take a look at mine for an example, which I am sure you can improve on: http://intertwingly.net/blog/index.atom - Sam Ruby
Re: mod_mbox and atom 1.0
Nick Kew wrote: > On Wednesday 12 October 2005 04:31, Paul Querna wrote: > >>>An outline of what needs to be done can be found here: >>> >>> http://intertwingly.net/stories/2005/09/28/xchar.rb > > Erm, no. We need to reencode from any incoming charset. > We don't need to reinvent any wheels by recreating individual > charset conversion tables. There are two special cases that merits consideration. If *after* you convert to unicode, you end up with 1) Characters that are outside the valid range for XML then they must be replaced: 0x9, 0xA, 0xD, (0x20..0xD7FF), (0xE000..0xFFFD), (0x1..0x10) The most common character that causes such a problem is a form-feed character, common in RFC's for example. 2) Characters in the range of (0x80..0x9F) are either reserved or are control characters. 27 of these characters were "embraced and extended" by our friends in Redmond. That's the single table that you so viscerally reacted to. The most common characters that cause such problems are the so-called smart-quotes. >>Right now mod_mbox does *no* encoding translation. We really need to be >>calling apr_xlate all over, and turning everything into UTF-8 First. >>Currently, each item is encoded in whatever the client program sent it >>as... which isn't good. > > Even the HTML is erroneously sent as iso-8859-1, so posts that arrive as > utf-8 (eg from wrowe) display incorrectly! As of now it's not really fit for > purpose. We should fix this for the benefit of all display formats, rather > than address html, atom, or indeed anything else in isolation. One possibility is to convert characters about 0xFF to numeric character references, like ’. Even though it it wrong to do so, people often consume feeds with regular expressions, "aggregate" bits from various places using the equivalent of strcat, and toss the results into a web page, leaving the default as iso-8859-1. Numeric character references have the benefit of meaning the same thing independent of whether the bytes are interpreted as iso-8859-1, utf-8, or even us-ascii. > Regarding the mail archives, the ideal solution would be to transcode > incoming messages to a homogenous utf-8 before storing them. To make > that useful, we'd need to transcode the existing archives too, though that > would just be a one-off script. I see a mod_smtpd filter thrashing around > that to-do list ... dammit, it's the long-awaited updates to charset_lite! Just mentioning in passing: if you have a message of uncertain encoding, there is a regular expression that can be used to determine if it is likely in utf-8 already. Given the design of utf-8, false positives are rare, and the chances drop as the length of the message increases. > The harder bit to deal with is _local_ encoding in a different charsets in > header lines. That's a PITA, and is AFAIK peculiar to SMTP. - Sam Ruby
