On Jun 17, 2010, at 12:30 PM, Henning Michael Møller Just wrote:
>> So it may be valid UTF-8, but why does it come out looking like crap? That
>> is, "LaurinaviÃ≥Ÿius"? I suppose there's an > argument that
>> "LaurinaviÄŸius" is correct and valid, if ugly. Maybe?
>
> I am unsure if this is the explanation you are looking for but here goes:
>
> I think the original data contained the character \x{010d}. In utf-8, that
> means that it should be represented as the bytes \x{c4} and \x{8d}. If those
> bytes are not marked as in fact being a two-byte utf-8 encoding of a single
> character, or if an application reading the data mistakenly thinks it is not
> encoded (both common errors), somewhere along the transmission an application
> may decide that it needs to re-encode the characters in utf-8.
>
> So the original character \x{010d} is represented by the bytes \x{c4} and
> \x{8d}, an application thinks those are in fact characters and encodes them
> again as \x{c3} + \x{84} and \x{c2} + \x{8d}, respectively. Which I believe
> is your broken data.
I see. That makes sense. FYI, the original source is at:
http://pipes.yahoo.com/pipes/pipe.run?Size=Medium&_id=f53b7bed8b88412fab9715a995629722&_render=rss&max=50&nsid=1025993%40N22
Look for "Tomas" in the output. If it doesn't show pu, change max=50 to max=75
or something.
> I think the error comes from Perl's handling of utf-8 data and that this
> handling has changed in subtle ways all the way since Perl 5.6. We have
> supported utf-8 in our applications since Perl 5.6 and have experienced this
> repeatedly. Any major upgrade of Perl or indeed the much needed upgrade of
> DBD::ODBC Martin Evans provided have given us a lot of work trying to sort
> out these troubles.
Maintaining the backwards compatibility from the pre-utf8 days must make it far
more difficult than it otherwise would be.
> I wonder if your code would work fine in Perl 5.8? We are "only" at 5.10(.1)
> but the upgrade from 5.8 to 5.10 also gave us some utf-8 trouble. If it works
> fine in Perl 5.8 maybe the error is in an assumption somewhere in XML::LibXML?
In my application, I finally got XML::LibXML to choke on the invalid
characters, and then found that the problem was that I was running
Encode::CP1252::zap_cp1252 against the string before passing it to XML::LibXML.
Once I removed that, it stopped choking. So clearly zap_cp1252 was changing
bytes it should not have. I now have it running fix_cp1252 *after* the parsing,
when everything is already UTF-8. Now that I think about it, though, I should
probably change it so that it searches on characters instead of bytes when
working on a utf8 string. Will have to look into that.
In the meantime, I'll just accept that sometimes the characters are valid UTF-8
and look like shit. Frankly, when I run the above feed through NetNewsWire, the
offending byte sequence displays as "Ä", just as it does in my app's output. So
I blame Yahoo.
Thanks for the detailed explanation, Henning, much appreciated.
Best,
David
