On Jun 16, 2010, at 9:05 AM, David E. Wheeler wrote:
> On Jun 16, 2010, at 2:34 AM, Michael Ludwig wrote:
>
>> Try passing the parser options as a hash reference:
>>
>> my $doc = $parser->parse_html_string($str, {encoding => 'utf-8'});
>
> WTF! That fixes it! I don't understand why it seems to be ignoring the
> encoding set in the constructor. But I've noticed the same thing with other
> options. Seems like there's some consistency to be worked out in XML::LibXML
> options, still.
Okay, a bit more information: this was not quite it, alas.
>> In order to print Unicode text strings (as opposed to octet strings)
>> correctly to a terminal (UTF-8 or not), add the following line before
>> the first output:
>>
>> binmode STDOUT, ':utf8';
>>
>> But note that STDOUT is global.
>
> Yes, I do this all the time. Surprisingly, I don't get warnings for this
> script, even though it is outputting multibyte characters.
This is key. If I set the binmode on STDOUT to :utf8, the bogus characters
print out bogus. If I set it to :raw, they come out right after processing by
both Encode and XML::LibXML (I'm assuming they're interpreted as latin-1).
So my question is this: Why isn't Encode dying when it runs into these
characters? They're not valid utf-8, AFAICT. Are they somehow valid utf8 (that
is, valid in Perl's internal format)? Why would they be?
I think what I need is some code to strip non-utf8 characters from a string --
even if that string has the utf8 bit switched on. I thought that Encode would
do that for me, but in this case apparently not. Anyone got an example?
Thanks,
David
