[EMAIL PROTECTED] writes:
> LWP seems to have issues with fetching pages that are utf-8 encoded.
>
> Using a simple script like
>
> use LWP::UserAgent;
> use Encode;
>
> my $ua = LWP::UserAgent->new();
> my $resp = $ua->get("http://bild.de";);
>
> if(Encode::is_utf8($resp->content)) {
> print "utf8\n";
> } else {
> print "no utf8\n";
> }
>
> shows
>
> "no utf8"
>
> (meaning that although the page is utf-8 encoded, the resulting Perl string
> isn't)
> and it prints the warning
>
> Parsing of undecoded UTF-8 will give garbage when decoding entities
> at .../LWP/Protocol.pm line 114.
>
> which seems to be related to a message I posted last year:
>
> http://www.nntp.perl.org/group/perl.libwww/2006/08/msg6801.html
>
> although there were no responses at the time.
This is indeed an outstanding bug that I don't have any good fix for
yet, but you can work around the warning by setting the 'parse_head'
attribute to FALSE:
my $ua = LWP::UserAgent->new(parse_head => 0);
Extracting the content using the:
$resp->decoded_content
method will decode the UTF-8 if there is a proper header to be found
in the HTTP reponse.
--Gisle
