Thomas Krichel wrote:
Ere Maijala writes
On 7.10.2010 15:17, Thomas Krichel wrote:
...
use Encode::Guess qw/latin-1/;
$decoded=decode("Guess", $dodgy_input);
$decoded then should be a utf-8 string with utf8 flag on.
Would that work for a predominantly proper utf-8 input with some
"mistakes" thrown in?
It will try to guess between UTF-8 and ISO-8859-1. This can be done
because UTF-8 has many invalid byte sequences. But say if you
wanted to guess between ISO-8859-1 and ISO-8859-2, you'd be out of
luck.
Not necessarily.
There are tools such as http://www.let.rug.nl/~vannoord/TextCat/ which
provide very reliable guessing of languages. A minor adaptation might be
needed to make it guess twice, once for each of ISO-8859-1 and
ISO-8859-2 and then you take the highest ranked.
cheers
stuart
--
Stuart Yeates
http://www.nzetc.org/ New Zealand Electronic Text Centre
http://researcharchive.vuw.ac.nz/ Institutional Repository
