Perl use the UTF8_ALLOW_ANYUV mask in functions that should not be restricted to only the valid Unicode code points. For some reason this mask currently include the UTF8_ALLOW_LONG flag. This seems totally wrong as there can't be a good reason to allow overlong sequences just because we don't want to restrict the valid values.
Perl's ord() function is for instance perfectly happy with an overlong NUL: $ perl -MEncode -wle '$a = "\xe0\x80\x80";Encode::_utf8_on($a);print ord($a)' 0 This patch fixes this problem: --- utf8.h.cur 2004-12-06 11:16:52.176181667 +0100 +++ utf8.h 2004-12-06 11:17:16.672129909 +0100 @@ -183,8 +183,7 @@ #define UTF8_ALLOW_FFFF 0x0040 /* Allows also FFFE. */ #define UTF8_ALLOW_LONG 0x0080 #define UTF8_ALLOW_ANYUV (UTF8_ALLOW_EMPTY|UTF8_ALLOW_FE_FF|\ - UTF8_ALLOW_SURROGATE|\ - UTF8_ALLOW_FFFF|UTF8_ALLOW_LONG) + UTF8_ALLOW_SURROGATE|UTF8_ALLOW_FFFF) #define UTF8_ALLOW_ANY 0x00FF #define UTF8_CHECK_ONLY 0x0200 With this patch the example above outputs: $ perl -MEncode -wle '$a = "\xe0\x80\x80";Encode::_utf8_on($a);print ord($a)' Malformed UTF-8 character (3 bytes, need 1, after start byte 0xe0) in ord at -e line 1. 0 Regards, Gisle