Hi KK,
On 5/7/2009 at 2:55 AM, KK wrote:
In some of the pages I'm getting some \ufffd chars which I think is
some sort of unmappable[by Java?] character, right?. Any idea on how
to handle this? Just replacing with blank char will not do [this
depends on the requirement, though].
From http://www.unicode.org/charts/PDF/UFFF0.pdf:
FFFD: REPLACEMENT CHARACTER: used to replace an
incoming character whose value is unknown or
unrepresentable in Unicode.
Also, from http://www.unicode.org/versions/Unicode5.1.0/:
Applications are free to use any of these noncharacter
code points internally but should never attempt to
exchange them. If a noncharacter is received in open
interchange, an application is not required to
interpret it in any way. It is good practice, however,
to recognize it as a noncharacter and to take
appropriate action, such as replacing it with U+FFFD
REPLACEMENT CHARACTER, to indicate the problem in the
text. It is not recommended to simply delete
noncharacter code points from such text, because of
the potential security issues caused by deleting
uninterpreted characters. (See conformance clause C7
in Section 3.2, Conformance Requirements, and Unicode
Technical Report #36, Unicode Security
Considerations.)
So if you're seeing \ufffd in text, you (or someone before you in the
processing chain) attempted to convert the text from some other encoding into
Unicode, but the encoding conversion failed (no target Unicode character
corresponding to the source character). This can happen when attempting to
convert from an incorrectly identified source encoding.
Steve
