Henri Sivonen on 2006-03-11:
I think it would be beneficial to additionally stipulate that
1. The meta element-based character encoding information declaration
is expected to work only if the Basic Latin range of characters maps
to the same bytes as in the US-ASCII encoding.
Is this realistic? I'm not really familiar enough with character
encodings to say if this is what happens in general.
I suppose it is realistic. See below.
Yes, for most encodings, the US-ASCII range is the same, and if you restrict
it a bit further (the "INVARIANT" charset in RFC 1345), it covers most of
the ambiguous encodings. The others can be easily detected as they usually
have very different bit patterns (EBCDIC) or word lengths (UTF-16, UTF-32).
2. If there is no external character encoding information nor a BOM
(see below), there MUST NOT be any non-ASCII bytes in the document
byte stream before the end of the meta element that declares the
character encoding. (In practice this would ban unescaped non-ASCII
class names on the html and [head] elements and non-ASCII comments at
the beginning of the document.)
Again, can we realistically require this? I need to do some studies of
non-latin pages, I guess.
As UA behavior, no. As a conformance requirement, maybe.
If you require browsers to switch on-the-fly, they can redo the decoding
when they find the <meta> anyway, and this is no longer a problem. There are
a lot of documents with non-ASCII-language comments and <title> tags that
are positioned before the <meta>.
Authors should avoid including inline character encoding
information. Character encoding information should instead be
included at the transport level (e.g. using the HTTP Content-Type
header).
I disagree.
With HTML with contemporary UAs, there is no real harm in including
the character encoding information both on the HTTP level and in the
meta as long as the information is not contradictory. On the contrary,
the author-provided internal information is actually useful when end
users save pages to disk using UAs that do not reserialize with
internal character encoding information.
...and it breaks everything when you have a transcoding proxy, or
similar.
Well, not until you save to disk, since HTTP takes precedence. However,
authors can escape this by using UTF-8. (Assuming here that tampering with
UTF-8 would be harmful, wrong and pointless.)
Interestingly, transcoding proxies tend to be brought up by residents of
Western Europe, North America or the Commonwealth. I have never seen a
Russion person living in Russia or a Japanese person living in Japan talk
about transcoding proxies in any online or offline discussion. That's why
I doubt the importance of transcoding proxies.
Transcoding is very popular, especially in Russia. With mod_charset in
Apache it will (AFAICT) use the information in the <meta> of the document to
determine the source encoding and then transcode it to an encoding it
believes the client can handle (based on browser sniffing). It transcodes on
a byte level, so the <meta> reamains unchanged, but is overridden by the
HTTP header.
The <meta> tag is really information to the server, it is the server that is
*supposed* to read it and post the data into the HTTP header. Unfortunately
not many servers support that, leaving us with having to parse them in the
browsers instead. Reading the <meta> tag for encoding information is
basically at the same level as guessing the encoding by frequency
analysis--The server didn't say anything so perhaps you can get lucky.
Character encoding information shouldn't be duplicated, IMHO, that's
just asking for trouble.
I suggest a mismatch be considered an easy parse error and, therefore,
reportable.
That will not work in the mod_charset case above.
For HTML, user agents must use the following algorithm in determining the
character encoding of a document:
1. If the transport layer specifies an encoding, use that.
Shouldn't there be a BOM-sniffing step here? (UTF-16 and UTF-8 only;
UTF-32 makes no practical sense for interchange on the Web.)
I don't know, should there?
I believe there should.
BOM-sniffing should be done *after* looking at the transport layer's
information. It might know something you don't. It's a part of the
"guessing-the-content" step.
Requirements I'd like to see:
Documents must specify a character encoding an must use an IANA-registered
encoding and must identify it using its preferred MIME name or use a BOM
(with UTF-8, UTF-16 or UTF-32). UAs must recognize the preferred MIME name
of every encoding they support that has a preferred MIME name. UAs should
recognize IANA-registered aliases.
That could be useful, the only problem being that the IANA list of encoding
labels is a bit difficult to read when you want to try figuring out which
name to write.
Documents must not use UTF-EBCDIC, BOCU-1, CESU-8, UTF-7, UTF-16BE (i.e.
BOMless), UTF-16LE, UTF-32BE, UTF-32LE or any encodings from the EBCDIC
family of encodings. Documents using the UTF-16 or UTF-32 encodings must
have a BOM.
I don't think forbidding BOCU-1 is a good idea. If there is ever a proper
specification written of it, it could be very useful as a compression format
for documents.
Encoding errors are easy parse errors. (Emit U+FFFD on bogus data.)
Yes, especially since encoding defintions tend to change over time.
Authors are adviced to use the UTF-8 encoding. Authors are adviced not to
use the UTF-32 encoding or legacy encodings. (Note: I think UTF-32 on the
Web is harmful and utterly pointless, but Firefox and Opera support it.
UTF-32 can be useful as an internal format, but I agree that it's not very
useful on the web. Not sure about the "harmful" bit, though.
--
\\//
Peter, software engineer, Opera Software
The opinions expressed are my own, and not those of my employer.
Please reply only by follow-ups on the mailing list.
