On 2012/11/17 9:45, Doug Ewell wrote:
If he is targeting HTML5, then none of this matters, because HTML5 says that ISO 8859-1 is really Windows-1252.
Yes. But unless Python wants to limit its use to HTML5, this should be handled on a separate level (mapping a "iso-8859-1" label to the Windows-1252 decoder logic), not by trying to change ISO-8859-1 itself.
Regards, Martin.
For example, there is no C1 control called NL in Windows-1252. There is only 0x85, which maps to U+2026 HORIZONTAL ELLIPSIS. -- Doug Ewell | Thornton, Colorado, USA http://www.ewellic.org | @DougEwell From: Philippe Verdy Sent: Friday, November 16, 2012 17:35 To: Whistler, Ken Cc: Buck Golemon ; unicode@unicode.org Subject: Re: latin1 decoder implementation In fact not really, because Unicode DOES assign more precise semantics to a few of these controls, notably for those given whitespace and newline properties (notably TAB, LF, CR in C0 controls and NL in C1 controls, with a few additional constraints for the CR+LF sequence) as they are part of almost all plain text protocols ; NUL also has a specific behavior which is so common that it cannot be mapped to anything else than a terminator or separator of plain text sequences. So even if the ISO/IEC 8859 standard does not specify a charecter mapping in C0 and C1 controls, the registered MIME types are doing so (but nothing is well defined for the C0 and C1 controls except NUL, TAB, CR, LF, NL, for MIME usages purpose). And then yes, the ISO/IEC 8859 standard is different (more restrictive) from the MIME charsets defined by the IETF in some RFC's (and registered in the IANA registry), simply because the ISO/IEC standard (encoded charset) was developed to be compatible with various encoding schemes, some of them defined by ISO, some others defined by other standard European or East-Asian bodies (including 7-bit schemes, using escape sequences, or shift in/out controls). By itself, the ISO/IEC 8859 is not a complete encoding scheme, it is just defining several encoded character sets, independantly of the encoding schme used to store or transport it (it is not even sufficient to represent any plain-text content). On the opposite, The MIME "charsets" named "ISO_8859-*" registered by the IETF in the IANA registry are "concrete" encoding schemes, based on the ISO/IEC 8859 standard, and suitable for representing a plain-text content, because the MIME charsets are also adding a text presentation protocol. In practice, almost nobody today uses the ISO/IEC 8859 standard alone : there's always an additional concrete protocol added on top of it (which generally makes use of the C0 and C1 controls, but not necessarily, and not always the same way). So plain-text documents never use the ISO/IEC 8859 standard, but the MIME charsets (plus a few specific or proprietary charsets that have not been registered in the IANA registry as they are bound to a non-open protocol).
