Tony,
Thank you for your proposals!
I will take them to heart and improve the docs accordingly. Some parts are
20 years old, so no surprise that they are not exactly correct any more...
Michael.
On Fri, 18 Aug 2017, Tony Whyman wrote:
There has been some heated discussion on the Lazarus lists on the
subject to character encodings etc. This has exposed several issues with
the FPC Manual that I wanted to record.
1. The char type
The manual says: "A Char is exactly 1 byte in size, and contains one
ASCII character. "
This was probably true when Pascal was first defined, but char is often
now used for any on-byte character set e.g. ISO 8859-1. Replace ASCII
with ANSI.
2. WideChar
The Manual says: "A WideChar is exactly 2 bytes in size, and contains
one UNICODE character in UTF-16 encoding. "
This seems to be wrong as UTF-16 is not limited to code points defined
using a single 16-bit code unit, but also permits code points comprising
two 16-bit code units. The definition should be updated to indicate that
a WideChar was really created for the obsolescent UCS-2 and is limited
to a UTF-16 subset (Unicode characters that can be expressed as a single
16-bit code unit).
Proposed replacement text: "A WideChar is exactly 2 bytes in size, and
contains one UNICODE character in UCS-2 encoding or UTF-16 encoding
limited to the Basic Multilingual Plane. Note that Unicode Characters
represented by a UTF-16 code points that require two 16-bit code units
cannot be contained in a single WideChar variable."
3. UnicodeStrings
The Manual says: "For multi-byte string types, the basic character has a
size of at least 2."
Proposed improvement:
"Multi-byte string types are used to represent Unicode characters
encoded as code points requiring two or four bytes".
As with UTF8String, the following caveat should also be added:
"Since a unicode character may require two or four bytes to be
represented in the UTF-16 encoding, there are 2 points to take care of
when using UnicodeString/WideString:
1. The character index – which retrieves a WideChar at a certain
position – must be used with care: the expression S[i] will not
necessarily be a valid character for a string S of type
UnicodeString/WideString.
2. The length of the string is not necessarily equal to the number of
elements in the array. The standard function length cannot be used to
get the character length of the string, it will always return the array
length.
--
Wiki Page on "Character and string Type"
1. This needs to start with a Health Warning on the use of the word
Unicode. Proposed Text (borrowing from Wikipedia):
"Free Pascal supports several character and string types. They range
from single ANSI characters to unicode strings and also include pointer
types. Differences also apply to encodings and reference counting. ANSI
is typically used to refer to single byte character encodings - although
FPC also uses AnsiStrings to hold Unicode UTF-8 encoded strings.
Unicode is a computing industry standard for the consistent encoding,
representation, and handling of text expressed in most of the world's
writing systems. Developed in conjunction with the Universal Coded
Character Set (UCS) standard and published as The Unicode Standard, the
latest version of Unicode contains a repertoire of 136,755 characters
covering 139 modern and historic scripts, as well as multiple symbol sets.
Unicode can be implemented by different character encodings. The Unicode
standard defines UTF-8, UTF-16, and UTF-32, and several other encodings
are in use. The most commonly used encodings are UTF-8, UTF-16 and
UCS-2, a precursor of UTF-16.
The original idea behind Unicode was to replace the typical
256-character encodings requiring 1 byte per character with an encoding
using 2^16 = 65,536 values requiring 2 bytes per character.The early
2-byte encoding was usually called "Unicode", but is now called "UCS-2".
UCS-2 differs from UTF-16 by being a constant length encoding and only
capable of encoding characters of Basic Multilingual Plane (BMP), it is
supported by many programs. However, "UCS-2 should now be considered
obsolete. It no longer refers to an encoding form in either 10646 or the
Unicode Standard.
Unfortunately, the term Unicode, in common usage, is still often used to
refer to the UCS-2 two byte encoding and this can give rise to much
confusion e.g. when Unicode is used when referring to the UTF-8 encoding."
2. The text on WideChar is too terse and needs to be expanded. Proposed
text:
"A variable of type WideChar, also referred to as UnicodeChar (which
derives from the archaic use of Unicode to mean UCS-2), is exactly 2
bytes in size, and usually contains either:
(a) a single UCS-2 code point, or
(b) a single UTF-16 code unit.
In case (b), this is sufficient for Unicode Characters that have a
UTF-16 code point that comprises a single
