On 11/3/26 5:06 PM, Steve Kargl wrote:
On 11/3/26 16:08, Jerry D wrote:
On 11/3/26 2:39 PM, Harald Anlauf wrote:
Dear all,
this patch seems to be a very special challenge:
The testcase has an UTF-8 byte order mark, which I do not see in the
patch unless I use "od". If you don't see it, then please see the
attachment in the PR for reference.
We used a too short line buffer when reading the testcase, leading
to valgrind invalid read/writes. The solution here is to extend
the line buffer, taking care of line length limits. I do hope I
got it right.
Regtested on x86_64-pc-linux-gnu. OK for mainline?
Thanks,
Harald
Yes, OK Harald.
I did a read of the standard and this is acceptable.
Is it?
6.1.1 Characters
The processor character set is processor dependent. Each character
in a processor character set is either a control character or a
graphic character. The set of graphic characters is further divided
into letters (6.1.2), digits (6.1.3), underscore (6.1.4), special
characters (6.1.5), and other characters (6.1.6).
The letters, digits, underscore, and special characters make up the
Fortran character set.
A UTF-8 BOM is not a member of the special characters (6.1.5). It can
be a member of the processor character set.
6.1.6 Other characters
Additional characters may be representable in the processor, but
shall appear only in comments (6.3.2.3, 6.3.3.2), character constants
(7.4.4), input/output records (12.2.2), and character string edit
descriptors (13.3.2).
I think it needs to be hidden behind an -fallow-utf-8-bom option
or some such knob. At a minimum, a warning should be issued.
I had this:
- §6.1.1 / Annex A: the processor character set is processor dependent. Only the
Fortran character set (letters, digits, underscore, special characters) is
required, and only those characters can form tokens. So identifiers, keywords
and operators must be ASCII-range.
- §6.1.6 Other characters: "Additional characters may be representable in the
processor, but shall appear only in comments, character constants, input/output
records, and character string edit descriptors." This is the clause that permits
non-ASCII text such as UTF-8 at all, and it confines it to those places.
- §6.3.1 ¶2: "A comment may contain any character that may occur in any
character context."
- §7.4.4.3 ¶3: a rep-char in a character literal is "any graphic character in
the processor-dependent character set" in free form, and any character in fixed
form. The kind prefix (e.g. ucs4_'...') picks the kind, but how source bytes map
to characters of that kind is not specified.
- §7.4.4.1 / §3.89: the ISO 10646 character kind is defined as UCS-4, not UTF-8.
That is the in-memory representation from SELECTED_CHAR_KIND('ISO_10646').
- Line length: free form allows "at most ten thousand characters" (§6.3.2.1).
Fixed form requires exactly 72 characters for a line of default-kind characters
only; if a line has other characters, the maximum is processor dependent
(§6.3.3.1). The standard doesn't say whether a multibyte UTF-8 sequence counts
as one character or several, which is why fixed-form column counting with UTF-8
is a processor choice.
I interpreted the last bullet to allow as processor dependent.
