On 11/3/26 19:17, Jerry D wrote:
On 11/3/26 5:06 PM, Steve Kargl wrote:
On 11/3/26 16:08, Jerry D wrote:
On 11/3/26 2:39 PM, Harald Anlauf wrote:
Dear all,
this patch seems to be a very special challenge:
The testcase has an UTF-8 byte order mark, which I do not see in the
patch unless I use "od". If you don't see it, then please see the
attachment in the PR for reference.
We used a too short line buffer when reading the testcase, leading
to valgrind invalid read/writes. The solution here is to extend
the line buffer, taking care of line length limits. I do hope I
got it right.
Regtested on x86_64-pc-linux-gnu. OK for mainline?
Thanks,
Harald
Yes, OK Harald.
I did a read of the standard and this is acceptable.
Is it?
6.1.1 Characters
The processor character set is processor dependent. Each character
in a processor character set is either a control character or a
graphic character. The set of graphic characters is further divided
into letters (6.1.2), digits (6.1.3), underscore (6.1.4), special
characters (6.1.5), and other characters (6.1.6).
The letters, digits, underscore, and special characters make up the
Fortran character set.
A UTF-8 BOM is not a member of the special characters (6.1.5). It can
be a member of the processor character set.
6.1.6 Other characters
Additional characters may be representable in the processor, but
shall appear only in comments (6.3.2.3, 6.3.3.2), character constants
(7.4.4), input/output records (12.2.2), and character string edit
descriptors (13.3.2).
I think it needs to be hidden behind an -fallow-utf-8-bom option
or some such knob. At a minimum, a warning should be issued.
I had this:
- §6.1.1 / Annex A: the processor character set is processor dependent.
Only the Fortran character set (letters, digits, underscore, special
characters) is required, and only those characters can form tokens. So
identifiers, keywords and operators must be ASCII-range.
- §6.1.6 Other characters: "Additional characters may be representable
in the processor, but shall appear only in comments, character
constants, input/output records, and character string edit descriptors."
This is the clause that permits non-ASCII text such as UTF-8 at all, and
it confines it to those places.
- §6.3.1 ¶2: "A comment may contain any character that may occur in any
character context."
- §7.4.4.3 ¶3: a rep-char in a character literal is "any graphic
character in the processor-dependent character set" in free form, and
any character in fixed form. The kind prefix (e.g. ucs4_'...') picks the
kind, but how source bytes map to characters of that kind is not specified.
- §7.4.4.1 / §3.89: the ISO 10646 character kind is defined as UCS-4,
not UTF-8. That is the in-memory representation from
SELECTED_CHAR_KIND('ISO_10646').
- Line length: free form allows "at most ten thousand characters"
(§6.3.2.1). Fixed form requires exactly 72 characters for a line of
default-kind characters only; if a line has other characters, the
maximum is processor dependent (§6.3.3.1). The standard doesn't say
whether a multibyte UTF-8 sequence counts as one character or several,
which is why fixed-form column counting with UTF-8 is a processor choice.
I interpreted the last bullet to allow as processor dependent.
I'll let you and Harald decide how to proceed as I have not
been too active in gfortran development lately. However, I
think 6.1.6 is clear. A utf-8 BOM shall not appear as the first
character in a line.
--
steve
