Re: Fw: Problems with compiling autogen with GCC8 or newer versions

2021-01-10 Thread Bruce Korb via Gcc

Hi,

On 1/10/21 3:56 PM, Martin Sebor wrote:

Sure.  I was confirming that based on the GCC dump there is no risk
of an overflow in the translation unit, and so there is no warning.


OK. :) I didn't understand the GCC dump. Despite having commit privs, 
I'm not actually a compiler guru.



It can /not/ overflow. Those compiler stats are not decipherable by me.


They indicate the minimum, likely, maximum, and unlikely maximum
number of bytes of output for each directive and the running totals
for the call (in parentheses).  The relevant lines are these:

  Directive 2 at offset 2: "%s"
    Result: 0, 255, 255, 255 (2, 257, 257, 257)

The result tells us that the length of the %s argument is between
0 and 255 bytes long.
It should be 1 to 255. 0 is actually impossible, but it would take crazy 
complicated sleuthing to figure it out, even though the "spn_*" 
functions should be inlined.


Since objsize (the size of the destination) is 520 there is no
buffer overflow.
The size of the destination is guaranteed to be between 263 and 518 
bytes. The "def_str" pointer will always point at least two bytes past 
the start of the 520 byte buffer.


The note in the forwarded message indicates that GCC computes
the destination size to be much smaller for some reason:

  note: 'sprintf' output between 4 and 259 bytes into a destination of 
size 255


I.e., it thinks it's just 255 bytes.  As I explained, such a small
size would trigger the warning by design.
Yep. If it can accurately figure out the minimum size remaining, that 
would be completely fine. "If."

  I can't really think of
a reason why GCC would compute a smaller size here (it looks far
from trivial).
If it can figure out that the minimum size is 263, that'd be great. If 
it can't, it needs to be quiet.

  We'd need to see the original poster's translation
unit and know the host and the target GCC was configured for.


OK. Not anything I can do. Thomas would have to send in his version of 
"gd.i".



Thank you!

Regards, Bruce



Re: Fw: Problems with compiling autogen with GCC8 or newer versions

2021-01-10 Thread Martin Sebor via Gcc

On 1/10/21 12:28 PM, Bruce Korb wrote:

Hi Martin,

On 1/10/21 11:01 AM, Martin Sebor wrote:

On 1/8/21 12:38 PM, Bruce Korb via Gcc wrote:
This is the code that must be confusing to GCC. "def_str" points to 
the second character in the 520 byte buffer. "def_scan" points to a 
character that we already know we're going to copy into the 
destination, so the "spn" function doesn't look at it:


 {
 char * end = spn_ag_char_map_chars(def_scan + 1, 31);
 size_t len = end - def_scan;
 if (len >= 256)
 goto fail_return;
 memcpy(def_str, def_scan, len);
 def_str += len;
 *def_str = '\0';
 def_scan = end;
 }


In the function preamble, "def_str" points to the first character 
(character "0") of a 520 byte buffer. Before this fragment, "*def_str" 
is set to an apostrophe and the pointer advanced. After execution passes 
through this fragment, "def_str" is pointing to a NUL byte that can be 
as far as 257 bytes into the buffer (character "257"). That leaves 263 
more bytes. The "offending" sprintf is:


sprintf(def_str, "  %s'", name_bf);


Sure.  I was confirming that based on the GCC dump there is no risk
of an overflow in the translation unit, and so there is no warning.



GCC correctly determines that "name_bf" cannot contain more than 255 
bytes. Add 3 bytes of text and a NUL byte and the sprintf will be 
dropping *AT MOST* 259 characters into the buffer. The buffer is 4 bytes 
longer than necessary.




GCC 8 also doesn't warn but it does determine the size.  Here's
the output for the relevant directive (from the output of
-fdump-tree-printf-return-value in GCC versions prior to 10, or
-fdump-tree-strlen in GCC 10 and later).  objsize is the size
of the destination, or 520 bytes here (this is in contrast to
the 255 in the originally reported message). The Result numbers
are the minimum and maximum size of the output (between 0 and
255 characters).

Computing maximum subobject size for def_str_146:
getdefs.c:275: sprintf: objsize = 520, fmtstr = "  %s'"
  Directive 1 at offset 0: "  ", length = 2
    Result: 2, 2, 2, 2 (2, 2, 2, 2)
  Directive 2 at offset 2: "%s"
    Result: 0, 255, 255, 255 (2, 257, 257, 257)
  Directive 3 at offset 4: "'", length = 1
    Result: 1, 1, 1, 1 (3, 258, 258, 258)
  Directive 4 at offset 5: "", length = 1

Besides provable overflow, it's worth noting that -Wformat-overflow

It can /not/ overflow. Those compiler stats are not decipherable by me.


They indicate the minimum, likely, maximum, and unlikely maximum
number of bytes of output for each directive and the running totals
for the call (in parentheses).  The relevant lines are these:

  Directive 2 at offset 2: "%s"
Result: 0, 255, 255, 255 (2, 257, 257, 257)

The result tells us that the length of the %s argument is between
0 and 255 bytes long.

Since objsize (the size of the destination) is 520 there is no
buffer overflow.

The note in the forwarded message indicates that GCC computes
the destination size to be much smaller for some reason:

  note: 'sprintf' output between 4 and 259 bytes into a destination of 
size 255


I.e., it thinks it's just 255 bytes.  As I explained, such a small
size would trigger the warning by design.  I can't really think of
a reason why GCC would compute a smaller size here (it looks far
from trivial).  We'd need to see the original poster's translation
unit and know the host and the target GCC was configured for.

Martin


also diagnoses a subset of cases where it can't prove that overflow
cannot happen.  One common case is:

  extern char a[8], b[8];
  sprintf (a, "a=%s", b);
My example would have the "a" array sized at 16 bytes and "b" provably 
not containing more than 7 characters (plus a NUL). There would be no 
overflow.

... The solution is to either use precision
to constrain the amount of output or in GCC 10 and later to assert
that b's length is less than 7.
See the fragment below to see where the characters in name_bf can 
*/NOT/* be more than 255. There is no need for either a precision 
constraint or an assertion, based on that code fragment.


So if in the autogen file def_str is ever less than 258 bytes/[259 -- 
NUL byte, too]/

I'd expect the warning to trigger with a message close to
the original.


It can not be less than 263. For the sake of those not reading the code, 
here is the fragment that fills in "name_bf[256]":



     {
     char * end = spn_ag_char_map_chars(def_scan + 1, 26);
     size_t len = end - def_scan;
     if (len >= 256)
     goto fail_return;
     memcpy(name_bf, def_scan, len);
     name_bf[len] = '\0';
     def_scan = end;
     }






Re: Fw: Problems with compiling autogen with GCC8 or newer versions

2021-01-10 Thread Bruce Korb via Gcc

Hi Martin,

On 1/10/21 11:01 AM, Martin Sebor wrote:

On 1/8/21 12:38 PM, Bruce Korb via Gcc wrote:
This is the code that must be confusing to GCC. "def_str" points to 
the second character in the 520 byte buffer. "def_scan" points to a 
character that we already know we're going to copy into the 
destination, so the "spn" function doesn't look at it:


 {
 char * end = spn_ag_char_map_chars(def_scan + 1, 31);
 size_t len = end - def_scan;
 if (len >= 256)
 goto fail_return;
 memcpy(def_str, def_scan, len);
 def_str += len;
 *def_str = '\0';
 def_scan = end;
 }


In the function preamble, "def_str" points to the first character 
(character "0") of a 520 byte buffer. Before this fragment, "*def_str" 
is set to an apostrophe and the pointer advanced. After execution passes 
through this fragment, "def_str" is pointing to a NUL byte that can be 
as far as 257 bytes into the buffer (character "257"). That leaves 263 
more bytes. The "offending" sprintf is:


sprintf(def_str, "  %s'", name_bf);

GCC correctly determines that "name_bf" cannot contain more than 255 
bytes. Add 3 bytes of text and a NUL byte and the sprintf will be 
dropping *AT MOST* 259 characters into the buffer. The buffer is 4 bytes 
longer than necessary.




GCC 8 also doesn't warn but it does determine the size.  Here's
the output for the relevant directive (from the output of
-fdump-tree-printf-return-value in GCC versions prior to 10, or
-fdump-tree-strlen in GCC 10 and later).  objsize is the size
of the destination, or 520 bytes here (this is in contrast to
the 255 in the originally reported message). The Result numbers
are the minimum and maximum size of the output (between 0 and
255 characters).

Computing maximum subobject size for def_str_146:
getdefs.c:275: sprintf: objsize = 520, fmtstr = "  %s'"
  Directive 1 at offset 0: "  ", length = 2
    Result: 2, 2, 2, 2 (2, 2, 2, 2)
  Directive 2 at offset 2: "%s"
    Result: 0, 255, 255, 255 (2, 257, 257, 257)
  Directive 3 at offset 4: "'", length = 1
    Result: 1, 1, 1, 1 (3, 258, 258, 258)
  Directive 4 at offset 5: "", length = 1

Besides provable overflow, it's worth noting that -Wformat-overflow

It can /not/ overflow. Those compiler stats are not decipherable by me.

also diagnoses a subset of cases where it can't prove that overflow
cannot happen.  One common case is:

  extern char a[8], b[8];
  sprintf (a, "a=%s", b);
My example would have the "a" array sized at 16 bytes and "b" provably 
not containing more than 7 characters (plus a NUL). There would be no 
overflow.

... The solution is to either use precision
to constrain the amount of output or in GCC 10 and later to assert
that b's length is less than 7.
See the fragment below to see where the characters in name_bf can 
*/NOT/* be more than 255. There is no need for either a precision 
constraint or an assertion, based on that code fragment.


So if in the autogen file def_str is ever less than 258 bytes/[259 -- 
NUL byte, too]/

I'd expect the warning to trigger with a message close to
the original.


It can not be less than 263. For the sake of those not reading the code, 
here is the fragment that fills in "name_bf[256]":



    {
    char * end = spn_ag_char_map_chars(def_scan + 1, 26);
    size_t len = end - def_scan;
    if (len >= 256)
    goto fail_return;
    memcpy(name_bf, def_scan, len);
    name_bf[len] = '\0';
    def_scan = end;
    }




Re: Fw: Problems with compiling autogen with GCC8 or newer versions

2021-01-10 Thread Martin Sebor via Gcc

On 1/8/21 12:38 PM, Bruce Korb via Gcc wrote:

    $ bash cc.sh
    + wrn+=' -Werror=format-overflow'
    + gcc -DHAVE_CONFIG_H -I. -I.. -I../autoopts
    -Wno-format-contains-nul -Wall -Werror -Wcast-align
    -Wmissing-prototypes -Wpointer-arith -Wshadow -Wstrict-prototypes
    -Wwrite-strings -Wstrict-aliasing=3 -Wextra -Wno-cast-qual
    -Werror=format-overflow -fno-strict-aliasing -g -O2 -MT gd.o -MD -MP
    -MF .deps/gd.Tpo -c -o gd.o gd.c
    $ bash cc.sh/(edited to do pre-processing only, to wit:)/
    + gcc -DHAVE_CONFIG_H -I. -I.. -I../autoopts -E -o gd.i gd.c
    $ gcc --version
    gcc (SUSE Linux) 7.5.0
    Copyright (C) 2017 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions. There
    is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
    PURPOSE.

buildPreamble() starts on line 10230.

This is the code that must be confusing to GCC. "def_str" points to the 
second character in the 520 byte buffer. "def_scan" points to a 
character that we already know we're going to copy into the destination, 
so the "spn" function doesn't look at it:


     {
     char * end = spn_ag_char_map_chars(def_scan + 1, 31);
     size_t len = end - def_scan;
     if (len >= 256)
     goto fail_return;
     memcpy(def_str, def_scan, len);
     def_str += len;
     *def_str = '\0';
     def_scan = end;
     }

"spn_ag_char_map_chars" skips over characters that are in the 31st 
character set. (It's generated.)
If there are too many characters in that set, this code fragment will go 
into failure mode.


*IF* this .i file doesn't generate the error, then Thomas' compiler must 
be yielding something weird in its preprocessing output. It compiles 
just fine with my 7.5.0 compiler.




In my testing GCC 7 doesn't warn for any of the sprintf callss
in the translation unit, at least in part because it determine
the size of the destination.

GCC 8 also doesn't warn but it does determine the size.  Here's
the output for the relevant directive (from the output of
-fdump-tree-printf-return-value in GCC versions prior to 10, or
-fdump-tree-strlen in GCC 10 and later).  objsize is the size
of the destination, or 520 bytes here (this is in contrast to
the 255 in the originally reported message). The Result numbers
are the minimum and maximum size of the output (between 0 and
255 characters).

Computing maximum subobject size for def_str_146:
getdefs.c:275: sprintf: objsize = 520, fmtstr = "  %s'"
  Directive 1 at offset 0: "  ", length = 2
Result: 2, 2, 2, 2 (2, 2, 2, 2)
  Directive 2 at offset 2: "%s"
Result: 0, 255, 255, 255 (2, 257, 257, 257)
  Directive 3 at offset 4: "'", length = 1
Result: 1, 1, 1, 1 (3, 258, 258, 258)
  Directive 4 at offset 5: "", length = 1

Besides provable overflow, it's worth noting that -Wformat-overflow
also diagnoses a subset of cases where it can't prove that overflow
cannot happen.  One common case is:

  extern char a[8], b[8];
  sprintf (a, "a=%s", b);

  warning: ‘%s’ directive writing up to 7 bytes into a region of size 6 
[-Wformat-overflow=]


Although b's length isn't known, GCC uses its size to warn about
the possible overflow.  The solution is to either use precision
to constrain the amount of output or in GCC 10 and later to assert
that b's length is less than 7.

So if in the autogen file def_str is ever less than 258 bytes
I'd expect the warning to trigger with a message close to
the original.

Martin


On 1/8/21 11:01 AM, Jeff Law wrote:

On 1/8/21 10:39 AM, Bruce Korb via Gcc wrote:

Also, GCC's code analysis is wrong. "name_bf" contains *NO MORE* than
MAXNAMELEN characters. That is provable.

"def_str" points into a buffer of size ((MAXNAMELEN * 2) + 8) and at
an offset maximum of MAXNAMELEN+1 (also provable), meaning that at a
minimum there are MAXNAMELEN+6 bytes left in the buffer.

That objected-to sprintf can add a maximum of MAXNAMELEN + 4 to where
"def_str" points.

GCC is wrong. It is unable to figure out how far into the buffer
"def_str" can point.
Can you get a .i file, command line and file a report.  It'd be 
appreciated.


jeff





Re: Fw: Problems with compiling autogen with GCC8 or newer versions

2021-01-08 Thread Jeff Law via Gcc



On 1/8/21 10:39 AM, Bruce Korb via Gcc wrote:
> Hi,
>
> You are supposed to be able to post once you've subscribed.
>
> Also, GCC's code analysis is wrong. "name_bf" contains *NO MORE* than
> MAXNAMELEN characters. That is provable.
>
> "def_str" points into a buffer of size ((MAXNAMELEN * 2) + 8) and at
> an offset maximum of MAXNAMELEN+1 (also provable), meaning that at a
> minimum there are MAXNAMELEN+6 bytes left in the buffer.
>
> That objected-to sprintf can add a maximum of MAXNAMELEN + 4 to where
> "def_str" points.
>
> GCC is wrong. It is unable to figure out how far into the buffer
> "def_str" can point.
Can you get a .i file, command line and file a report.  It'd be appreciated.

jeff



Re: Fw: Problems with compiling autogen with GCC8 or newer versions

2021-01-08 Thread Bruce Korb via Gcc

Hi,

You are supposed to be able to post once you've subscribed.

Also, GCC's code analysis is wrong. "name_bf" contains *NO MORE* than 
MAXNAMELEN characters. That is provable.


"def_str" points into a buffer of size ((MAXNAMELEN * 2) + 8) and at an 
offset maximum of MAXNAMELEN+1 (also provable), meaning that at a 
minimum there are MAXNAMELEN+6 bytes left in the buffer.


That objected-to sprintf can add a maximum of MAXNAMELEN + 4 to where 
"def_str" points.


GCC is wrong. It is unable to figure out how far into the buffer 
"def_str" can point.


On 1/8/21 2:26 AM, Oppe, Thomas C ERDC-RDE-ITL-MS Contractor wrote:

Dear Sir:

I would like to post the following message to the mailing list 
"autogen-us...@lists.sourceforge.net".  Could you please add me to this list?

I am an HPC user at ERDC DSRC in Vicksburg, MS.  One of my projects is building GCC 
snapshots and releases using various software prerequisite packages necessary in the 
"make check" phase.  One of these packages is autogen-5.18.16.

Thank you for your consideration.

Tom Oppe


-
Thomas C. Oppe
HPCMP Benchmarking Team
HITS Team SAIC
thomas.c.o...@erdc.dren.mil
Work:  (601) 634-2797
Cell:(601) 642-6391
-


From: Oppe, Thomas C ERDC-RDE-ITL-MS Contractor
Sent: Friday, January 8, 2021 12:32 AM
To: autogen-us...@lists.sourceforge.net
Subject: Problems with compiling autogen with GCC8 or newer versions

Dear Sir:

When compiling autogen-5.18.16 with gcc8 or newer, I am getting format overflow errors 
like the following during the "make" step:

top_builddir=".." top_srcdir=".." VERBOSE="" /bin/bash "../build-aux/run-ag.sh" 
-MFstamp-opts -MTstamp-opts -MP ./opts.def
gcc -DHAVE_CONFIG_H -I. -I..  -I.. -I../autoopts  -g -O2 
-I/p/home/oppe/gcc/10.2.0/include -Wno-format-contains-nul -fno-strict-aliasing 
-Wall -Werror -Wcast-align -Wmissing-prototypes -Wpointer-arith -Wshadow 
-Wstrict-prototypes -Wwrite-strings -Wstrict-aliasing=3 -Wextra -Wno-cast-qual 
-g -O2 -I/p/home/oppe/gcc/10.2.0/include -Wno-format-contains-nul 
-fno-strict-aliasing -c -o gd.o gd.c
In file included from gd.c:11:
getdefs.c: In function 'buildDefinition':
getdefs.c:451:29: error: '%s' directive writing up to 255 bytes into a region 
of size 253 [-Werror=format-overflow=]
   451 | sprintf(def_str, "  %s'", name_bf);
   | ^~~~~
getdefs.c:451:9: note: 'sprintf' output between 4 and 259 bytes into a 
destination of size 255
   451 | sprintf(def_str, "  %s'", name_bf);
   | ^~
cc1: all warnings being treated as errors
make[2]: *** [gd.o] Error 1
make[2]: Leaving directory `/p/work1/oppe/autogen-5.18.16/getdefs'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/p/work1/oppe/autogen-5.18.16'
make: *** [all] Error 2

Do I just add "-Wno-error=format-overflow" to the compile options?  Do you have 
a fix?

Tom Oppe

-
Thomas C. Oppe
HPCMP Benchmarking Team
HITS Team SAIC
thomas.c.o...@erdc.dren.mil
Work:  (601) 634-2797
Cell:(601) 642-6391
-