Re: Fw: Problems with compiling autogen with GCC8 or newer versions
Hi, On 1/10/21 3:56 PM, Martin Sebor wrote: Sure. I was confirming that based on the GCC dump there is no risk of an overflow in the translation unit, and so there is no warning. OK. :) I didn't understand the GCC dump. Despite having commit privs, I'm not actually a compiler guru. It can /not/ overflow. Those compiler stats are not decipherable by me. They indicate the minimum, likely, maximum, and unlikely maximum number of bytes of output for each directive and the running totals for the call (in parentheses). The relevant lines are these: Directive 2 at offset 2: "%s" Result: 0, 255, 255, 255 (2, 257, 257, 257) The result tells us that the length of the %s argument is between 0 and 255 bytes long. It should be 1 to 255. 0 is actually impossible, but it would take crazy complicated sleuthing to figure it out, even though the "spn_*" functions should be inlined. Since objsize (the size of the destination) is 520 there is no buffer overflow. The size of the destination is guaranteed to be between 263 and 518 bytes. The "def_str" pointer will always point at least two bytes past the start of the 520 byte buffer. The note in the forwarded message indicates that GCC computes the destination size to be much smaller for some reason: note: 'sprintf' output between 4 and 259 bytes into a destination of size 255 I.e., it thinks it's just 255 bytes. As I explained, such a small size would trigger the warning by design. Yep. If it can accurately figure out the minimum size remaining, that would be completely fine. "If." I can't really think of a reason why GCC would compute a smaller size here (it looks far from trivial). If it can figure out that the minimum size is 263, that'd be great. If it can't, it needs to be quiet. We'd need to see the original poster's translation unit and know the host and the target GCC was configured for. OK. Not anything I can do. Thomas would have to send in his version of "gd.i". Thank you! Regards, Bruce
Re: Fw: Problems with compiling autogen with GCC8 or newer versions
On 1/10/21 12:28 PM, Bruce Korb wrote: Hi Martin, On 1/10/21 11:01 AM, Martin Sebor wrote: On 1/8/21 12:38 PM, Bruce Korb via Gcc wrote: This is the code that must be confusing to GCC. "def_str" points to the second character in the 520 byte buffer. "def_scan" points to a character that we already know we're going to copy into the destination, so the "spn" function doesn't look at it: { char * end = spn_ag_char_map_chars(def_scan + 1, 31); size_t len = end - def_scan; if (len >= 256) goto fail_return; memcpy(def_str, def_scan, len); def_str += len; *def_str = '\0'; def_scan = end; } In the function preamble, "def_str" points to the first character (character "0") of a 520 byte buffer. Before this fragment, "*def_str" is set to an apostrophe and the pointer advanced. After execution passes through this fragment, "def_str" is pointing to a NUL byte that can be as far as 257 bytes into the buffer (character "257"). That leaves 263 more bytes. The "offending" sprintf is: sprintf(def_str, " %s'", name_bf); Sure. I was confirming that based on the GCC dump there is no risk of an overflow in the translation unit, and so there is no warning. GCC correctly determines that "name_bf" cannot contain more than 255 bytes. Add 3 bytes of text and a NUL byte and the sprintf will be dropping *AT MOST* 259 characters into the buffer. The buffer is 4 bytes longer than necessary. GCC 8 also doesn't warn but it does determine the size. Here's the output for the relevant directive (from the output of -fdump-tree-printf-return-value in GCC versions prior to 10, or -fdump-tree-strlen in GCC 10 and later). objsize is the size of the destination, or 520 bytes here (this is in contrast to the 255 in the originally reported message). The Result numbers are the minimum and maximum size of the output (between 0 and 255 characters). Computing maximum subobject size for def_str_146: getdefs.c:275: sprintf: objsize = 520, fmtstr = " %s'" Directive 1 at offset 0: " ", length = 2 Result: 2, 2, 2, 2 (2, 2, 2, 2) Directive 2 at offset 2: "%s" Result: 0, 255, 255, 255 (2, 257, 257, 257) Directive 3 at offset 4: "'", length = 1 Result: 1, 1, 1, 1 (3, 258, 258, 258) Directive 4 at offset 5: "", length = 1 Besides provable overflow, it's worth noting that -Wformat-overflow It can /not/ overflow. Those compiler stats are not decipherable by me. They indicate the minimum, likely, maximum, and unlikely maximum number of bytes of output for each directive and the running totals for the call (in parentheses). The relevant lines are these: Directive 2 at offset 2: "%s" Result: 0, 255, 255, 255 (2, 257, 257, 257) The result tells us that the length of the %s argument is between 0 and 255 bytes long. Since objsize (the size of the destination) is 520 there is no buffer overflow. The note in the forwarded message indicates that GCC computes the destination size to be much smaller for some reason: note: 'sprintf' output between 4 and 259 bytes into a destination of size 255 I.e., it thinks it's just 255 bytes. As I explained, such a small size would trigger the warning by design. I can't really think of a reason why GCC would compute a smaller size here (it looks far from trivial). We'd need to see the original poster's translation unit and know the host and the target GCC was configured for. Martin also diagnoses a subset of cases where it can't prove that overflow cannot happen. One common case is: extern char a[8], b[8]; sprintf (a, "a=%s", b); My example would have the "a" array sized at 16 bytes and "b" provably not containing more than 7 characters (plus a NUL). There would be no overflow. ... The solution is to either use precision to constrain the amount of output or in GCC 10 and later to assert that b's length is less than 7. See the fragment below to see where the characters in name_bf can */NOT/* be more than 255. There is no need for either a precision constraint or an assertion, based on that code fragment. So if in the autogen file def_str is ever less than 258 bytes/[259 -- NUL byte, too]/ I'd expect the warning to trigger with a message close to the original. It can not be less than 263. For the sake of those not reading the code, here is the fragment that fills in "name_bf[256]": { char * end = spn_ag_char_map_chars(def_scan + 1, 26); size_t len = end - def_scan; if (len >= 256) goto fail_return; memcpy(name_bf, def_scan, len); name_bf[len] = '\0'; def_scan = end; }
Re: Fw: Problems with compiling autogen with GCC8 or newer versions
Hi Martin, On 1/10/21 11:01 AM, Martin Sebor wrote: On 1/8/21 12:38 PM, Bruce Korb via Gcc wrote: This is the code that must be confusing to GCC. "def_str" points to the second character in the 520 byte buffer. "def_scan" points to a character that we already know we're going to copy into the destination, so the "spn" function doesn't look at it: { char * end = spn_ag_char_map_chars(def_scan + 1, 31); size_t len = end - def_scan; if (len >= 256) goto fail_return; memcpy(def_str, def_scan, len); def_str += len; *def_str = '\0'; def_scan = end; } In the function preamble, "def_str" points to the first character (character "0") of a 520 byte buffer. Before this fragment, "*def_str" is set to an apostrophe and the pointer advanced. After execution passes through this fragment, "def_str" is pointing to a NUL byte that can be as far as 257 bytes into the buffer (character "257"). That leaves 263 more bytes. The "offending" sprintf is: sprintf(def_str, " %s'", name_bf); GCC correctly determines that "name_bf" cannot contain more than 255 bytes. Add 3 bytes of text and a NUL byte and the sprintf will be dropping *AT MOST* 259 characters into the buffer. The buffer is 4 bytes longer than necessary. GCC 8 also doesn't warn but it does determine the size. Here's the output for the relevant directive (from the output of -fdump-tree-printf-return-value in GCC versions prior to 10, or -fdump-tree-strlen in GCC 10 and later). objsize is the size of the destination, or 520 bytes here (this is in contrast to the 255 in the originally reported message). The Result numbers are the minimum and maximum size of the output (between 0 and 255 characters). Computing maximum subobject size for def_str_146: getdefs.c:275: sprintf: objsize = 520, fmtstr = " %s'" Directive 1 at offset 0: " ", length = 2 Result: 2, 2, 2, 2 (2, 2, 2, 2) Directive 2 at offset 2: "%s" Result: 0, 255, 255, 255 (2, 257, 257, 257) Directive 3 at offset 4: "'", length = 1 Result: 1, 1, 1, 1 (3, 258, 258, 258) Directive 4 at offset 5: "", length = 1 Besides provable overflow, it's worth noting that -Wformat-overflow It can /not/ overflow. Those compiler stats are not decipherable by me. also diagnoses a subset of cases where it can't prove that overflow cannot happen. One common case is: extern char a[8], b[8]; sprintf (a, "a=%s", b); My example would have the "a" array sized at 16 bytes and "b" provably not containing more than 7 characters (plus a NUL). There would be no overflow. ... The solution is to either use precision to constrain the amount of output or in GCC 10 and later to assert that b's length is less than 7. See the fragment below to see where the characters in name_bf can */NOT/* be more than 255. There is no need for either a precision constraint or an assertion, based on that code fragment. So if in the autogen file def_str is ever less than 258 bytes/[259 -- NUL byte, too]/ I'd expect the warning to trigger with a message close to the original. It can not be less than 263. For the sake of those not reading the code, here is the fragment that fills in "name_bf[256]": { char * end = spn_ag_char_map_chars(def_scan + 1, 26); size_t len = end - def_scan; if (len >= 256) goto fail_return; memcpy(name_bf, def_scan, len); name_bf[len] = '\0'; def_scan = end; }
Re: Fw: Problems with compiling autogen with GCC8 or newer versions
On 1/8/21 12:38 PM, Bruce Korb via Gcc wrote: $ bash cc.sh + wrn+=' -Werror=format-overflow' + gcc -DHAVE_CONFIG_H -I. -I.. -I../autoopts -Wno-format-contains-nul -Wall -Werror -Wcast-align -Wmissing-prototypes -Wpointer-arith -Wshadow -Wstrict-prototypes -Wwrite-strings -Wstrict-aliasing=3 -Wextra -Wno-cast-qual -Werror=format-overflow -fno-strict-aliasing -g -O2 -MT gd.o -MD -MP -MF .deps/gd.Tpo -c -o gd.o gd.c $ bash cc.sh/(edited to do pre-processing only, to wit:)/ + gcc -DHAVE_CONFIG_H -I. -I.. -I../autoopts -E -o gd.i gd.c $ gcc --version gcc (SUSE Linux) 7.5.0 Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. buildPreamble() starts on line 10230. This is the code that must be confusing to GCC. "def_str" points to the second character in the 520 byte buffer. "def_scan" points to a character that we already know we're going to copy into the destination, so the "spn" function doesn't look at it: { char * end = spn_ag_char_map_chars(def_scan + 1, 31); size_t len = end - def_scan; if (len >= 256) goto fail_return; memcpy(def_str, def_scan, len); def_str += len; *def_str = '\0'; def_scan = end; } "spn_ag_char_map_chars" skips over characters that are in the 31st character set. (It's generated.) If there are too many characters in that set, this code fragment will go into failure mode. *IF* this .i file doesn't generate the error, then Thomas' compiler must be yielding something weird in its preprocessing output. It compiles just fine with my 7.5.0 compiler. In my testing GCC 7 doesn't warn for any of the sprintf callss in the translation unit, at least in part because it determine the size of the destination. GCC 8 also doesn't warn but it does determine the size. Here's the output for the relevant directive (from the output of -fdump-tree-printf-return-value in GCC versions prior to 10, or -fdump-tree-strlen in GCC 10 and later). objsize is the size of the destination, or 520 bytes here (this is in contrast to the 255 in the originally reported message). The Result numbers are the minimum and maximum size of the output (between 0 and 255 characters). Computing maximum subobject size for def_str_146: getdefs.c:275: sprintf: objsize = 520, fmtstr = " %s'" Directive 1 at offset 0: " ", length = 2 Result: 2, 2, 2, 2 (2, 2, 2, 2) Directive 2 at offset 2: "%s" Result: 0, 255, 255, 255 (2, 257, 257, 257) Directive 3 at offset 4: "'", length = 1 Result: 1, 1, 1, 1 (3, 258, 258, 258) Directive 4 at offset 5: "", length = 1 Besides provable overflow, it's worth noting that -Wformat-overflow also diagnoses a subset of cases where it can't prove that overflow cannot happen. One common case is: extern char a[8], b[8]; sprintf (a, "a=%s", b); warning: ‘%s’ directive writing up to 7 bytes into a region of size 6 [-Wformat-overflow=] Although b's length isn't known, GCC uses its size to warn about the possible overflow. The solution is to either use precision to constrain the amount of output or in GCC 10 and later to assert that b's length is less than 7. So if in the autogen file def_str is ever less than 258 bytes I'd expect the warning to trigger with a message close to the original. Martin On 1/8/21 11:01 AM, Jeff Law wrote: On 1/8/21 10:39 AM, Bruce Korb via Gcc wrote: Also, GCC's code analysis is wrong. "name_bf" contains *NO MORE* than MAXNAMELEN characters. That is provable. "def_str" points into a buffer of size ((MAXNAMELEN * 2) + 8) and at an offset maximum of MAXNAMELEN+1 (also provable), meaning that at a minimum there are MAXNAMELEN+6 bytes left in the buffer. That objected-to sprintf can add a maximum of MAXNAMELEN + 4 to where "def_str" points. GCC is wrong. It is unable to figure out how far into the buffer "def_str" can point. Can you get a .i file, command line and file a report. It'd be appreciated. jeff
Re: Fw: Problems with compiling autogen with GCC8 or newer versions
On 1/8/21 10:39 AM, Bruce Korb via Gcc wrote: > Hi, > > You are supposed to be able to post once you've subscribed. > > Also, GCC's code analysis is wrong. "name_bf" contains *NO MORE* than > MAXNAMELEN characters. That is provable. > > "def_str" points into a buffer of size ((MAXNAMELEN * 2) + 8) and at > an offset maximum of MAXNAMELEN+1 (also provable), meaning that at a > minimum there are MAXNAMELEN+6 bytes left in the buffer. > > That objected-to sprintf can add a maximum of MAXNAMELEN + 4 to where > "def_str" points. > > GCC is wrong. It is unable to figure out how far into the buffer > "def_str" can point. Can you get a .i file, command line and file a report. It'd be appreciated. jeff
Re: Fw: Problems with compiling autogen with GCC8 or newer versions
Hi, You are supposed to be able to post once you've subscribed. Also, GCC's code analysis is wrong. "name_bf" contains *NO MORE* than MAXNAMELEN characters. That is provable. "def_str" points into a buffer of size ((MAXNAMELEN * 2) + 8) and at an offset maximum of MAXNAMELEN+1 (also provable), meaning that at a minimum there are MAXNAMELEN+6 bytes left in the buffer. That objected-to sprintf can add a maximum of MAXNAMELEN + 4 to where "def_str" points. GCC is wrong. It is unable to figure out how far into the buffer "def_str" can point. On 1/8/21 2:26 AM, Oppe, Thomas C ERDC-RDE-ITL-MS Contractor wrote: Dear Sir: I would like to post the following message to the mailing list "autogen-us...@lists.sourceforge.net". Could you please add me to this list? I am an HPC user at ERDC DSRC in Vicksburg, MS. One of my projects is building GCC snapshots and releases using various software prerequisite packages necessary in the "make check" phase. One of these packages is autogen-5.18.16. Thank you for your consideration. Tom Oppe - Thomas C. Oppe HPCMP Benchmarking Team HITS Team SAIC thomas.c.o...@erdc.dren.mil Work: (601) 634-2797 Cell:(601) 642-6391 - From: Oppe, Thomas C ERDC-RDE-ITL-MS Contractor Sent: Friday, January 8, 2021 12:32 AM To: autogen-us...@lists.sourceforge.net Subject: Problems with compiling autogen with GCC8 or newer versions Dear Sir: When compiling autogen-5.18.16 with gcc8 or newer, I am getting format overflow errors like the following during the "make" step: top_builddir=".." top_srcdir=".." VERBOSE="" /bin/bash "../build-aux/run-ag.sh" -MFstamp-opts -MTstamp-opts -MP ./opts.def gcc -DHAVE_CONFIG_H -I. -I.. -I.. -I../autoopts -g -O2 -I/p/home/oppe/gcc/10.2.0/include -Wno-format-contains-nul -fno-strict-aliasing -Wall -Werror -Wcast-align -Wmissing-prototypes -Wpointer-arith -Wshadow -Wstrict-prototypes -Wwrite-strings -Wstrict-aliasing=3 -Wextra -Wno-cast-qual -g -O2 -I/p/home/oppe/gcc/10.2.0/include -Wno-format-contains-nul -fno-strict-aliasing -c -o gd.o gd.c In file included from gd.c:11: getdefs.c: In function 'buildDefinition': getdefs.c:451:29: error: '%s' directive writing up to 255 bytes into a region of size 253 [-Werror=format-overflow=] 451 | sprintf(def_str, " %s'", name_bf); | ^~~~~ getdefs.c:451:9: note: 'sprintf' output between 4 and 259 bytes into a destination of size 255 451 | sprintf(def_str, " %s'", name_bf); | ^~ cc1: all warnings being treated as errors make[2]: *** [gd.o] Error 1 make[2]: Leaving directory `/p/work1/oppe/autogen-5.18.16/getdefs' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/p/work1/oppe/autogen-5.18.16' make: *** [all] Error 2 Do I just add "-Wno-error=format-overflow" to the compile options? Do you have a fix? Tom Oppe - Thomas C. Oppe HPCMP Benchmarking Team HITS Team SAIC thomas.c.o...@erdc.dren.mil Work: (601) 634-2797 Cell:(601) 642-6391 -