On Fri, 12 Jul 2024, Jakub Jelinek wrote:
> Hi!
>
> Nick has implemented a new .base64 directive in gas (to be shipped in
> the upcoming binutils 2.43; big thanks for that).
> See https://sourceware.org/bugzilla/show_bug.cgi?id=31964
>
> The following patch adjusts default_elf_asm_output_ascii (i.e.
> ASM_OUTPUT_ASCII elfos.h implementation) to use it if it detects binary
> data and gas supports it.
>
> Without this patch, we emit stuff like:
> .string "\177ELF\002\001\001\003"
> .string ""
> .string ""
> .string ""
> .string ""
> .string ""
> .string ""
> .string ""
> .string "\002"
> .string ">"
> ...
> .string "\324\001\236
> 0FS\202\002E\n0@\203\004\005&\202\021\337)\021\203C\020A\300\220I\004\t\b\206(\234\0132l\004b\300\bK\006\220$0\303\020P$\233\211\002D\f"
> etc., with this patch more compact
> .base64
> "f0VMRgIBAQMAAAAAAAAAAAIAPgABAAAAABf3AAAAAABAAAAAAAAAAACneB0AAAAAAAAAAEAAOAAOAEAALAArAAYAAAAEAAAAQAAAAAAAAABAAEAAAAAAAEAAQAAAAAAAEAMAAAAAAAAQAwAAAAAAAAgAAAAAAAAAAwAAAAQAAABQAwAAAAAAAFADQAAAAAAAUANAAAAAAAAcAAAAAAAAABwAAAAAAAAAAQAAAAAAAAABAAAABAAAAAAAAAAAAAAAAABAAAAAAAAAAEAAAAAAADBwOQAAAAAAMHA5AAAAAAAAEAAAAAAAAAEAAAAFAAAAAIA5AAAAAAAAgHkAAAAA"
> .base64
> "AACAeQAAAAAAxSSgAgAAAADFJKACAAAAAAAQAAAAAAAAAQAAAAQAAAAAsNkCAAAAAACwGQMAAAAAALAZAwAAAADMtc0AAAAAAMy1zQAAAAAAABAAAAAAAAABAAAABgAAAGhmpwMAAAAAaHbnAwAAAABoducDAAAAAOAMAQAAAAAA4MEeAAAAAAAAEAAAAAAAAAIAAAAGAAAAkH2nAwAAAACQjecDAAAAAJCN5wMAAAAAQAIAAAAAAABAAgAAAAAAAAgAAAAAAAAABAAAAAQAAABwAwAAAAAAAHADQAAAAAAAcANAAAAAAABAAAAAAAAAAEAAAAAAAAAACAAAAAAA"
> .base64
> "AAAEAAAABAAAALADAAAAAAAAsANAAAAAAACwA0AAAAAAACAAAAAAAAAAIAAAAAAAAAAEAAAAAAAAAAcAAAAEAAAAaGanAwAAAABoducDAAAAAGh25wMAAAAAAAAAAAAAAAAQAAAAAAAAAAgAAAAAAAAAU+V0ZAQAAABwAwAAAAAAAHADQAAAAAAAcANAAAAAAABAAAAAAAAAAEAAAAAAAAAACAAAAAAAAABQ5XRkBAAAAAw/WAMAAAAADD+YAwAAAAAMP5gDAAAAAPy7CgAAAAAA/LsKAAAAAAAEAAAAAAAAAFHldGQGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
> .base64
> "AAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAUuV0ZAQAAABoZqcDAAAAAGh25wMAAAAAaHbnAwAAAACYGQAAAAAAAJgZAAAAAAAAAQAAAAAAAAAvbGliNjQvbGQtbGludXgteDg2LTY0LnNvLjIAAAAAAAQAAAAwAAAABQAAAEdOVQACgADABAAAAAEAAAAAAAAAAQABwAQAAAAJAAAAAAAAAAIAAcAEAAAAAwAAAAAAAAAEAAAAEAAAAAEAAABHTlUAAAAAAAMAAAACAAAAAAAAAAOAAACsqAAAgS0AAOJWAAAjNwAAXjAAAAAAAAAAAAAAF1gAAHsxAABBBwAA"
> .base64
> "G0kAALGmAACwoAAAAAAAAAAAAACQhAAAAAAAAOw1AACNYgAAAAAAAFQoAAAAAAAAx3UAALZAAAAAAAAAiIUAALGeAABBlAAAWEsAAPmRAACmOgAAAAAAADh3AAAAAAAAlCAAAAAAAABymgAAaosAAMIjAAAKMQAAMkIAADU0AAAAAAAA5ZwAAAAAAAAAAAAAAAAAAFIdAAAIGQAAAAAAAMFbAAAoTQAAGDcAAIRgAAA6HgAAlxwAAAAAAADOlgAAAAAAAEhPAAARiwAAMGgAAOVtAADMFgAAAAAAAAAAAACrjgAAYl4AACZVAAA/HgAAAAAAAAAAAABqPwAAAAAA"
> The patch attempts to juggle between readability and compactness, so
> if it detects some hunk of the initializer that would be shorter to be
> emitted as .string/.ascii directive, it does so, but if it previously
> used .base64 directive it switches mode only if there is a 16+ char
> ASCII-ish string.
>
> On my #embed testcase from yesterday
> unsigned char a[] = {
> #embed "cc1plus"
> };
> without this patch it emits 2.4GB of assembly, while with this
> patch 649M.
> Compile times (trunk, so yes,rtl,extra checking) are:
> time ./xgcc -B ./ -S -std=c23 -O2 embed-11.c
>
>
>
>
>
> real 0m13.647s
>
>
> user 0m7.157s
>
>
> sys 0m2.597s
>
>
> time ./xgcc -B ./ -c -std=c23 -O2 embed-11.c
>
>
>
>
>
> real 0m28.649s
>
>
> user 0m26.653s
>
>
> sys 0m1.958s
>
>
> without the patch and
> time ./xgcc -B ./ -S -std=c23 -O2 embed-11.c
>
> real 0m4.283s
> user 0m2.288s
> sys 0m0.859s
> time ./xgcc -B ./ -c -std=c23 -O2 embed-11.c
>
> real 0m6.888s
> user 0m5.876s
> sys 0m1.002s
> with the patch, so that feels like significant improvement.
> The resulting embed-11.o is identical between the two ways of expressing
> the mostly binary data in the assembly. But note that there are portions
> like:
> .base64
> "nAAAAAAAAAAvZRcAIgAOAFAzMwEAAAAABgAAAAAAAACEQBgAEgAOAFBHcAIAAAAA7AAAAAAAAAAAX19nbXB6X2dldF9zaQBtcGZyX3NldF9zaV8yZXhwAG1wZnJfY29zaABtcGZyX3RhbmgAbXBmcl9zZXRfbmFuAG1wZnJfc3ViAG1wZnJfdGFuAG1wZnJfc3RydG9mcgBfX2dtcHpfc3ViX3VpAF9fZ21wX2dldF9tZW1vcnlfZnVuY3Rpb25zAF9fZ21wel9zZXRfdWkAbXBmcl9wb3cAX19nbXB6X3N1YgBfX2dtcHpfZml0c19zbG9uZ19wAG1wZnJfYXRh"
> .base64
> "bjIAX19nbXB6X2RpdmV4YWN0AG1wZnJfc2V0X2VtaW4AX19nbXB6X3NldABfX2dtcHpfbXVsAG1wZnJfY2xlYXIAbXBmcl9sb2cAbXBmcl9hdGFuaABfX2dtcHpfc3dhcABtcGZyX2FzaW5oAG1wZnJfYXNpbgBtcGZyX2NsZWFycwBfX2dtcHpfbXVsXzJleHAAX19nbXB6X2FkZG11bABtcGZyX3NpbmgAX19nbXB6X2FkZF91aQBfX2dtcHFfY2xlYXIAX19nbW9uX3N0YXJ0X18AbXBmcl9hY29zAG1wZnJfc2V0X2VtYXgAbXBmcl9jb3MAbXBmcl9zaW4A"
> .string "__gmpz_ui_pow_ui"
> .string "mpfr_get_str"
> .string "mpfr_acosh"
> .string "mpfr_sub_ui"
> .string "__gmpq_set_ui"
> .string "mpfr_set_inf"
> ...
> .string "GLIBC_2.14"
> .string "GLIBC_2.11"
> .base64
> "AAABAAIAAQADAAMAAwADAAMAAwAEAAUABgADAAEAAQADAAMABwABAAEAAwADAAMAAwAIAAEAAwADAAEAAwABAAMAAwABAAMAAQADAAMAAwADAAMAAwADAAYAAwADAAEAAQAIAAMAAwADAAMAAwABAAMAAQADAAMAAQABAAEAAwAIAAEAAwADAAEAAwABAAMAAQADAAEABgADAAMAAQAHAAMAAwADAAMAAwABAAMAAQABAAMAAwADAAkAAQABAAEAAwAKAAEAAwADAAMAAQABAAMAAwALAAEAAwADAAEAAQADAAMAAwABAAMAAwABAAEAAwADAAMABwABAAMAAwAB"
> .base64
> "AAEAAwADAAEAAwABAAMAAQADAAMAAwADAAEAAQABAAEAAwADAAMAAQABAAEAAQABAAEAAQADAAMAAwADAAMAAQABAAwAAwADAA0AAwADAAMAAwADAAEAAQADAAMAAQABAAMAAwADAAEAAwADAAEAAwAIAAMAAwADAAMABgABAA4ACwAGAAEAAQADAAEAAQADAAEAAwABAAMAAwABAAEAAwABAAMAAwABAAEAAwADAAMAAwABAAMAAQABAAEAAQABAAMADwABAAMAAQADAAMAAwABAAEAAQAIAAEADAADAAMAAQABAAMAAwADAAEAAQABAAEAAQADAAEAAwADAAEA"
> .base64
> "AwABAAMAAQADAAMAAQABAAEAAwADAAMAAwADAAMAAQADAAMACAAQAA8AAQABAAEAAQABAAEAAQABAAEAAQABAAEAAQABAAEAAQABAAEAAQABAAEAAQABAAEAAQABAAEAAQABAAEAAQABAAEAAQABAAEAAQABAAEAAQABAAEAAQABAAEAAQABAAEAAQA="
> so it isn't all just totally unreadable stuff.
>
> Tested on top of the #embed patchset (just to get the above
> data) and then with full bootstrap/regtest on top of vanilla trunk,
> on x86_64-linux against trunk binutils prepended into $PATH (HAVE_GAS_BASE64
> is true) and i686-linux (against older binutils, HAVE_GAS_BASE64 not
> defined), ok for trunk?
OK.
Do we want to have a command-line switch controlling the use of
.base64? It might be the 16+ char limit is too large for the
readability purpose and/or with wide-strings, etc. .base64 is
unwanted to some people at least when debugging? (does gas
support sth like .wstring or UTF-8/16/32 input?) I don't have
a good suggestion for the flag name, a quick scan for assembler
related options doesn't reveal a nice "pattern"; -f[no-]base64-in-asm?
Thanks,
Richard.
> 2024-07-12 Jakub Jelinek <ja...@redhat.com>
>
> * configure.ac (HAVE_GAS_BASE64): New check.
> * config/elfos.h (BASE64_ASM_OP): Define if HAVE_GAS_BASE64 is
> defined.
> * varasm.cc (assemble_string): Bump maximum from 2000 to 16384 if
> BASE64_ASM_OP is defined.
> (default_elf_asm_output_limited_string): Emit opening '"' together
> with STRING_ASM_OP.
> (default_elf_asm_output_ascii): Use BASE64_ASM_OP if defined and
> beneficial. Remove UB when last_null is NULL.
> * configure: Regenerate.
> * config.in: Regenerate.
>
> --- gcc/configure.ac.jj 2024-07-01 11:28:22.435241431 +0200
> +++ gcc/configure.ac 2024-07-12 10:28:38.038804308 +0200
> @@ -3054,6 +3054,11 @@ case "${target}" in
> ;;
> esac
>
> +gcc_GAS_CHECK_FEATURE([.base64], gcc_cv_as_base64,,
> + [ .section .rodata
> + .base64
> "Tm9uIHB1ZG9yIGVzdCBuaWwgc2NpcmUsIHB1ZG9yIG5pbCBkaXNjZXJlIHZlbGxlLgo="],,
> +[AC_DEFINE(HAVE_GAS_BASE64, 1, [Define if your assembler supports
> .base64.])])
> +
> # gnu_indirect_function type is an extension proposed at
> # http://groups.google/com/group/generic-abi/files. It allows dynamic runtime
> # selection of function implementation
> --- gcc/config/elfos.h.jj 2024-01-03 11:51:43.966558606 +0100
> +++ gcc/config/elfos.h 2024-07-12 10:40:59.721200959 +0200
> @@ -444,6 +444,10 @@ see the files COPYING3 and COPYING.RUNTI
>
> #define STRING_ASM_OP "\t.string\t"
>
> +#ifdef HAVE_GAS_BASE64
> +#define BASE64_ASM_OP "\t.base64\t"
> +#endif
> +
> /* The routine used to output NUL terminated strings. We use a special
> version of this for most svr4 targets because doing so makes the
> generated assembly code more compact (and thus faster to assemble)
> --- gcc/varasm.cc.jj 2024-07-08 11:00:54.723980651 +0200
> +++ gcc/varasm.cc 2024-07-12 13:28:39.934619478 +0200
> @@ -2101,7 +2101,19 @@ void
> assemble_string (const char *p, int size)
> {
> int pos = 0;
> +#if defined(BASE64_ASM_OP) \
> + && BITS_PER_UNIT == 8 \
> + && CHAR_BIT == 8 \
> + && 'A' == 65 \
> + && 'a' == 97 \
> + && '0' == 48 \
> + && '+' == 43 \
> + && '/' == 47 \
> + && '=' == 61
> + int maximum = 16384;
> +#else
> int maximum = 2000;
> +#endif
>
> /* If the string is very long, split it up. */
>
> @@ -8469,8 +8481,7 @@ default_elf_asm_output_limited_string (F
> int escape;
> unsigned char c;
>
> - fputs (STRING_ASM_OP, f);
> - putc ('"', f);
> + fputs (STRING_ASM_OP "\"", f);
> while (*s != '\0')
> {
> c = *s;
> @@ -8504,9 +8515,11 @@ default_elf_asm_output_ascii (FILE *f, c
> {
> const char *limit = s + len;
> const char *last_null = NULL;
> + const char *last_base64 = s;
> unsigned bytes_in_chunk = 0;
> unsigned char c;
> int escape;
> + bool prev_base64 = false;
>
> for (; s < limit; s++)
> {
> @@ -8519,7 +8532,7 @@ default_elf_asm_output_ascii (FILE *f, c
> bytes_in_chunk = 0;
> }
>
> - if (s > last_null)
> + if ((uintptr_t) s > (uintptr_t) last_null)
> {
> for (p = s; p < limit && *p != '\0'; p++)
> continue;
> @@ -8528,6 +8541,104 @@ default_elf_asm_output_ascii (FILE *f, c
> else
> p = last_null;
>
> +#if defined(BASE64_ASM_OP) \
> + && BITS_PER_UNIT == 8 \
> + && CHAR_BIT == 8 \
> + && 'A' == 65 \
> + && 'a' == 97 \
> + && '0' == 48 \
> + && '+' == 43 \
> + && '/' == 47 \
> + && '=' == 61
> + if (s >= last_base64)
> + {
> + unsigned cnt = 0;
> + const char *t;
> + for (t = s; t < limit && (t - s) < (long) ELF_STRING_LIMIT - 1; t++)
> + {
> + if (t == p && t != s)
> + {
> + if (cnt <= (t - s + 1 + 2) / 3 * 4
> + && (!prev_base64 || (t - s) >= 16)
> + && ((t - s) > 1 || cnt <= 2))
> + {
> + last_base64 = p;
> + goto no_base64;
> + }
> + }
> + c = *t;
> + escape = ELF_ASCII_ESCAPES[c];
> + switch (escape)
> + {
> + case 0:
> + ++cnt;
> + break;
> + case 1:
> + if (c == 0)
> + cnt += 2 + strlen (STRING_ASM_OP) + 1;
> + else
> + cnt += 4;
> + break;
> + default:
> + cnt += 2;
> + break;
> + }
> + }
> + if (cnt > (t - s + 2) / 3 * 4 && (t - s) >= 3)
> + {
> + if (bytes_in_chunk > 0)
> + {
> + putc ('\"', f);
> + putc ('\n', f);
> + bytes_in_chunk = 0;
> + }
> +
> + unsigned char buf[(ELF_STRING_LIMIT + 2) / 3 * 4 + 3];
> + unsigned j = 0;
> + static const char base64_enc[] =
> + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
> + "0123456789+/";
> +
> + fputs (BASE64_ASM_OP "\"", f);
> + while (s < t)
> + {
> + unsigned char a = *s;
> + unsigned char b = 0, c = 0;
> + if (s < t - 1)
> + b = s[1];
> + if (s < t - 2)
> + c = s[2];
> + unsigned long v = ((((unsigned long) a) << 16)
> + | (((unsigned long) b) << 8)
> + | c);
> + buf[j++] = base64_enc[(v >> 18) & 63];
> + buf[j++] = base64_enc[(v >> 12) & 63];
> + buf[j++] = base64_enc[(v >> 6) & 63];
> + buf[j++] = base64_enc[v & 63];
> + if (s >= t - 2)
> + {
> + buf[j - 1] = '=';
> + if (s >= t - 1)
> + buf[j - 2] = '=';
> + break;
> + }
> + s += 3;
> + }
> + memcpy (buf + j, "\"\n", 3);
> + fputs ((const char *) buf, f);
> + s = t - 1;
> + prev_base64 = true;
> + continue;
> + }
> + last_base64 = t;
> + no_base64:
> + prev_base64 = false;
> + }
> +#else
> + (void) last_base64;
> + (void) prev_base64;
> +#endif
> +
> if (p < limit && (p - s) <= (long) ELF_STRING_LIMIT)
> {
> if (bytes_in_chunk > 0)
> --- gcc/configure.jj 2024-07-01 11:28:22.432241469 +0200
> +++ gcc/configure 2024-07-12 10:28:44.843716198 +0200
> @@ -25859,6 +25859,39 @@ case "${target}" in
> ;;
> esac
>
> +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking assembler for .base64" >&5
> +$as_echo_n "checking assembler for .base64... " >&6; }
> +if ${gcc_cv_as_base64+:} false; then :
> + $as_echo_n "(cached) " >&6
> +else
> + gcc_cv_as_base64=no
> + if test x$gcc_cv_as != x; then
> + $as_echo ' .section .rodata
> + .base64
> "Tm9uIHB1ZG9yIGVzdCBuaWwgc2NpcmUsIHB1ZG9yIG5pbCBkaXNjZXJlIHZlbGxlLgo="' >
> conftest.s
> + if { ac_try='$gcc_cv_as $gcc_cv_as_flags -o conftest.o conftest.s >&5'
> + { { eval echo "\"\$as_me\":${as_lineno-$LINENO}: \"$ac_try\""; } >&5
> + (eval $ac_try) 2>&5
> + ac_status=$?
> + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
> + test $ac_status = 0; }; }
> + then
> + gcc_cv_as_base64=yes
> + else
> + echo "configure: failed program was" >&5
> + cat conftest.s >&5
> + fi
> + rm -f conftest.o conftest.s
> + fi
> +fi
> +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $gcc_cv_as_base64" >&5
> +$as_echo "$gcc_cv_as_base64" >&6; }
> +if test $gcc_cv_as_base64 = yes; then
> +
> +$as_echo "#define HAVE_GAS_BASE64 1" >>confdefs.h
> +
> +fi
> +
> +
> # gnu_indirect_function type is an extension proposed at
> # http://groups.google/com/group/generic-abi/files. It allows dynamic runtime
> # selection of function implementation
> --- gcc/config.in.jj 2024-06-17 18:50:45.791187189 +0200
> +++ gcc/config.in 2024-07-12 10:28:41.994753122 +0200
> @@ -1431,6 +1431,12 @@
> #endif
>
>
> +/* Define if your assembler supports .base64. */
> +#ifndef USED_FOR_TARGET
> +#undef HAVE_GAS_BASE64
> +#endif
> +
> +
> /* Define 0/1 if your assembler supports CFI directives. */
> #undef HAVE_GAS_CFI_DIRECTIVE
>
>
> Jakub
>
>
--
Richard Biener <rguent...@suse.de>
SUSE Software Solutions Germany GmbH,
Frankenstrasse 146, 90461 Nuernberg, Germany;
GF: Ivo Totev, Andrew McDonald, Werner Knoblich; (HRB 36809, AG Nuernberg)
