Enabling LTO for target libraries (e.g., libgo, libstdc++)
I am attempting to convince GCC to build target libraries with link-time optimizations enabled. I am primarily interested in libgo, but this discussion seems like it would be applicable to libstdc++, libgfortran, etc. The benchmarking I've done suggests that LTOing libgo yields a 5-20% speedup on various Go programs, which is quite substantial. The trouble is convincing GCC's build system to apply the various LTO flags to the correct places. Ian Taylor suggested the following to plumb -flto into libgo compilation: $ make GOCFLAGS_FOR_TARGET="-g -O3 -flto" This nearly works, and I believe there are analogous options that would apply to the other target libraries that GCC builds. The trouble is that while building libgo, the build system uses ar and ranlib directly from binutils, without providing them with the LTO plugin that was built earlier. This means that the LTO information is dropped on the floor, and attempting to link with the built libgo archive will fail. I have a simple patch to the top-level configure.ac that resolves the issue by teaching the build system to use the gcc-ar and gcc-ranlib wrappers which were built earlier and know how to pass the linker plugin to the underlying ar/ranlib commands. The patch is small enough that I've included it at the end of this email. My question is whether this is a reasonable thing to do. It seems like using the gcc-ar and gcc-ranlib wrappers strictly improves the situation, and won't impact compilations that don't specify -flto. But I'm not familiar enough with the build system to say for sure. Does anyone have advice to offer? Has anyone tried convincing the build system to compile some of the other target libraries (like libstdc++ or libgfortran) with -flto? diff --git a/configure.ac b/configure.ac index 87f2aee05008..1c38ac5979ff 100644 --- a/configure.ac +++ b/configure.ac @@ -3400,7 +3400,8 @@ ACX_CHECK_INSTALLED_TARGET_TOOL(WINDMC_FOR_TARGET, windmc) RAW_CXX_FOR_TARGET="$CXX_FOR_TARGET" -GCC_TARGET_TOOL(ar, AR_FOR_TARGET, AR, [binutils/ar]) +GCC_TARGET_TOOL(ar, AR_FOR_TARGET, AR, + [gcc/gcc-ar -B$$r/$(HOST_SUBDIR)/gcc/]) GCC_TARGET_TOOL(as, AS_FOR_TARGET, AS, [gas/as-new]) GCC_TARGET_TOOL(cc, CC_FOR_TARGET, CC, [gcc/xgcc -B$$r/$(HOST_SUBDIR)/gcc/]) dnl see comments for CXX_FOR_TARGET_FLAG_TO_PASS @@ -3424,7 +3425,8 @@ GCC_TARGET_TOOL(nm, NM_FOR_TARGET, NM, [binutils/nm-new]) GCC_TARGET_TOOL(objcopy, OBJCOPY_FOR_TARGET, OBJCOPY, [binutils/objcopy]) GCC_TARGET_TOOL(objdump, OBJDUMP_FOR_TARGET, OBJDUMP, [binutils/objdump]) GCC_TARGET_TOOL(otool, OTOOL_FOR_TARGET, OTOOL) -GCC_TARGET_TOOL(ranlib, RANLIB_FOR_TARGET, RANLIB, [binutils/ranlib]) +GCC_TARGET_TOOL(ranlib, RANLIB_FOR_TARGET, RANLIB, + [gcc/gcc-ranlib -B$$r/$(HOST_SUBDIR)/gcc/]) GCC_TARGET_TOOL(readelf, READELF_FOR_TARGET, READELF, [binutils/readelf]) GCC_TARGET_TOOL(strip, STRIP_FOR_TARGET, STRIP, [binutils/strip-new]) GCC_TARGET_TOOL(windres, WINDRES_FOR_TARGET, WINDRES, [binutils/windres])
Re: Enabling LTO for target libraries (e.g., libgo, libstdc++)
On January 25, 2019 7:22:36 AM GMT+01:00, Nikhil Benesch wrote: >I am attempting to convince GCC to build target libraries with >link-time >optimizations enabled. I am primarily interested in libgo, but this >discussion >seems like it would be applicable to libstdc++, libgfortran, etc. The >benchmarking I've done suggests that LTOing libgo yields a 5-20% >speedup on >various Go programs, which is quite substantial. > >The trouble is convincing GCC's build system to apply the various LTO >flags to >the correct places. Ian Taylor suggested the following to plumb -flto >into >libgo compilation: > >$ make GOCFLAGS_FOR_TARGET="-g -O3 -flto" > >This nearly works, and I believe there are analogous options that would >apply to >the other target libraries that GCC builds. > >The trouble is that while building libgo, the build system uses ar and >ranlib >directly from binutils, without providing them with the LTO plugin that >was >built earlier. This means that the LTO information is dropped on the >floor, and >attempting to link with the built libgo archive will fail. > >I have a simple patch to the top-level configure.ac that resolves the >issue by >teaching the build system to use the gcc-ar and gcc-ranlib wrappers >which were >built earlier and know how to pass the linker plugin to the underlying >ar/ranlib >commands. The patch is small enough that I've included it at the end of >this >email. > >My question is whether this is a reasonable thing to do. It seems like >using >the gcc-ar and gcc-ranlib wrappers strictly improves the situation, and >won't >impact compilations that don't specify -flto. But I'm not familiar >enough with >the build system to say for sure. > >Does anyone have advice to offer? Has anyone tried convincing the build >system >to compile some of the other target libraries (like libstdc++ or >libgfortran) >with -flto? Using the built gcc-ar and ranlib sounds good but the patch looks not Form a quick look. I think we want to change the top level makefile to pass down the appropriate ar/ranlib_FOR_TARGET similar to how we pass CC_FOR_tARGET. Richard. >diff --git a/configure.ac b/configure.ac >index 87f2aee05008..1c38ac5979ff 100644 >--- a/configure.ac >+++ b/configure.ac >@@ -3400,7 +3400,8 @@ >ACX_CHECK_INSTALLED_TARGET_TOOL(WINDMC_FOR_TARGET, windmc) > > RAW_CXX_FOR_TARGET="$CXX_FOR_TARGET" > >-GCC_TARGET_TOOL(ar, AR_FOR_TARGET, AR, [binutils/ar]) >+GCC_TARGET_TOOL(ar, AR_FOR_TARGET, AR, >+ [gcc/gcc-ar -B$$r/$(HOST_SUBDIR)/gcc/]) > GCC_TARGET_TOOL(as, AS_FOR_TARGET, AS, [gas/as-new]) >GCC_TARGET_TOOL(cc, CC_FOR_TARGET, CC, [gcc/xgcc >-B$$r/$(HOST_SUBDIR)/gcc/]) > dnl see comments for CXX_FOR_TARGET_FLAG_TO_PASS >@@ -3424,7 +3425,8 @@ GCC_TARGET_TOOL(nm, NM_FOR_TARGET, NM, >[binutils/nm-new]) >GCC_TARGET_TOOL(objcopy, OBJCOPY_FOR_TARGET, OBJCOPY, >[binutils/objcopy]) >GCC_TARGET_TOOL(objdump, OBJDUMP_FOR_TARGET, OBJDUMP, >[binutils/objdump]) > GCC_TARGET_TOOL(otool, OTOOL_FOR_TARGET, OTOOL) >-GCC_TARGET_TOOL(ranlib, RANLIB_FOR_TARGET, RANLIB, [binutils/ranlib]) >+GCC_TARGET_TOOL(ranlib, RANLIB_FOR_TARGET, RANLIB, >+ [gcc/gcc-ranlib -B$$r/$(HOST_SUBDIR)/gcc/]) >GCC_TARGET_TOOL(readelf, READELF_FOR_TARGET, READELF, >[binutils/readelf]) > GCC_TARGET_TOOL(strip, STRIP_FOR_TARGET, STRIP, [binutils/strip-new]) >GCC_TARGET_TOOL(windres, WINDRES_FOR_TARGET, WINDRES, >[binutils/windres])
Re: Enabling LTO for target libraries (e.g., libgo, libstdc++)
On Fri, 25 Jan 2019, Nikhil Benesch wrote: > I am attempting to convince GCC to build target libraries with link-time > optimizations enabled. I am primarily interested in libgo, but this discussion Note that as far as I know issues with host-dependencies of LTO bytecode (bug 41526) may still exist, so this wouldn't work with Canadian cross compilers. > I have a simple patch to the top-level configure.ac that resolves the issue by > teaching the build system to use the gcc-ar and gcc-ranlib wrappers which were > built earlier and know how to pass the linker plugin to the underlying > ar/ranlib > commands. The patch is small enough that I've included it at the end of this > email. Will those wrappers properly wrap round tools for the target that were configured using --with-build-time-tools? -- Joseph S. Myers jos...@codesourcery.com
Re: Enabling LTO for target libraries (e.g., libgo, libstdc++)
On January 25, 2019 6:17:54 PM GMT+01:00, Joseph Myers wrote: >On Fri, 25 Jan 2019, Nikhil Benesch wrote: > >> I am attempting to convince GCC to build target libraries with >link-time >> optimizations enabled. I am primarily interested in libgo, but this >discussion > >Note that as far as I know issues with host-dependencies of LTO >bytecode >(bug 41526) may still exist, so this wouldn't work with Canadian cross >compilers. I was expecting the LTO byte code not to be retained in the built libraries. >> I have a simple patch to the top-level configure.ac that resolves the >issue by >> teaching the build system to use the gcc-ar and gcc-ranlib wrappers >which were >> built earlier and know how to pass the linker plugin to the >underlying ar/ranlib >> commands. The patch is small enough that I've included it at the end >of this >> email. > >Will those wrappers properly wrap round tools for the target that were >configured using --with-build-time-tools?
Re: Enabling LTO for target libraries (e.g., libgo, libstdc++)
Richard, regarding your first mail, I'm afraid I'm not sure what you mean. Isn't this exactly the section of configure.ac that handles CC_FOR_TARGET? The Makefile is dumb and just passes the autoconf-substituted value for CC_FOR_TARGET directly through. (I agree the patch will need some tweaks to work for all the build/host/target combination possibilities.) On Fri, Jan 25, 2019 at 12:18 PM Joseph Myers wrote: > > On Fri, 25 Jan 2019, Nikhil Benesch wrote: > > > I am attempting to convince GCC to build target libraries with link-time > > optimizations enabled. I am primarily interested in libgo, but this > > discussion > > Note that as far as I know issues with host-dependencies of LTO bytecode > (bug 41526) may still exist, so this wouldn't work with Canadian cross > compilers. Ack, thanks for the heads up. For now I'm only interested in turning on LTO during native builds. > > I have a simple patch to the top-level configure.ac that resolves the issue > > by > > teaching the build system to use the gcc-ar and gcc-ranlib wrappers which > > were > > built earlier and know how to pass the linker plugin to the underlying > > ar/ranlib > > commands. The patch is small enough that I've included it at the end of this > > email. > > Will those wrappers properly wrap round tools for the target that were > configured using --with-build-time-tools? That's a good question. I think the answer is no. I'll investigate further. On Fri, Jan 25, 2019 at 12:39 PM Richard Biener wrote: > > On January 25, 2019 6:17:54 PM GMT+01:00, Joseph Myers > wrote: > >On Fri, 25 Jan 2019, Nikhil Benesch wrote: > > > >> I am attempting to convince GCC to build target libraries with > >link-time > >> optimizations enabled. I am primarily interested in libgo, but this > >discussion > > > >Note that as far as I know issues with host-dependencies of LTO > >bytecode > >(bug 41526) may still exist, so this wouldn't work with Canadian cross > >compilers. > > I was expecting the LTO byte code not to be retained in the built libraries. > > >> I have a simple patch to the top-level configure.ac that resolves the > >issue by > >> teaching the build system to use the gcc-ar and gcc-ranlib wrappers > >which were > >> built earlier and know how to pass the linker plugin to the > >underlying ar/ranlib > >> commands. The patch is small enough that I've included it at the end > >of this > >> email. > > > >Will those wrappers properly wrap round tools for the target that were > >configured using --with-build-time-tools? >
Re: Enabling LTO for target libraries (e.g., libgo, libstdc++)
On Freitag, 25. Januar 2019 07:22:36 CET Nikhil Benesch wrote: > Does anyone have advice to offer? Has anyone tried convincing the build > system to compile some of the other target libraries (like libstdc++ or > libgfortran) with -flto? > Make sure the static versions of the libraries are partially linked before being archived so they do not have LTO code in the final libraries. gcc -r -o libname.o ar x libname.a libname.o Never done it with gcc libraries though. 'Allan
