On Thu, Oct 05, 2023 at 07:20:35AM -0400, Ben Boeckel wrote: > On Wed, Oct 04, 2023 at 22:19:32 +0100, Sergei Trofimovich via Gcc wrote: > > The prototype that creates equivalent of the following commands does > > work for smaller packages: > > > > > > -fmacro-prefix-map=/nix/store/y8wfrgk7br5rfz4221lfb9v8w3n0cnyd-glibc-2.37-8-dev=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-glibc-2.37-8-dev > > > > -fmacro-prefix-map=/nix/store/8n240jfdmsb3lnc2qa2vb9dwk638j1lp-gmp-with-cxx-6.3.0-dev=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gmp-with-cxx-6.3.0-dev > > > > -fmacro-prefix-map=/nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nlohmann_json-3.11.2 > > ... > > > > The above works for small amount of options (like, 100). But around 1000 > > options we start hitting linux limits on the single environment variable > > or real-world packages like `qemu` with a ton of input depends. > > Are you trying to pass this through via `CFLAGS` and friends?
Roughly via `CFLAGS`. `nixpkgs` uses it's own private `NIX_CFLAGS_COMPILE` variable which gets extended in `gcc-wrapper` shel wrapper as expcit list of arguments to real `gcc-binary`. It's almost like `CFLAGS` but is expected to be transaparent to most build systems. > > The command-line limitations are in various places: > > - `gcc` limitation of lifting all command line options into a single > > environment variable: https://gcc.gnu.org/PR111527 > > - `linux` limitation of constraining single environ variable to a value > > way below than full available environment space: > > https://lkml.org/lkml/2023/9/24/381 > > > > `linux` fix would buy us 50x more budged (A Lot) but it will not help > > much other operating systems like `Darwin` where absolute environment > > limit is a lot lower than `linux`. > > > > I already implemented [1.] in https://github.com/NixOS/nixpkgs/pull/255192 > > (also attached `mangle-NIX_STORE-in-__FILE__.patch` 3.5K patch against > > `master` as a proof of concept). > > > > What would be the best way to scale up `-fmacro-prefix-map=` up to NixOS > > needs for `gcc`? I would like to implement something sensible I could > > upstream. > > How about `CFLAGS=@macro_prefix_map.args` and writing that file in the > same codepath where you generate the flags today. It'll work with just > about every compiler and tools like `ccache` will understand that it is > an input that affects the build and properly take the file's contents > into account. That was my initial attempt. I'll duplicate my response from https://gcc.gnu.org/pipermail/gcc/2023-October/242639.html here as is: """ Yeah, in theory response files would extend the limit. In practice `gcc` always extends response files internally into a single `COLLECT_GCC_OPTIONS` option and hits the environment variable limit very early: https://gcc.gnu.org/PR111527 Example reproducer: $ for i in `seq 1 1000`; do printf -- "-fmacro-prefix-map=%0*d=%0*d\n" 200 1 200 2; done > a.rsp $ touch a.c; gcc @a.rsp -c a.c gcc: fatal error: cannot execute 'cc1': execv: Argument list too long compilation terminated. And if you want to look at the gory details: $ strace -f -etrace=execve -s 1000000 -v -v -v gcc @a.rsp -c a.c ... [pid 78] execve("cc1", ["cc1", "-quiet", "a.c", "-quiet", "-dumpbase", "a.c", "-dumpbase-ext", ".c", "-mtune=generic", "-march=x86-64", "-fmacro-prefix-map=...=...", "-fmacro-prefix-map=...=...", ...], [..., "COLLECT_GCC=gcc", "COLLECT_GCC_OPTIONS='-fmacro-prefix-map=...=...' '-fmacro-prefix-map=...=...' ... '-c' '-mtune=generic' '-march=x86-64'"]) = -1 E2BIG (Argument list too long) Note how `gcc` not only expands response file into an argument list (that is not too bad) but also duplicates the whole list as a single `COLLECT_GCC_OPTIONS=...` environment variable with added quoting on top. Would be nice if `gcc` just passed response files around as is :) """ -- Sergei
