Re: Build reproducibility of gcc @ NixOS
Hi Arthur, On Fri, Apr 2, 2021 at 6:45 PM Arthur Gautier wrote: > > On Fri, Apr 2, 2021 at 4:32 PM Tadeus Prastowo > wrote: > > > > Hi Arthur, > > > > On Fri, Apr 2, 2021 at 5:04 PM Arthur Gautier > > wrote: > > > > > > Hi Tadeus, > > > > > > On Fri, Apr 2, 2021 at 9:07 AM Tadeus Prastowo > > > wrote: > [...] > > > > By "the manual", do you refer to > > https://gcc.gnu.org/install/build.html#Building-with-profile-feedback > > ? > yes > > > > Quoting the page: > > When ‘make profiledbootstrap’ is run, it will first build a stage1 > > compiler. > > This compiler is used to build a stageprofile compiler instrumented > > to collect execution counts of instruction and branch probabilities. > > Training run is done by building stagetrain compiler. > > Finally a stagefeedback compiler is built using the information collected. > > End quote. > > > > Based on the quote, a reproducible build is to expected for the > > following compilers: > > 1. The stage1 compiler. > > 2. The stageprofile compiler, which is built by the stage1 compiler. > > 3. The stagetrain compiler, which is built by the stageprofile compiler. > > > > Then, a reproducible build is expected for the stagefeedback compiler > > on the condition that the same information, which was collected by the > > stageprofile compiler when building the stagegrain compiler, is used. > > > > Do you agree with that reasoning? > Yes, and as far as I can tell, up to the stageprofile I get the same result. > Only the output of the stagetrain compilation changes, which affects > the stagefeedback compilation. By saying "up to the stageprofile I get the same result", do you mean that you obtain the same stageprofile compilers on two different architectures? Or, do you mean that you obtain the same stageprofile compilers on the same machine by building GCC one after another in different build directories? And by saying "the output of the stagetrain compilation changes", do you mean that the stageprofile compiler __running on the same machine__ produces different stagetrain compilers? Or, do you mean that the stageprofile compiler running on the same machine obtains different statistics that will later be used to build the stagefeedback compiler? Or, something else? > > > And I would expect, given > > > the same input provided in the same order, two different architectures > > > to take the same branch, and not observe any difference. > > > > In other words, you expect that branch statistics depends only on the > > given source code. Correct? > That would be my understanding (although very limited). Could you tell me what you actually mean by the word "architectures", please? It is because in my understanding different architectures mean different instruction sets. > What I'm trying to understand is: what "local behavior" is injected in > my build, and see if I could get rid of that, and only keep branch > statistics/execution counts, which I expect to be reproducible. That currently I don't know because I haven't fully understood your actual situation. > > > I understand > > > that with autoprofiled builds, the local architecture behavior is > > > injected in the build, but I don't use that. > > > I'm not using any -march in the build either (as far as I can > > > understand/tell). So I do not expect the build to change its > > > instruction set either. > > > > > > Is that normal that two different architectures would issue two > > > different "execution counts of instruction and branch probabilities"? > > > > I guess that it would be the case. > > > > > Or is there something more? > > > > Perhaps you can have the reproducible build that you want by first > > isolating the information that is collected by the stageprofile > > compiler when building the stagegrain compiler and then reusing the > > same information when building every other stagefeedback compiler. > > Yeah, but said information is not reproducible itself (that would > defeat the purpose of the effort). I expect that running the same stageprofile compiler on the same machine twice, once to build stagetrain compiler A and and then once again to build stagetrain compiler B, should obtain A = B. While you imply that it was not the case, I am not sure because you don't say whether in your case you did the same (i.e., running the same stageprofile compiler on the same machine twice to obtain A and B). -- Best regards, Tadeus
Re: Build reproducibility of gcc @ NixOS
Hi Arthur, On Fri, Apr 2, 2021 at 5:04 PM Arthur Gautier wrote: > > Hi Tadeus, > > On Fri, Apr 2, 2021 at 9:07 AM Tadeus Prastowo > wrote: [...] > > Since an optimized build is likely to be machine-dependent regardless > > of any intended injection (e.g., different instructions used in GCC > > binaries depending on /proc/cpuinfo), I don't understand why an > > optimized build should be reproducible on different machines, unless > > of course every channel that GCC uses to find out about the machine > > (e.g., /proc/cpuinfo) is under your total control. So, do you mean to > > ask a list of all channels that GCC uses to find out about the > > machine? > > This is where I'm getting confused. According to the manual, > stagetrain only record branch statistics. By "the manual", do you refer to https://gcc.gnu.org/install/build.html#Building-with-profile-feedback ? Quoting the page: When ‘make profiledbootstrap’ is run, it will first build a stage1 compiler. This compiler is used to build a stageprofile compiler instrumented to collect execution counts of instruction and branch probabilities. Training run is done by building stagetrain compiler. Finally a stagefeedback compiler is built using the information collected. End quote. Based on the quote, a reproducible build is to expected for the following compilers: 1. The stage1 compiler. 2. The stageprofile compiler, which is built by the stage1 compiler. 3. The stagetrain compiler, which is built by the stageprofile compiler. Then, a reproducible build is expected for the stagefeedback compiler on the condition that the same information, which was collected by the stageprofile compiler when building the stagegrain compiler, is used. Do you agree with that reasoning? > And I would expect, given > the same input provided in the same order, two different architectures > to take the same branch, and not observe any difference. In other words, you expect that branch statistics depends only on the given source code. Correct? > I understand > that with autoprofiled builds, the local architecture behavior is > injected in the build, but I don't use that. > I'm not using any -march in the build either (as far as I can > understand/tell). So I do not expect the build to change its > instruction set either. > > Is that normal that two different architectures would issue two > different "execution counts of instruction and branch probabilities"? I guess that it would be the case. > Or is there something more? Perhaps you can have the reproducible build that you want by first isolating the information that is collected by the stageprofile compiler when building the stagegrain compiler and then reusing the same information when building every other stagefeedback compiler. > Thank you for your reply! My pleasure. > -- > Arthur -- Best regards, Tadeus
Re: Build reproducibility of gcc @ NixOS
Hi Arthur, On Fri, Apr 2, 2021 at 5:56 AM Arthur Gautier wrote: > > Dear GCC development team, > > We've been trying to build reproducibly the minimal NixOS image, and > gcc was one of the last issues we had. > We found that disabling profiled bootstrap compilation of GCC allowed > us to get a reproducible build of gcc. > Our efforts can be followed here: https://github.com/NixOS/nixpkgs/pull/112928 > > But I measured disabling this optimization to cost around 7-12% > depending on the build. That is expected as mentioned in the manual: https://gcc.gnu.org/install/build.html#Building-with-profile-feedback And, I have reproduced it as well as reported here: https://gcc.gnu.org/ml/gcc-help/2019-05/msg00118.html, the last questions that remain being here: https://gcc.gnu.org/legacy-ml/gcc-help/2019-07/msg00053.html > Because of this performance regression, we're trying to find a middle > ground. Ideally we'd like to keep the performance of gcc as untouched > as possible (even if that costs us on compilation time of gcc itself). > > Compiling gcc twice on the same machine gets us the same output, but > compiling on a different architecture gets us a different result. > Reading the documentation, it would seem that autoprofiledback > bootstrap would use machine metrics and injects them in the build (and > we don't use autoprofiledback), But I would not expect the stagetrain > of profiledbootstrap to do that. > I tried disabling concurrency of the stagetrain without luck. > > It feels like I'm missing something. > Would anyone have any idea what could inject the host's behavior here? Since an optimized build is likely to be machine-dependent regardless of any intended injection (e.g., different instructions used in GCC binaries depending on /proc/cpuinfo), I don't understand why an optimized build should be reproducible on different machines, unless of course every channel that GCC uses to find out about the machine (e.g., /proc/cpuinfo) is under your total control. So, do you mean to ask a list of all channels that GCC uses to find out about the machine? > Thank you for your help! No worries. > Best, > -- > Arthur -- Best regards, Tadeus
Re: How to pack small type to big type without memory load/store ?
On Tue, Aug 4, 2020 at 5:10 AM Jojo R wrote: > > Hi, > > Form My ABI, float register is used by function call, > I want to pack float a and b into double v without any memory load/store, > > The flowing is my demo: > > Typedef union { > float ff[2]; > double v; > } double_u; > > Double pack_float (float a, float b) { > > double_u tmp; > > tmp.ff[0] = a; > tmp.ff[1] = b; > > return tmp.v; > } > > There is memory store in these statement: > > tmp.ff[0] = a; > tmp.ff[1] = b; > > Could someone give me some hints to avoid memory load/store ? Use inline assembly (https://gcc.gnu.org/onlinedocs/gcc/Using-Assembly-Language-with-C.html) because AFAIK, C/C++ cannot force its compiler to place a variable in a register. > Jojo -- Best regards, Tadeus