Re: Build time
On Friday, 23 July 2021 at 18:53:06 UTC, JG wrote: Any suggestion on how to try and improve the build time. You could try some of the tools listed on the wiki for build time profiling: https://wiki.dlang.org/Development_tools#Build_time_profiling (intentional bump to aid search results, as apparently this list is not very well known)
Re: Build time
One thing to add: With LDC you can add this flag ``-linkonce-templates`` to get faster link time when using templates a lot
Re: Build time
On Saturday, 24 July 2021 at 09:12:15 UTC, JG wrote: On Saturday, 24 July 2021 at 08:26:39 UTC, JG wrote: On Friday, 23 July 2021 at 20:03:22 UTC, Dennis wrote: On Friday, 23 July 2021 at 18:53:06 UTC, JG wrote: [...] You can try profiling it with LDC 1.25 or later. Add this to dub.sdl: [...] Thanks for this suggestion. Unfortunately this makes the compile use too much memory for my system and so it gets killed before the end and no my-trace.tracy file is produced. I will try building on parts of the program with this and see if I can see what is going on. Got this to work after removing part of the program, the slowest parts are in library code (sumtype match to be precise). I will look into whether my usage can be improved. I should also mention that what I said about compile time was a little inaccurate, some of that time linking (which involves llvm). LLVM is known to be slow to compile If you are on windows i suggest to use DMD for iterating quickly, i get much faster compile time with it (probably on linux too, but i'm not a linux user)
Re: Build time
On Saturday, 24 July 2021 at 09:12:15 UTC, JG wrote: the slowest parts are in library code This is really common. A lot of libraries do really weird things... if you want to keep quick builds it is best to use the language directly.
Re: Build time
On Saturday, 24 July 2021 at 09:12:15 UTC, JG wrote: On Saturday, 24 July 2021 at 08:26:39 UTC, JG wrote: On Friday, 23 July 2021 at 20:03:22 UTC, Dennis wrote: [...] Thanks for this suggestion. Unfortunately this makes the compile use too much memory for my system and so it gets killed before the end and no my-trace.tracy file is produced. I will try building on parts of the program with this and see if I can see what is going on. Got this to work after removing part of the program, the slowest parts are in library code (sumtype match to be precise). I will look into whether my usage can be improved. I should also mention that what I said about compile time was a little inaccurate, some of that time linking (which involves llvm). Thanks very much to everyone for the help. With a few minor changes so far I have halved the compile time.
Re: Build time
On Saturday, 24 July 2021 at 08:26:39 UTC, JG wrote: On Friday, 23 July 2021 at 20:03:22 UTC, Dennis wrote: On Friday, 23 July 2021 at 18:53:06 UTC, JG wrote: [...] You can try profiling it with LDC 1.25 or later. Add this to dub.sdl: [...] Thanks for this suggestion. Unfortunately this makes the compile use too much memory for my system and so it gets killed before the end and no my-trace.tracy file is produced. I will try building on parts of the program with this and see if I can see what is going on. Got this to work after removing part of the program, the slowest parts are in library code (sumtype match to be precise). I will look into whether my usage can be improved. I should also mention that what I said about compile time was a little inaccurate, some of that time linking (which involves llvm).
Re: Build time
On Friday, 23 July 2021 at 20:03:22 UTC, Dennis wrote: On Friday, 23 July 2021 at 18:53:06 UTC, JG wrote: [...] You can try profiling it with LDC 1.25 or later. Add this to dub.sdl: [...] Thanks for this suggestion. Unfortunately this makes the compile use too much memory for my system and so it gets killed before the end and no my-trace.tracy file is produced. I will try building on parts of the program with this and see if I can see what is going on.
Re: Build time
On Friday, 23 July 2021 at 19:09:02 UTC, JG wrote: I am not sure how relevant it is but it is a compiler that I have been writing, not something serious (yet - if ever). Compile time optimizations are a bit weird compared to runtime ones - things that would fast at run time may actually be very slow at compile time, so eyeballing the code might help me point out something to consider.
Re: Build time
On Friday, 23 July 2021 at 19:32:08 UTC, H. S. Teoh wrote: And avoid doing too much work in CTFE, which is known to be slow. Well it very much depends HOW you do it. Like the ~= operation in ctfe is awfully slow and wastes a lot of memory depending on the size of the string, but if you preallocate and copy memory in chunks it isn't too bad at all. And if you use format!"str"() in ctfe that alone can be a real speed killer. That's why I wanna see the code, it is possible there's a fairly simple bottleneck to look at.
Re: Build time
On Friday, 23 July 2021 at 18:53:06 UTC, JG wrote: Any suggestion on how to try and improve the build time. I am currently using dub. You can try profiling it with LDC 1.25 or later. Add this to dub.sdl: ``` dflags "--ftime-trace" platform="ldc" dflags "--ftime-trace-file=./my-trace.json" platform="ldc" postBuildCommands "import-chrome ./my-trace.json ./my-trace.tracy" platform="ldc" ``` You can [get import-chrome and tracy here](https://github.com/wolfpld/tracy). There's binaries for Windows, and on Ubuntu/Debian Linux you can install as follows: ``` # Install required packages (there may be more, but this was the only one I didn't have installed yet) sudo apt install libcapstone-dev # Go to your code folder and clone tracy: git clone https://github.com/wolfpld/tracy # Build the import-chrome tool cd tracy/import-chrome/build/unix/ make # Copy it to the /usr/local/bin folder so it can be invoked from anywhere sudo cp import-chrome-release /usr/local/bin/import-chrome # Back to the root of the repo, now build the profiler cd ../../../ cd profiler/build/unix/ make sudo cp Tracy-release /usr/local/bin/tracy ``` Then do: ``` dub build --compiler=ldc2 tracy my-trace.tracy ``` And you can inspect what functions the compiler spends the most time on compiling. Sometimes there's one dumb thing taking a lot of time, so you can improve build times by working around that. E.g. I once replaced `std.conv: text` with `snprintf` to reduce my build time from 2.1 to 1.6 seconds.
Re: Build time
On Fri, Jul 23, 2021 at 06:53:06PM +, JG via Digitalmars-d-learn wrote: [...] > The program I writing is around 3000 loc and recently I noticed a > large slow down in compile time which after investigation seemed to be > caused by my computer running out of memory. The compile was using > more than 15GB memory. I tried using lowmem and that did solve the > memory problem but the compile still takes around 1 minute. Any > suggestion on how to try and improve the build time. I am currently > using dub. 3000 loc and 1 minute build time? Sounds like you're using too many nested templates / CTFE. > Of course one could try to use fewer templates and less meta > programming but that seems to defeat the purpose of using d. I wouldn't say use fewer templates / less meta-programming. But I'd say look into how deeply nested templates are, and whether some templates parameters may be unnecessary. If you have recursive templates, consider refactoring it so that it uses linear expansion instead. Shallowly-nested templates generally don't run into performance problems. (Confession: I wrote the variadic version of cartesianProduct in std.algorithm with recursive templates. It uses an exponential number of template expansions, and so quickly brings the compiler to its knees when you try to take 4 or more cartesian products in a row. Eventually, I refactored the most common case (no infinite ranges among its arguments) to use a linear expansion with a nested loop instead. Compile times improved by a HUGE margin.) And avoid doing too much work in CTFE, which is known to be slow. But not as slow as overly-deeply nested templates. Another way is to have a separate build step for expanding the most heavy templates, so that you only incur that heavy expansion once in a while when you change the relevant code. I had a Vibe.d project where Diet templates slowed me down too much (they are super template-heavy), so I split it into several different build targets with a separate link step, so that when I'm not changing the Diet templates it doesn't slow me down so much. Dub unfortunately won't help you here (unless you use subpackages -- but I doubt it will win much) -- I recommend using a better build system like CMake or SCons. Dub's architecture simply does not play well with staged compilation. Alternatively, use a separate pre-compilation stage for generating code (e.g., write a utility that emits D code that then gets compiled in a subsequent step). As much as I love D's compile-time capabilities, there comes a time when it's simply more practical to just `writefln` some D code snippets into a file and compile that into the main program, instead of trying to tame the memory-guzzling beast that is the D compiler. T -- I am Ohm of Borg. Resistance is voltage over current.
Re: Build time
On Friday, 23 July 2021 at 18:57:46 UTC, Adam D Ruppe wrote: On Friday, 23 July 2021 at 18:53:06 UTC, JG wrote: The program I writing is around 3000 loc what's the code? I am not sure how relevant it is but it is a compiler that I have been writing, not something serious (yet - if ever).
Re: Build time
On Friday, 23 July 2021 at 18:53:06 UTC, JG wrote: The program I writing is around 3000 loc what's the code?
Build time
Hi, The program I writing is around 3000 loc and recently I noticed a large slow down in compile time which after investigation seemed to be caused by my computer running out of memory. The compile was using more than 15GB memory. I tried using lowmem and that did solve the memory problem but the compile still takes around 1 minute. Any suggestion on how to try and improve the build time. I am currently using dub. Of course one could try to use fewer templates and less meta programming but that seems to defeat the purpose of using d.
Re: x64 build time 3x slower?
On Tuesday, 8 August 2017 at 00:01:05 UTC, Johnson Jones wrote: On Monday, 7 August 2017 at 22:56:37 UTC, Moritz Maxeiner wrote: On Monday, 7 August 2017 at 22:19:57 UTC, Johnson Jones wrote: Why would that be. Program take about 4 seconds to compile and 12 for x64. There is fundamentally no difference between the two versions. I do link in gtk x86 and gtk x64 depending on version, and that's it as far as I can tell. Debug x86 4 x64 12 Release x86 3 x64 5 The timings are pretty steady. Split up the build time in compile time and link time and see how the difference is distributed between the two. If it's distributed overwhelmingly to the link time it could be that you're using Microsoft's linker for x64 and OPTLINK for x86? Yeah, I guess that is probably it. Forgot that two different linkers were used... although, Not much is being linked. Visual D doesn't split up the two easily, I suppose I can't separate them. It should probably do individual profiling between the two. Usually the build times are pretty close and I don't recall it being slow when I first started building, and my code hasn't changed much... But I did modify a few options such as debug diagnostics and such as those were causing visual studio to freeze. I guess I could perfmon it to see what exactly it is doing. It takes about 3 seconds to write the map file(I have it generating cross references, not sure what it actually does but I thought it might help with debugging issues. I have it enabled for both x86 and x64. I thought I disabled it with no differences, but maybe I missed it). It takes another 4 seconds to create the pdb. But it seems that about 6 seconds are done doing thing 7:13:14.7148555 PM link.exe 9860 QueryStandardInformationFile C:\D\Libraries\x64\gtkd.lib SUCCESS AllocationSize: 84,443,136, EndOfFile: 84,439,450, NumberOfLinks: 1, DeletePending: False, Directory: False 7:13:14.7148665 PM link.exe 9860 CreateFileMapping C:\D\Libraries\x64\gtkd.lib SUCCESS SyncType: SyncTypeOther 7:13:15.7801685 PM ServiceHub.IdentityHost.exe 4952 Thread Exit SUCCESS Thread ID: 11224, User Time: 0.000, Kernel Time: 0.000 7:13:17.2481064 PM ServiceHub.VSDetouredHost.exe 5004 Thread Create SUCCESS Thread ID: 10976 7:13:17.4141043 PM devenv.exe 1040 Thread Create SUCCESS Thread ID: 11200 7:13:17.8990951 PM DParserCOMServer.exe 4492 Thread Create SUCCESS Thread ID: 9176 7:13:18.7481654 PM ServiceHub.VSDetouredHost.exe 5004 Thread Exit SUCCESS Thread ID: 10976, User Time: 0.000, Kernel Time: 0.000 7:13:18.9163321 PM devenv.exe 1040 Thread Exit SUCCESS Thread ID: 11200, User Time: 0.000, Kernel Time: 0.000 7:13:19.4011479 PM DParserCOMServer.exe 4492 Thread Exit SUCCESS Thread ID: 9176, User Time: 0.000, Kernel Time: 0.000 7:13:19.9681080 PM ServiceHub.VSDetouredHost.exe 5004 Thread Create SUCCESS Thread ID: 10416 Not sure what is happening in there but it seems like Visual D or Visual Studio issue rather than dmd ;/ I'll look in to it some more to see what I can find.
Re: x64 build time 3x slower?
On Monday, 7 August 2017 at 22:56:37 UTC, Moritz Maxeiner wrote: On Monday, 7 August 2017 at 22:19:57 UTC, Johnson Jones wrote: Why would that be. Program take about 4 seconds to compile and 12 for x64. There is fundamentally no difference between the two versions. I do link in gtk x86 and gtk x64 depending on version, and that's it as far as I can tell. Debug x86 4 x64 12 Release x86 3 x64 5 The timings are pretty steady. Split up the build time in compile time and link time and see how the difference is distributed between the two. If it's distributed overwhelmingly to the link time it could be that you're using Microsoft's linker for x64 and OPTLINK for x86? Yeah, I guess that is probably it. Forgot that two different linkers were used... although, Not much is being linked. Visual D doesn't split up the two easily, I suppose I can't separate them. It should probably do individual profiling between the two. Usually the build times are pretty close and I don't recall it being slow when I first started building, and my code hasn't changed much... But I did modify a few options such as debug diagnostics and such as those were causing visual studio to freeze. I guess I could perfmon it to see what exactly it is doing.
Re: x64 build time 3x slower?
On Monday, 7 August 2017 at 22:19:57 UTC, Johnson Jones wrote: Why would that be. Program take about 4 seconds to compile and 12 for x64. There is fundamentally no difference between the two versions. I do link in gtk x86 and gtk x64 depending on version, and that's it as far as I can tell. Debug x86 4 x64 12 Release x86 3 x64 5 The timings are pretty steady. Split up the build time in compile time and link time and see how the difference is distributed between the two. If it's distributed overwhelmingly to the link time it could be that you're using Microsoft's linker for x64 and OPTLINK for x86?
x64 build time 3x slower?
Why would that be. Program take about 4 seconds to compile and 12 for x64. There is fundamentally no difference between the two versions. I do link in gtk x86 and gtk x64 depending on version, and that's it as far as I can tell. Debug x86 4 x64 12 Release x86 3 x64 5 The timings are pretty steady.
