On Wed, Jun 27, 2018 at 04:00:37PM +0000, Mario Silva via Digitalmars-d wrote: > Hello, > > Our code base has been growing steadily and it's currently at a point > where my 16GB machine just freezes when we're compiling our code. This > happens because DMD just consumes all my memory for a while.
How are you compiling the code? Dub? Makefile? Are you compiling everything all at once? DMD is known to prefer compilation speed over memory efficiency, a choice I personally don't quite agree with, but it is what it is. That means past a certain size, you're going to have to split up your project into multiple separate compilations. The usual convention is to compile packages together, e.g., if you have the source tree structure: main/main.d main/common.d pkg1/package.d pkg1/mod1.d pkg1/mod2.d pkg2/package.d pkg2/mod1.d pkg2/mod2.d then you'd compile main/* first, say into main.a (or main.lib), and compile pkg1/* into pkg1.a, and compile pkg2/* into pkg2.a, then finally link them all together with a final invocation of the compiler. This should reduce the total amount of memory used by any single invocation of dmd, which should keep memory usage sane. It may also reduce compilation time if only one package needs to be recompiled (though template-heavy code may not save that much, depending on what you do with it). [..] > Are there any plans to work on compiler performance in terms of memory > consumption and compilation time? Not that I know of, though it'd be really nice if somebody worked on that. Given Walter's past choice of compilation speed over memory efficiency, I'm not sure how far such an effort would get. But I'd vote for it. There *is* the option of just turning on the GC, now that dmd is bootstrapping, but the last time I tried there were bugs that caused crashes, so it may not (yet) be a viable option. > Any tips on how to code in a way that minimizes both compilation times > and memory consumption when compiling? [...] In general: (1) avoid template bloat, and (2) use separate compilation in strategic ways. (1) implies: - Avoid using (or reduce usage of) libraries / modules that are heavy on templates, except where absolutely necessary. If you're using Phobos, std.format comes to mind (though in a sufficiently large project I'd expect it doesn't play that big of a role, but the principle holds). If you're using vibe.d, Diet templates can be a source of heavy template usage. More generally, avoid recursive templates, or templates instantiation chains that nest too deeply. If not done carefully, you may end up with a quadratic or an exponential number of template instantiations, which will quickly eat up all your memory and CPU cycles (in addition to generating bloated executables). - Avoid doing too much in CTFE, because the current CTFE engine is slow (and also very memory-inefficient). We're all waiting for newCTFE to land, but from what I can tell that's still a ways away. (Note: not to be confused with other compile-time features that are not directly related to CTFE. Using static if's and the like won't add much overhead unless you're using recursive templates, in which case see point #1.) If template-heavy or CTFE-heavy code is essential to your project, consider splitting them into their own self-contained (sub)modules, and separately compile them, so that recompilations that don't touch those parts of the code will be reasonably fast. In one of my vibe.d projects, I basically separated out the "business logic" of the program into a module outside the code that directly interfaces with vibe.d (Diet templates being the primary target). This has sped up recompilations when only the "business logic" parts of the code needs to change. Furthermore, because Diet templates tend to get very heavy, I also split those up into different compilation groups (e.g., new user / account management pages, error/miscellaneous pages, main pages) so that if I'm working on Diet templates in, say, the new user creation pages, I only have to recompile the Diet templates related to account management, not every other Diet template in the project. This also saves a lot of time waiting for compilation. T -- It is impossible to make anything foolproof because fools are so ingenious. -- Sammy