Re: Compile-time memory footprint of std.algorithm
On Sat, Jun 21, 2014 at 09:34:35PM +0100, Iain Buclaw via Digitalmars-d wrote: > On 22 April 2014 19:09, Iain Buclaw via Digitalmars-d > wrote: > > Testing a 2.065 pre-release snapshot against GDC. I see that > > std.algorithm now surpasses 2.1GBs of memory consumption when > > compiling unittests. This is bringing my laptop down to its knees > > for a painful 2/3 minutes. > > > > This is time that could be better spent if the unittests where > > simply broken down/split up. > > > The final nail in the coffin was when my laptop locked up building > phobos development using dmd. > > Went out and bought an SSD disk and replaced my crippled HDD drive - > expecting no further problems in the near future... It's long past due for std.algorithm to be broken up. And this isn't the first time problems like this came up, either. I vaguely recall someone working on an algorithms module, potentially splitting up some of the stuff from the current std.algorithm; whatever happened with that? In fact, splitting std.algorithm has been mentioned so many times, that I feel like I should just shut up and submit a PR for it instead. Even if it gets rejected, at least it gets things moving instead of everyone talking about it yet nothing ever comes of it. T -- Perhaps the most widespread illusion is that if we were in power we would behave very differently from those who now hold it---when, in truth, in order to get power we would have to become very much like them. -- Unknown
Re: Compile-time memory footprint of std.algorithm
On 22 April 2014 19:09, Iain Buclaw via Digitalmars-d wrote: > Testing a 2.065 pre-release snapshot against GDC. I see that std.algorithm > now surpasses 2.1GBs of memory consumption when compiling unittests. This > is bringing my laptop down to its knees for a painful 2/3 minutes. > > This is time that could be better spent if the unittests where simply broken > down/split up. The final nail in the coffin was when my laptop locked up building phobos development using dmd. Went out and bought an SSD disk and replaced my crippled HDD drive - expecting no further problems in the near future...
Re: Compile-time memory footprint of std.algorithm
24-Apr-2014 05:12, Marco Leise пишет: Am Wed, 23 Apr 2014 21:23:17 +0400 schrieb Dmitry Olshansky : 23-Apr-2014 20:56, Daniel Murphy пишет: "Dmitry Olshansky" wrote in message news:lj7mrr$1p5s$1...@digitalmars.com... At a times I really don't know why can't we just drop in a Boehm GC (the stock one, not homebrew stuff) and be done with it. Speed? There is no point in speed if it leaks that much. Or you know, switch to D and use druntime's GC. Good point. Can't wait to see D-only codebase. Hmm. DMD doesn't use a known and tried, imprecise GC because it is a lot slower. No it doesn't. It used a precursor of D's GC and that turned out to be slow. See Walter's post. How is DMD written in D using the druntime GC going to help that ? GC is that easier to reach, every enhancement to D's GC becomes instantly available. Wanna make compiler faster - make D's runtime faster! ;) I wondered about this ever since there was talk about DDMD. I'm totally expecting compile times to multiply by 1.2 or so. Since memory management is going to stay the same with disabled GC (at least for starters), I doubt things will change radically. If they will then it'll just highlight perf problems in D's runtime that need work. -- Dmitry Olshansky
Re: Compile-time memory footprint of std.algorithm
On 4/24/2014 7:16 AM, Iain Buclaw via Digitalmars-d wrote: On 24 April 2014 12:01, Ary Borenszweig via Digitalmars-d It wasn't IIRC. 'Twas in-house GC, no? It was with the C++ version of the original D collector.
Re: Compile-time memory footprint of std.algorithm
On Thursday, 24 April 2014 at 06:16:05 UTC, Jacob Carlborg wrote: On 23/04/14 08:39, Walter Bright wrote: I made a build of dmd with a collector in it. It destroyed the speed. Took it out. Isn't that bad advertisement for the GC in D? Or has it something to do with DMD not being designed with a GC in mind? Well, keep in mind we are comparing using the GC versus "doing nothing". I'd be interested in knowing the speed with *any* memory management model in DMD.
Re: Compile-time memory footprint of std.algorithm
On 24 April 2014 12:01, Ary Borenszweig via Digitalmars-d wrote: > On 4/24/14, 3:16 AM, Jacob Carlborg wrote: >> >> On 23/04/14 08:39, Walter Bright wrote: >> >>> I made a build of dmd with a collector in it. It destroyed the speed. >>> Took it out. >> >> >> Isn't that bad advertisement for the GC in D? Or has it something to do >> with DMD not being designed with a GC in mind? > > > dmd is written in C++, the collector must have been boehm > It wasn't IIRC. 'Twas in-house GC, no?
Re: Compile-time memory footprint of std.algorithm
On 4/24/14, 3:16 AM, Jacob Carlborg wrote: On 23/04/14 08:39, Walter Bright wrote: I made a build of dmd with a collector in it. It destroyed the speed. Took it out. Isn't that bad advertisement for the GC in D? Or has it something to do with DMD not being designed with a GC in mind? dmd is written in C++, the collector must have been boehm
Re: Compile-time memory footprint of std.algorithm
On Wednesday, 23 April 2014 at 08:12:42 UTC, Walter Bright wrote: tl;dr: allocation is a critical speed issue with dmd. Using the bump-pointer method is very fast, and it matters. Alternatively we could replace heap on size threshold.
Re: Compile-time memory footprint of std.algorithm
On 23 April 2014 22:24, Iain Buclaw wrote: > On 23 April 2014 21:55, "Nordlöw" wrote: >>> Maybe we should investigate where the memory is going first before >>> planning our attack :-) >> >> >> I agree. Tool anyone? > > I'm using valgrind - may take a while to process and merge them all. > I'll post an update in the morning. I was amazed to see some small losses in the glue (that I'll be dealing with), but by and large the worst culprits were all the syntaxCopy'ing done in Template semantic analysis. The resultant assembly file emitted by gdc is 83MB in size, so I think it is impossible to not have a large memory consumption here. The stats file is 100MB (39k reported leaks) and I'm not sure just what to do with it yet.
Re: Compile-time memory footprint of std.algorithm
On 23/04/14 08:39, Walter Bright wrote: I made a build of dmd with a collector in it. It destroyed the speed. Took it out. Isn't that bad advertisement for the GC in D? Or has it something to do with DMD not being designed with a GC in mind? -- /Jacob Carlborg
Re: Compile-time memory footprint of std.algorithm
Am Wed, 23 Apr 2014 21:23:17 +0400 schrieb Dmitry Olshansky : > 23-Apr-2014 20:56, Daniel Murphy пишет: > > "Dmitry Olshansky" wrote in message > > news:lj7mrr$1p5s$1...@digitalmars.com... > >> At a times I really don't know why can't we just drop in a Boehm GC > >> (the stock one, not homebrew stuff) and be done with it. Speed? There > >> is no point in speed if it leaks that much. > > > > Or you know, switch to D and use druntime's GC. > > Good point. Can't wait to see D-only codebase. Hmm. DMD doesn't use a known and tried, imprecise GC because it is a lot slower. How is DMD written in D using the druntime GC going to help that ? I wondered about this ever since there was talk about DDMD. I'm totally expecting compile times to multiply by 1.2 or so. -- Marco
Re: Compile-time memory footprint of std.algorithm
On Wednesday, 23 April 2014 at 23:19:20 UTC, Jussi Jumppanen wrote: In that project compiling the lexer.d file causes a massive increase in compiler memory usage. The code is actually located here: https://github.com/Hackerpilot/Dscanner If you want to make DMD cry, compile it with "-O -inline -release".
Re: Compile-time memory footprint of std.algorithm
On Wednesday, 23 April 2014 at 20:04:09 UTC, Peter Alexander wrote: Maybe we should investigate where the memory is going first before planning our attack :-) FWIW one hint might be found in the DCD project found here: https://github.com/Hackerpilot/DCD/ In that project compiling the lexer.d file causes a massive increase in compiler memory usage. More details found here: https://github.com/Hackerpilot/DCD/issues/93 NOTE: That was DMD running on a 32 bit Windows XP machine.
Re: Compile-time memory footprint of std.algorithm
On 23 April 2014 21:55, "Nordlöw" wrote: >> Maybe we should investigate where the memory is going first before >> planning our attack :-) > > > I agree. Tool anyone? I'm using valgrind - may take a while to process and merge them all. I'll post an update in the morning.
Re: Compile-time memory footprint of std.algorithm
I agree. Tool anyone? https://stackoverflow.com/questions/23255043/finding-unexercised-bits-of-allocated-data Massif may give some clues.
Re: Compile-time memory footprint of std.algorithm
Maybe we should investigate where the memory is going first before planning our attack :-) I agree. Tool anyone?
Re: Compile-time memory footprint of std.algorithm
On Wednesday, 23 April 2014 at 19:54:29 UTC, Nordlöw wrote: tl;dr: allocation is a critical speed issue with dmd. Using the bump-pointer method is very fast, and it matters. What about packing DMD structure members such as integers and enums more efficiently? We could start with making enums __attribute__((packed)). Is there any free static/dynamic tool to check for unexercized bits? How does Clang do to save so much space compared to GCC? Do they pack gentlier or use deallocation? A much higher-hanging fruit is to switch from using pointers to 32-bit handles on 64-bit CPUs to reference tokens, sub-expressions etc. But I guess that is a big undertaking getting type-safe and may give performance hits. Maybe we should investigate where the memory is going first before planning our attack :-)
Re: Compile-time memory footprint of std.algorithm
tl;dr: allocation is a critical speed issue with dmd. Using the bump-pointer method is very fast, and it matters. What about packing DMD structure members such as integers and enums more efficiently? We could start with making enums __attribute__((packed)). Is there any free static/dynamic tool to check for unexercized bits? How does Clang do to save so much space compared to GCC? Do they pack gentlier or use deallocation? A much higher-hanging fruit is to switch from using pointers to 32-bit handles on 64-bit CPUs to reference tokens, sub-expressions etc. But I guess that is a big undertaking getting type-safe and may give performance hits.
Re: Compile-time memory footprint of std.algorithm
On Wednesday, 23 April 2014 at 15:46:00 UTC, Steven Schveighoffer wrote: The time it takes to compile a program where the compiler consumes 2G of ram on a 2G machine is infinite ;) (nitpick: not necessarily given good swap behaviour!)
Re: Compile-time memory footprint of std.algorithm
On Wednesday, 23 April 2014 at 17:16:40 UTC, Walter Bright wrote: On 4/23/2014 2:00 AM, Dmitry Olshansky wrote: If the prime reason compilation is fast is because we just throw away memory, we must be doing something wrong, very wrong. I've tried adding a collector to DMD with poor results. If you'd like to give it a try as well, please do so. The thing is, I work all day every day on D. I cannot do more. If people want more things done, like redesigning memory allocation in the compiler, redesigning D to do ARC, etc., they'll need to pitch in. Well said Walter!
Re: Compile-time memory footprint of std.algorithm
23-Apr-2014 21:16, Walter Bright пишет: On 4/23/2014 2:00 AM, Dmitry Olshansky wrote: If the prime reason compilation is fast is because we just throw away memory, we must be doing something wrong, very wrong. I've tried adding a collector to DMD with poor results. If you'd like to give it a try as well, please do so. I'll give it a spin then. The thing is, I work all day every day on D. I cannot do more. That is understood, thanks for honesty. If people want more things done, like redesigning memory allocation in the compiler, redesigning D to do ARC, etc., they'll need to pitch in. True. -- Dmitry Olshansky
Re: Compile-time memory footprint of std.algorithm
On 4/23/14, 1:56 PM, Daniel Murphy wrote: "Dmitry Olshansky" wrote in message news:lj7mrr$1p5s$1...@digitalmars.com... At a times I really don't know why can't we just drop in a Boehm GC (the stock one, not homebrew stuff) and be done with it. Speed? There is no point in speed if it leaks that much. Or you know, switch to D and use druntime's GC. But that will be slow. Walter's point is that if you introduce a GC it will be slower. Of course, you won't be able to compile big stuff. But developers usually have good machines, so it's not that a big deal.
Re: Compile-time memory footprint of std.algorithm
23-Apr-2014 20:56, Daniel Murphy пишет: "Dmitry Olshansky" wrote in message news:lj7mrr$1p5s$1...@digitalmars.com... At a times I really don't know why can't we just drop in a Boehm GC (the stock one, not homebrew stuff) and be done with it. Speed? There is no point in speed if it leaks that much. Or you know, switch to D and use druntime's GC. Good point. Can't wait to see D-only codebase. -- Dmitry Olshansky
Re: Compile-time memory footprint of std.algorithm
On 4/23/2014 2:00 AM, Dmitry Olshansky wrote: If the prime reason compilation is fast is because we just throw away memory, we must be doing something wrong, very wrong. I've tried adding a collector to DMD with poor results. If you'd like to give it a try as well, please do so. The thing is, I work all day every day on D. I cannot do more. If people want more things done, like redesigning memory allocation in the compiler, redesigning D to do ARC, etc., they'll need to pitch in.
Re: Compile-time memory footprint of std.algorithm
"Dmitry Olshansky" wrote in message news:lj7mrr$1p5s$1...@digitalmars.com... At a times I really don't know why can't we just drop in a Boehm GC (the stock one, not homebrew stuff) and be done with it. Speed? There is no point in speed if it leaks that much. Or you know, switch to D and use druntime's GC.
Re: Compile-time memory footprint of std.algorithm
On Wed, 23 Apr 2014 02:39:05 -0400, Walter Bright wrote: On 4/22/2014 11:33 PM, Dmitry Olshansky wrote: At a times I really don't know why can't we just drop in a Boehm GC (the stock one, not homebrew stuff) and be done with it. Speed? There is no point in speed if it leaks that much. I made a build of dmd with a collector in it. It destroyed the speed. Took it out. The time it takes to compile a program where the compiler consumes 2G of ram on a 2G machine is infinite ;) There must be some compromise between slow-but-perfect memory management and invoking the OOM killer. -Steve
Re: Compile-time memory footprint of std.algorithm
23-Apr-2014 12:12, Walter Bright пишет: On 4/23/2014 12:20 AM, Kagamin wrote: On Wednesday, 23 April 2014 at 06:39:04 UTC, Walter Bright wrote: I made a build of dmd with a collector in it. It destroyed the speed. Took it out. Is it because of garbage collections? Then allow people configure collection threshold, say, collect garbage only when the heap is bigger than 16GB. It's more than that. I invite you to read the article I wrote on DrDobbs a while back about changes to the allocator to improve speed. tl;dr: allocation is a critical speed issue with dmd. Using the bump-pointer method is very fast, and it matters. This stinks it's not even half-serious. A x2 speed increase was due to scraping the old allocator on Win32 altogether and using plain HeapAPI. If the prime reason compilation is fast is because we just throw away memory, we must be doing something wrong, very wrong. -- Dmitry Olshansky
Re: Compile-time memory footprint of std.algorithm
On 4/22/2014 11:56 PM, Dmitry Olshansky wrote: Getting more practical - any chance to use it selectively in CTFE and related stuff that is KNOWN to generate garbage? Using it there only will require a rewrite of interpret.c.
Re: Compile-time memory footprint of std.algorithm
On 4/23/2014 12:20 AM, Kagamin wrote: On Wednesday, 23 April 2014 at 06:39:04 UTC, Walter Bright wrote: I made a build of dmd with a collector in it. It destroyed the speed. Took it out. Is it because of garbage collections? Then allow people configure collection threshold, say, collect garbage only when the heap is bigger than 16GB. It's more than that. I invite you to read the article I wrote on DrDobbs a while back about changes to the allocator to improve speed. tl;dr: allocation is a critical speed issue with dmd. Using the bump-pointer method is very fast, and it matters.
Re: Compile-time memory footprint of std.algorithm
On Wednesday, 23 April 2014 at 06:39:04 UTC, Walter Bright wrote: I made a build of dmd with a collector in it. It destroyed the speed. Took it out. Is it because of garbage collections? Then allow people configure collection threshold, say, collect garbage only when the heap is bigger than 16GB.
Re: Compile-time memory footprint of std.algorithm
23-Apr-2014 10:39, Walter Bright пишет: On 4/22/2014 11:33 PM, Dmitry Olshansky wrote: At a times I really don't know why can't we just drop in a Boehm GC (the stock one, not homebrew stuff) and be done with it. Speed? There is no point in speed if it leaks that much. I made a build of dmd with a collector in it. It destroyed the speed. Took it out. Getting more practical - any chance to use it selectively in CTFE and related stuff that is KNOWN to generate garbage? -- Dmitry Olshansky
Re: Compile-time memory footprint of std.algorithm
On 4/22/2014 11:33 PM, Dmitry Olshansky wrote: At a times I really don't know why can't we just drop in a Boehm GC (the stock one, not homebrew stuff) and be done with it. Speed? There is no point in speed if it leaks that much. I made a build of dmd with a collector in it. It destroyed the speed. Took it out.
Re: Compile-time memory footprint of std.algorithm
23-Apr-2014 01:00, Iain Buclaw via Digitalmars-d пишет: On 22 April 2014 21:43, Peter Alexander via Digitalmars-d wrote: On Tuesday, 22 April 2014 at 18:09:12 UTC, Iain Buclaw wrote: Testing a 2.065 pre-release snapshot against GDC. I see that std.algorithm now surpasses 2.1GBs of memory consumption when compiling unittests. This is bringing my laptop down to its knees for a painful 2/3 minutes. My (ancient) laptop only has 2GB of RAM :-) Has anyone looked into why it is using so much? Is it all the temporary allocations created by CTFE that are never cleaned up? I blame Kenji and all the semanticTiargs and other template-related copying and discarding of memory around the place. :o) At a times I really don't know why can't we just drop in a Boehm GC (the stock one, not homebrew stuff) and be done with it. Speed? There is no point in speed if it leaks that much. -- Dmitry Olshansky
Re: Compile-time memory footprint of std.algorithm
On 22 April 2014 21:43, Peter Alexander via Digitalmars-d wrote: > On Tuesday, 22 April 2014 at 18:09:12 UTC, Iain Buclaw wrote: >> >> Testing a 2.065 pre-release snapshot against GDC. I see that std.algorithm >> now surpasses 2.1GBs of memory consumption when compiling unittests. This >> is bringing my laptop down to its knees for a painful 2/3 minutes. > > > My (ancient) laptop only has 2GB of RAM :-) > > Has anyone looked into why it is using so much? Is it all the temporary > allocations created by CTFE that are never cleaned up? I blame Kenji and all the semanticTiargs and other template-related copying and discarding of memory around the place. :o)
Re: Compile-time memory footprint of std.algorithm
On Tuesday, 22 April 2014 at 18:09:12 UTC, Iain Buclaw wrote: Testing a 2.065 pre-release snapshot against GDC. I see that std.algorithm now surpasses 2.1GBs of memory consumption when compiling unittests. This is bringing my laptop down to its knees for a painful 2/3 minutes. My (ancient) laptop only has 2GB of RAM :-) Has anyone looked into why it is using so much? Is it all the temporary allocations created by CTFE that are never cleaned up?
Re: Compile-time memory footprint of std.algorithm
On Tue, Apr 22, 2014 at 06:09:11PM +, Iain Buclaw via Digitalmars-d wrote: > Testing a 2.065 pre-release snapshot against GDC. I see that > std.algorithm now surpasses 2.1GBs of memory consumption when > compiling unittests. This is bringing my laptop down to its knees for > a painful 2/3 minutes. > > This is time that could be better spent if the unittests where simply > broken down/split up. Didn't we say (many months ago!) that we wanted to split up std.algorithm into more manageable chunks? I see that that hasn't happened yet. :-( T -- "Real programmers can write assembly code in any language. :-)" -- Larry Wall
