How to save RAM in D programs (on zero initialized buffers)

2012-02-07 Thread Marco Leise
Hi, this is me again with some "size matters" topic. This time, it's not the executable size, no! Instead I want to discuss a runtime memory footprint and speed issue that affects everyone, and how to improve the situation dramatically. In D we allocate memory through the GC, that is initia

Re: How to save RAM in D programs (on zero initialized buffers)

2012-02-07 Thread Marco Leise
Aw Opera fooled me again into answering a post, instead of creating a new one - ignore this, I'll repost a proper thread.

How to save RAM in D programs (on zero initialized buffers): Reloaded

2012-02-07 Thread Marco Leise
Hi, this is me again with some "size matters" topic. This time, it's not the executable size, no! Instead I want to discuss a runtime memory footprint and speed issue that affects everyone, and how to improve the situation dramatically. In D we allocate memory through the GC, that is initialized

Re: How to save RAM in D programs (on zero initialized buffers)

2012-02-07 Thread Marco Leise
Am 07.02.2012, 21:11 Uhr, schrieb Nick Sabalausky : Is void initialization not good enough? IIRC it's something like: ubyte[] buf = void; That gives me a) no buffer, who's pointer is b) not initialized to null. I want instead a defined pointer, to a valid array, that is initialized to zero

Re: How to save RAM in D programs (on zero initialized buffers)

2012-02-07 Thread Michel Fortin
On 2012-02-07 20:24:40 +, "Marco Leise" said: Am 07.02.2012, 21:11 Uhr, schrieb Nick Sabalausky : Is void initialization not good enough? IIRC it's something like: ubyte[] buf = void; That gives me a) no buffer, who's pointer is b) not initialized to null. I want instead a defined poi

Re: How to save RAM in D programs (on zero initialized buffers)

2012-02-07 Thread Martin Nowak
On Tue, 07 Feb 2012 21:24:40 +0100, Marco Leise wrote: Am 07.02.2012, 21:11 Uhr, schrieb Nick Sabalausky : Is void initialization not good enough? IIRC it's something like: ubyte[] buf = void; That gives me a) no buffer, who's pointer is b) not initialized to null. I want instead a define

Re: How to save RAM in D programs (on zero initialized buffers)

2012-02-07 Thread Martin Nowak
On Tue, 07 Feb 2012 21:37:03 +0100, Michel Fortin wrote: On 2012-02-07 20:24:40 +, "Marco Leise" said: Am 07.02.2012, 21:11 Uhr, schrieb Nick Sabalausky : Is void initialization not good enough? IIRC it's something like: ubyte[] buf = void; That gives me a) no buffer, who's pointe

Re: How to save RAM in D programs (on zero initialized buffers)

2012-02-07 Thread Marco Leise
Am 08.02.2012, 04:37 Uhr, schrieb Manfred Nowak : Marco Leise wrote: I'm not aware of any caveats, are there any? The tests only cover a very small fraction of an unknown data structure: the allocation phase. Of course one can want to make a bad design running faster. Especially if one need

Re: How to save RAM in D programs (on zero initialized buffers)

2012-02-07 Thread Jacob Carlborg
On 2012-02-07 21:37, Michel Fortin wrote: On 2012-02-07 20:24:40 +, "Marco Leise" said: Am 07.02.2012, 21:11 Uhr, schrieb Nick Sabalausky : Is void initialization not good enough? IIRC it's something like: ubyte[] buf = void; That gives me a) no buffer, who's pointer is b) not initia

Re: How to save RAM in D programs (on zero initialized buffers)

2012-02-08 Thread Manfred Nowak
Marco Leise wrote: > In D we allocate memory through the GC [...] > Let's assume, we have a program that allocates some buffers in > advance, that it may not use fully. [...] > there is really no alternative to calloc. 1) calloc implements a strategie for allocating memory. If this strategy is u

Re: How to save RAM in D programs (on zero initialized buffers)

2012-02-08 Thread Marco Leise
Am 08.02.2012, 14:30 Uhr, schrieb Manfred Nowak : Marco Leise wrote: In D we allocate memory through the GC [...] Let's assume, we have a program that allocates some buffers in advance, that it may not use fully. [...] there is really no alternative to calloc. 1) calloc implements a stra

Re: How to save RAM in D programs (on zero initialized buffers)

2012-02-08 Thread Michel Fortin
On 2012-02-08 07:30:29 +, Jacob Carlborg said: On 2012-02-07 21:37, Michel Fortin wrote: On 2012-02-07 20:24:40 +, "Marco Leise" said: Am 07.02.2012, 21:11 Uhr, schrieb Nick Sabalausky : Is void initialization not good enough? IIRC it's something like: ubyte[] buf = void; That

Re: How to save RAM in D programs (on zero initialized buffers)

2012-02-08 Thread Manfred Nowak
Marco Leise wrote: >> For the case that a + b exceeds available memory > Here I can't follow you. A typo: I wanted to write "a + c". As far as I understand your messages I misunderstand what you write and will verify that at this weekend. -manfred

Re: How to save RAM in D programs (on zero initialized buffers)

2012-02-08 Thread Manfred Nowak
Marco Leise wrote: > That sounds a bit vague. Andrei has written a paper on allocation: http://erdani.com/publications/cuj-2005-12.pdf -manfred

Re: How to save RAM in D programs (on zero initialized buffers)

2012-02-09 Thread Marco Leise
Am 09.02.2012, 03:56 Uhr, schrieb Manfred Nowak : Marco Leise wrote: That sounds a bit vague. Andrei has written a paper on allocation: http://erdani.com/publications/cuj-2005-12.pdf -manfred Oh ok, that's farther than I ever digged into memory management. My current problem with the

Re: How to save RAM in D programs (on zero initialized buffers)

2012-02-09 Thread Kagamin
I guess, calloc will reuse blocks too, so if you run the compressing function twice, it will reuse the memory block used and freed previously and zero it out honestly.

Re: How to save RAM in D programs (on zero initialized buffers)

2012-02-09 Thread Marco Leise
Am 09.02.2012, 11:55 Uhr, schrieb Kagamin : I guess, calloc will reuse blocks too, so if you run the compressing function twice, it will reuse the memory block used and freed previously and zero it out honestly. You don't understand how it works. calloc gives you exactly 0 KB of memory. T

Re: How to save RAM in D programs (on zero initialized buffers): Reloaded

2012-02-07 Thread Nick Sabalausky
Is void initialization not good enough? IIRC it's something like: ubyte[] buf = void;

Re: How to save RAM in D programs (on zero initialized buffers): Reloaded

2012-02-07 Thread Kapps
On 07/02/2012 2:11 PM, Nick Sabalausky wrote: Is void initialization not good enough? IIRC it's something like: ubyte[] buf = void; This example would be uninitializedArray!(ubyte[])(1024 * 1024). I would guess that it gives significantly better performance. There's also minimallyInitiali

Re: How to save RAM in D programs (on zero initialized buffers): Reloaded

2012-02-07 Thread Iain Buclaw
On 7 February 2012 19:39, Marco Leise wrote: > Hi, this is me again with some "size matters" topic. This time, it's not > the executable size, no! Instead I want to discuss a runtime memory > footprint and speed issue that affects everyone, and how to improve the > situation dramatically. > > In D

Re: How to save RAM in D programs (on zero initialized buffers): Reloaded

2012-02-07 Thread F i L
Can't you just write a custom allocator using calloc for performance critical structures (http://dlang.org/memory.html#newdelete), or do what Iain said.

Re: How to save RAM in D programs (on zero initialized buffers): Reloaded

2012-02-07 Thread Timon Gehr
On 02/08/2012 12:01 AM, F i L wrote: Can't you just write a custom allocator using calloc for performance critical structures (http://dlang.org/memory.html#newdelete), or do what Iain said. The solution with the best performance and least memory requirements obviously must be the default.

Re: How to save RAM in D programs (on zero initialized buffers): Reloaded

2012-02-07 Thread Artur Skawina
On 02/08/12 00:01, F i L wrote: > Can't you just write a custom allocator using calloc for performance critical > structures (http://dlang.org/memory.html#newdelete), or do what Iain said. That won't help - the compiler will defeat the optimization by "initializing" the area... artur

Re: How to save RAM in D programs (on zero initialized buffers): Reloaded

2012-02-07 Thread F i L
Artur Skawina wrote: That won't help - the compiler will defeat the optimization by "initializing" the area... I see. Timon Gehr wrote: The solution with the best performance and least memory requirements obviously must be the default. No argument here. Only, if calloc is an all-around bet

Re: How to save RAM in D programs (on zero initialized buffers): Reloaded

2012-02-07 Thread Manfred Nowak
Marco Leise wrote: > I'm not aware of any caveats, are there any? The tests only cover a very small fraction of an unknown data structure: the allocation phase. Of course one can want to make a bad design running faster. Especially if one need to allocate 0.5 TB main memory and can allocate th

Re: How to save RAM in D programs (on zero initialized buffers): Reloaded

2012-02-07 Thread F i L
F i L wrote: Only, if calloc is an all-around better allocation method than malloc, why is malloc even used? Note-to-self: google before asking stupid questions...

Re: How to save RAM in D programs (on zero initialized buffers): Reloaded

2012-02-07 Thread Jose Armando Garcia
On Tue, Feb 7, 2012 at 5:39 PM, Marco Leise wrote: > Hi, this is me again with some "size matters" topic. This time, it's not > the executable size, no! Instead I want to discuss a runtime memory > footprint and speed issue that affects everyone, and how to improve the > situation dramatically. >

Re: How to save RAM in D programs (on zero initialized buffers): Reloaded

2012-02-07 Thread Marco Leise
Am 08.02.2012, 04:40 Uhr, schrieb Jose Armando Garcia : Special? What do you mean by special? Most OS use Virtual Memory so sure they can say here is a page and yet not have that page backed by physical memory. To my knowledge, they can only do this if the allocated memory points to an unmapped

Re: How to save RAM in D programs (on zero initialized buffers): Reloaded

2012-02-07 Thread Marco Leise
Am 07.02.2012, 22:15 Uhr, schrieb Iain Buclaw : o zero out a memory block <-- !!! What about these functions? import std.array; byte[][ALLOCS] a, b; writeln("** uninitializedArray!(ubyte[])(1024*1024)"); foreach(i; 0 .. ALLOCS) b[i] = uninitializedArray!(ubyte[])(1024 * 102