Re: How to reuse the space of RAM of an array of strings?
On Friday, 21 December 2018 at 22:31:26 UTC, Steven Schveighoffer wrote: On 12/21/18 3:41 PM, Giovanni Di Maria wrote: [...] Note: alloca is a builtin intrinsic, so I wouldn't use that as a function name. Don't think it's affecting your program, but I wanted to point that out. [...] This does NOT free the ram, it simply resets vec to null. If you want to free the memory, use GC.free(vec.ptr); vec = null; [...] Note, this allocates a lot of smaller arrays on its way up to the really big array. You are better off doing: vec.reserve(5_000_000); which will pre-allocate the capacity needed. This will make you only allocate once. -Steve Hi Steve Ok, thank you very much. Ciao Giovanni
Re: How to reuse the space of RAM of an array of strings?
On 12/21/18 3:41 PM, Giovanni Di Maria wrote:
Hi. Can you help me please?
I asked also to "General Group" but i haven't solved my problem.
I have tried many experiments but without success.
I have a dynamic array of strings:
string[] vec;
After to have inserted dynamically 5_000_000 strings to array, for six
times, i measure the RAM with an utility (for example Wise Memory
Optimizer), deleting the array every time.
Every time I destroy the array, but the RAM is always less.
In particular:
- at beginning the FREE RAM is 1564 MB;
- after first loop the FREE RAM IS 1480 MB
- after second loop the FREE RAM IS 1415 MB
- after third loop the FREE RAM IS 1402 MB
- after forth loop the FREE RAM IS 1338 MB
- after fifth loop the FREE RAM IS 1280 MB
- after sixth loop the FREE RAM IS 1200 MB
- at end the FREE RAM returns to 1564 MB
I want to reuse the dynamic array.
This is the program:
import std.stdio;
string[] vec;
void main()
{
alloca();
alloca();
alloca();
alloca();
alloca();
alloca();
}
void alloca()
Note: alloca is a builtin intrinsic, so I wouldn't use that as a
function name. Don't think it's affecting your program, but I wanted to
point that out.
{
vec.destroy;
This does NOT free the ram, it simply resets vec to null.
If you want to free the memory, use
GC.free(vec.ptr); vec = null;
writeln("Filling .");
for (int i = 0; i < 500; i++)
vec ~= "1234567890ABCDEFGHIL123456";
Note, this allocates a lot of smaller arrays on its way up to the really
big array. You are better off doing:
vec.reserve(5_000_000);
which will pre-allocate the capacity needed. This will make you only
allocate once.
-Steve
Re: How to reuse the space of RAM of an array of strings?
On Friday, 21 December 2018 at 21:28:14 UTC, H. S. Teoh wrote: [...] Thank you very much for your replay. I will meditate about your words. Thank you Giovanni
Re: How to reuse the space of RAM of an array of strings?
On Fri, Dec 21, 2018 at 08:41:10PM +, Giovanni Di Maria via
Digitalmars-d-learn wrote:
[...]
> After to have inserted dynamically 5_000_000 strings to array, for six
> times, i measure the RAM with an utility (for example Wise Memory
> Optimizer), deleting the array every time.
>
> Every time I destroy the array, but the RAM is always less.
> In particular:
>
> - at beginning the FREE RAM is 1564 MB;
> - after first loop the FREE RAM IS 1480 MB
> - after second loop the FREE RAM IS 1415 MB
> - after third loop the FREE RAM IS 1402 MB
> - after forth loop the FREE RAM IS 1338 MB
> - after fifth loop the FREE RAM IS 1280 MB
> - after sixth loop the FREE RAM IS 1200 MB
> - at end the FREE RAM returns to 1564 MB
>
> I want to reuse the dynamic array.
[...]
You need to understand that generally speaking, the GC does not return
memory to the OS; it retains that memory for future use, e.g., when you
allocate a new object. Also, not all objects may be freed immediately
if the GC can still find references to it somewhere, including on the
stack. And if you're in a 32-bit environment, there's a higher chance
you might run into the problem of false pointers (non-pointer values
that look like a valid pointer into an allocated object, causing the GC
to think the object is still live when it's actually already dead).
Furthermore, when allocating large numbers of objects the GC may create
additional internal data structures to keep track of said objects, and
may continue to retain these structures after the objects have been
collected.
If you want deterministic deallocation, your best bet is to use
malloc/free instead of the GC (that means you'll need to avoid
allocating operators like ~, ~=, array literals, and so on). Using @nogc
may help here.
A more reliable test is if you let your loop run forever, and see if the
memory usage eventually plateaus. It should stop increasing at a certain
point once you've reached steady state; if not, that would be stronger
proof of memory leakage.
Furthermore, if you want to reuse existing memory for the array, you
should just write to it directly instead of deallocating / reallocating.
I.e., instead of doing this:
foreach (i; 0 .. 1_000_000)
{
auto arr = new int[1_000_000];
... // do stuff with arr
arr = null; // or arr.destroy or whatever
}
do this:
auto arr = new int[1_000_000]; // only allocate once
foreach (i; 0 .. 1_000_000)
{
... // overwrite any previous contents of arr with new data
... // do stuff with arr
}
For your array of strings, you probably want to use a string buffer /
pool somewhere, i.e., a contiguous block of memory that you allocate
once, and take slices of as you populate your string array, instead of
allocating once per array element. This will give you better control
over your memory usage, and also reduce GC load (so collection cycles
will generally be faster, since there's less stuff that needs to be
marked and collected).
T
--
Give a man a fish, and he eats once. Teach a man to fish, and he will sit
forever.
How to reuse the space of RAM of an array of strings?
Hi. Can you help me please?
I asked also to "General Group" but i haven't solved my problem.
I have tried many experiments but without success.
I have a dynamic array of strings:
string[] vec;
After to have inserted dynamically 5_000_000 strings to array,
for six times, i measure the RAM with an utility (for example
Wise Memory Optimizer), deleting the array every time.
Every time I destroy the array, but the RAM is always less.
In particular:
- at beginning the FREE RAM is 1564 MB;
- after first loop the FREE RAM IS 1480 MB
- after second loop the FREE RAM IS 1415 MB
- after third loop the FREE RAM IS 1402 MB
- after forth loop the FREE RAM IS 1338 MB
- after fifth loop the FREE RAM IS 1280 MB
- after sixth loop the FREE RAM IS 1200 MB
- at end the FREE RAM returns to 1564 MB
I want to reuse the dynamic array.
This is the program:
import std.stdio;
string[] vec;
void main()
{
alloca();
alloca();
alloca();
alloca();
alloca();
alloca();
}
void alloca()
{
vec.destroy;
writeln("Filling .");
for (int i = 0; i < 500; i++)
vec ~= "1234567890ABCDEFGHIL123456";
writeln("Array occupation: ", vec.capacity);
writeln("Press ");
readln();
}
P.S.
I have tried: free(), destroy(), minimize(), GC, collect(), etc
but without success.
Can you help me please?
Thank you very much
Giovanni Di Maria
