Cleaning/Releasing large arrays from memory

[Clinton via Digitalmars-d-learn]

Hi guys, I have a question on how to free large arrays in D after 
they're no longer needed.


Let's say I have this:


SomeKey[] getKeys() {
  SomeKey[] n;

  foreach(batching stuff...) {
   SomeKey newstuff = db.select!(SomeKey[])(...); // gets around 
6M of these from db


   redis.send("newItems", newStuff.length);

   n ~= newStuff;

   destroy(newStuff);
  }

  return n;
}

void main() {
  SomeKey[] myarray = getKeys();
  ulong[string] mappedValues;

  foreach(i, item; myArray) {
   mappedValues[item.name] = i;
  }

  // No longer need myarray, and we need the memory back
  destory(myarray);
  GC.collect;

  // Another memory intensive operation below
}


This seems to work for me to a point, but I notice my program 
still holds a lot in memory even after certain memory intensive 
arrays should have been collected.


Is this just a hit and miss with the GC or am I doing something 
wrong? Uses around 40% less memory forcing the GC and cleaning 
arrays but seems like a lot is left over, especially for threads 
that have already finished(I've expect a thread to free all it's 
memory). I'm not sharing anything across threads besides a config 
struct.


This is a simplified version of my app. At the moment when at 
around 9GB at it's peak instead of 19GB like before.

Re: Cleaning/Releasing large arrays from memory

[Clinton via Digitalmars-d-learn]

On Monday, 24 July 2017 at 14:51:04 UTC, rikki cattermole wrote:

Stuff the GC.
You don't need it to care about collecting (or destroying for 
that matter).


Tell it to free[0] the array directly.

```D
T[] array;

GC.free(array.ptr);
```

Normally I would suggest to create your own buffer, but because 
of the DB library probably doesn't support that, no point 
trying to force it.


[0] http://dlang.org/phobos/core_memory.html#.GC.free


Thanks! I didn't know I could do that with the GC enabled.

Re: Cleaning/Releasing large arrays from memory

[Clinton via Digitalmars-d-learn]

On Monday, 24 July 2017 at 14:51:04 UTC, rikki cattermole wrote:

Stuff the GC.
You don't need it to care about collecting (or destroying for 
that matter).


Tell it to free[0] the array directly.

```D
T[] array;

GC.free(array.ptr);
```

Normally I would suggest to create your own buffer, but because 
of the DB library probably doesn't support that, no point 
trying to force it.


[0] http://dlang.org/phobos/core_memory.html#.GC.free


BTW, how would I do this with associative arrays?

Re: Cleaning/Releasing large arrays from memory

[Clinton via Digitalmars-d-learn]

On Monday, 24 July 2017 at 18:01:19 UTC, ketmar wrote:

Clinton wrote:

On Monday, 24 July 2017 at 14:51:04 UTC, rikki cattermole 
wrote:

Stuff the GC.
You don't need it to care about collecting (or destroying for 
that matter).


Tell it to free[0] the array directly.

```D
T[] array;

GC.free(array.ptr);
```

Normally I would suggest to create your own buffer, but 
because of the DB library probably doesn't support that, no 
point trying to force it.


[0] http://dlang.org/phobos/core_memory.html#.GC.free


BTW, how would I do this with associative arrays?


you can't. you can `.clear` 'em, though, but this won't 
immediately free the internal storage.


Meh.. Okay, well using these two should make things a lot better. 
Thanks for the help and clearing up my confusion.

How to make AA key a pointer

[Clinton via Digitalmars-d-learn]

Hi all, I need advice from better developers on this concern.

I'm using an AA to reference another array for quicker access:

[code]
alias contactId = string;
bool[contactId][] matches;
ulong[contactId] idsToMatches;

bool[string] matchesForId(string id) {
  return matches.get(idsToMatches[id], bool[string].init);
}
[/code]

Just wondering, how do I set the keys to avoid copying the id 
string? So, let's say ids come from another array of structs(e.g. 
Contact[]). I want to avoid having two copies of the string id 
value in both of these AAs above (to avoid using extra memory 
usage).


The reason this is a potential issue is because these arrays can 
get extremely large. This is to match duplicate contacts and 
group them in matches. The reason I use bool is because it's the 
smallest size type and I don't think I can use void.


I guess my question is: does dmd already create pointers to the 
id from the AA, or is each new key a new allocation?

Re: How to make AA key a pointer

[Clinton via Digitalmars-d-learn]

On Monday, 19 February 2018 at 14:55:01 UTC, Clinton wrote:

Hi all, I need advice from better developers on this concern.

I'm using an AA to reference another array for quicker access:

[...]


Sorry, on second look my explanation isn't very clear. I want to 
know if:


bool[string] myAA;

myAA[contact.id] = true; // Does this copy contact.id or is this 
a pointer to contact.id?

Re: How to make AA key a pointer

[Clinton via Digitalmars-d-learn]

On Monday, 19 February 2018 at 15:02:29 UTC, ketmar wrote:

Clinton wrote:


On Monday, 19 February 2018 at 14:55:01 UTC, Clinton wrote:

[...]


Sorry, on second look my explanation isn't very clear. I want 
to know if:


bool[string] myAA;

myAA[contact.id] = true; // Does this copy contact.id or is 
this a pointer to contact.id?


there is absolutely no reason to copy `string` ever, as it is 
`immutable`. and compiler knows that. anyway, why don't you 
just check it by writing the code first?


import std.stdio;
void main () {
int[string] a;
string s = "test";
writefln("%08x", s.ptr);
a[s] = 666;
s = "test1";
writefln("%08x", s.ptr);
a[s] = 42;
foreach (string k; a.byKey) writefln("%08x", k.ptr);
}


Thanks. I actually did a similar test a little while ago and 
found it out. Thanks for confirming. I still struggle a bit with 
these basic things.