Re: What is the memory usage of my app?

[Adil via Digitalmars-d-learn]

On Thursday, 16 April 2015 at 17:13:25 UTC, Laeeth Isharc wrote:
Fwiw, I have been working on something similar.  Others will 
have more experience on the GC, but perhaps you might find this 
interesting.

For CSV files, what I found is that parsing is quite slow (and 
memory intensive).  So rather than parse the same data every 
time, I found it helpful to do so once in a batch that runs on 
a cron job, and write out to msgpack format.

I am not a GC expert, but what happens if you run GC.collect() 
once you are done parsing?

auto loadGiltPrices()
{
        auto data=cast(ubyte[])std.file.read("/hist/msgpack/dmo.pack");
        return cast(immutable)data.unpack!(GiltPriceFromDMO[][string]);
}

struct GiltPriceFromDMO
{
        string name;
        string ISIN;
        KPDateTime redemptionDate;
        KPDateTime closeDate;
        int indexLag;
        double cleanPrice;
        double dirtyPrice;
        double accrued;
        double yield;
        double modifiedDuration;
}

void main(string[] args)
{
        auto gilts=readCSVDMO();
        ubyte[] data=pack(gilts);
        std.file.write("dmo.pack",data);
        writefln("* done");
        data=cast(ubyte[])std.file.read("dmo.pack");
}

On Thursday, 16 April 2015 at 12:17:24 UTC, Adil wrote:
I've written a simple socket-server app that securities (stock
market shares) data and allows clients to query over them. The
app starts by loading instrument information from a CSV file 
into
some structs, then listens on a socket responding to queries. 
It
doesn't mutate the data or allocate anything substantial.

There are 2 main structs in the app. One stores security data,
and the other groups together securities. They are defined as
follows :

````
__gshared Securities securities;

struct Security
{
         string RIC;
         string TRBC;
         string[string] fields;
         double[string] doubles;

         @nogc @property pure size_t bytes()
         {
             size_t bytes;

             bytes = RIC.sizeof + RIC.length;
             bytes += TRBC.sizeof + TRBC.length;

             foreach(k,v; fields) {
                 bytes += (k.sizeof + k.length + v.sizeof +
v.length);
             }

             foreach(k, v; doubles) {
                 bytes += (k.sizeof + k.length + v.sizeof);
             }

             return bytes + Security.sizeof;
         }
}

struct Securities
{
         Security[] securities;
         private size_t[string] rics;

         // Store offsets for each TRBC group
         ulong[2][string] econSect;
         ulong[2][string] busSect;
         ulong[2][string] IndGrp;
         ulong[2][string] Ind;

         @nogc @property pure size_t bytes()
         {
             size_t bytes;

             foreach(Security s; securities) {
                 bytes += s.sizeof + s.bytes;
             }

             foreach(k, v; rics) {
                 bytes += k.sizeof + k.length + v.sizeof;
             }

             foreach(k, v; econSect) {
                 bytes += k.sizeof + k.length + v.sizeof;
             }

             foreach(k, v; busSect) {
                 bytes += k.sizeof + k.length + v.sizeof;
             }

             foreach(k, v; IndGrp) {
                 bytes += k.sizeof + k.length + v.sizeof;
             }

             foreach(k, v; Ind) {
                 bytes += k.sizeof + k.length + v.sizeof;
             }

             return bytes + Securities.sizeof;
         }
}
````

Calling Securities.bytes shows "188 MB", but "ps" shows about 
591
MB of Resident memory. Where is the memory usage coming from?
What am i missing?

Laeeth,

GC.collect() made no difference. It seems the memory is being 
held by the data structures above. I think i may not be 
accounting for hash table usage properly, or it could be 
something else.

I only need to work with interday data for now, so the CSV load 
speed doesn't bother me atm. Great idea on using the mmfile w 
msgpack! I will try that out.

Adil