Re: DUB: Only fetch and cache packages in dub.json without running build
On Tuesday, 10 April 2018 at 15:31:41 UTC, John Colvin wrote: On Tuesday, 10 April 2018 at 13:50:38 UTC, Clinton wrote: [...] As far as I understand it, `dub describe` fetches everything. Then you can cache `~/.dub/packages/`. Alternatively you can do `dub describe --cache=local` to put the packages in the current directory. You could then use `dub add-path .` or add `--cache=local` to all future calls to use those locally fetched packages. Even better: % mkdir cache % cd cache % dub describe --root=../ --cache=local and then either % dub build --root=../ --cache=local or % dub add-path . % cd ../ % dub build which keeps things nice and clean Just tried this on my CircleCI setup. Works perfectly! Thanks everyone!
Re: DUB: Only fetch and cache packages in dub.json without running build
On Tuesday, 10 April 2018 at 15:31:41 UTC, John Colvin wrote: On Tuesday, 10 April 2018 at 13:50:38 UTC, Clinton wrote: [...] As far as I understand it, `dub describe` fetches everything. Then you can cache `~/.dub/packages/`. Alternatively you can do `dub describe --cache=local` to put the packages in the current directory. You could then use `dub add-path .` or add `--cache=local` to all future calls to use those locally fetched packages. Even better: % mkdir cache % cd cache % dub describe --root=../ --cache=local and then either % dub build --root=../ --cache=local or % dub add-path . % cd ../ % dub build which keeps things nice and clean Wow, this is helpful! I thought describe only showed which packages were currently cached or just output based on the dub.json. I'm going to try this out.
Re: DUB: Only fetch and cache packages in dub.json without running build
On Tuesday, 10 April 2018 at 13:50:38 UTC, Clinton wrote: Hi all, I'm setting up a CircleCI config for my project. Right now I'm trying to cache dependencies before running builds. This way I can run "dub build --nodeps" immediately after the packages are cached to avoid extra network calls and speed it up. [...] Thanks. That's a good idea. It would be great to have "dub fetch --all" or something like that to fetch all packages from the project dub.json.
DUB: Only fetch and cache packages in dub.json without running build
Hi all, I'm setting up a CircleCI config for my project. Right now I'm trying to cache dependencies before running builds. This way I can run "dub build --nodeps" immediately after the packages are cached to avoid extra network calls and speed it up. I'm wondering if there's a way to isolate the part that downloads and caches all of the dependencies in dub.json without running the build. Right now I have to run the build twice. First one, for the sake of downloading the dependencies and later for running if the cache exists. Having the build run the first time takes up a lot of time. I'm going for the "npm install" type of effect. The reason the cache needs to be rebuilt each time is because CircleCI runs docker images. Once the deployment is finished, the image fs is destroyed so there's no way to hold onto the cache for future builds.
Re: How to make AA key a pointer
On Monday, 19 February 2018 at 15:02:29 UTC, ketmar wrote:
Clinton wrote:
On Monday, 19 February 2018 at 14:55:01 UTC, Clinton wrote:
[...]
Sorry, on second look my explanation isn't very clear. I want
to know if:
bool[string] myAA;
myAA[contact.id] = true; // Does this copy contact.id or is
this a pointer to contact.id?
there is absolutely no reason to copy `string` ever, as it is
`immutable`. and compiler knows that. anyway, why don't you
just check it by writing the code first?
import std.stdio;
void main () {
int[string] a;
string s = "test";
writefln("%08x", s.ptr);
a[s] = 666;
s = "test1";
writefln("%08x", s.ptr);
a[s] = 42;
foreach (string k; a.byKey) writefln("%08x", k.ptr);
}
Thanks. I actually did a similar test a little while ago and
found it out. Thanks for confirming. I still struggle a bit with
these basic things.
Re: How to make AA key a pointer
On Monday, 19 February 2018 at 14:55:01 UTC, Clinton wrote: Hi all, I need advice from better developers on this concern. I'm using an AA to reference another array for quicker access: [...] Sorry, on second look my explanation isn't very clear. I want to know if: bool[string] myAA; myAA[contact.id] = true; // Does this copy contact.id or is this a pointer to contact.id?
How to make AA key a pointer
Hi all, I need advice from better developers on this concern.
I'm using an AA to reference another array for quicker access:
[code]
alias contactId = string;
bool[contactId][] matches;
ulong[contactId] idsToMatches;
bool[string] matchesForId(string id) {
return matches.get(idsToMatches[id], bool[string].init);
}
[/code]
Just wondering, how do I set the keys to avoid copying the id
string? So, let's say ids come from another array of structs(e.g.
Contact[]). I want to avoid having two copies of the string id
value in both of these AAs above (to avoid using extra memory
usage).
The reason this is a potential issue is because these arrays can
get extremely large. This is to match duplicate contacts and
group them in matches. The reason I use bool is because it's the
smallest size type and I don't think I can use void.
I guess my question is: does dmd already create pointers to the
id from the AA, or is each new key a new allocation?
Looking for part time Dlang dev
Hi all, I fell in love with Dlang a few years back and built my technology product with it. Now I'm slowly moving away from being the only dev and looking for some Dlang developers. Looking for some devs that are fine with starting very gradually, doing some part time contracts and later learning more about the entire platform I built. Doesn't matter where you live. Anyone interested, please send me a msg at clinton.d.skakun --at-- dedupe.ly so we can chat.
Re: Cleaning/Releasing large arrays from memory
On Monday, 24 July 2017 at 18:01:19 UTC, ketmar wrote: Clinton wrote: On Monday, 24 July 2017 at 14:51:04 UTC, rikki cattermole wrote: Stuff the GC. You don't need it to care about collecting (or destroying for that matter). Tell it to free[0] the array directly. ```D T[] array; GC.free(array.ptr); ``` Normally I would suggest to create your own buffer, but because of the DB library probably doesn't support that, no point trying to force it. [0] http://dlang.org/phobos/core_memory.html#.GC.free BTW, how would I do this with associative arrays? you can't. you can `.clear` 'em, though, but this won't immediately free the internal storage. Meh.. Okay, well using these two should make things a lot better. Thanks for the help and clearing up my confusion.
Re: Cleaning/Releasing large arrays from memory
On Monday, 24 July 2017 at 14:51:04 UTC, rikki cattermole wrote: Stuff the GC. You don't need it to care about collecting (or destroying for that matter). Tell it to free[0] the array directly. ```D T[] array; GC.free(array.ptr); ``` Normally I would suggest to create your own buffer, but because of the DB library probably doesn't support that, no point trying to force it. [0] http://dlang.org/phobos/core_memory.html#.GC.free BTW, how would I do this with associative arrays?
Re: Cleaning/Releasing large arrays from memory
On Monday, 24 July 2017 at 14:51:04 UTC, rikki cattermole wrote: Stuff the GC. You don't need it to care about collecting (or destroying for that matter). Tell it to free[0] the array directly. ```D T[] array; GC.free(array.ptr); ``` Normally I would suggest to create your own buffer, but because of the DB library probably doesn't support that, no point trying to force it. [0] http://dlang.org/phobos/core_memory.html#.GC.free Thanks! I didn't know I could do that with the GC enabled.
Cleaning/Releasing large arrays from memory
Hi guys, I have a question on how to free large arrays in D after
they're no longer needed.
Let's say I have this:
SomeKey[] getKeys() {
SomeKey[] n;
foreach(batching stuff...) {
SomeKey newstuff = db.select!(SomeKey[])(...); // gets around
6M of these from db
redis.send("newItems", newStuff.length);
n ~= newStuff;
destroy(newStuff);
}
return n;
}
void main() {
SomeKey[] myarray = getKeys();
ulong[string] mappedValues;
foreach(i, item; myArray) {
mappedValues[item.name] = i;
}
// No longer need myarray, and we need the memory back
destory(myarray);
GC.collect;
// Another memory intensive operation below
}
This seems to work for me to a point, but I notice my program
still holds a lot in memory even after certain memory intensive
arrays should have been collected.
Is this just a hit and miss with the GC or am I doing something
wrong? Uses around 40% less memory forcing the GC and cleaning
arrays but seems like a lot is left over, especially for threads
that have already finished(I've expect a thread to free all it's
memory). I'm not sharing anything across threads besides a config
struct.
This is a simplified version of my app. At the moment when at
around 9GB at it's peak instead of 19GB like before.
