Re: multithread/concurrency/parallel methods and performance
On Monday, 19 February 2018 at 14:57:22 UTC, SrMordred wrote: On Monday, 19 February 2018 at 05:54:53 UTC, Dmitry Olshansky wrote: The operation is trivial and dataset is rather small. In such cases SIMD with eg array ops is the way to go: result[] = values[] * values2[]; Yes, absolutely right :) I make a simple example to understand why the threads are not scaling in the way i thought they would. Yeah, the world is ugly place where trivial math sometimes doesn’t work. I suggest to: - run with different number of threads from 1 to n - vary sizes from 100k to 10m For your numbers - 400ms / 64 is ~ 6ms, if we divide by # cores it’s 6/7 ~ 0.86ms which is a deal smaller then a CPU timeslice. In essence a single core runs fast b/c it doesn’t wait for all others to complete via join easily burning its quota in one go. In MT I bet some of overhead comes from not all threads finishing (and starting) at once, so the join block in the kernel. You could run your MT code with strace to see if it hits the futex call or some such, if it does that’s where you are getting delays. (that’s assuming you are on Linux) std.parallel version is a bit faster b/c I think it caches created threadpool so you don’t start threads anew on each run. I imagine that, if one core work is done in 200ms a 4 core work will be done in 50ms, plus some overhead, since they are working on separate block of memory, without need of sync, and without false sharing, etc (at least I think i don´t have this problem here). If you had a long queue of small tasks like that and you don’t wait to join all threads untill absolutely required you get near perfect scalability. (Unless hitting other bottlenecks like RAM).
Re: Building from source on FreeBSD
On Monday, 19 February 2018 at 12:01:31 UTC, Jonathan M Davis wrote: I've never been able to figure this problem out, so I've I always just edited the makefile so that the two programs in CURL_TOOLS aren't built. IIRC, if you use -v to see the full linker command, it's even actually linking to the curl library that's there but somehow not succeeding. It's all quite weird. It's been a while since I dug into it though. - Jonathan M Davis ok... so I decided to dig into it a little further. seems the problem relates to a single line, in dget.d pragma(lib, "curl"); I just commented out the one line in dget.d, and it all seems to build ok (including dget). So no need to comment out tools in posix.mak anymore.
Re: std.traits.isBoolean
On Monday, 19 February 2018 at 17:22:04 UTC, Nathan S. wrote: On Monday, 19 February 2018 at 15:12:15 UTC, Tony wrote: But, assuming there is a use case for it, what if you want to restrict to a type that is either boolean, or a struct/class that can substitute for boolean - how do you do that without using the "private" BooleanTypeOf thing? In that case you can just write `is(T : bool)`. Thanks. Assuming it would substitute, that should probably be used on this page in place of BooleanTypeOf since BooleanTypeOf is not supposed to be public: https://dlang.org/library/std/traits/is_boolean.html "enum isBoolean(T) = is(BooleanTypeOf!T) && !isAggregateType!T;"
Re: Could that bug be catch using D's way?
On 02/19/2018 05:33 AM, rikki cattermole wrote: https://dlang.org/phobos/std_experimental_checkedint.html#.Checked.min Accompanying presentations: DConf 2017: https://www.youtube.com/watch?v=29h6jGtZD-U Google Tel Aviv: https://www.youtube.com/watch?v=es6U7WAlKpQ Andrei likes the second one more. Ali
Re: Could that bug be catch using D's way?
On Monday, 19 February 2018 at 21:34:04 UTC, Simen Kjærås wrote: On Monday, 19 February 2018 at 14:20:16 UTC, Basile B. wrote: I had never used Checked and i discover that strangely there's no hook for opAssign. onLowerBound and onUpperBound works for +=, -=, *=, /=, %=, ^^=, &=, |=, ^=, <<=, >>=, and >>>=. But since init is 0, += works: Ah, thanks. Filed a bug: https://issues.dlang.org/show_bug.cgi?id=18471 -- Simen I think this is a design choice, it's too obvious to be an omission (it's even documented that opAssign is like the ctor). I'm curious to know what's the rationale though.
Re: Could that bug be catch using D's way?
On Monday, 19 February 2018 at 14:20:16 UTC, Basile B. wrote: I had never used Checked and i discover that strangely there's no hook for opAssign. onLowerBound and onUpperBound works for +=, -=, *=, /=, %=, ^^=, &=, |=, ^=, <<=, >>=, and >>>=. But since init is 0, += works: Ah, thanks. Filed a bug: https://issues.dlang.org/show_bug.cgi?id=18471 -- Simen
Re: Building from source on FreeBSD
On Monday, February 19, 2018 14:41:21 Seb via Digitalmars-d-learn wrote: > On Monday, 19 February 2018 at 12:01:31 UTC, Jonathan M Davis > > wrote: > > On Monday, February 19, 2018 11:43:26 psychoticRabbit via > > > > Digitalmars-d- learn wrote: > >> [...] > > > > I've never been able to figure this problem out, so I've I > > always just edited the makefile so that the two programs in > > CURL_TOOLS aren't built. IIRC, if you use -v to see the full > > linker command, it's even actually linking to the curl library > > that's there but somehow not succeeding. It's all quite weird. > > It's been a while since I dug into it though. > > > > - Jonathan M Davis > > FYI: https://github.com/dlang/tools/pull/316 It looks like that might actually fix the tools build for FreeBSD on my machine if the bit that he currently has versioned for DragonFly is used for FreeBSD as well (and it probably needs to be there for every BSD). I've never understood why the tools repo builds fine for releases but has never built properly for me locally. - Jonathan M Davis
Re: std.traits.isBoolean
On Monday, 19 February 2018 at 15:12:15 UTC, Tony wrote: But, assuming there is a use case for it, what if you want to restrict to a type that is either boolean, or a struct/class that can substitute for boolean - how do you do that without using the "private" TypeOfBoolean thing? In that case you can just write `is(T : bool)`.
Re: std.traits.isBoolean
On Monday, 19 February 2018 at 13:47:15 UTC, Basile B. wrote: Indeed but Phobos maintainers don't want the ...TypeOf family to be documented. (https://github.com/dlang/phobos/pull/5747) Ok, thanks. But, assuming there is a use case for it, what if you want to restrict to a type that is either boolean, or a struct/class that can substitute for boolean - how do you do that without using the "private" TypeOfBoolean thing?
Re: How to make AA key a pointer
Clinton wrote:
On Monday, 19 February 2018 at 14:55:01 UTC, Clinton wrote:
Hi all, I need advice from better developers on this concern.
I'm using an AA to reference another array for quicker access:
[...]
Sorry, on second look my explanation isn't very clear. I want to know if:
bool[string] myAA;
myAA[contact.id] = true; // Does this copy contact.id or is this a
pointer to contact.id?
there is absolutely no reason to copy `string` ever, as it is `immutable`.
and compiler knows that. anyway, why don't you just check it by writing the
code first?
import std.stdio;
void main () {
int[string] a;
string s = "test";
writefln("%08x", s.ptr);
a[s] = 666;
s = "test1";
writefln("%08x", s.ptr);
a[s] = 42;
foreach (string k; a.byKey) writefln("%08x", k.ptr);
}
Re: How to make AA key a pointer
On Monday, 19 February 2018 at 14:57:47 UTC, Clinton wrote: On Monday, 19 February 2018 at 14:55:01 UTC, Clinton wrote: Hi all, I need advice from better developers on this concern. I'm using an AA to reference another array for quicker access: [...] Sorry, on second look my explanation isn't very clear. I want to know if: bool[string] myAA; myAA[contact.id] = true; // Does this copy contact.id or is this a pointer to contact.id? It's a pointer. In D, string is an alias to immutable(char)[] (Slice of immutable characters). A slice is a combination of pointer and length.
Re: How to make AA key a pointer
On Monday, 19 February 2018 at 14:55:01 UTC, Clinton wrote: Hi all, I need advice from better developers on this concern. I'm using an AA to reference another array for quicker access: [...] Sorry, on second look my explanation isn't very clear. I want to know if: bool[string] myAA; myAA[contact.id] = true; // Does this copy contact.id or is this a pointer to contact.id?
Re: multithread/concurrency/parallel methods and performance
On Monday, 19 February 2018 at 05:54:53 UTC, Dmitry Olshansky wrote: The operation is trivial and dataset is rather small. In such cases SIMD with eg array ops is the way to go: result[] = values[] * values2[]; Yes, absolutely right :) I make a simple example to understand why the threads are not scaling in the way i thought they would. I imagine that, if one core work is done in 200ms a 4 core work will be done in 50ms, plus some overhead, since they are working on separate block of memory, without need of sync, and without false sharing, etc (at least I think i don´t have this problem here).
Re: multithread/concurrency/parallel methods and performance
On Monday, 19 February 2018 at 05:49:54 UTC, Nicholas Wilson wrote: As SIZE=1024*1024 (i.e. not much, possibly well within L2 cache for 32bit) it may be that dealing with the concurrency overhead adds a significant amount of overhead. That 'concurrency overhead' is what i´m not getting. Since the array is big, dividing it into 6, 7 cores will not trash L1 since they are very far from each other, right? Or L2 cache trashing is also a problem in this case? _base : 150 ms, 728 μs, and 5 hnsecs _parallel : 120 ms, 78 μs, and 5 hnsecs _concurrency : 134 ms, 787 μs, and 4 hnsecs _thread : 129 ms, 476 μs, and 2 hnsecs Yes, on my PC I was using -release. Yet, 150ms for 1 core. 120-134ms of X cores. Shouldn´t be way faster? I´m trying to understand where the overhead is, and if is possible to get rid of it (perfect thread scaling).
How to make AA key a pointer
Hi all, I need advice from better developers on this concern.
I'm using an AA to reference another array for quicker access:
[code]
alias contactId = string;
bool[contactId][] matches;
ulong[contactId] idsToMatches;
bool[string] matchesForId(string id) {
return matches.get(idsToMatches[id], bool[string].init);
}
[/code]
Just wondering, how do I set the keys to avoid copying the id
string? So, let's say ids come from another array of structs(e.g.
Contact[]). I want to avoid having two copies of the string id
value in both of these AAs above (to avoid using extra memory
usage).
The reason this is a potential issue is because these arrays can
get extremely large. This is to match duplicate contacts and
group them in matches. The reason I use bool is because it's the
smallest size type and I don't think I can use void.
I guess my question is: does dmd already create pointers to the
id from the AA, or is each new key a new allocation?
Re: Building from source on FreeBSD
On Monday, 19 February 2018 at 12:01:31 UTC, Jonathan M Davis wrote: On Monday, February 19, 2018 11:43:26 psychoticRabbit via Digitalmars-d- learn wrote: [...] I've never been able to figure this problem out, so I've I always just edited the makefile so that the two programs in CURL_TOOLS aren't built. IIRC, if you use -v to see the full linker command, it's even actually linking to the curl library that's there but somehow not succeeding. It's all quite weird. It's been a while since I dug into it though. - Jonathan M Davis FYI: https://github.com/dlang/tools/pull/316
Re: Could that bug be catch using D's way?
On Monday, 19 February 2018 at 13:51:50 UTC, Simen Kjærås wrote:
On Monday, 19 February 2018 at 13:33:34 UTC, rikki cattermole
wrote:
https://dlang.org/phobos/std_experimental_checkedint.html#.Checked.min
Can't seem to get that to work, so I assumed it's not meant to
be used that way:
import std.experimental.checkedint;
struct MyHook {
enum min(T) = 3;
enum max(T) = 15;
static B onLowerBound(T, B)(T value, B bound)
{
assert(0);
}
static B onUpperBound(T, B)(T value, B bound)
{
assert(0);
}
}
unittest
{
Checked!(int, MyHook) a;
a = 22;
assert(a != 22); // This assert triggers, not the others.
}
--
Simen
I had never used Checked and i discover that strangely there's no
hook for opAssign. onLowerBound and onUpperBound works for +=,
-=, *=, /=, %=, ^^=, &=, |=, ^=, <<=, >>=, and >>>=. But since
init is 0, += works:
struct MyHook {
enum min(T) = 3;
enum max(T) = 15;
static B onLowerBound(T, B)(T value, B bound)
{
assert(0);
}
static B onUpperBound(T, B)(T value, B bound)
{
assert(0);
}
}
unittest
{
Checked!(int, MyHook) a;
a += 16; // triggers the onUpperBound assert
}
Re: Could that bug be catch using D's way?
On Monday, 19 February 2018 at 13:33:34 UTC, rikki cattermole
wrote:
https://dlang.org/phobos/std_experimental_checkedint.html#.Checked.min
Can't seem to get that to work, so I assumed it's not meant to be
used that way:
import std.experimental.checkedint;
struct MyHook {
enum min(T) = 3;
enum max(T) = 15;
static B onLowerBound(T, B)(T value, B bound)
{
assert(0);
}
static B onUpperBound(T, B)(T value, B bound)
{
assert(0);
}
}
unittest
{
Checked!(int, MyHook) a;
a = 22;
assert(a != 22); // This assert triggers, not the others.
}
--
Simen
Re: std.traits.isBoolean
On Monday, 19 February 2018 at 13:07:08 UTC, Tony wrote: It doesn't appear that BooleanTypeof is documented on dlang.org (outside of it's placement on the isBooleanType page). At least it isn't coming up in a "BooleanTypeOf site:dlang.org" search and not on the traits page: https://dlang.org/library/std/traits.html Indeed but Phobos maintainers don't want the ...TypeOf family to be documented. (https://github.com/dlang/phobos/pull/5747)
Re: Could that bug be catch using D's way?
On 19/02/2018 1:24 PM, Simen Kjærås wrote:
On Monday, 19 February 2018 at 12:58:45 UTC, Marc wrote:
I'm pretty sure something could be done with Ada's type range but what
we could do using D?
We can easily define a range type in D. The simple example below
probably has awful performance and many holes, but outlines the basic
idea. It will not have compile-time bounds checks, but can be used as an
int when it needs to, and enforces at run-time that the value is within
the specified bounds.
import std.format : format;
import std.exception : enforce;
struct Range(T, T min, T max, T defaultValue = 0)
if (defaultValue >= min && defaultValue <= max)
{
private T payload;
this(T value)
{
this = value;
}
T get() { return payload; }
alias get this;
Range opBinary(string op)(T value)
{
return Range(mixin("payload "~op~" value"));
}
ref Range opOpAssing(string op)(T value)
{
this = Range(mixin("payload "~op~" value"));
return this;
}
ref Range opAssign(T value, string file = __FILE__, int line =
__LINE__)
{
enforce(value <= max && value >= min, format("Value needs to be
between %s and %s, not %s.", min, max, value), file, line);
payload = value;
return this;
}
}
unittest
{
Range!(int, 3, 15, 3) a = 4;
a = 16; // foo.d(38): Value needs to be between 3 and 15, not 16.
int n = a;
}
It currently does no sensible testing of what operators it allows, and
does not support unary operators. Other potential improvements include
merging of bounds when the other operand is a range as well (so
Range!(int, 3, 6) + Range!(int, 2, 8) returns a Range!(int, 5, 14)).
Perhaps a more sensible default value would be min(maxValue, max(0,
minValue)).
--
Simen
https://dlang.org/phobos/std_experimental_checkedint.html#.Checked.min
Re: Could that bug be catch using D's way?
On Monday, 19 February 2018 at 12:58:45 UTC, Marc wrote:
I'm pretty sure something could be done with Ada's type range
but what we could do using D?
We can easily define a range type in D. The simple example below
probably has awful performance and many holes, but outlines the
basic idea. It will not have compile-time bounds checks, but can
be used as an int when it needs to, and enforces at run-time that
the value is within the specified bounds.
import std.format : format;
import std.exception : enforce;
struct Range(T, T min, T max, T defaultValue = 0)
if (defaultValue >= min && defaultValue <= max)
{
private T payload;
this(T value)
{
this = value;
}
T get() { return payload; }
alias get this;
Range opBinary(string op)(T value)
{
return Range(mixin("payload "~op~" value"));
}
ref Range opOpAssing(string op)(T value)
{
this = Range(mixin("payload "~op~" value"));
return this;
}
ref Range opAssign(T value, string file = __FILE__, int line
= __LINE__)
{
enforce(value <= max && value >= min, format("Value needs
to be between %s and %s, not %s.", min, max, value), file, line);
payload = value;
return this;
}
}
unittest
{
Range!(int, 3, 15, 3) a = 4;
a = 16; // foo.d(38): Value needs to be between 3 and 15, not
16.
int n = a;
}
It currently does no sensible testing of what operators it
allows, and does not support unary operators. Other potential
improvements include merging of bounds when the other operand is
a range as well (so Range!(int, 3, 6) + Range!(int, 2, 8) returns
a Range!(int, 5, 14)). Perhaps a more sensible default value
would be min(maxValue, max(0, minValue)).
--
Simen
Re: std.traits.isBoolean
On Sunday, 18 February 2018 at 15:12:50 UTC, Mike Parker wrote:
Generally, no. But with alias this, it can be:
=
import std.traits : BooleanTypeOf;
import std.stdio : writeln;
struct NoBool {
int x;
}
struct AliasThisBool {
bool b;
alias b this;
}
void main()
{
static if(is(BooleanTypeOf!NoBool)) writeln("NoBool");
static if(is(BooleanTypeOf!AliasThisBool))
writeln("AliasThisBool");
}
Thanks!
It doesn't appear that BooleanTypeof is documented on dlang.org
(outside of it's placement on the isBooleanType page). At least
it isn't coming up in a "BooleanTypeOf site:dlang.org" search and
not on the traits page:
https://dlang.org/library/std/traits.html
Could that bug be catch using D's way?
I'm pretty sure something could be done with Ada's type range but what we could do using D?
Re: Building from source on FreeBSD
On Monday, 19 February 2018 at 12:01:31 UTC, Jonathan M Davis wrote: I've never been able to figure this problem out, so I've I always just edited the makefile so that the two programs in CURL_TOOLS aren't built. IIRC, if you use -v to see the full linker command, it's even actually linking to the curl library that's there but somehow not succeeding. It's all quite weird. It's been a while since I dug into it though. - Jonathan M Davis Thanks for the quick response. So I took your advice and just commented out (as per below), and now it all builds fine - AFAICT ;-) -FROM 61:all: $(TOOLS) $(CURL_TOOLS) $(ROOT)/dustmite 76:$(TOOLS) $(DOC_TOOLS) $(CURL_TOOLS) $(TEST_TOOLS): $(ROOT)/%: %.d -TO 61:all: $(TOOLS) $(ROOT)/dustmite 76:$(TOOLS) $(DOC_TOOLS) $(TEST_TOOLS): $(ROOT)/%: %.d
Re: Building from source on FreeBSD
On Monday, February 19, 2018 11:43:26 psychoticRabbit via Digitalmars-d- learn wrote: > So I finally got around to building from source. > > I have builds working just fine on a variety of linux machines, > it's just a FreeBSD problem I'm having. > > So, on FreeBSD, I can build the dmd directory, and the phobos > directory ok. > > When building tools directory, most tools get built ok, but the > build process stops during something to do with curl.. > > > > > gmake -f posix.mak -j8 (yeah on FreeBSD use gmake not make) > > . > .. > ... (bunch of stuff builds ok.. then this next line) > /usr/bin/ld: cannot find -lcurl > cc: error: linker command failed with exit code 1 (use -v to see > invocation) > Error: linker exited with status 1 > gmake: *** [posix.mak:77: generated/freebsd/64/dget] Error 1 > gmake: *** Waiting for unfinished jobs > > > Yes, I do have the FreeBSD curl package installed - curl-7.56.0 > > Yes, curl is in the path> which curl -> /usr/local/bin/curl > > So I can only assume my FreeBSD machine is missing some curl > related library/package?? > > Any help would be appreciated. I've never been able to figure this problem out, so I've I always just edited the makefile so that the two programs in CURL_TOOLS aren't built. IIRC, if you use -v to see the full linker command, it's even actually linking to the curl library that's there but somehow not succeeding. It's all quite weird. It's been a while since I dug into it though. - Jonathan M Davis
Building from source on FreeBSD
So I finally got around to building from source. I have builds working just fine on a variety of linux machines, it's just a FreeBSD problem I'm having. So, on FreeBSD, I can build the dmd directory, and the phobos directory ok. When building tools directory, most tools get built ok, but the build process stops during something to do with curl.. gmake -f posix.mak -j8 (yeah on FreeBSD use gmake not make) . .. ... (bunch of stuff builds ok.. then this next line) /usr/bin/ld: cannot find -lcurl cc: error: linker command failed with exit code 1 (use -v to see invocation) Error: linker exited with status 1 gmake: *** [posix.mak:77: generated/freebsd/64/dget] Error 1 gmake: *** Waiting for unfinished jobs Yes, I do have the FreeBSD curl package installed - curl-7.56.0 Yes, curl is in the path> which curl -> /usr/local/bin/curl So I can only assume my FreeBSD machine is missing some curl related library/package?? Any help would be appreciated.
Re: Size threshold replace complex probing with linear search for small hash tables
On Monday, 19 February 2018 at 10:22:12 UTC, Nordlöw wrote: I'm currently developing a combined HashMap and HashSet with open addressing You might want to consider using Robin Hood hashing to reduce the worst-case length of collision chains, regardless of what kind of probing scheme you use.
Size threshold replace complex probing with linear search for small hash tables
I'm currently developing a combined HashMap and HashSet with open addressing at https://github.com/nordlow/phobos-next/blob/master/src/open_hashmap_or_hashset.d with probing using steps of triangular numbers when length is a power of two at https://github.com/nordlow/phobos-next/blob/master/src/probing.d I've read that for small tables (where the size of the entire array Element[] is smaller than a certain threshold) a linear search is usually faster. Is this threshold somehow related to the sizes of cache-line? Suggestions for compile-time or run-time logic that decides when to use linear search are very welcome! My current proposal is to use linear search when ElementType.sizeof*length <= byte-size of a cache-line.
Re: std.array.array for immutable data types
On Monday, February 19, 2018 07:25:07 Nicholas Wilson via Digitalmars-d- learn wrote: > On Monday, 19 February 2018 at 07:08:49 UTC, Fra Mecca wrote: > > Is there a way to avoid using to! conversion here? > > > > immutable string[] dst = to!(immutable > > string[])(array(pipe.readEnd.byLineCopy)); > > assumeUnique. > > immutable string[] dst = > pipe.readEnd.byLineCopy.array.assumeUnique; Arguably, to!(immutable string[]) should work on the result of byLineCopy, but it doesn't seem to. If it did, then that would eliminate the need for assumeUnique, and it's always better to avoid assumeUnique if you can. However, given the current limitation with to and byLineCopy, it probably is the best solution. - Jonathan M Davis
Re: Trying to forward unwrapped opDispatch names to alias this
On Monday, 19 February 2018 at 01:00:23 UTC, Adam D. Ruppe wrote:
On Monday, 19 February 2018 at 00:42:05 UTC, aliak wrote:
struct B(T) {
T t;
A a;
alias a this;
auto opDispatch(string name)() if (hasMember!(T, name)) {
return mixin("t." ~ name);
Did you perhaps mean `A` instead of `T` here? cuz in your code
T is int, not the struct.
I don't think I did :p
T is the wrapped type. So if T has a member (in the example it's
the built in field "max") then forward that. If member not there
then I figured the alias this would be used. I.e. in the example
b.p should call A.p.
I assume this should work because rules for alias this (as I
understand) are to basically try if there's a member name that
resolves the call, else forward to alias this.
