Re: binding to C++
On Friday, 26 May 2017 at 15:17:08 UTC, drug wrote: Trying to bind to cpp code I stop at some moment having undefined reference to some cpp function. But objdump -Ct cpplibrary.so shows me that this cpp function exists in the library. linker message about cpp function is _identical_ to objdump message so I don't know where is the difference. For example linker says: `undefined reference to `Namespace::StructName::method_name(Namespace::Otherstruct const*)` And ojbdump called by command: ``` objdump -Ct libLibrary.so | method_name ``` shows the following: ``` 00026ed0 wF .text 0025 Namespace::StructName::method_name(Namespace::Otherstruct const*) ``` That is two outputs are identical. Could someone help what to do to investigate the problem? Thanks in advance If you were compiling C++ code, you could get such error if method_name was declared const in the library, and you tried to link it as non-const. == Library == struct StructName { void method_name(...) const { ... } }; == Your Code == struct StructName { void method_name(...); /* NOTE: no const here */ }; Don't know if that is the issue in your case.
Re: purity question
On Monday, 29 May 2017 at 01:36:24 UTC, Jonathan M Davis wrote: A simple example: anything that has a malloc/free pair. Yeah, if you do it right, you should be fine, but you have to do it right, and it's very easy to miss some detail that makes it wrong to insist to the compiler that what you're doing is pure. If malloc were marked as pure, wouldn't that mean it must return the same pointer every time you call it with the same size?
Error: func(const(A) a) is not callable using argument types (const(A)
What does that even mean? Scenario: bool func(const ImVec2 label_size) { return true; } void main() { //first attempt: const ImVec2 label_size = CalcTextSize(label.ptr, null, true); //Error: cannot implicitly convert expression (CalcTextSize(cast(immutable(char)*)label, null, true, -1F)) of type ImVec2 to const(ImVec2) //second attempt const ImVec2 label_size = cast(const)CalcTextSize(label.ptr, null, true); // Error: cannot implicitly convert expression (CalcTextSize(cast(immutable(char)*)label, null, true, -1F)) of type const(ImVec2) to const(ImVec2) //third attempt const auto label_size = CalcTextSize(label.ptr, null, true); //Okay: don't know why the other two didn't work but I can keep going for now func(label_size); //Error: function imgui_d.func (const(ImVec2) label_size) is not callable using argument types (const(ImVec2)) }
Re: Error: func(const(A) a) is not callable using argument types (const(A)
It seems there are two different ImVec2 types. So ImVec2 is not same as ImVec2 :) On Tue, May 30, 2017 at 12:09 PM, Andrew Edwards via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote: > What does that even mean? > > Scenario: > > bool func(const ImVec2 label_size) > { > return true; > } > > void main() > { > //first attempt: > const ImVec2 label_size = CalcTextSize(label.ptr, null, true); > //Error: cannot implicitly convert expression > (CalcTextSize(cast(immutable(char)*)label, null, true, -1F)) of type > ImVec2 to const(ImVec2) > > //second attempt > const ImVec2 label_size = cast(const)CalcTextSize(label.ptr, null, > true); > // Error: cannot implicitly convert expression > (CalcTextSize(cast(immutable(char)*)label, null, true, -1F)) of type > const(ImVec2) to const(ImVec2) > > //third attempt > const auto label_size = CalcTextSize(label.ptr, null, true); > //Okay: don't know why the other two didn't work but I can keep going > for now > > func(label_size); > //Error: function imgui_d.func (const(ImVec2) label_size) is not > callable using argument types (const(ImVec2)) > } >
Re: Error: func(const(A) a) is not callable using argument types (const(A)
Sorry, rough day. Could someone please explain what this means and how do go about resolving it? Thanks, Andrew
Re: Error: func(const(A) a) is not callable using argument types (const(A)
On Tuesday, 30 May 2017 at 10:09:50 UTC, Andrew Edwards wrote: What does that even mean? Scenario: bool func(const ImVec2 label_size) { return true; } void main() { //first attempt: const ImVec2 label_size = CalcTextSize(label.ptr, null, true); //Error: cannot implicitly convert expression (CalcTextSize(cast(immutable(char)*)label, null, true, -1F)) of type ImVec2 to const(ImVec2) //second attempt const ImVec2 label_size = cast(const)CalcTextSize(label.ptr, null, true); // Error: cannot implicitly convert expression (CalcTextSize(cast(immutable(char)*)label, null, true, -1F)) of type const(ImVec2) to const(ImVec2) //third attempt const auto label_size = CalcTextSize(label.ptr, null, true); //Okay: don't know why the other two didn't work but I can keep going for now func(label_size); //Error: function imgui_d.func (const(ImVec2) label_size) is not callable using argument types (const(ImVec2)) } My immediate thought is 'is that the same ImVec2?' Do you have two definitions of ImVec2 laying about? What's the output of this code, if you insert it somewhere in the above? import std.traits; pragma(msg, fqn!ImVec2); pragma(msg, fqn!(typeof(CalcTextSize(label.ptr, null, true; -- Biotronic
Re: Error: func(const(A) a) is not callable using argument types (const(A)
import std.traits : fqn = fullyQualifiedName; On Tue, May 30, 2017 at 12:24 PM, Biotronic via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote: > On Tuesday, 30 May 2017 at 10:09:50 UTC, Andrew Edwards wrote: > >> What does that even mean? >> >> Scenario: >> >> bool func(const ImVec2 label_size) >> { >> return true; >> } >> >> void main() >> { >> //first attempt: >> const ImVec2 label_size = CalcTextSize(label.ptr, null, true); >> //Error: cannot implicitly convert expression >> (CalcTextSize(cast(immutable(char)*)label, null, true, -1F)) of type >> ImVec2 to const(ImVec2) >> >> //second attempt >> const ImVec2 label_size = cast(const)CalcTextSize(label.ptr, null, >> true); >> // Error: cannot implicitly convert expression >> (CalcTextSize(cast(immutable(char)*)label, null, true, -1F)) of type >> const(ImVec2) to const(ImVec2) >> >> //third attempt >> const auto label_size = CalcTextSize(label.ptr, null, true); >> //Okay: don't know why the other two didn't work but I can keep going >> for now >> >> func(label_size); >> //Error: function imgui_d.func (const(ImVec2) label_size) is not >> callable using argument types (const(ImVec2)) >> } >> > > My immediate thought is 'is that the same ImVec2?' Do you have two > definitions of ImVec2 laying about? > > What's the output of this code, if you insert it somewhere in the above? > > import std.traits; > > pragma(msg, fqn!ImVec2); > pragma(msg, fqn!(typeof(CalcTextSize(label.ptr, null, > >> true; >> > > -- > Biotronic >
Re: Error: func(const(A) a) is not callable using argument types (const(A)
On Tuesday, 30 May 2017 at 10:20:53 UTC, Andrew Edwards wrote: Sorry, rough day. Could someone please explain what this means and how do go about resolving it? Thanks, Andrew If you want to resolve it just do const label_size = CalcTextSize(...); but as others have mentioned make sure it is the right type (although if they are functionally identical it won't matter).
Re: Error: func(const(A) a) is not callable using argument types (const(A)
On Tuesday, 30 May 2017 at 10:31:24 UTC, Daniel Kozak wrote: import std.traits : fqn = fullyQualifiedName; Darnit. I just googled the template and got a result talking about fqn!T. So yeah - this code: import std.traits; pragma(msg, fullyQualifiedName!ImVec2); pragma(msg, fullyQualifiedName!(typeof(CalcTextSize(label.ptr, null, true; -- Biotronic
Re: Error: func(const(A) a) is not callable using argument types (const(A)
On Tuesday, 30 May 2017 at 10:37:58 UTC, Biotronic wrote: On Tuesday, 30 May 2017 at 10:31:24 UTC, Daniel Kozak wrote: import std.traits : fqn = fullyQualifiedName; Darnit. I just googled the template and got a result talking about fqn!T. So yeah - this code: import std.traits; pragma(msg, fullyQualifiedName!ImVec2); pragma(msg, fullyQualifiedName!(typeof(CalcTextSize(label.ptr, null, true; -- Biotronic This is exactly the cause. Output is follows: internal.ImVec2 types.ImVec2 I'm leveraging types as I try to do my own port of the lib so CalcTextSize is returning an instance of ImVec2 as defined types and I'm trying to assign to one I have declared in internal. Thanks.
Rosetta Commatizing numbers
I ran into a Rosetta code solution in D that had obvious errors. It's like the author or the previous editor wasn't even trying to do it right, like a protest against how many detailed rules the task had. I assumed that's not the way we want to do things in D. Then I spent all day fixing it. Once I thought I had it working, I added some unittests, and realized it wasn't really correct, and struggled with the more complicated regex I had written, until it was simplified, and I really fixed it. So I pushed it, hard. The earlier version of the page made D look more error prone than other languages, but short. Now my solution is as long as some of the other language's solutions, but it's well commented and tested, I think. Now I doubt any of the solutions in other languages are as correct or potentially useful or informative. Something I removed was there was a commented out /* pure nothrow */ someone had left there, and a @safe attribute, because those don't work with using the parts of the library that I used. It seems like there's a tangle of features in the library that would take a lot more usage examples than what can fit in the library doc itself to sort out for people. Is it normal to add a bunch of wished-for future attributes to a function, then comment them out because they don't compile yet? Maybe that's something that happens often with D code, but I didn't think that should be how D is advertised. An extra that I could add, if I should push again, is using the compile time regex function ctRegex, but I found out that's not enough to make the function nothrow, by far. Does anyone have any thoughts about this? Did I do right by D? http://rosettacode.org/wiki/Commatizing_numbers
Re: Error: func(const(A) a) is not callable using argument types (const(A)
On Tuesday, 30 May 2017 at 10:46:12 UTC, Andrew Edwards wrote: On Tuesday, 30 May 2017 at 10:37:58 UTC, Biotronic wrote: On Tuesday, 30 May 2017 at 10:31:24 UTC, Daniel Kozak wrote: import std.traits : fqn = fullyQualifiedName; Darnit. I just googled the template and got a result talking about fqn!T. So yeah - this code: import std.traits; pragma(msg, fullyQualifiedName!ImVec2); pragma(msg, fullyQualifiedName!(typeof(CalcTextSize(label.ptr, null, true; -- Biotronic This is exactly the cause. Output is follows: internal.ImVec2 types.ImVec2 I'm leveraging types as I try to do my own port of the lib so CalcTextSize is returning an instance of ImVec2 as defined types and I'm trying to assign to one I have declared in internal. Thanks. Pleasure. :) I don't know why you have two different ImVec2s, but you may have good reasons to. If they need to be separate, you'll need to write a conversion function between the two for the cases where you have one and want the other. This could be the constructor or opAssign, or a standalone function if you want to be more explicit about it. I'd argue that error message could be improved upon, btw. -- Simen
Re: Error: func(const(A) a) is not callable using argument types (const(A)
http://forum.dlang.org/post/xpmpakmusudanwuzz...@forum.dlang.org https://issues.dlang.org/show_bug.cgi?id=9631 On Tue, May 30, 2017 at 1:07 PM, Biotronic via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote: > On Tuesday, 30 May 2017 at 10:46:12 UTC, Andrew Edwards wrote: > >> On Tuesday, 30 May 2017 at 10:37:58 UTC, Biotronic wrote: >> >>> On Tuesday, 30 May 2017 at 10:31:24 UTC, Daniel Kozak wrote: >>> import std.traits : fqn = fullyQualifiedName; >>> >>> Darnit. I just googled the template and got a result talking about >>> fqn!T. So yeah - this code: >>> >>> import std.traits; >>> >>> pragma(msg, fullyQualifiedName!ImVec2); >>> pragma(msg, fullyQualifiedName!(typeof(CalcTextSize(label.ptr, null, >>> true; >>> >>> -- >>> Biotronic >>> >> >> This is exactly the cause. Output is follows: >> >> internal.ImVec2 >> types.ImVec2 >> >> I'm leveraging types as I try to do my own port of the lib so >> CalcTextSize is returning an instance of ImVec2 as defined types and I'm >> trying to assign to one I have declared in internal. >> >> Thanks. >> > > Pleasure. :) > > I don't know why you have two different ImVec2s, but you may have good > reasons to. If they need to be separate, you'll need to write a conversion > function between the two for the cases where you have one and want the > other. This could be the constructor or opAssign, or a standalone function > if you want to be more explicit about it. > > I'd argue that error message could be improved upon, btw. > > -- > Simen >
Re: purity question
Rene Zwanenburg wrote: On Monday, 29 May 2017 at 01:36:24 UTC, Jonathan M Davis wrote: A simple example: anything that has a malloc/free pair. Yeah, if you do it right, you should be fine, but you have to do it right, and it's very easy to miss some detail that makes it wrong to insist to the compiler that what you're doing is pure. If malloc were marked as pure, wouldn't that mean it must return the same pointer every time you call it with the same size? of course. but D "pure" is not what other world knows as "pure". we love to mess with words.
Re: Best way for handle missing args in REST interface in vibed
I had post question here http://forum.rejectedsoftware.com/groups/rejectedsoftware.vibed/thread/43511/
Re: purity question
On Tuesday, 30 May 2017 at 11:34:52 UTC, ketmar wrote: If malloc were marked as pure, wouldn't that mean it must return the same pointer every time you call it with the same size? of course. but D "pure" is not what other world knows as "pure". we love to mess with words. Well, there's the ability to modify non-const reference parameters from a pure function, but that's not applicable to malloc. Are there any other special rules?
Re: purity question
Rene Zwanenburg wrote: On Tuesday, 30 May 2017 at 11:34:52 UTC, ketmar wrote: If malloc were marked as pure, wouldn't that mean it must return the same pointer every time you call it with the same size? of course. but D "pure" is not what other world knows as "pure". we love to mess with words. Well, there's the ability to modify non-const reference parameters from a pure function, but that's not applicable to malloc. Are there any other special rules? "pure" methods can mutate object state.
Re: purity question
On Tuesday, 30 May 2017 at 13:45:07 UTC, Rene Zwanenburg wrote: On Tuesday, 30 May 2017 at 11:34:52 UTC, ketmar wrote: If malloc were marked as pure, wouldn't that mean it must return the same pointer every time you call it with the same size? of course. but D "pure" is not what other world knows as "pure". we love to mess with words. Well, there's the ability to modify non-const reference parameters from a pure function, but that's not applicable to malloc. Are there any other special rules? The rules[0] are: 0) Can't call functions not marked pure. 1) Can't touch anything mutable that's not explicitly passed to the pure function. 1b) Except GC internal state - i.e. memory can be allocated via the GC. 1c) And floating-point exception flags and modes. 2) Can't do I/O (can be seen as a special case of 1 and 0). There's a few more details, but that's the important stuff. For a good article on the subject, I recommend David Nadlinger's Purity in D: http://klickverbot.at/blog/2012/05/purity-in-d/ [0]: https://dlang.org/spec/function.html#pure-functions
Re: purity question
On 05/30/2017 11:12 AM, Rene Zwanenburg wrote: If malloc were marked as pure, wouldn't that mean it must return the same pointer every time you call it with the same size? D's `pure` mostly means: "does not access mutable state, and does not do input/output". There is never a requirement that a function must return the same value for the same input. But a compiler is allowed to memoize the result of a `pure` function when it has no mutable indirections in its parameter and return types. Such a function is "strongly pure". When there are mutable indirections, the function is "weakly pure". Weakly pure functions are not assumed to be memoizable, but "weakly pure" still has meaning: * Can call weakly pure functions from strongly pure ones. * When a weakly pure function has mutable indirections in the return type but not in the parameters (like malloc), then it must be returning freshly allocated memory. That means, the result cannot be referenced from anywhere else. So it can be converted to const/immutable/shared implicitly. The spec calls that a "pure factory function". Repeating Biotronic's links, the spec and David Nadlinger's article are the go-to resources for D's take on purity: https://dlang.org/spec/function.html#pure-functions http://klickverbot.at/blog/2012/05/purity-in-d/
Finding the index of the maximum value in an associative array
Hi, I have an associative array of type int[ulong] and I'm trying to get the index of the maximum value, like this: int[ulong] aa = [1UL: 2000, 2UL: 5000, 5UL: 1000]; writeln(aa.maxIndex); // should print 2 Can anyone help me out? Many thanks, Lyle
Re: What is the postfix for min long value?
Jonathan, I saw this answered in another post: http://forum.dlang.org/post/gtaublmskqrhnbhoe...@forum.dlang.org I.e., you can do long(-9223372036854775808UL) :) Or long l = -9223372036854775808UL;
Re: What is the postfix for min long value?
On Tuesday, 6 October 2015 at 15:16:13 UTC, tcak wrote: While writing max ulong value, I added the "u" postfix. So compiler accepted it as ulong value (That's my interpretation if correct on compiler's side). writeln( 18_446_744_073_709_551_615u ); But when I try to print out minimum value of long, compiler says Error: signed integer overflow writeln( -9_223_372_036_854_775_808 ); In case that is the wrong value, I checked it with writeln( long.min ); already. Do I need to put a postfix for that number? I checked documentation by searching "dlang integer" etc, but couldn't have found any information/anything about postfix at all. See above ^^ :D
Re: Finding the index of the maximum value in an associative array
On Tuesday, 30 May 2017 at 17:57:04 UTC, Lyle wrote: Hi, I have an associative array of type int[ulong] and I'm trying to get the index of the maximum value, like this: int[ulong] aa = [1UL: 2000, 2UL: 5000, 5UL: 1000]; writeln(aa.maxIndex); // should print 2 Can anyone help me out? Many thanks, Lyle Simple enough: get all key/values pairs in the AA, and ask for the element which is the maximum relatively to the value, then take its key. auto maxIndex(int[ulong] aa) { import std.algorithm; return aa.byKeyValue .maxElement!(x => x.value) .key; }
Re: purity question
On Tuesday, May 30, 2017 16:54:13 ag0aep6g via Digitalmars-d-learn wrote: > On 05/30/2017 11:12 AM, Rene Zwanenburg wrote: > > If malloc were marked as pure, wouldn't that mean it must return the > > same pointer every time you call it with the same size? > > D's `pure` mostly means: "does not access mutable state, and does not do > input/output". > > There is never a requirement that a function must return the same value > for the same input. But a compiler is allowed to memoize the result of a > `pure` function when it has no mutable indirections in its parameter and > return types. Such a function is "strongly pure". > > When there are mutable indirections, the function is "weakly pure". > Weakly pure functions are not assumed to be memoizable, but "weakly > pure" still has meaning: > > * Can call weakly pure functions from strongly pure ones. > > * When a weakly pure function has mutable indirections in the return > type but not in the parameters (like malloc), then it must be returning > freshly allocated memory. That means, the result cannot be referenced > from anywhere else. So it can be converted to const/immutable/shared > implicitly. The spec calls that a "pure factory function". > > Repeating Biotronic's links, the spec and David Nadlinger's article are > the go-to resources for D's take on purity: > > https://dlang.org/spec/function.html#pure-functions > http://klickverbot.at/blog/2012/05/purity-in-d/ Yeah, basically, D's pure was originally what is now sometimes called "strongly pure," which is quite close to functionally pure in that the same input results in the same output (it still allows alocating memory and returning it though, so you can have equal but different objects if the same function is called multiple times with the same arguments in different contexts where the compiler doesn't memoize it). However, pure was so restrictive as to be borderline useless, because it could only pass types that were immutable or implicitly convertible to immutable. So, it was expanded to include any function which could not access global, mutable state, because such functions can be called from a "strongly" pure function without violating the strongly pure function's guarantees - and thus "weakly" pure functions were born, making it so that D's pure is really more like @noglobal than functionally pure. It's a critical building block in functional purity, and strongly pure functions still get all of the benefits that they got before (and now they can actually be useful, because they can call many more functions), but _most_ pure functions are weakly pure and thus don't get the full benefits of functional purity - and that's why some folks like Ketmar tend to get annoyed with D's pure. But the way it is is actually quite useful even if it's initially confusing. And even when there are no strongly pure functions in your program, knowing that a function cannot access global variables except through its arguments means a lot in and of itself. - Jonathan M Davis
Re: Finding the index of the maximum value in an associative array
On Tue, May 30, 2017 at 05:57:04PM +, Lyle via Digitalmars-d-learn wrote: > Hi, > > I have an associative array of type int[ulong] and I'm trying to get > the index of the maximum value, like this: > > int[ulong] aa = [1UL: 2000, > 2UL: 5000, > 5UL: 1000]; > > writeln(aa.maxIndex); // should print 2 > > Can anyone help me out? [...] Try this: void main() { import std.algorithm.iteration : fold; import std.array : byPair; import std.stdio : writeln; int[ulong] aa = [1UL: 2000, 2UL: 5000, 5UL: 1000]; writeln(aa.byPair .fold!((a,b) => a[1] < b[1] ? b : a)[0]); } Note that an associative array really isn't the best data structure if you'll be performing this operation frequently, because you'll be iterating over the entire contents each time. You may want to consider using a sorted container like a RedBlackTree in std.container or something similar instead. T -- Ignorance is bliss... until you suffer the consequences!
Re: purity question
On Tue, May 30, 2017 at 11:10:19AM -0700, Jonathan M Davis via Digitalmars-d-learn wrote: [...] > Yeah, basically, D's pure was originally what is now sometimes called > "strongly pure," which is quite close to functionally pure in that the > same input results in the same output (it still allows alocating > memory and returning it though, so you can have equal but different > objects if the same function is called multiple times with the same > arguments in different contexts where the compiler doesn't memoize > it). However, pure was so restrictive as to be borderline useless, > because it could only pass types that were immutable or implicitly > convertible to immutable. So, it was expanded to include any function > which could not access global, mutable state, because such functions > can be called from a "strongly" pure function without violating the > strongly pure function's guarantees - and thus "weakly" pure functions > were born, making it so that D's pure is really more like @noglobal > than functionally pure. It's a critical building block in functional > purity, and strongly pure functions still get all of the benefits that > they got before (and now they can actually be useful, because they can > call many more functions), but _most_ pure functions are weakly pure > and thus don't get the full benefits of functional purity - and that's > why some folks like Ketmar tend to get annoyed with D's pure. But the > way it is is actually quite useful even if it's initially confusing. > And even when there are no strongly pure functions in your program, > knowing that a function cannot access global variables except through > its arguments means a lot in and of itself. [...] In retrospect, we could have named "weakly pure" something like @noglobal, and "strongly pure" as pure. But be that as it may, I think D's pure system is actually extremely ingenious. Look at it this way: in the classical sense of functional purity, which is what you get in (pure) functional languages, it can get quite difficult to implement the functionality you want, because the language enforces that every primitive you use in your implementation must be pure. The reasoning is that if you're only allowed to use pure primitives, then the resulting function is guaranteed to be pure. It's nice and simple. However, it's also rather cumbersome, especially in the context of an imperative language like D. For example, in a functional language you cannot assign new values to variables, and you cannot write for-loops, because the loop index is not allowed to mutate. You cannot mutate anything, because mutation makes the code impure. So instead of straightforward loops, you need to resort to things like tail recursion; instead of mutation, you need to use monads, and so on. I'm not saying this is a bad thing, but it's just cumbersome because you, the programmer, cannot simply implement a function with a straightforward algorithm, but you have to work harder to express the algorithm in functional terms using recursion and other functional (pure) constructs. The first insight in D's purity system is the observation that, given some function f(), from the caller's POV all they care about is that (1) the function always returns the same value given the same arguments, and (2) there are no visible side-effects. Point (2) is where the insight lies: in a sense, it *doesn't matter* if f() does all kinds of impure stuff in order to compute its return value, as long as the outside world cannot see it. It may be internally impure, but externally, as far as the outside world is concerned, it's pure, because the caller can't tell the difference. So D allows things like variables, mutation, for-loops, and all kinds of stuff inside (strongly) pure functions -- as long as no global state is touched, and as long as the function arguments aren't modified (from the caller's POV), f() is pure. In other words, as long as f() keeps its dirty laundry to itself and leaves the outside world untouched, it is externally pure, even if it's internally impure. The next insight is this: suppose f() calls g(), and g() is impure. In the traditional functional language purity system, this is outright illegal, because a pure function, by definition, cannot call an impure function, otherwise it is itself impure. However, suppose g() does not modify any global state. It *may* modify stuff through its arguments -- which makes it impure. But if f() is not allowed to modify its arguments and has no global state, then at worst, it can only pass its internal, non-global state to g(). Therefore, it is impossible for g() to reach global state through its arguments. Thus, f()'s (external) purity is preserved. Note the fine distinction here, that g()'s external purity is keyed on being called from inside a strongly-pure function f(). If f() is impure, then there is no guarantee that g() may modify global state (because f() may pass a reference to a global variable
howto count lines - fast
After reading https://dlang.org/blog/2017/05/24/faster-command-line-tools-in-d/ , i was wondering how fast one can do a simple "wc -l" in D. So i made a couple short implementations and found myself confronted with slow results compared to "/usr/bin/wc -l". How would a implementation look like in D, which is fast?
Re: howto count lines - fast
On Tuesday, 30 May 2017 at 20:02:38 UTC, Nitram wrote: After reading https://dlang.org/blog/2017/05/24/faster-command-line-tools-in-d/ , i was wondering how fast one can do a simple "wc -l" in D. So i made a couple short implementations and found myself confronted with slow results compared to "/usr/bin/wc -l". How would a implementation look like in D, which is fast? Not sure if this is the fastest, but anyway void main(){ import std.file : read; import std.conv : to; import std.algorithm : count; auto data = cast(ubyte[])read("somefile.txt"); auto lc = data.count('\n'.to!ubyte); } Jordan
Re: howto count lines - fast
On Tuesday, 30 May 2017 at 20:37:44 UTC, Jordan Wilson wrote: On Tuesday, 30 May 2017 at 20:02:38 UTC, Nitram wrote: After reading https://dlang.org/blog/2017/05/24/faster-command-line-tools-in-d/ , i was wondering how fast one can do a simple "wc -l" in D. So i made a couple short implementations and found myself confronted with slow results compared to "/usr/bin/wc -l". How would a implementation look like in D, which is fast? Not sure if this is the fastest, but anyway void main(){ import std.file : read; import std.conv : to; import std.algorithm : count; auto data = cast(ubyte[])read("somefile.txt"); auto lc = data.count('\n'.to!ubyte); } Jordan I should say, if you don't care about storing the data, File("somefile.txt","r").byLine.count is probably more idiomatic, and probably just as fast. Jordan
Re: howto count lines - fast
I do not know this is my first attempt and it is almost same fast as wc on my pc: int main(string[] args) { import std.stdio : writeln, writefln, File; import std.array : uninitializedArray; auto f = File("data"); size_t c = 0; auto buffer = uninitializedArray!(ubyte[])(1024); foreach (chunk; f.byChunk(buffer)) { foreach (ch; chunk) if (ch == '\n') ++c; } writeln(c); return 0; } And I belive it could be faster when iopipe is used Dne 30.5.2017 v 22:02 Nitram via Digitalmars-d-learn napsal(a): After reading https://dlang.org/blog/2017/05/24/faster-command-line-tools-in-d/ , i was wondering how fast one can do a simple "wc -l" in D. So i made a couple short implementations and found myself confronted with slow results compared to "/usr/bin/wc -l". How would a implementation look like in D, which is fast?
ushort calls byte overload
Hello. I have this code import std.stdio; void foo(byte a) { writeln(typeof(a).stringof); } void foo(short a) { writeln(typeof(a).stringof); } void foo(int a) { writeln(typeof(a).stringof); } void main() { foo(0); // int, and byte if not define foo(int) foo(ushort(0)); // byte (unexpected for me) foo(cast(ushort)0); // byte (unexpected too) foo(cast(short)0); // short foo(short(0)); // short ushort x = 0; foo(x); // short } Is this a bug or I don't understand something?
Re: ushort calls byte overload
and this is unexpected for me too immutable ushort y = 0; foo(y); // byte
Re: ushort calls byte overload
Dne 30.5.2017 v 23:16 Oleg B via Digitalmars-d-learn napsal(a): Hello. I have this code import std.stdio; void foo(byte a) { writeln(typeof(a).stringof); } void foo(short a) { writeln(typeof(a).stringof); } void foo(int a) { writeln(typeof(a).stringof); } void main() { foo(0); // int, and byte if not define foo(int) foo(ushort(0)); // byte (unexpected for me) foo(cast(ushort)0); // byte (unexpected too) foo(cast(short)0); // short foo(short(0)); // short ushort x = 0; foo(x); // short } Is this a bug or I don't understand something? It is "not" a bug, and it is how compiler works. Compiler do many assumptions (it is sometimes useful). So if it see something like immutable ushort y = 0 (compile time value) or just "0" (still compile time value) it knows its value so it knows it can be saftly represent in byte, ubyte and so on. However ushort x = 0; foo(x); // here x is runtime value so compilere does not know the size, so it will use the type. Lets look to an another example immutable ushort y = 0; byte x = y; this will compile because of current rules. but this: ushort y = 0; byte x = y; will produce this error: Error: cannot implicitly convert expression (y) of type ushort to byte And in this example is everything ok: import std.stdio; void f(ushort u) { writeln("ushort"); } void f(ubyte u) { writeln("ubyte"); } void main() { ushort y = 0; immutable ushort x = 0; f(y); f(x); } RESULT IS: ushort ushort but obviously in follow example it could be misleading import std.stdio; void f(short u) { writeln("short"); } void f(ubyte u) // or byte u { writeln("ubyte or byte"); } void main() { ushort y = 0; immutable ushort x = 0; f(y); f(x); } RESULT IS: ushort [u]byte But it make sense, compiler will select the best match in this case it is byte or ubyte but if we change x to something like 128 wi will get different results for byte and ubyte
Re: ushort calls byte overload
On Tuesday, 30 May 2017 at 21:42:03 UTC, Daniel Kozak wrote: Compiler do many assumptions (it is sometimes useful). but if compiler find one-to-one correspondence it don't make assumptions, like here? import std.stdio; void f(ushort u) { writeln("ushort"); } void f(ubyte u) { writeln("ubyte"); } void main() { ushort y = 0; immutable ushort x = 0; f(y); f(x); } RESULT IS: ushort ushort or it can?
Re: Finding the index of the maximum value in an associative array
On Tuesday, 30 May 2017 at 18:05:02 UTC, cym13 wrote: On Tuesday, 30 May 2017 at 17:57:04 UTC, Lyle wrote: Hi, I have an associative array of type int[ulong] and I'm trying to get the index of the maximum value, like this: int[ulong] aa = [1UL: 2000, 2UL: 5000, 5UL: 1000]; writeln(aa.maxIndex); // should print 2 Can anyone help me out? Many thanks, Lyle Simple enough: get all key/values pairs in the AA, and ask for the element which is the maximum relatively to the value, then take its key. auto maxIndex(int[ulong] aa) { import std.algorithm; return aa.byKeyValue .maxElement!(x => x.value) .key; } Thank you! I used this solution in the end. I had looked at the .byKeyValue and .byPair methods but they didn't seem to work - turns out I was using an old DMD version!
Re: howto count lines - fast
On Tue, May 30, 2017 at 08:02:38PM +, Nitram via Digitalmars-d-learn wrote: > After reading > https://dlang.org/blog/2017/05/24/faster-command-line-tools-in-d/ , i > was wondering how fast one can do a simple "wc -l" in D. > > So i made a couple short implementations and found myself confronted > with slow results compared to "/usr/bin/wc -l". > > How would a implementation look like in D, which is fast? This little challenge piqued my interest. So I decided to take a shot at seeing if I could beat my system's /usr/bin/wc -l. First order of business: whenever it comes to performance, always choose the right compiler for the job. While dmd is the reference compiler and has the latest and greatest features, it doesn't do too well in the performance department. So the first thing I did was to use gdc instead. Specifically, gdc -O3, to get the most aggressive optimizations the gcc backend can give me. Second order of business: the naïve algorithm of reading 1 character at a time from the file is obviously suboptimal, so I didn't even consider it. Third order of business: I constructed a large text file containing 10634420 lines by concatenating 5 copies of a large CSV file I obtained online. The sample had to be large enough so that overhead like program startup times and background system noise don't bias the test results too much. With this setup, I measured the performance of my system's wc -l as the baseline for comparing the performance of the D solutions. Here's a typical output of my shell's `time wc -l` command: real0m0.344s user0m0.153s sys 0m0.191s Since we're talking about beating wc, and others have already posted the obvious solutions of loading into a buffer and scanning the buffer, the most obvious next step up is to use std.mmfile to memory-map the file so that we don't waste effort managing file buffers ourselves: let the OS do it for us. So here are the algorithms I tried: 1) lineCount1(): use std.mmfile to memory-map the file so that we can scan it as a single, contiguous array without any fussy buffer-management details. Result: pretty fast, but still loses to wc. Typical measurements: real0m0.422s user0m0.366s sys 0m0.054s 2) lineCount2(): just as a comparison, I did a read-into-buffer algorithm so that we have a reference as to how it performs. As expected, it's worse than the std.mmfile solution. Typical measurements: real0m0.519s user0m0.320s sys 0m0.198s 3) lineCount3(): In lineCount1(), I used std.algorithm.searching.count for counting newlines; so I thought, what if the range abstraction is introducing too much overhead? So I decided to write a simple foreach loop instead, in hope that the simpler code will allow the gcc optimizer do a better job. Sadly, the result is pretty much the same as lineCount1: real0m0.421s user0m0.379s sys 0m0.042s (The good news, though, is that this shows that using std.algorithm does not introduce excessive overhead compared to a hand-written loop. This proves that the high-level range abstraction does not necessarily equate to slower performance.) 4) lineCount4(): Having failed to beat wc thus far, it's time to bring out the big guns. Since we're essentially counting newlines in the input, who says we have to process the data sequentially from start to end? Counting from front to back or back to front gives the same answer, as does counting from middle to end, then front to middle. In particular, counting from middle to end *in parallel* with counting from front to middle, then summing the subtotals, gives the same answer. I.e., time to bust out std.parallelism. :-) So, in my 4th algorithm, I split the input into 16KB chunks, and then scan them for newlines in parallel, creating file_size/16384 subtotals. Then I sum the subtotals in the end. Here's the result, tested on my AMD Phenom 6-core CPU: real0m0.242s user0m1.151s sys 0m0.057s Woohoo!!! Finally, we beat the system's wc -l!! And by a pretty fair margin, too. Eat your heart out, wc!!! (The large user time is because we're using all 6 cores at once. But the actual elapsed time is shorter.) Here's the code for all of the above: -snip- import std.stdio; // real0m0.422s // user0m0.366s // sys 0m0.054s size_t lineCount1(string filename) { import std.algorithm.searching : count; import std.mmfile; auto f = new MmFile(filename); auto data = cast(ubyte[]) f[]; return data.count('\n'); } // real0m0.519s // user0m0.320s // sys 0m0.198s size_t lineCount2(string filename) { import std.algorithm.searching : count; auto f = File(filename); ubyte[] buf; size_t c = 0; buf.length = 32768; do { auto d = f.rawRead(buf); c += d.count('\n'); } while (!f.eof()); return c; } // real0m0.421
Re: howto count lines - fast
On 05/30/2017 01:02 PM, Nitram wrote: After reading https://dlang.org/blog/2017/05/24/faster-command-line-tools-in-d/ , i was wondering how fast one can do a simple "wc -l" in D. So i made a couple short implementations and found myself confronted with slow results compared to "/usr/bin/wc -l". How would a implementation look like in D, which is fast? I could not make the D program come close to wc's performance when the data was piped from stdin. A typical run with Daniel Kozak's program: $ time cat deleteme.txt | wc -l 5062176 real0m0.086s user0m0.068s sys 0m0.048s $ time cat deleteme.txt | ./deneme 5062176 real0m0.174s user0m0.152s sys 0m0.072s Ali
Re: howto count lines - fast
P.S. After I posted the code, I took a closer look at the disassembly and found that gdc wasn't generating the best code for the parallel foreach loop body. I haven't fully traced the cause yet, but I did find a simple optimization (arguably a micro-optimization): updating the subtotal inside the inner loop is a bit inefficient, because the subtotal is outside the loop body and, due to the way std.parallelism is implemented, is passed by reference to the loop body. This caused gdc not to enregister the subtotal, so incrementing it requires a cache roundtrip at best, a full RAM roundtrip at worst. So I introduced a local count variable for incrementing, and only assign to the subtotal array at the end of the block: ---snip--- // real0m0.240s // user0m1.045s // sys 0m0.059s size_t lineCount4(string filename) { import std.algorithm.comparison : min; import std.algorithm.iteration : sum; import std.mmfile; import std.parallelism; auto f = new MmFile(filename); auto data = cast(ubyte[]) f[]; size_t[] subtotals; enum blockSize = 16384; if (data.length == 0) return 0; subtotals.length = 1 + (data.length - 1) / blockSize; foreach (j, ref subtotal; parallel(subtotals)) { size_t end = min(blockSize*(j+1), data.length); size_t c = 0; foreach (i; blockSize*j .. end) { if (data[i] == '\n') c++; } subtotal = c; } return subtotals.sum; } ---snip--- A look at the disassembly showed that gdc has enregistered the subtotal, and thereby able to eliminate a branch from the inner loop using a sete instruction and an add for the if statement. The measured performance is only marginally better, though, and doesn't change the overall performance in a significant way. But I thought this might be a useful tip for people who want to squeeze out the last drop of juice from their CPUs. ;-) Next, I'll have to look into why gdc didn't unroll the inner loop, like I thought it would. But I'm out of time now, so that will have to wait for another day. T -- The peace of mind---from knowing that viruses which exploit Microsoft system vulnerabilities cannot touch Linux---is priceless. -- Frustrated system administrator.
Re: howto count lines - fast
On Tuesday, 30 May 2017 at 23:41:01 UTC, H. S. Teoh wrote: This little challenge piqued my interest. So I decided to take a shot at seeing if I could beat my system's /usr/bin/wc -l. First order of business: whenever it comes to performance, always choose the right compiler for the job... Woohoo!!! Finally, we beat the system's wc -l!! And by a pretty fair margin, too. Eat your heart out, wc!!! (The large user time is because we're using all 6 cores at once. But the actual elapsed time is shorter.) Hm... I cheated a little bit: took your program and compiled with `ldc2 -release -O3 -mcpu=skylake`. For data I took std.datetime concatenated 1000 times (35446000 lines). Results (minimum of 10 runs each): wc -l real 0.50 user 0.40 sys 0.09 lineCount1 real 0.23 user 0.19 sys 0.04 lineCount2 real 0.29 user 0.17 sys 0.12 lineCount3 real 0.23 user 0.18 sys 0.04 lineCount4 real 0.22 user 1.52 sys 0.04 Seems like all of them beat wc, so machine and compiler matter a great deal in this comparison. Thing is, on small files wc is going to win pretty much always, due to D's clunky startup. But with larger ones - wc falls far behind. Interestingly, both lineCount1 and lineCount3 on this machine are nearly even with lineCount4 :)
Re: Rosetta Commatizing numbers
On Tuesday, 30 May 2017 at 10:54:49 UTC, Solomon E wrote: The earlier version of the page made D look more error prone than other languages, but short. Now my solution is as long as some of the other language's solutions, but it's well commented and tested, I think. Now I doubt any of the solutions in other languages are as correct or potentially useful or informative. Regardless to solution length...one place to make code a little shorter could be when you check for special prefix; maybe replace the two foreach loops? Something like: if (specials != null) { // There may be special prefixed formats that use different separators. // Any format with a longer prefix should override a shorter one. auto pairs = specials.byKeyValue .array .sort!((a,b) => a.key.length < b.key.length); auto preAnyDigit = matchNum.pre.stripRight('0'); pairs.filter!(a => preAnyDigit.length >= a.key.length) .filter!(a => a.key == preAnyDigit[$ - a.key.length .. $]) .each!(a => ins = a.value); } Jordan
Re: Rosetta Commatizing numbers
On Tuesday, 30 May 2017 at 10:54:49 UTC, Solomon E wrote: I ran into a Rosetta code solution in D that had obvious errors. It's like the author or the previous editor wasn't even trying to do it right, like a protest against how many detailed rules the task had. I assumed that's not the way we want to do things in D. ... Does anyone have any thoughts about this? Did I do right by D? I'd say the previous version (by bearophile) suited the task much better, but both aren't perfect. As a general note, consider the following paragraph of the problem statement: "Some of the commatizing rules (specified below) are arbitrary, but they'll be a part of this task requirements, if only to make the results consistent amongst national preferences and other disciplines." This literally means that, while there are complex rules in the real world for commatizing numbers, the problem is kept simple by enforcing strict rules. The minute concerns of the Real World, like "Current New Zealand dollar format overrides old Zimbabwe dollar format", are irrelevant to the formal problem being solved. Perhaps the example inputs section ("Strings to be used as a minimum") gets misleading, but that's what they are: examples, not general rules. By the way, as it's a wiki page, problem statement text could also be improved ;) . Why? For example, look at Indian numbering system where commatizing is visibly different (https://en.wikipedia.org/wiki/Indian_numbering_system) - and we don't know whether the string should use it or not without the context. Or consider that hexadecimal numbers are usually split in groups of four digits, not three - and we don't know whether a [0-9]+ number is decimal or hexadecimal without the context. See, trying to provide an ultimate solution to real-world commatizing, while keeping it a single function without the context, can't possibly succeed. What can be done, then? Well, the page authors already did the difficult part for us: they extracted the essence of a complex real-world problem into a small set of formal rules, which are now the formal problem statement. Now comes the easy part: to do exactly what is asked in the problem statement. The flexibility comes from having function parameters. If we have a solution to a formal problem, using it for the real-world version of the problem is either just specifying the right parameters (hopefully), or changing the function if the real world gets too complex for it. In the latter case, the more short and readable the existing solution is, the faster can we change the function to suit our real-world case. - Now, where is the old version wrong? Turns out it just calls the function with default parameters for every line of input - which is wrong since the first two input lines need to be handled specially. Well, that's what the function parameters are for. To have a correct solution, we have to use custom parameters for the first two lines of input. The function itself is fine. Your solution addresses this problem by special-casing the inputs inside the function, perhaps because of the misleading inputs section in the problem statement. That's a wrong approach. First, it introduces magic numbers 33 and 36 into the code, which is a bad programming practice (see here: https://en.wikipedia.org/wiki/Magic_number_(programming)#Unnamed_numerical_constants). Second, it's plain wrong. According to the problem statement, we don't have these rules for every possible line of >33 standalone decimals, or >36 characters in total. We just have to call our function with a concrete set of custom parameters for one concrete example, and other set of parameters for another example. That's to demonstrate that our function accepts and makes proper use of custom parameters! Special-casing example inputs inside the function is not a solution: if we go down this path, the perfect solution would be a bunch of "if" statements for every possible example input producing the respective example outputs, and empty function for all other possible inputs. So, how do we call with special parameters? Currently, we can look at every other language except C# as inspiration: ALGOL 68, J, Java, Perl 6, Phix, Racket, and REXX. Your solution also has a good way to check example inputs: a unittest block. It even shows one of D's strengths compared to other languages. And there, you do use custom parameters to check that the function works. A good approach would be to put all the examples in the unittest instead of reading them from a file. This way, the program will be immediately usable and runnable: no need to create an additional arbitrarily-named file just to test it. - All in all, the only thing I'd change in bearophile's solution is to remove the file reading loop, add the unittest block from your solution instead, and place all the examples there. Printing the result does not see
Re: Rosetta Commatizing numbers
On Wednesday, 31 May 2017 at 04:31:14 UTC, Ivan Kazmenko wrote: Now, where is the old version wrong? ... Actually, it also changes every number in the string, not only the first one as required. Because of that, it also fails the "do not touch the exponent" requirement. Sadly, both are not covered by the examples. The program is perhaps easily fixed by changing replaceAll into replaceFirst. As for adding examples to better check the requirements, I don't know Rosettacode's policy for that.
Re: ushort calls byte overload
On Tuesday, 30 May 2017 at 21:16:26 UTC, Oleg B wrote: Hello. I have this code import std.stdio; void foo(byte a) { writeln(typeof(a).stringof); } void foo(short a) { writeln(typeof(a).stringof); } void foo(int a) { writeln(typeof(a).stringof); } void main() { foo(0); // int, and byte if not define foo(int) foo(ushort(0)); // byte (unexpected for me) foo(cast(ushort)0); // byte (unexpected too) foo(cast(short)0); // short foo(short(0)); // short ushort x = 0; foo(x); // short } Is this a bug or I don't understand something? Hm, interesting. I think what you're seeing here is an unexpected application of value range propagation: http://www.drdobbs.com/tools/value-range-propagation/229300211 None of these functions can be called with ushort without conversions. As the manual says: The function with the best match is selected. The levels of matching are: no match match with implicit conversions match with conversion to const exact match All of your functions require some implicit conversions (the ushort here can be converted to byte because of value range propagation). So the next rule is in effect - the functions are ordered and the "most specialized" is chozen. The byte function is the most specialized, so it is called. Or something like that :) but if compiler find one-to-one correspondence it don't make assumptions, like here? Apparently then it just chooses the function that is "exact match".
Re: howto count lines - fast
Dne 31.5.2017 v 02:13 Ali Çehreli via Digitalmars-d-learn napsal(a): I could not make the D program come close to wc's performance when the data was piped from stdin. A typical run with Daniel Kozak's program: $ time cat deleteme.txt | wc -l 5062176 real0m0.086s user0m0.068s sys0m0.048s $ time cat deleteme.txt | ./deneme 5062176 real0m0.174s user0m0.152s sys0m0.072s Ali How do you compile it? When I use ldc2 -O3 -release -mcpu=bdver1 lc.d my code is even faster than wc