Re: static array with inferred size
On Wednesday, 20 September 2017 at 00:47:25 UTC, Steven Schveighoffer wrote: This needs to happen. e.g.: char[$] arr = "hello"; // syntax up for debate, but I like this. Yes, hope to see this one day as a language feature, not a library solution. --Ilya
Re: 24-bit int
On Sunday, 3 September 2017 at 23:30:43 UTC, EntangledQuanta wrote: On Sunday, 3 September 2017 at 04:01:34 UTC, Ilya Yaroshenko wrote: On Saturday, 2 September 2017 at 03:29:20 UTC, EntangledQuanta wrote: On Saturday, 2 September 2017 at 02:49:41 UTC, Ilya Yaroshenko wrote: [...] Thanks. Seems useful. Just added `bytegroup` topology. Released in v0.6.12 (will be available in DUB after few minutes.) http://docs.algorithm.dlang.io/latest/mir_ndslice_topology.html#bytegroup It is faster for your task then `bitpack`. Best regards, Ilya Thanks! I might end up using this. Is this basically just a logical mapping(cast(int)bytes[i*3]) & 0xFF) type of stuff or is there more of a performance hit? I could do the mapping myself if that is the case as I do not need much of a general solution. I'll probably be using in a just a few lines of code. It just needs to be nearly as fast as direct access. The implementation can be found here https://github.com/libmir/mir-algorithm/blob/master/source/mir/ndslice/iterator.d It uses unions and byte loads. The speed should be almost the same as direct access.
Re: C++ / Why Iterators Got It All Wrong
On Sunday, 3 September 2017 at 16:33:04 UTC, Moritz Maxeiner wrote: On Sunday, 3 September 2017 at 14:19:19 UTC, Ilya Yaroshenko wrote: Ranges requires for 25% more space for canonical matrixes then iterators. We do have to differentiate between a container and the API with which to iterate over its elements. D's dynamic arrays (a pointer+length) require more space then using only a raw pointer, of course. If that's more than an iterator depends on the type of iterator you use. Most common iterator implementations I know consist of a begin and an end pointer, essentially requiring the same space as D's dynamic arrays. In the case you describe, though, we aren't talking about the iteration API, we are talking about what's used internally for storage. Maybe I should call it cursors or generic pointers instead of iterators.
Re: Generalized Linear Models and Stochastic Gradient Descent in D
On Sunday, 11 June 2017 at 13:36:09 UTC, jmh530 wrote: On Sunday, 11 June 2017 at 12:36:19 UTC, 9il wrote: I ported few large complex Matlab scripts using Lubeck and Mir-Algorithm (closed source). It works perfectly and results are the same as Matlab original! All functions from Lubeck was used in this work. Mir Algorithm has over then 2K downloads, and its downloads rates grows fast. I wasn't familiar with Lubeck, so thanks for bringing attention to it. Can I ask why this isn't part of mir? It uses mir-blas and mir-lapack packages. They are low level and satisfy betterC requirements. Mir is a libraries collection to write more high level easy to use libraries. Mir should provide basic data types, patterns, and low level API. Lubeck is high level library for scripting like programming. I like that it is located at Kaleidic GitHub. It is very important for D to have successful open source commercial projects. mir-blas, mir-lapack, and Lubeck are sponsored by Symmetry Investments and Kaleidic Associates. Another example is ASDF by Tamedia Digital. BTW, I need to write announce )
Re: Recently added ndslice API
On Friday, 17 March 2017 at 17:35:33 UTC, jmh530 wrote: On Saturday, 11 March 2017 at 10:53:35 UTC, Ilya Yaroshenko wrote: Github: https://github.com/libmir/mir-algorithm Every time I look at that mir-alogrithm main page with the Scheme image, I'm just like what the hell is going on. Maybe you can just link to it instead of featuring it so prominently... Agreed)
Re: Please change alias ReplaceArrayWithPointer = Flag!"replaceArrayWithPointer"
On Friday, 22 January 2016 at 17:09:01 UTC, Jacob Carlborg wrote: On 2016-01-21 20:31, Andrei Alexandrescu wrote: [...] Can we just implement a basic form of named parameters that remove the ugly workaround that Flag is. void a(int x); a(x: 3); // error, cannot be called with named parameters void b(int x:); b(3); // ok b(x: 4); // ok void c(int x:, int y:); c(x: 3, y: 4); // ok c(y: 4, x: 4); // error, named parameters out of order The first error is to avoid making parameter names public API by default. The second error is to not change how overloading works. DIP please! --Ilya
Re: Collapsing n-dimensional array to linear (1 dimensional)
On Friday, 22 January 2016 at 12:07:11 UTC, abad wrote: Let's say I have an array like this: int[][][] array; And I want to generate a linear int[] based on its data. Is there a standard library method for achieving this, or must I iterate over the array manually? What I'm thinking of is something like this: int[] onedim = std.array.collapse(array); You may want to look at upcoming Multidimensional Random Access Ranges, `std.experimental.ndslice` package. They are available with DMD 2.070 beta. Pacakge has `reshape` and `byElement` functions. Docs: http://dlang.org/phobos-prerelease/std_experimental_ndslice.html Ilya
Re: Please change alias ReplaceArrayWithPointer = Flag!"replaceArrayWithPointer"
On Thursday, 21 January 2016 at 19:31:19 UTC, Andrei Alexandrescu wrote: The correct idiom involving Flag is: * Use the name Flag!"frob" for the type of the flag * Use Yes.frob and No.frob for the flag values * Do NOT alias Flag!"frob" to a new name. This is unnecessary, unhelpful, and wasteful. Can somebody please change the respective code in std.experimental.ndslice? Thanks, Andrei OK --Ilya
Re: opIndex, opSlice, length for joiner?
On Wednesday, 20 January 2016 at 17:36:29 UTC, Maverick Chardet
wrote:
Hi everyone,
After reading the thread "Distributed Memory implementation":
http://forum.dlang.org/post/oqcfifzolmolcvyup...@forum.dlang.org
And more precisely the suggestion of Dragos Carp on page 2:
http://forum.dlang.org/post/txgabyyoutitketlp...@forum.dlang.org
I looked at the joiner source code
(std.algorithm.iteration.joiner) and I have several ideas to
make it have the opIndex, opSlice and length methods under
certain conditions.
First question: what do you think of making this possible if
all the considered ranges have opIndex, opSlice and length? I
think that all the ranges types don't have to all implement the
three methods in all cases (for instance length would actually
only require ElementType!RoR and Separator to implement length)
but we can discuss all that later...
The most important issue that comes to my mind is that the
operations would not take constant time... A trivial
implementation would be in O(k) where k is the number of joined
ranges, and with some precomputation in O(k) time and space we
can make length O(1) and opIndex and opSlice O(log k)... Would
that be a problem?
Thank you in advance for your remarks!
O(K) looks OK for joiner. However this will cause additional
performance problem because a couple of Phobos algorithm assumes
that opIndex is fast as front+popFront. Perhaps a special UDA can
be introduced, something like @optimized("forward range").
See also `byElement` for multidimensional random access slices.
It has opIndex and slicing.
https://github.com/D-Programming-Language/phobos/blob/v2.070.0-b2/std/experimental/ndslice/selection.d#L893
Ilya
Re: medianOfMedians
On Wednesday, 20 January 2016 at 02:26:35 UTC, Timon Gehr wrote: On 01/20/2016 02:20 AM, Andrei Alexandrescu wrote: [...] This is not an implementation of the median of medians algorithm. [...] The approximate medianOfMedians algorithm can be used for topN. --Ilya
Re: topN using a heap
On Monday, 18 January 2016 at 23:27:19 UTC, Ivan Kazmenko wrote: On Monday, 18 January 2016 at 23:18:03 UTC, Ilya wrote: A RNGs don't improve worst case. It only changes an permutation for worst case. --Ilya Still, use of RNG makes it impossible to construct the worst case beforehand, once and for all. In that sense, this is a regression. No, it is definitely possible, because RNGs are Pseudo-RNGs. --Ilya
Re: topN using a heap
On Tuesday, 19 January 2016 at 00:02:08 UTC, Timon Gehr wrote: On 01/19/2016 12:55 AM, Ilya wrote: On Monday, 18 January 2016 at 23:53:53 UTC, Timon Gehr wrote: On 01/19/2016 12:50 AM, Ilya wrote: ... 1. Yes, probability of hitting the worst case repeatedly is is practically zero. But RNGs do not change this probability. 2. It is possible to build attack for our RNGs, because they are Pseudo-RNGs. --Ilya You also need to predict the seed. How do you do that? We can not use unpredictable seed (like system clock) in pure functions. --Ilya Clearly. The point is, there already was an impure implementation of topN whose expected linear running time is still specified in the documentation. The current implementation does not fulfill this bound. There is already implementation with predictable seed. Proof: https://github.com/D-Programming-Language/phobos/blob/master/std/random.d#L1151 --Ilya
Re: topN using a heap
On Tuesday, 19 January 2016 at 01:04:03 UTC, Andrei Alexandrescu wrote: On 1/18/16 7:46 PM, Ilya wrote: On Tuesday, 19 January 2016 at 00:38:14 UTC, Timon Gehr wrote: On 01/19/2016 01:12 AM, Ilya wrote: There is already implementation with predictable seed. Proof: https://github.com/D-Programming-Language/phobos/blob/master/std/random.d#L1151 --Ilya The default RNG is seeded with unpredictableSeed. What is the point you are trying to make? unpredictableSeed is initialized only once and can be easily estimated. --Ilya https://github.com/D-Programming-Language/phobos/blob/v2.069.2/std/random.d#L1120 unpredictableSeed uses MinstdRand0. The sources of randomness used for initializing MinstdRand0 are the PID, the thread ID, and the system time at the moment of the seeding. Then at each subsequent call to unpredictableSeed, the time of the call is XORed with the current value of the MinstdRand0. How do you think things could be improved? Andrei A good variant with minimal overhead is to call unpredictableSeed for sorting big arrays each time (one seed per array): a. A hacker would not able to estimate a seed using other API calls. For example, "give me a set of random numbers". b. A hacker would not be able to estimate a seed using a small arrays because they don't use RNGs. (and they have not any overhead!) c. A hacker would not be able to estimate a seed for big arrays, because attack based on time measurement would not work for big arrays. Ilya
Re: topN using a heap
On Tuesday, 19 January 2016 at 01:04:03 UTC, Andrei Alexandrescu wrote: On 1/18/16 7:46 PM, Ilya wrote: On Tuesday, 19 January 2016 at 00:38:14 UTC, Timon Gehr wrote: On 01/19/2016 01:12 AM, Ilya wrote: There is already implementation with predictable seed. Proof: https://github.com/D-Programming-Language/phobos/blob/master/std/random.d#L1151 --Ilya The default RNG is seeded with unpredictableSeed. What is the point you are trying to make? unpredictableSeed is initialized only once and can be easily estimated. --Ilya https://github.com/D-Programming-Language/phobos/blob/v2.069.2/std/random.d#L1120 unpredictableSeed uses MinstdRand0. The sources of randomness used for initializing MinstdRand0 are the PID, the thread ID, and the system time at the moment of the seeding. Then at each subsequent call to unpredictableSeed, the time of the call is XORed with the current value of the MinstdRand0. How do you think things could be improved? Andrei A good variant with minimal overhead is to call unpredictableSeed for sorting big arrays each time (one seed per array): a. A hacker would not able to estimate a seed using other API calls. For example, "give me a set of random numbers". b. A hacker would not be able to estimate a seed using a small arrays because they don't use RNGs. (and they have not any overhead!) c. A hacker would not be able to estimate a seed for big arrays, because attack based on time measurement would not work for big arrays. EDIT: c. ... because each big array has own seed and time measurement would not work for big arrays with different seeds. Ilya
Re: topN using a heap
On Monday, 18 January 2016 at 20:45:56 UTC, Ivan Kazmenko wrote: On Monday, 18 January 2016 at 12:00:10 UTC, Ivan Kazmenko wrote: Here goes the test which shows quadratic behavior for the new version: http://dpaste.dzfl.pl/e4b3bc26c3cf (dpaste kills the slow code before it completes the task) Correction: this is the result of removing a uniform call in pull 3921. Since we are supposed to discuss the improvements related to pull 3934 here, I created a separate entry for the issue: https://issues.dlang.org/show_bug.cgi?id=15583. A RNGs don't improve worst case. It only changes an permutation for worst case. --Ilya
Re: topN using a heap
On Monday, 18 January 2016 at 23:49:36 UTC, Andrei Alexandrescu wrote: On 1/18/16 6:44 PM, Ilya wrote: On Monday, 18 January 2016 at 23:27:19 UTC, Ivan Kazmenko wrote: On Monday, 18 January 2016 at 23:18:03 UTC, Ilya wrote: A RNGs don't improve worst case. It only changes an permutation for worst case. --Ilya Still, use of RNG makes it impossible to construct the worst case beforehand, once and for all. In that sense, this is a regression. No, it is definitely possible, because RNGs are Pseudo-RNGs. --Ilya unpredictableSeed uses the system clock as a source of randomness, so we're good there. -- Andrei Would this work for pure functions? --Ilya
Re: topN using a heap
On Monday, 18 January 2016 at 23:39:02 UTC, Andrei Alexandrescu wrote: On 1/18/16 6:18 PM, Ilya wrote: On Monday, 18 January 2016 at 20:45:56 UTC, Ivan Kazmenko wrote: On Monday, 18 January 2016 at 12:00:10 UTC, Ivan Kazmenko wrote: Here goes the test which shows quadratic behavior for the new version: http://dpaste.dzfl.pl/e4b3bc26c3cf (dpaste kills the slow code before it completes the task) Correction: this is the result of removing a uniform call in pull 3921. Since we are supposed to discuss the improvements related to pull 3934 here, I created a separate entry for the issue: https://issues.dlang.org/show_bug.cgi?id=15583. A RNGs don't improve worst case. It only changes an permutation for worst case. --Ilya Well it does improve things. The probability of hitting the worst case repeatedly is practically zero, and it's impossible to create an attack input. I'm not sure whether we should worry about this. Probably we do because sort attacks are a press favorite. The nice thing about not relying on randomness is that pure functions can call sort, topN etc. As a sort of a compromise I was thinking of seeding the RNG with not only the length of the range, but also the integral representation of the address of the first element. This would still allow an attack if the input is always at the same address. Thoughts? Andrei 1. Yes, probability of hitting the worst case repeatedly is is practically zero. But RNGs do not change this probability. 2. It is possible to build attack for our RNGs, because they are Pseudo-RNGs. --Ilya
Re: topN using a heap
On Tuesday, 19 January 2016 at 00:11:40 UTC, Andrei Alexandrescu wrote: On 1/18/16 6:51 PM, Ilya wrote: On Monday, 18 January 2016 at 23:49:36 UTC, Andrei Alexandrescu wrote: On 1/18/16 6:44 PM, Ilya wrote: On Monday, 18 January 2016 at 23:27:19 UTC, Ivan Kazmenko wrote: On Monday, 18 January 2016 at 23:18:03 UTC, Ilya wrote: A RNGs don't improve worst case. It only changes an permutation for worst case. --Ilya Still, use of RNG makes it impossible to construct the worst case beforehand, once and for all. In that sense, this is a regression. No, it is definitely possible, because RNGs are Pseudo-RNGs. --Ilya unpredictableSeed uses the system clock as a source of randomness, so we're good there. -- Andrei Would this work for pure functions? --Ilya Of course not. I think this back-and-forth takes away from the gist of things. So let me summarize what has happened: 1. topN was reportedly slow. It was using a random pivot. I made it use getPivot (deterministic) instead of a random pivot in https://github.com/D-Programming-Language/phobos/pull/3921. getPivot is also what sort uses. 2. Now that both functions use getPivot, I set out to improve it in https://github.com/D-Programming-Language/phobos/pull/3922. The problem is I couldn't make getPivot impure; pure functions already call sort, so making it impure would have been a regression. 3. So I made getPivot use just a bit of randomness taken from the length of the range. 4. sort was and is attackable before all of these changes 5. So now we have pure topN and sort (subject to the range offering pure primitives) but they are both attackable. 6. PRNGs don't have any other downside than making functions impure. The way I see it we have these solutions: (a) make topN still use a random pivot. That way there's no regression. Then proceed and make sort() avoid quadratic cases in other ways, e.g. switch to heapsort if performance degrades. (b) Improve sort() first, then apply a similar strategy to improving topN. Do not use RNGs at all. (c) Seek randomness in other places, e.g. address of elements, data hashes etc. Come with a solution that may still be attacked in narrow cases but make that unlikely enough to be a real concern. Andrei (a) Hope no (b) Yes (c) Memory addresses may not work with user defined ranges and hashes could be slow. (d) Make PRNGs optional. --Ilya
Re: topN using a heap
On Tuesday, 19 January 2016 at 00:38:14 UTC, Timon Gehr wrote: On 01/19/2016 01:12 AM, Ilya wrote: There is already implementation with predictable seed. Proof: https://github.com/D-Programming-Language/phobos/blob/master/std/random.d#L1151 --Ilya The default RNG is seeded with unpredictableSeed. What is the point you are trying to make? unpredictableSeed is initialized only once and can be easily estimated. --Ilya
Re: topN using a heap
On Tuesday, 19 January 2016 at 01:04:03 UTC, Andrei Alexandrescu wrote: On 1/18/16 7:46 PM, Ilya wrote: On Tuesday, 19 January 2016 at 00:38:14 UTC, Timon Gehr wrote: On 01/19/2016 01:12 AM, Ilya wrote: There is already implementation with predictable seed. Proof: https://github.com/D-Programming-Language/phobos/blob/master/std/random.d#L1151 --Ilya The default RNG is seeded with unpredictableSeed. What is the point you are trying to make? unpredictableSeed is initialized only once and can be easily estimated. --Ilya https://github.com/D-Programming-Language/phobos/blob/v2.069.2/std/random.d#L1120 unpredictableSeed uses MinstdRand0. The sources of randomness used for initializing MinstdRand0 are the PID, the thread ID, and the system time at the moment of the seeding. Then at each subsequent call to unpredictableSeed, the time of the call is XORed with the current value of the MinstdRand0. How do you think things could be improved? Andrei A good variant with minimal overhead is to call unpredictableSeed for sorting big arrays each time (one seed per array): a. A hacker would not be able to estimate a seed using other API calls. For example, "give me a set of random numbers". b. A hacker would not be able to estimate a seed using a small arrays because they don't use RNGs. (and they have not any overhead!) c. A hacker would not be able to estimate a seed for big arrays, because attack based on time measurement would not work for big arrays. EDIT: c. ... because each big array has own seed and time measurement would not work for big arrays with different seeds. __EDIT2__: To be fair (c) is not really correct. To do (c) correct a thread local global seed should be used to generate unpredictableSeed for each big arrays. Ilya
Re: topN using a heap
On Monday, 18 January 2016 at 23:55:38 UTC, Timon Gehr wrote: On 01/19/2016 12:51 AM, Ilya wrote: On Monday, 18 January 2016 at 23:49:36 UTC, Andrei Alexandrescu wrote: On 1/18/16 6:44 PM, Ilya wrote: On Monday, 18 January 2016 at 23:27:19 UTC, Ivan Kazmenko wrote: [...] No, it is definitely possible, because RNGs are Pseudo-RNGs. --Ilya unpredictableSeed uses the system clock as a source of randomness, so we're good there. -- Andrei Would this work for pure functions? --Ilya Only if they accept the RNG as an additional argument. Exactly :) So, I hope we would not add Pseudo-RNGs in std.algorithm. --Ilya
Re: topN using a heap
On Monday, 18 January 2016 at 23:53:53 UTC, Timon Gehr wrote: On 01/19/2016 12:50 AM, Ilya wrote: ... 1. Yes, probability of hitting the worst case repeatedly is is practically zero. But RNGs do not change this probability. 2. It is possible to build attack for our RNGs, because they are Pseudo-RNGs. --Ilya You also need to predict the seed. How do you do that? We can not use unpredictable seed (like system clock) in pure functions. --Ilya
Re: Scale-Hierarchy on ndslice
On Wednesday, 13 January 2016 at 20:11:55 UTC, Timothee Cour wrote: On Wed, Jan 13, 2016 at 10:13 AM, jmh530 via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote: On Tuesday, 12 January 2016 at 21:48:39 UTC, Per Nordlöw wrote: Have anybody been thinking about adding a scale-hierarchy structure on top of ndslice? What is a scale-hierarchy structure? https://en.wikipedia.org/wiki/Mipmap but isn't it out of scope? Looks like it would belong more to an image processing library (building on top of ndslice) For example, interpolation algorithms like cubic can be generalized. They can be used in physics. In addition, interpolation can be lazy. https://en.wikipedia.org/wiki/Bicubic_interpolation https://en.wikipedia.org/wiki/Tricubic_interpolation --Ilya
Re: std.experimental.yesnogc
On Thursday, 14 January 2016 at 00:35:53 UTC, Andrei Alexandrescu wrote: Hey folks, I want to push things forward with artifacts dedicated to avoiding the GC, and of course my main worry is finding the right name. An obvious choice is std.experimental.nogc but we know from Marketing 101 that expressing something as a positive is better than a negative. Another possibility is std.experimental.rc, but that's imprecise because the artifacts in there will contain a variety of things in addition to reference counting-related artifacts. Ideas? Andrei std.memory --Ilya
Re: Tutorials section on vibed.org
On Monday, 4 January 2016 at 09:53:45 UTC, Sönke Ludwig wrote: I've just added a sub page on vibed.org to collect links to all existing vibe.d tutorials [1]. If you know of any additional ones, or would like to have an existing one removed, please leave a quick comment: http://forum.rejectedsoftware.com/groups/rejectedsoftware.vibed/thread/29242/ Thanks! [1]: https://vibed.org/tutorials http://d.readthedocs.org/en/latest/examples.html#web-application
Re: sliced().array compatibility with parallel?
On Sunday, 10 January 2016 at 02:43:05 UTC, Jay Norwood wrote: On Sunday, 10 January 2016 at 01:54:18 UTC, Jay Norwood wrote: [...] The parallel time for this case is about a 2x speed-up on my corei5 laptop, debug build in windows32, dmd. [...] I will add significantly faster pairwise summation based on SIMD instructions into the future std.las. --Ilya
Re: shouldn't this throw ? iota(5).sliced(2,2)
On Saturday, 9 January 2016 at 04:15:08 UTC, Timothee Cour wrote: ok good, 'reshape' behaves as it should; but what's the rationale for not throwing on 'iota(5).sliced(2,2) ' ? in light of what i wrote above, it's surprising behavior and will cause bugs. What are the advantages of allowing it to not throw? On Fri, Jan 8, 2016 at 6:09 PM, Ilya via Digitalmars-d < digitalmars-d@puremagic.com> wrote: On Saturday, 9 January 2016 at 00:15:10 UTC, Timothee Cour wrote: I'm not sure I understand your argument. My problem is that iota(5) has 5 elements which is more than 2*2, so I would expect iota(5).sliced(2,2) or iota(7).sliced(2,3).sliced(1,2) to throw, as in pretty much any other tensor library: [...] auto a = iota(5).sliced(5); auto b = a.reshape(2, 2); // throws ReshapeException --ilya Agreed. I will add optional exception (by default sliced will trow). --Ilya
Re: ndslice: allow: a[ _, 9, R(2, $, -3)] as matlab's a[: ,9, reverse(2:3:end)) ]
On Saturday, 9 January 2016 at 00:26:42 UTC, Timothee Cour wrote: i wrote my own tensor library a while ago and it allows the more convenient matlab/python+numpy like syntax: a[ _, 9, R(2, $, -3)] these are allowed: R() or _: full range for an index R(a,b): a..b R(a,b,s): when s>0, iota(a,b,s) R(a,b,s): when s<0, iota(a,b,-s).reverse Could we have this in ndslice? Is `a[ _, 9, R(2, $, -3)]` equals to `a[0..$, 9, 2..$].reversed!2.strided!2(3)` or `a[0..$, 9, 2..$].strided!2(3).reversed!2` ?
Re: shouldn't this throw ? iota(5).sliced(2,2)
On Saturday, 9 January 2016 at 00:15:10 UTC, Timothee Cour wrote: I'm not sure I understand your argument. My problem is that iota(5) has 5 elements which is more than 2*2, so I would expect iota(5).sliced(2,2) or iota(7).sliced(2,3).sliced(1,2) to throw, as in pretty much any other tensor library: [...] auto a = iota(5).sliced(5); auto b = a.reshape(2, 2); // throws ReshapeException --ilya
Re: DMD now does Dwarf style exception handling!
On Saturday, 2 January 2016 at 21:16:38 UTC, Walter Bright wrote: What I've been working on for the last month or so. https://github.com/D-Programming-Language/dmd/pull/5324 For Linux 64 anyway. Anyone who wants to do PRs to extend it to Linux 32, OSX and FreeBSD, feel free! Unfortunately, this is of no help with Win64, which uses its own unique system. DMD for Win32 already can catch C++ exceptions, but Win32 is a dead platform. This is an enabling technology that will confer on D the ability to catch C++ exceptions, which is key to interaction with native C++ libraries. Next up: actually catching C++ exceptions! Thanks to everyone who helped out with this. Is this hack required for dmd 2.070 to get stack-trace on Linux 64? -- import etc.linux.memoryerror; static if (is(typeof(registerMemoryErrorHandler))) registerMemoryErrorHandler(); -- -- Ilya
Re: Using D and std.ndslice as a Numpy Replacement
On Sunday, 3 January 2016 at 23:18:16 UTC, Andrei Alexandrescu
wrote:
On 1/2/16 6:24 PM, Ilya Yaroshenko wrote:
On Saturday, 2 January 2016 at 23:23:38 UTC, Ilya Yaroshenko
wrote:
On Saturday, 2 January 2016 at 19:49:05 UTC, Jack Stouffer
wrote:
http://jackstouffer.com/blog/nd_slice.html
https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/
I just wanted to write to you that dip80-ndslice was moved to
mir
http://code.dlang.org/packages/mir
"dependencies": {
"dip80-ndslice": "~>0.8.7"
},
Ilya
EDIT:
"dependencies": {
"mir": "~>0.9.0-beta"
}
What is the relationship between mir and
std.experimental.ndslice? -- Andrei
1. mir.ndslice is a developer version of std.experimental.ndslice
2. mir can be used with DMD front end >= 2.068, so ndslice can be
used with LDC 0.17.0-alpha. It is important for benchmarks.
3. mir is going to be a testing package for the future std.la
(generic BLAS implementation)
-- Ilya
CyberShadow/DAutoTest — Merge failed
In the same time auto-tester works well (it may be not shown after the latest rebase). https://github.com/D-Programming-Language/phobos/pull/3397
Re: Voting For std.experimental.ndslice
On Wednesday, 30 December 2015 at 22:46:28 UTC, John Colvin wrote: On Wednesday, 30 December 2015 at 21:39:54 UTC, Ilya Yaroshenko wrote: On Tuesday, 29 December 2015 at 18:08:52 UTC, Andrei Alexandrescu wrote: On 12/29/2015 11:28 AM, Robert burner Schadek wrote: [...] Hopefully this is something that you or someone else could help by creating pull requests. Any volunteers? -- Andrei Does it means that the PR can be merged? --Ilya If there's a time constraint, perhaps we could merge it for 2.070 but keep adding documentation updates to both master and release branches? 2.070 would be merged into stable soon.
Re: Phobos and D runtime build are broken.
On Wednesday, 30 December 2015 at 23:54:22 UTC, tsbockman wrote: All pull requests for Phobos and the D runtime are failing the auto tester now: Exception - file 'lexer.h': lexer.h: No such file or directory Exception - file 'parse.h': parse.h: No such file or directory Error - file 'doc.d' contains trailing whitespace at line 2526 Exception - file 'gluestub.d': gluestub.d: No such file or directory https://github.com/D-Programming-Language/dmd/pull/5327/files
Re: Voting For std.experimental.ndslice
On Wednesday, 30 December 2015 at 01:03:39 UTC, Ilya Yaroshenko wrote: On Wednesday, 30 December 2015 at 00:24:38 UTC, Ilya Yaroshenko wrote: On Tuesday, 29 December 2015 at 21:19:19 UTC, Jack Stouffer wrote: On Tuesday, 29 December 2015 at 17:38:06 UTC, Ilya Yaroshenko wrote: On Tuesday, 29 December 2015 at 17:17:05 UTC, Jack Stouffer wrote: [...] Thank you for doing this! First draft: http://jackstouffer.com/hidden/nd_slice.html Please critique. Awesome! Please find my notes below. [...] - You may want to test both DMD and LDC 0.17.0 alpha1 for bechmarks https://github.com/ldc-developers/ldc/releases/tag/v0.17.0-alpha1 LDC 0.17.0 alpha1 works well with dip80-ndslice v0.8.7. [...] Ilya ldmd2/ldc2 flag -mcpu=native will optimise code for your CPU. -- Ilya The paragraph "This function does not calculate border cases in which a window overlaps the image partially. However, the function can still be used to carry out such calculations. That can be done by creating an amplified image, with the edges reflected from the original image, and then applying the given function to the new file. Note:" ends with "Note:". --Ilya
pragma(inline)
Hi, Does `pragma(inline, true)` force DMD compiler to inline function when `-inline` was _not_ defined? I am failing to get a good disassembled code with obj2asm/otool :-( Best, Ilya
Re: ndslice and limits of debug info and autocompletion
On Tuesday, 22 December 2015 at 00:21:16 UTC, Jay Norwood wrote:
I'm trying to determine if the debugger autocompletion would be
useful in combination with ndslice. I find that using visualD
I get offered no completion to select core_ctr or epu_ctr where
epu_ctr is used in the writeln below.
I take it this either means that there is some basic limitation
in the debug info, or else VisualD just punts after some number
of array subscripts.
The code builds and executes correctly ... but I was hoping the
debugger completion would help out with an exploratory mode
using compiled code.
import std.stdio;
import std.experimental.ndslice;
import std.experimental.ndslice.iteration: transposed;
struct sample{
ulong [10] core_ctr;
ulong [32] epu_ctr;
}
void main() {
auto a1 = new sample[60];
auto t3 = a1.sliced!(ReplaceArrayWithPointer.no)(3,4,5);
writeln(t3[0][0][0].epu_ctr);
}
I don't know how VisualD works, but to provide auto-completion it
should be able to compile a source code for autocompletion
information, because simple analyzer would not work with variadic
templates like opIndex(Indexes...)(Indexes indexes).
Nitpick: t3[0][0][0] is much slower than t3[0, 0, 0].
Best,
Ilya
Re: Formal Review of std.range.ndslice
On Friday, 11 December 2015 at 19:31:14 UTC, Stefan Frijters
wrote:
Today I've made an abortive attempt at replacing my code's [1]
dependence on unstd.multidimarray [2] with ndslice.
I'm guessing it's just me being stupid, but could anyone supply
with some hints on how to do the conversion with a minimum of
fuss?
Basically I have an N-dimensional array (N is known at compile
time) of type T wrapped in a struct to carry some additional
information.
I then address it using size_t[N] fixed-size arrays, and loop
over it a lot with foreach, which also uses size_t[N] as index.
So it looks something like this:
struct Field(T, uint N) {
alias arr this;
MultidimArray!(T, N) arr; // is there any way to supply the
correct type here with ndslice? I cannot use auto, right?
Slice!(N, T*) arr;
this (in size_t[N] lengths) {
arr = multidimArray!T(lengths);
// compute length
// more flexible construtors would be added after
// allocatrs support for ndslice
size_t len = 1;
foreach(l; lengths)
len *= l;
arr = new T[len].sliced(lengths);
}
}
and then things like
foreach(immutable p, ref pop; someField) {
pop = foo(someOtherField[bar(p)]);
...
}
std.experimental.ndslice.selection: indexSlice, byElement;
foreach(p; someField.shape.indexSlice.byElement) {
someField[p] = foo(someOtherField[bar(p)]);
...
}
where p is of type size_t[N].
I tried using ndarray in conjunction with the
std.experimental.allocator, but I don't particularly care about
memory management;
these are large arrays that are allocated once and kept around
for the duration of the program.
Any help would be appreciated.
[1] https://github.com/SFrijters/DLBC
[2] https://bitbucket.org/SFrijters/unstandard
See also updated docs:
http://dtest.thecybershadow.net/artifact/website-13cbdcf17d84fc31328c3f517a56bea783c418d6-d9c63e815273f0906309088334e7dfb1/web/phobos-prerelease/std_experimental_ndslice.html
Ilya
Re: Formal Review of std.range.ndslice
On Friday, 11 December 2015 at 22:56:15 UTC, Ilya wrote:
On Friday, 11 December 2015 at 19:31:14 UTC, Stefan Frijters
wrote:
Today I've made an abortive attempt at replacing my code's [1]
dependence on unstd.multidimarray [2] with ndslice.
I'm guessing it's just me being stupid, but could anyone
supply with some hints on how to do the conversion with a
minimum of fuss?
Basically I have an N-dimensional array (N is known at compile
time) of type T wrapped in a struct to carry some additional
information.
I then address it using size_t[N] fixed-size arrays, and loop
over it a lot with foreach, which also uses size_t[N] as index.
So it looks something like this:
struct Field(T, uint N) {
alias arr this;
MultidimArray!(T, N) arr; // is there any way to supply the
correct type here with ndslice? I cannot use auto, right?
Slice!(N, T*) arr;
this (in size_t[N] lengths) {
arr = multidimArray!T(lengths);
// compute length
// more flexible construtors would be added after
// allocatrs support for ndslice
size_t len = 1;
foreach(l; lengths)
len *= l;
arr = new T[len].sliced(lengths);
}
}
and then things like
foreach(immutable p, ref pop; someField) {
pop = foo(someOtherField[bar(p)]);
...
}
std.experimental.ndslice.selection: indexSlice, byElement;
foreach(p; someField.shape.indexSlice.byElement) {
someField[p] = foo(someOtherField[bar(p)]);
...
}
faster version:
std.experimental.ndslice.selection: byElement;
for(auto r = someField.arr.byEleemnt; r.popFront) {
r.front = foo(someOtherField[bar(r.index)]);
...
}
where p is of type size_t[N].
Ilya
Re: MurmurHash3
On Friday, 11 December 2015 at 22:43:00 UTC, Guillaume Chatelet wrote: On Friday, 11 December 2015 at 01:51:09 UTC, Ilya wrote: http://dpaste.dzfl.pl/1b94ed0aa96e#line-222 - seed is uint, can it be ulong? Done Mutmur hash has three stages: 1. Computation of hash for blocks (32bit or 128bit) 2. Compitation of hash for tail (remainder) 3. Finalization. I will be very happy, if step 1 will be represented as an output range. Then it can be used directly like reduce aggregator for ranges and multidimensional slices. Done Not thoroughly tested but updated for range and taking an ulong seed for MurmurHash3_x64_128: http://dpaste.dzfl.pl/1b94ed0aa96e Not sure I got what you meant about the optimized version. For the return value ? I haven't done any benchmarking yet. Current version is suitable for arrays but not ranges or types. Few examples: 1. Compute hash of ulong. 2. Compute hash of all elements in matrix column (element are in different arrays). I have created output range API draft http://dpaste.dzfl.pl/a24050042758 Ilya
Re: MurmurHash3
On Thursday, 10 December 2015 at 22:25:21 UTC, Guillaume Chatelet wrote: Here is an implementation of MurmurHash [1] for D. http://dpaste.dzfl.pl/1b94ed0aa96e I'll do a proper pull request later on for addition to std.digest if the community feels like it's a valuable addition. Guillaume -- 1 - https://en.wikipedia.org/wiki/MurmurHash http://dpaste.dzfl.pl/1b94ed0aa96e#line-222 - seed is uint, can it be ulong? Mutmur hash has three stages: 1. Computation of hash for blocks (32bit or 128bit) 2. Compitation of hash for tail (remainder) 3. Finalization. I will be very happy, if step 1 will be represented as an output range. Then it can be used directly like reduce aggregator for ranges and multidimensional slices.
Re: MurmurHash3
On Thursday, 10 December 2015 at 22:25:21 UTC, Guillaume Chatelet wrote: Here is an implementation of MurmurHash [1] for D. http://dpaste.dzfl.pl/1b94ed0aa96e I'll do a proper pull request later on for addition to std.digest if the community feels like it's a valuable addition. Guillaume -- 1 - https://en.wikipedia.org/wiki/MurmurHash Great! Could you please add an optimized interface to compute hashes for `uint` , `ulong` and `ulong[2]`? It would very good both for upcoming containers and multidimensional slices https://github.com/D-Programming-Language/phobos/pull/3397 .
Re: Release D 2.069.2
On Thursday, 3 December 2015 at 03:36:10 UTC, Martin Nowak wrote: Glad to announce D 2.069.2. http://dlang.org/download.html This point release fixes a few issues over 2.069.1, see the changelog for more details. http://dlang.org/changelog/2.069.2.html -Martin I am planing to open voting on ndslice package next week. Hope to get it in 2.070
Compile time strings auto concatenation!?
Can DMD frontend optimize string concatenation ``` enum Double(S) = S ~ S; assert(condition, "Text " ~ Double!"+" ~ ___FUNCTION__); ``` to ``` assert(condition, "Text ++_function_name_"); ``` ?
Re: Formal Review of std.range.ndslice
On Friday, 20 November 2015 at 19:21:45 UTC, Jack Stouffer wrote: On Wednesday, 18 November 2015 at 10:26:13 UTC, Ilya Yaroshenko wrote: On Wednesday, 18 November 2015 at 08:52:11 UTC, Robert burner Schadek wrote: [...] I have added makeSlice, however this function can be easily implemented by user: allocate array -> pass array to `sliced`. updated docs http://dtest.thecybershadow.net/artifact/website-7a646fdea76569e009844cdee5c93edab10980ca-ce47155797f387348826317811c4af0c/web/phobos-prerelease/std_experimental_range_ndslice.html --Ilya I spoke with Andrei over email and his opinion was that if this module makes any allocation that can't be avoided, then std.experimental.allocator should be integrated. Great! Added to PR's TODO list :-)
Re: Formal Review of std.range.ndslice
On Friday, 20 November 2015 at 21:23:40 UTC, Jack Stouffer wrote: On Friday, 20 November 2015 at 20:33:27 UTC, Ilya wrote: Great! Added to PR's TODO list :-) This will be interesting because there is no defined idiomatic usage of std.allocator. I imagine std.range.ndslice will be the trend-setter for Phobos in this regard. Slice is only a shell over a range/array. There is no reason to add any kind of counters or special stuff entities. All this idioms work out of the box because Slice holds an array. It is already works with std.container.array, and it will work with the future Array container based on allocators. -- Ilya
Re: range.save
On Friday, 20 November 2015 at 02:47:17 UTC, Rikki Cattermole wrote: On 20/11/15 10:30 AM, Freddy wrote: Does anyone else think range.save is a hack? I often find myself forgetting to call range.save in my generic code with my unittests working fine. Also, working with a range of ranges may forget to call range.save.(Ex: [1,2,4].repeat) Not really. If you forget it, then things won't work since it'll be empty ;) If range is class. For most of structs like Iota range.save returns just itself.
Re: Formal Review of std.range.ndslice
On Wednesday, 18 November 2015 at 18:40:40 UTC, Jack Stouffer wrote: On Wednesday, 18 November 2015 at 08:52:11 UTC, Robert burner Schadek wrote: [...] I know it's bad practice for regular Phobos code to import from std.experimental, but what's the protocol for code inside std.experimental importing other code in std.experimental? It would be nice to get Andrei's opinion on this. Another problem with using std.experimental.allocator is it ties the time table of this going into stable with std.allocator. Worse case scenario, it could be a year before std.allocator is stable, where as std.range.ndslice is much simpler code and could be moved much faster. Possible solutions: 1. Mark `makeSlice` with red "Experimental" 2. Move `makeSlice` to unittest, so user could copy-past it.
Re: Wanted: Review manager for std.experimental.range.ndslice
On Tuesday, 17 November 2015 at 13:45:38 UTC, Andrea Fontana wrote: On Monday, 16 November 2015 at 19:17:34 UTC, Ilya Yaroshenko wrote: w[1..$, 1..$].strided!0(3).strided!1(3)[] = m; Impressive :) Another way: w[1..2, 1..2] = m; --Ilya
Re: Wanted: Review manager for std.experimental.range.ndslice
On Monday, 16 November 2015 at 19:56:01 UTC, Jack Stouffer wrote: On Monday, 16 November 2015 at 03:05:03 UTC, Ilya Yaroshenko wrote: Hello, Review manager for N-dimensional ranges is wanted What does the job entail? It is very close draft: http://wiki.dlang.org/Review/Process
Re: Please vote for the DConf logo
3
