Re: Optional equivalent in D?
On Friday, November 15, 2013 23:39:38 Jacek Furmankiewicz wrote: > Many other languages are starting to frown on returning null > values from methods (due to NullPointerException risks, etc) and > wrapping them instead in an Optional like in > > Scala: > http://blog.danielwellman.com/2008/03/using-scalas-op.html > > Google Guava for Java: (now rolled into the base JDK for Java 8): > https://code.google.com/p/guava-libraries/wiki/UsingAndAvoidingNullExplained I really don't understand this. Optional is one of the most useless ideas that I've ever seen in Java. Just use null. It's built into the language. It works just fine. And wrapping it in a class isn't going to make it go away. Just learn to deal with null properly. I've known many programmers who have no problems with null whatsoever, and I'd be worried about any programmer who is so scared of it that they feel the need to wrap nullable objects in another type which has its own concept of null. The only types which aren't nullable in Java are the primitive types, and if you use them in generics (like Optional does), then you get a class that boxes the primitive type rather than using the primitive type directly, and that object is of course nullable. So, you might as well just use the class that boxes the primitive type directly and set its reference to null when you need it to be null. And Optional doesn't even protect against null, since it's perfectly possible to make its contents null. So, as far as I can see, Optional is utterly pointless. IMHO, it's outright bad software design. > Is there a similar approach in D? Or maybe an equivalent is in a > commonly used external library? We have std.typecons.Nullable. However, it's real value is in making it possible to have value types be null. I very much hope that no one is using it like Optional to set nullable types to null without actually using null. The _only_ nullable type in D that I would consider reasonable to use with Nullable would be arrays, and that's because of how null arrays aren't properly distinguishable from empty arrays, making using "is null" with them a bit iffy. - Jonathan M Davis
Re: Efficient string concatenation?
Thanks for the book! I printed it, all 673 pages of it. Immense work you have there.
Re: Efficient string concatenation?
On 11/15/2013 02:35 PM, Jacek Furmankiewicz wrote: > "Programming in D" PDF and he did not show this in his initial chapter on > Strings. Sorry about that. :) As I was targeting novices to programming, I tried to give as much as needed but as little as possible, so that the reader would not be overwhelmed. (I have to admit that the chapters about arrays, strings, and slices can be arranged in a better way.) In doing so, some important concepts have either been totally forgotten :p or appeared too late in the book. I just checked, Appender is all the way down in the Ranges chapter and not even as a proper section! Oops! :) Ali
Re: Optional equivalent in D?
Justin Whear: No, Nullable adds a potential null state to value types, e.g. it allows you to null an int. In std.typecons there is another version of Nullable, that uses a state as "null". So you can use it as nullable class reference. Is that good enough for the OP? Bye, bearophile
Re: Optional equivalent in D?
On Fri, 15 Nov 2013 23:41:38 +0100, Brad Anderson wrote: > On Friday, 15 November 2013 at 22:39:40 UTC, Jacek Furmankiewicz wrote: >> Many other languages are starting to frown on returning null values >> from methods (due to NullPointerException risks, etc) >> and wrapping them instead in an Optional like in >> >> Scala: >> http://blog.danielwellman.com/2008/03/using-scalas-op.html >> >> Google Guava for Java: (now rolled into the base JDK for Java 8): >> https://code.google.com/p/guava-libraries/wiki/ UsingAndAvoidingNullExplained >> >> Is there a similar approach in D? Or maybe an equivalent is in a >> commonly used external library? > > Sounds like std.typecons.Nullable to me. No, Nullable adds a potential null state to value types, e.g. it allows you to null an int. To the best of my knowledge there is no Maybe/ Optional type in Phobos, but they're extremely easy to implement yourself. Here's a Maybe type that I just threw together: http:// dpaste.dzfl.pl/e4b762ed
Re: Efficient string concatenation?
On Friday, 15 November 2013 at 22:35:48 UTC, Jacek Furmankiewicz wrote: I am learning D by going through Ali Cehreli's otherwise excellent "Programming in D" PDF and he did not show this in his initial chapter on Strings. Well, Appender is not string specific. D feels like being in a different league in terms of generic programming. Many stdlib stuff is implemented more abstract and generic, which makes it harder to find. Looking for some string operations, you might find it in std.algorithm, std.array, std.range, or std.string. Maybe it is different to C++ programmer, who are used to this from the STL.
Re: Optional equivalent in D?
On Friday, 15 November 2013 at 22:41:40 UTC, Brad Anderson wrote: On Friday, 15 November 2013 at 22:39:40 UTC, Jacek Furmankiewicz wrote: Many other languages are starting to frown on returning null values from methods (due to NullPointerException risks, etc) and wrapping them instead in an Optional like in Scala: http://blog.danielwellman.com/2008/03/using-scalas-op.html Google Guava for Java: (now rolled into the base JDK for Java 8): https://code.google.com/p/guava-libraries/wiki/UsingAndAvoidingNullExplained Is there a similar approach in D? Or maybe an equivalent is in a commonly used external library? Sounds like std.typecons.Nullable to me. I recommend having a look around std.typecons while you are in there. It's got a lot of gems that seem to go unnoticed by a lot of people.
Re: Optional equivalent in D?
Thanks! std.typecons definitely looks like something I need to dig into.
Re: Optional equivalent in D?
On Friday, 15 November 2013 at 22:39:40 UTC, Jacek Furmankiewicz wrote: Many other languages are starting to frown on returning null values from methods (due to NullPointerException risks, etc) and wrapping them instead in an Optional like in Scala: http://blog.danielwellman.com/2008/03/using-scalas-op.html Google Guava for Java: (now rolled into the base JDK for Java 8): https://code.google.com/p/guava-libraries/wiki/UsingAndAvoidingNullExplained Is there a similar approach in D? Or maybe an equivalent is in a commonly used external library? Sounds like std.typecons.Nullable to me.
Re: Efficient string concatenation?
Thank you all. I am learning D by going through Ali Cehreli's otherwise excellent "Programming in D" PDF and he did not show this in his initial chapter on Strings.
Optional equivalent in D?
Many other languages are starting to frown on returning null values from methods (due to NullPointerException risks, etc) and wrapping them instead in an Optional like in Scala: http://blog.danielwellman.com/2008/03/using-scalas-op.html Google Guava for Java: (now rolled into the base JDK for Java 8): https://code.google.com/p/guava-libraries/wiki/UsingAndAvoidingNullExplained Is there a similar approach in D? Or maybe an equivalent is in a commonly used external library?
Re: Efficient string concatenation?
On Fri, 15 Nov 2013 22:30:35 +, Justin Whear wrote: > std.array has an Appender type that can be used to build up a string (or > any other array type) efficiently. Oh, and if you have an idea of how large the result might grow, be sure to use the reserve() method on the appender.
Re: Efficient string concatenation?
On Friday, 15 November 2013 at 22:26:20 UTC, Jacek Furmankiewicz wrote: Since D strings are immutable (like in most other languages), string concatenation is usually pretty inefficient due to the need to create a new copy of the string every time. I presume string concatenation using the typical array syntax can be optimized by the compiler to do all of this in one shot, e..g string newString = string1 ~ string2 ~ string3; but what happens if I need to concatenante a large string in a loop? I tried looking through Phobos for a StringBuilder class (since that is the common solution in Java and C#), but did not find anything similar. What is the D way of doing efficient string concatenation (especially if it spans multiple statements, e.g. while in a loop)? Appender in std.array is probably what you are looking for. std.algorithm.joiner is also useful (no allocations at all even) but the use case is a bit different.
Re: Efficient string concatenation?
On Fri, 15 Nov 2013 23:26:19 +0100, Jacek Furmankiewicz wrote:
> Since D strings are immutable (like in most other languages), string
> concatenation is usually pretty inefficient due to the need to create a
> new copy of the string every time.
>
> I presume string concatenation using the typical array syntax can be
> optimized by the compiler to do all of this in one shot, e..g
>
> string newString = string1 ~ string2 ~ string3;
>
> but what happens if I need to concatenante a large string in a loop?
>
> I tried looking through Phobos for a StringBuilder class (since that is
> the common solution in Java and C#), but did not find anything similar.
>
> What is the D way of doing efficient string concatenation (especially if
> it spans multiple statements, e.g. while in a loop)?
std.array has an Appender type that can be used to build up a string (or
any other array type) efficiently. E.g.:
auto strBuilder = appender!string;
while (...)
{
str.put("foo");
}
// Get the array out:
string str = strBuilder.data;
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
Sohow does Facebook handle it with their new D code? No GC at all, explicit memory management?
Efficient string concatenation?
Since D strings are immutable (like in most other languages), string concatenation is usually pretty inefficient due to the need to create a new copy of the string every time. I presume string concatenation using the typical array syntax can be optimized by the compiler to do all of this in one shot, e..g string newString = string1 ~ string2 ~ string3; but what happens if I need to concatenante a large string in a loop? I tried looking through Phobos for a StringBuilder class (since that is the common solution in Java and C#), but did not find anything similar. What is the D way of doing efficient string concatenation (especially if it spans multiple statements, e.g. while in a loop)?
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
On Thursday, 14 November 2013 at 23:10:58 UTC, Jacek Furmankiewicz wrote: While looking a D, I am just trying to focus on the parts which I know would be a showstopper for us on day one...and this particular issue is it. Yes, I also think for long-running memory-hungry server-stuff the current conservative GC is a show stopper. Some people are working on a concurrent and a precise GC. Then parallel and concurrent and incremental and generational and whatnot GCs will be explored. The point where it gets interesting for these server-apps is not clear though. Regarding the GC, I've seen some slides on DConf about other garbage collectors available. Is there any resource/tutorial that shows how you can swap out the default GC for those alternative implementations? As far as I know those other GCs are not ready for prime time yet. For sure, there is no other GC shipped with the current D 2.064.2. Oh, and keep nagging! We need to hear about showstoppers, so we can fix them! ;)
Re: Best data structure for a particle system?
On Friday, 15 November 2013 at 14:01:36 UTC, Chris Cain wrote: On Friday, 15 November 2013 at 13:32:38 UTC, Mikko Ronkainen wrote: Ok, thanks! That linked list cache thrashing was just the thing I knew that I don't know :) Let's say I just use dynamic array and grow it when adding new particles to the system, and when particles die, just set their dead flag on. This means that, when adding a new particle, I need to scan the array first for possible dead particles that can be reused. Is there some trick that could be used here to reduce excessive searching/iteration? Instead of having a "dead flag", you could swap ( http://dlang.org/phobos/std_algorithm.html#swap ) the dying particle with the last particle in the list and then decrement the list's length. This swapping might even speed up the normal particle processing, because with a mix of dead and alive particles the flag checking could confuse the branch predictor. http://stackoverflow.com/questions/11227809/why-is-processing-a-sorted-array-faster-than-an-unsorted-array Ultimately, if flag or swap is faster must be measured. Profiling and bencharking are your friends, if you want to optimize your particle system. For a start use an array and either method.
Re: Best data structure for a particle system?
Am Fri, 15 Nov 2013 21:56:15 +0900 schrieb Mike Parker : > On 11/15/2013 9:19 PM, Mikko Ronkainen wrote: > > > > If relevant, doubly-linked might work better? dcollections LinkList, or > > maybe DList? I'm asking mostly because I'd like the container to avoid > > memory allocations while in use. > > For a particle system, I would avoid lists. A list of particles needs to > be iterated every frame. A linked list is going to kill you as the > particle count increases. Since the particles will most like not be > contiguous in memory, you'll be thrashing your cache to hell and back. > > "Caches love linear access" is a quote to remember. When you need to do > frequent iteration of a list, your cache will love you if all of the > items are in a block of contiguous memory. So it's almost always better > to use an array for this sort of thing. Make your particle object a > struct and use a dynamic array of particles that you can grow as needed > or, if it makes sense, a fixed size static array. The point is that > arrays of structs will be lined up one next to the other in memory so > that you can make good use of the cache. Of course, the size of your > particle object also has a role to play here. > > Google for "linked list cache thrashing" or "cache-friendly data > structures" or some such for ideas. It is true, that these days you optimize for cache locality more than for anything else. How about this: - use an array - keep alive particles at the front and dead particles at the back - when an alive particle dies, you swap it with the last alive particle in the array and mark it as dead This way you always have a compact array from 0..aliveCount and don't need if-else to check for dead ones. (Which is otherwise another slowdown factor). -- Marco
Thrift maintained..?
I thrid compiling thrift 0.9.1 from github with d support, but there's a bug in the makefile it seems. $(addprefix.log: $(addprefix @p='$(addprefix'; \ b='$(addprefix'; \ $(am__check_pre) $(LOG_DRIVER) --test-name "$$f" \ --log-file $$b.log --trs-file $$b.trs \ $(am__common_driver_flags) $(AM_LOG_DRIVER_FLAGS) $(LOG_DRIVER_FLAGS) -- $(LOG_COMPILE) \ "$$tst" $(AM_TESTS_FD_REDIRECT) The error is reported at the first line with the message: Makefile:1206: *** unterminated variable reference. Stop. Can anyone spot the error? (And preferably send a pull request so I don't take the credit :) )
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
On Thursday, 14 November 2013 at 17:36:09 UTC, Jacek Furmankiewicz wrote: Could anyone point me to what would be the closest D equivalents (maybe in an external library if not part of Phobos) so we can playing around with them? Much appreciated Jacek In such cases the easiest route is to find some C/C++ library for such tasks, make a C interface in the latter case, and link with it. That would require a bit of extra work but much less than writing your own performant implementation from scratch. E.g. I once wrote a simple wrapper for the Kyoto Cabinet key-value store: https://github.com/lomereiter/kyoto-d/blob/master/kyotocabinet.d
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
Thank you Russell for the explanation. Always a chance to learn something new.
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
On Fri, 2013-11-15 at 20:10 +0100, Jacek Furmankiewicz wrote: > if I recall from the initial Go discussion, the Go folks were > saying that for close to realtime SLAs the goroutine/channel > approach may have some scalability limits...which is why they > started recommending the RW mutex approach in the end. > > Now, that was a few months ago, since then Go 1.1 (and soon 1.2) > came out, so that may be a false statement at this time. I guess they were hinting at scheduling issues where there are many more goroutines than kernel threads available. Not a problem for Web services and applications but a potential problem for hard real-time. I don't think 12.0, 1.1, 1.2,… will change the core issue – though it will change the code generation, which is getting better. Though gccgo already produces very efficient code, much faster execution than the main Go system. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Re: Why can't I copy a const struct?
On 11/15/2013 10:37 AM, Ali Çehreli wrote:
> > this(this) {
> > a = a.dup;
>
> That line can work only if a is mutable. The trouble is, the type of a
> is const(int[]) there.
I lied! :) The type of a is int[] in there. The actual trouble is, to be
able to start executing this(this), the left-hand side a should have
already been a reference to the right-hand side a.
What fails is that initialization of the mutable reference on the
left-hand side from the const reference on the right-hand side.
Ali
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
if I recall from the initial Go discussion, the Go folks were saying that for close to realtime SLAs the goroutine/channel approach may have some scalability limits...which is why they started recommending the RW mutex approach in the end. Now, that was a few months ago, since then Go 1.1 (and soon 1.2) came out, so that may be a false statement at this time.
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
On Friday, 15 November 2013 at 17:46:41 UTC, Russel Winder wrote: If D programmers are being told to use locks in applications code, then the D programming model and library are failing. Or the advice is wrong ;-) I don't really buy it. It is good from simplicity/safety point of view (just use library stuff and your code is thread-safe) but not performance. Back in C++ days we have almost always resorted to writing own concurrent data structures to abuse domain specifics and application architecture as much as possible and thus minimize actual concurrent locking frequency. And most of those solutions were completely unsuitable as generic ones.
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
On Fri, 2013-11-15 at 18:55 +0100, Jacek Furmankiewicz wrote: > Yes, that is what they say in Go...but it doesn't scale either. > :-) I don't follow. CSP scales very well and Go implements CSP. (Well an updated version from Hoare's 1978 CSP.) > I had the exact same discussion on the Go forums a while back and > the conclusion was basically the same...roll your own maps with > RW locks: > > https://groups.google.com/forum/?fromgroups#!searchin/golang-nuts/furmankiewicz/golang-nuts/jjjvXG4HdUw/ffWytKQ7X9YJ > > But...at the end someone actually built lock-free data structures > in Go out of this: > > https://github.com/zond/gotomic This is, of course, how ConcurrentHashMap arrived in Java, Java didn't have a shared access, thread safe map so someone created it. Go has no shared access, thread safe map and no-one has created one that is in the standard library. Of course Java is a shared-memory multithreading language whereas Go is a CSP one, so the idea of a shared access memory safe data structure is actually anathema. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Re: Why can't I copy a const struct?
The short answer to the question in the subject line is because D does
not have copy constructors.
On 11/15/2013 07:12 AM, Atila Neves wrote:
> private struct DummyStruct {
> int[] a;
>
> this(this) {
> a = a.dup;
That line can work only if a is mutable. The trouble is, the type of a
is const(int[]) there.
> }
> }
>
>
> void main() {
> const dummy1 = DummyStruct();
> DummyStruct dummy2 = dummy1;
> }
>
> struct_copy.d(12): Error: conversion error from const(DummyStruct)
> to DummyStruct
>
> Surely I should be able to copy it and use the mutable version as I
> please, no? I'd understand if the postblit constructor wasn't defined...
That makes sense but the compiler would not analyze the code and ensure
that the code obeys const. (It probably can in most situations though.)
> Atila
One workaround is a constructor that takes by ref const:
import std.stdio;
private struct DummyStruct {
int[] a;
this (int[] a) {
this.a = a.dup;
}
this(ref const(DummyStruct) that) {
writeln("copy");
this.a = that.a.dup;
}
}
void main() {
const dummy1 = DummyStruct([ 1 ]);
auto dummy2 = DummyStruct(dummy1); // <-- note different syntax
assert(dummy2.a.ptr != dummy1.a.ptr);
}
Ali
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
On Fri, 2013-11-15 at 19:05 +0100, ilya-stromberg wrote: > On Friday, 15 November 2013 at 17:46:41 UTC, Russel Winder wrote: > > If D programmers are being told to use locks in applications > > code, then > > the D programming model and library are failing. Or the advice > > is > > wrong ;-) > > It's possible to implement lock-free data structures in D, you > can use core.atomic > http://dlang.org/phobos/core_atomic.html > > But it's REALLY difficult to implement and it can be SLOWER than > Mutex version (not only in D, it depends from usage situation). I didn't intend to imply that core data structures had to be lock free, it is clear that creators of thread and process safe data structures should be free to use locks if it makes things faster and more efficient. My point was about applications built on the language platform: the platform should provide all the things needed so that applications code never mention locks. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
On Friday, 15 November 2013 at 18:16:17 UTC, Jacek Furmankiewicz wrote: taskPool looks like the closest equivalent in D that I could find. Yes, that's sad truth: if you want to use D, be ready make something yourself. BTW, why did you decide to migrate to D? Any problems with Java?
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
No, we didn't decide to migrate to D. Java is working out fine for us. I am however always interested in what is out there, 'cause you never know if there may not be a better solution. And from what I've seen so far I really like D in terms of pure language features. Go is cool too, but it has made some choices which to me are questionable (error codes instead of exceptions, lack of templates/generics). Coupled with vibe.d, dub, etc. I see some really interesting stuff going on in the D community that seems to have been greatly under the radar. Definitely plan to spend more time with D on my own, even if I cannot use it at work.
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
On Friday, 15 November 2013 at 17:46:41 UTC, Russel Winder wrote: The trend in the JVM-verse is very much "if you use synchronized or an explicit lock, and you are not creating a core library data structure, you are doing it wrong". The background is that the whole purpose of a lock it to control concurrency and thus stop parallelism. Applications programmers should never have to use a lock. ConcurrentHashMap, and thread safe queues are two consequences of all this. True, concurrency in Java is really simple these days (especially with the Executors framework that Python 3 pretty much copies verbatim). taskPool looks like the closest equivalent in D that I could find.
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
On Friday, 15 November 2013 at 17:46:41 UTC, Russel Winder wrote: If D programmers are being told to use locks in applications code, then the D programming model and library are failing. Or the advice is wrong ;-) It's possible to implement lock-free data structures in D, you can use core.atomic http://dlang.org/phobos/core_atomic.html But it's REALLY difficult to implement and it can be SLOWER than Mutex version (not only in D, it depends from usage situation).
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
Yes, that is what they say in Go...but it doesn't scale either. :-) I had the exact same discussion on the Go forums a while back and the conclusion was basically the same...roll your own maps with RW locks: https://groups.google.com/forum/?fromgroups#!searchin/golang-nuts/furmankiewicz/golang-nuts/jjjvXG4HdUw/ffWytKQ7X9YJ But...at the end someone actually built lock-free data structures in Go out of this: https://github.com/zond/gotomic
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
On Fri, 2013-11-15 at 18:03 +0100, ilya-stromberg wrote: > On Friday, 15 November 2013 at 16:36:56 UTC, Jacek Furmankiewicz > wrote: > > How can you achieve lock-free reads with the synchronized MyMap > > approach? > > In this case you can use Readers-writer lock > http://en.wikipedia.org/wiki/Readers%E2%80%93writer_lock > > It allows multiple reads and single write. I think that the > easiest way is use OS spesific function, for example > `pthread_rwlock_t` for POSX. Note that D supports C ABI, so you > can call any C function from D. > > I don't know any D implementation of Readers-writer lock, but you > can ask this question - maybe it already exist. Sorry to come in late on this one (and miss everything that comes before). The trend in the JVM-verse is very much "if you use synchronized or an explicit lock, and you are not creating a core library data structure, you are doing it wrong". The background is that the whole purpose of a lock it to control concurrency and thus stop parallelism. Applications programmers should never have to use a lock. ConcurrentHashMap, and thread safe queues are two consequences of all this. In the Go-verse the attitude is basically the same, you should use channels and communications – the synchronization is managed by the data structure. If D programmers are being told to use locks in applications code, then the D programming model and library are failing. Or the advice is wrong ;-) -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
On Friday, 15 November 2013 at 17:09:54 UTC, Dicebot wrote: On Friday, 15 November 2013 at 17:03:15 UTC, ilya-stromberg wrote: I don't know any D implementation of Readers-writer lock, but you can ask this question - maybe it already exist. http://dlang.org/phobos/core_sync_rwmutex.html Thank you. I just never use it.
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
On Friday, 15 November 2013 at 17:03:15 UTC, ilya-stromberg wrote: I don't know any D implementation of Readers-writer lock, but you can ask this question - maybe it already exist. http://dlang.org/phobos/core_sync_rwmutex.html
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
On Friday, 15 November 2013 at 16:36:56 UTC, Jacek Furmankiewicz wrote: How can you achieve lock-free reads with the synchronized MyMap approach? In this case you can use Readers-writer lock http://en.wikipedia.org/wiki/Readers%E2%80%93writer_lock It allows multiple reads and single write. I think that the easiest way is use OS spesific function, for example `pthread_rwlock_t` for POSX. Note that D supports C ABI, so you can call any C function from D. I don't know any D implementation of Readers-writer lock, but you can ask this question - maybe it already exist.
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
So, if you add a read() method to MyMap for those threads, would that be synchronized as well? That is what we would not want due performance impact. How can you achieve lock-free reads with the synchronized MyMap approach?
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
On Friday, 15 November 2013 at 15:21:59 UTC, Jacek Furmankiewicz wrote: So what happens when the "write" operation is doing map[1] = map[1].editData(5); and at the same time 50 threads are simultaneously reading the value in map[1]?. Is that reassignment operation thread safe? Or would I get corrupted reads with potentially a partially overriden value? Jacek Yes, this is thread safe. Put attention to the `MyMap` class definition, it's marked as `synchronized`. It means that all class functions use the same Mutex. So, "write" operation will block map and all 50 threads will wait.
Re: Compiling an app to a single binary - possible?
Thank you for the quick response, that is great news. Cheers Jacek
Re: Compiling an app to a single binary - possible?
On Friday, 15 November 2013 at 15:27:45 UTC, Jacek Furmankiewicz wrote: One of the nice features of Go is that when you compile an app, it pulls in ALL the dependencies (i.e. the full SDK + all libraries your app depends on) and generates a single binary (around 2 MB for a Hello World app). This is extremely useful for deployment purposes, since it is so straightforward to just copy the app to multiple servers without having to worry if every one of them has all the required dependencies installed / updated, etc. Does D offer something similar (maybe via some dmd switches)? For example, If I am creating a vibe.d app, would I need to deploy the vibe.d libraries separately with my app on all the servers in production? Thanks Jacek Go provides only static linking. D provides both and it is up to developer to decide. Often it is simply a matter of what version of library is installed in the system - static or dynamic one. You can use `ldd` on Linux to check dynamic dependencies for any given binary. On my Arch Linux installation output for simple vibe.d app looks like this: $ ldd ./test linux-vdso.so.1 (0x7fba8f53b000) libevent_pthreads-2.0.so.5 => /usr/lib/libevent_pthreads-2.0.so.5 (0x7fba8f11b000) libevent-2.0.so.5 => /usr/lib/libevent-2.0.so.5 (0x7fba8eed3000) libssl.so.1.0.0 => /usr/lib/libssl.so.1.0.0 (0x7fba8ec66000) libcrypto.so.1.0.0 => /usr/lib/libcrypto.so.1.0.0 (0x7fba8e85e000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x7fba8e64) libm.so.6 => /usr/lib/libm.so.6 (0x7fba8e33d000) librt.so.1 => /usr/lib/librt.so.1 (0x7fba8e135000) libc.so.6 => /usr/lib/libc.so.6 (0x7fba8dd8a000) /lib64/ld-linux-x86-64.so.2 (0x7fba8f31e000) libdl.so.2 => /usr/lib/libdl.so.2 (0x7fba8db86000) libz.so.1 => /usr/lib/libz.so.1 (0x7fba8d97) List may differ between distros / operating systems. Note that you can't "force" static linking if only shared libraries are present in the system and AFAIK Go does no magic here - it will simply fail to compile/link in such situation. tl; dr: yes, if you use static versions of required libraries
Re: genetically modified slices - is it possible?
On Friday, 15 November 2013 at 15:12:23 UTC, Alexandr Druzhinin wrote: 15.11.2013 22:09, Adam D. Ruppe пишет: You could make it work like this: auto slice3 = array[ slice1.length + (slice1.ptr - array.ptr) .. (slice2.ptr - array.ptr)]; Since the slices all start into array somewhere, subtracting the pointers gives their start index. Thank you very much! I forget about .ptr. What about ranges in general? They haven't .ptr. Ranges in general are not arrays, they're objects - you can't get what's in between two objects.
Re: Compiling an app to a single binary - possible?
On Friday, 15 November 2013 at 15:27:45 UTC, Jacek Furmankiewicz wrote: One of the nice features of Go is that when you compile an app, it pulls in ALL the dependencies (i.e. the full SDK + all libraries your app depends on) and generates a single binary (around 2 MB for a Hello World app). This is extremely useful for deployment purposes, since it is so straightforward to just copy the app to multiple servers without having to worry if every one of them has all the required dependencies installed / updated, etc. Does D offer something similar (maybe via some dmd switches)? For example, If I am creating a vibe.d app, would I need to deploy the vibe.d libraries separately with my app on all the servers in production? Thanks Jacek D, like many languages compiled to native code (C, C++) supports dynamically loaded libraries (.dll and .so files) as an option. You have an option to use DLLs (in which case you'll have to deploy them) but it's not a requirement - you can build just like in Go as long as you don't use the DDL feature. You should check vibe.d build options whenever it's possible to build it without DLL dependency.
Re: Best data structure for a particle system?
On Friday, 15 November 2013 at 14:01:36 UTC, Chris Cain wrote: By default (using the default GC and everything), D does not reallocate a dynamic array every time you change the length (even increasing it), so this will still be okay with allocations. Not exactly so. If you decrease the length, the capacity is set to 0. If you then try to increase it, you must use assumeSafeAppend on the array, or it will be reallocated. Alternatively, you may choose to allocate a large enough array once and then create a wrapper struct to store the currently used length. See the explanation of how array slices work in D: http://dlang.org/d-array-article.html Or a thread I started when learning to use D arrays as stacks and queues without wrappers: http://forum.dlang.org/thread/yrxspdrpusrrijmfy...@forum.dlang.org?page=1 Ivan Kazmenko.
Re: Compiling an app to a single binary - possible?
On Friday, 15 November 2013 at 15:27:45 UTC, Jacek Furmankiewicz wrote: One of the nice features of Go is that when you compile an app, it pulls in ALL the dependencies (i.e. the full SDK + all libraries your app depends on) and generates a single binary (around 2 MB for a Hello World app). This is the default in D. The main trouble comes when you use outside C libraries (gtk, sdl, curl, etc.), which might need to be installed, and I don't think they can easily be bundled in a single binary, but D apps and libs generally are all statically linked. I don't know for sure about vibe.d specifically though.
Compiling an app to a single binary - possible?
One of the nice features of Go is that when you compile an app, it pulls in ALL the dependencies (i.e. the full SDK + all libraries your app depends on) and generates a single binary (around 2 MB for a Hello World app). This is extremely useful for deployment purposes, since it is so straightforward to just copy the app to multiple servers without having to worry if every one of them has all the required dependencies installed / updated, etc. Does D offer something similar (maybe via some dmd switches)? For example, If I am creating a vibe.d app, would I need to deploy the vibe.d libraries separately with my app on all the servers in production? Thanks Jacek
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
On Friday, 15 November 2013 at 07:42:22 UTC, ilya-stromberg wrote:
On Thursday, 14 November 2013 at 22:12:10 UTC, Jacek
Furmankiewicz wrote:
On Thursday, 14 November 2013 at 21:36:46 UTC, ilya-stromberg
wrote:
On Thursday, 14 November 2013 at 21:31:52 UTC, Jacek
Furmankiewicz wrote:
How often do you change the data? Probably, you should use
`immutable` variables.
Customer specific. It may change once a year. It may change
multiple times per second for a while, then nothing again for
weeks.
Others may do mass loads of business rules, hence do mass
changes every few hours.
Next to impossible to predict.
You can use `immutable` variables. It allows you to share the
data without any synchronization. Like this:
class MyData
{
int data1;
string data2;
//creates new object
this(int data1, string data2)
{
this.data1 = data1;
this.data2 = data2;
}
//modify the data
immutable(MyData) editData(int i) const
{
//copy this object - we can't change immutable variables
MyData dataCopy = new MyData(this.data1, this.data2)
//modify the data copy
dataCopy.data1 += i;
//assume that `dataCopy` is immutable
return cast(immutable(MyData)) dataCopy;
}
}
shared myMap;
//map implementation
synchronized class MyMap
{
HashMap!(int, immutable(MyData)) map;
void foo()
{
map[1] = new immutable MyData(1, "data");
}
void bar()
{
map[1] = map[1].editData(5);
}
}
//init map
shared static this()
{
myMap = new MyMap();
}
void main()
{
myMap.foo();
myMap.bar();
}
So what happens when the "write" operation is doing
map[1] = map[1].editData(5);
and at the same time 50 threads are simultaneously reading the
value in map[1]?.
Is that reassignment operation thread safe?
Or would I get corrupted reads with potentially a partially
overriden value?
Jacek
Re: Best data structure for a particle system?
On Friday, 15 November 2013 at 14:08:20 UTC, bearophile wrote: The situation is a little more complex, there is a capacity field that I think is kept in a cold place of the array, it's also cached, but only if you append to just one array, etc. An alternative might be hold an array and manage a slice to that array. "Appending" would just be reslicing the array. If the array is long you are accessing a cold part of it to swap with the end. Sure. But on a long array, the time taken to iterate over the entire array looking for a dead particle to recycle would take time as well. Not to mention that removing the dead particle by swapping it with the end would likely keep the overall array smaller and, thus, more likely to fit completely in the cache. There's a lot to be said about keeping memory usage as compact as possible. Is there any empirical data to suggest either approach is better? There's factors that could suggest either one might be faster depending on the situation.
Re: genetically modified slices - is it possible?
15.11.2013 22:13, bearophile пишет: Alexandr Druzhinin: A simple solution is to keep two indexes, and use them to find the slices when you need them. I did it the first. But then I decided to make it more D-ish and stumbled upon the issue. Wasn't it wrong decision and would be better to stay with indices?
Re: genetically modified slices - is it possible?
Alexandr Druzhinin: I'd like to get slice that's consist of begining one other slice and end of yet another slice (all slices belong to the same array of course). Is it possible? With iterators it's simple, but I can't manage do it with slices. http://dpaste.dzfl.pl/443cd4a1 A simple solution is to keep two indexes, and use them to find the slices when you need them. If you want to keep only slices around, then the third slice could be computed from the other two slices using a small function written by you, using the .ptr pointer of the two slices and some pointer arithmetic, plus some run-time preconditions to assert at run-time they are from the same array. Bye, bearophile
Re: genetically modified slices - is it possible?
15.11.2013 22:09, Adam D. Ruppe пишет: You could make it work like this: auto slice3 = array[ slice1.length + (slice1.ptr - array.ptr) .. (slice2.ptr - array.ptr)]; Since the slices all start into array somewhere, subtracting the pointers gives their start index. Thank you very much! I forget about .ptr. What about ranges in general? They haven't .ptr.
Why can't I copy a const struct?
private struct DummyStruct {
int[] a;
this(this) {
a = a.dup;
}
}
void main() {
const dummy1 = DummyStruct();
DummyStruct dummy2 = dummy1;
}
struct_copy.d(12): Error: conversion error from
const(DummyStruct) to DummyStruct
Surely I should be able to copy it and use the mutable version as
I please, no? I'd understand if the postblit constructor wasn't
defined...
Atila
Re: genetically modified slices - is it possible?
You could make it work like this: auto slice3 = array[ slice1.length + (slice1.ptr - array.ptr) .. (slice2.ptr - array.ptr)]; Since the slices all start into array somewhere, subtracting the pointers gives their start index.
genetically modified slices - is it possible?
I'd like to get slice that's consist of begining one other slice and end of yet another slice (all slices belong to the same array of course). Is it possible? With iterators it's simple, but I can't manage do it with slices. http://dpaste.dzfl.pl/443cd4a1
Re: Strange error
Go to bugzilla.
Re: Best data structure for a particle system?
Chris Cain: Instead of having a "dead flag", you could swap ( http://dlang.org/phobos/std_algorithm.html#swap ) the dying particle with the last particle in the list and then decrement the list's length. If the array is long you are accessing a cold part of it to swap with the end. By default (using the default GC and everything), D does not reallocate a dynamic array every time you change the length (even increasing it), so this will still be okay with allocations. The situation is a little more complex, there is a capacity field that I think is kept in a cold place of the array, it's also cached, but only if you append to just one array, etc. Bye, bearophile
Re: Best data structure for a particle system?
On Friday, 15 November 2013 at 13:32:38 UTC, Mikko Ronkainen wrote: Ok, thanks! That linked list cache thrashing was just the thing I knew that I don't know :) Let's say I just use dynamic array and grow it when adding new particles to the system, and when particles die, just set their dead flag on. This means that, when adding a new particle, I need to scan the array first for possible dead particles that can be reused. Is there some trick that could be used here to reduce excessive searching/iteration? Instead of having a "dead flag", you could swap ( http://dlang.org/phobos/std_algorithm.html#swap ) the dying particle with the last particle in the list and then decrement the list's length. Two assumptions: 1, you don't have pointers to particles anywhere. 2, when a particle "dies" it knows about the list and its position in the list. By default (using the default GC and everything), D does not reallocate a dynamic array every time you change the length (even increasing it), so this will still be okay with allocations.
Re: Best data structure for a particle system?
Mikko Ronkainen: Let's say I just use dynamic array and grow it when adding new particles to the system, and when particles die, just set their dead flag on. This means that, when adding a new particle, I need to scan the array first for possible dead particles that can be reused. Is there some trick that could be used here to reduce excessive searching/iteration? Generally avoid resizing the array. Just leave some many ones in the array. Bye, bearophile
Re: Best data structure for a particle system?
Ok, thanks! That linked list cache thrashing was just the thing I knew that I don't know :) Let's say I just use dynamic array and grow it when adding new particles to the system, and when particles die, just set their dead flag on. This means that, when adding a new particle, I need to scan the array first for possible dead particles that can be reused. Is there some trick that could be used here to reduce excessive searching/iteration?
Re: Best data structure for a particle system?
On Friday, 15 November 2013 at 11:52:44 UTC, Mikko Ronkainen wrote: What would be the best data structure for handling particles in a particle system in D2? Here's some thoughts: Particles are simple structs. Two lists, one for alive particles, one for dead ones. Memory allocations should be avoided, preallocate everything, no allocations when moving between lists. Keep alive list as short as possible for fast iteration -> move dead particles off during iteration. Removal and addition of single items only, and it should be fast. Maybe a single-linked list, std.container.SList? Is there any gotchas? Or some better container for this scenario? Linked lists are very cache unfriendly, so they should be avoided. Generally try to use arrays first, then try again using arrays, and if you fail then try to use something different. I suggest to use a dynamic array of particle structs, that contain a boolean that represents if a particle is alive or dead. This is very simple to implement and use. Once you have implemented and benchmarked that, if the code is *not fast enough* then you could try adding a pointer field (or uint/ushort index) to each struct that contains the pointer to the next active cell. But you have to start using and updating such pointers only when the _estimate_ percentage of active cells is very low. An alternative strategy to try is to compact the array of particle structs (copying only the active particles as you scan it to use it). This is especially good if you don't need to keep their order stable and if the structs are small (like a size_t). Bye, bearophile
Re: Best data structure for a particle system?
On 11/15/2013 9:19 PM, Mikko Ronkainen wrote: If relevant, doubly-linked might work better? dcollections LinkList, or maybe DList? I'm asking mostly because I'd like the container to avoid memory allocations while in use. For a particle system, I would avoid lists. A list of particles needs to be iterated every frame. A linked list is going to kill you as the particle count increases. Since the particles will most like not be contiguous in memory, you'll be thrashing your cache to hell and back. "Caches love linear access" is a quote to remember. When you need to do frequent iteration of a list, your cache will love you if all of the items are in a block of contiguous memory. So it's almost always better to use an array for this sort of thing. Make your particle object a struct and use a dynamic array of particles that you can grow as needed or, if it makes sense, a fixed size static array. The point is that arrays of structs will be lined up one next to the other in memory so that you can make good use of the cache. Of course, the size of your particle object also has a role to play here. Google for "linked list cache thrashing" or "cache-friendly data structures" or some such for ideas.
Re: Best data structure for a particle system?
Use just one list with a flag in the particle to see whether the particle is alive or dead, saves swapping between lists and you can use a simple array for fast access. Yes, simplicity, that's a good idea :) I was just wondering how much time would be saved if just iterating over the active particles instead of everything every time (say a system of 1 particles). Maybe that's relevant, maybe not. If relevant, doubly-linked might work better? dcollections LinkList, or maybe DList? I'm asking mostly because I'd like the container to avoid memory allocations while in use.
Re: Best data structure for a particle system?
On Friday, 15 November 2013 at 11:52:44 UTC, Mikko Ronkainen wrote: What would be the best data structure for handling particles in a particle system in D2? Here's some thoughts: Particles are simple structs. Two lists, one for alive particles, one for dead ones. Memory allocations should be avoided, preallocate everything, no allocations when moving between lists. Keep alive list as short as possible for fast iteration -> move dead particles off during iteration. Removal and addition of single items only, and it should be fast. Maybe a single-linked list, std.container.SList? Is there any gotchas? Or some better container for this scenario? Use just one list with a flag in the particle to see whether the particle is alive or dead, saves swapping between lists and you can use a simple array for fast access.
Best data structure for a particle system?
What would be the best data structure for handling particles in a particle system in D2? Here's some thoughts: Particles are simple structs. Two lists, one for alive particles, one for dead ones. Memory allocations should be avoided, preallocate everything, no allocations when moving between lists. Keep alive list as short as possible for fast iteration -> move dead particles off during iteration. Removal and addition of single items only, and it should be fast. Maybe a single-linked list, std.container.SList? Is there any gotchas? Or some better container for this scenario?
Re: What is the closest to ConcurrentHashMap and NavigableMap in Java?
15-Nov-2013 03:35, Charles Hixson пишет: On 11/14/2013 01:36 PM, ilya-stromberg wrote: On Thursday, 14 November 2013 at 21:31:52 UTC, Jacek Furmankiewicz wrote: hashmap per thread is not an option. The cache may be a few GBs of data, there is no way we can duplicate that data per thread. Not to mention the start up time when we have to warm up the cache. How often do you change the data? Probably, you should use `immutable` variables. Immutable variables are nice when they can be used. Often, however, they can't. I think that for the "concurrent hashmap" the best answer is probably to run the map in a thread, with message passing access whether for read or write. Would be slow unless batched. At least in D message passing involves locking/unlocking a queue of messages. Sending back and forth you get 2 lock-wait-unlock and correspondingly context switches. And I wouldn't be surprised if that's how Java's concurrent hashmap is implemented under the covers. (OTOH, I haven't ever debugged such a setup. Someone who has may have a better answer.) As stated in Oracle's documentation somewhere it's implemented with fine grained locking (a lock per bucket of a hash map), some operations still lock the whole map. Rehashing still locks the whole thing I bet. -- Dmitry Olshansky
Strange error
This code (http://dpaste.dzfl.pl/4c499303) causes strange error description: class Bar had semantic errors when compiling On win32 it causes AV: Error: class Bar had semantic errors when compiling Assertion failure: '0' on line 1215 in file 'glue.c'
Re: What does it mean void[]?
On Friday, 15 November 2013 at 09:19:04 UTC, Orfeo wrote:
I have found in the module
https://github.com/NCrashed/serial-port/blob/master/source/serial/device.d
this function:
void write(const(void[]) arr) {
...
What exactly is void[]? An array of pointers? An array of
"anything"?
Thank you
It's semantics is "an array of octets". Similar, to "void *" in
C++, except, the overall length in bytes is known.
What does it mean void[]?
I have found in the module
https://github.com/NCrashed/serial-port/blob/master/source/serial/device.d
this function:
void write(const(void[]) arr) {
...
What exactly is void[]? An array of pointers? An array of
"anything"?
Thank you
Re: Deimos rules?
On 2013-11-15 05:30, Jonathan M Davis wrote: Deimos is specifically for bindings to C libraries and _not_ for D-ified wrappers. And that's the stance that Walter has taken when it's come up. But with dub and code.dlang.org, it should be simple enough to put a D-ified wrapper in a place where folks can find it. Personally I would mix the wrappers and bindings in the same project, as along as you could use the bindings completely separate without the wrappers. -- /Jacob Carlborg
Re: interface and class inheritance
On 2013-11-14 23:17, Oleg B wrote:
Oh sorry i mean
interface A {
void funcA();
}
class B : A {
final void funcA() { writeln( "B.funcA()" ); }
}
class C : B {
}
we can't change anything in class B
I would have moved "final void funcA()" to a template and mixin it in
both B and C, but if you cannot change B that won't work.
--
/Jacob Carlborg
Re: Figuring out the returntype opDipatch
On 2013-11-14 22:02, TheFlyingFiddle wrote:
After looking at the DIP some more i can see that my suggestion
implementation does not make any sense (and i missed some of the
syntax). If it can be done with AST's i don't have a sugestion for it.
If you have the an example like:
int a = foo.baz("Hello");
And the context parameter allows you do determine the type of "a", then
it would be possible.
But you would still have problems with usages like these:
auto a = foo.baz("Hello");
And:
void bar (T) (T t) { }
bar(foo.baz("Hello"));
Both of these requires some fallback type.
--
/Jacob Carlborg
Re: Figuring out the returntype opDipatch
On 2013-11-14 21:39, TheFlyingFiddle wrote: Might this be something you could solve using the DIP50 AST macros? Possibly, if the context parameter provides enough information about where the macro is used. -- /Jacob Carlborg
Re: pitfalls of enum
On Thu, Nov 14, 2013 at 5:02 PM, bearophile wrote:
> Ali Çehreli:
>
>
> > When is an enum *better* than a normal (static
>> const/immutable) constant?
>>
>> Good question. :)
>>
>
> When you can or want to compute something at compile-time, when you need
> values to feed to templates, etc.
>
>
but you can *can* do that with static const /const/immutable variables too:
auto fun2(int a)(int b){...}
immutable a=fun(2);//static const /const/immutable/enum
auto b=fun2!a(3);
Re: Linking from source-code
On 2013-11-14 14:49, Andrea Fontana wrote: BTW: you say "used like header files". You mean files that are not "compiled" but "imported"? Yes. When you have a pre-compiled library and need to declarations. -- /Jacob Carlborg
