Thread-Multiplexing for D
Hello, I once had a look at Go and D and can't see a reason to choose Go except for language built-in CSP and thread-multiplexing. But thread-multiplexing in Go is really a killer feature, especially when developing server-side applications that need to scale with the number of requests (from the Internet). Several startups chose therefore Go. It could have been D. So my point is that it would be good to have thread multiplexing in D as well. It exists for Java, see http://hawtdispatch.fusesource.org/. And then there is the original from Apple: http://en.wikipedia.org/wiki/Grand_Central_Dispatch. I think server-side applications that need to scale and are not for enterprise computing (aka company internal Java clusters) is a new kind of application for which people could also decide to choose D for just like Go. Putting in some thread-multiplexing into D is not easy and I won't have the time for doing that. But I would like to suggest to start an effort in that direction or think it over whether that would be something to have. Regards, Bienlein
Re: Thread-Multiplexing for D
Am 12.09.2013 16:12, schrieb Bienlein: Hello, I once had a look at Go and D and can't see a reason to choose Go except for language built-in CSP and thread-multiplexing. But thread-multiplexing in Go is really a killer feature, especially when developing server-side applications that need to scale with the number of requests (from the Internet). Several startups chose therefore Go. It could have been D. So my point is that it would be good to have thread multiplexing in D as well. It exists for Java, see http://hawtdispatch.fusesource.org/. And then there is the original from Apple: http://en.wikipedia.org/wiki/Grand_Central_Dispatch. I think server-side applications that need to scale and are not for enterprise computing (aka company internal Java clusters) is a new kind of application for which people could also decide to choose D for just like Go. Putting in some thread-multiplexing into D is not easy and I won't have the time for doing that. But I would like to suggest to start an effort in that direction or think it over whether that would be something to have. Regards, Bienlein There is an existing implementation already: http://vibed.org/ I didn't hear the term thread-multiplexing before, though. Maybe that should be added somewhere to make it more searchable.
Re: Thread-Multiplexing for D
Ah! Cool :-). They are saying that they are using libevent, see http://vibed.org/features#performance. I see... Although only shitty software comes from my country such as SAP, this is a company that seems to develop some cool stuff. So I hope that D will also pop up in job ads over here some day ;-). -- Bienlein
Re: Thread-Multiplexing for D
About thread-multiplexing... You find a lot in Google when searching for socket multplexing, but not when searching for thread-multiplexing. Maybe I coined the term myself (don't know any more) when reading the section here: http://golang.org/doc/effective_go.html#goroutines "Goroutines are multiplexed onto multiple OS threads so if one should block, such as while waiting for I/O, others continue to run. Their design hides many of the complexities of thread creation and management." -- Bienlein
Re: Thread-Multiplexing for D
On Sep 12, 2013, at 8:34 AM, Bienlein wrote: > About thread-multiplexing... You find a lot in Google when > searching for socket multplexing, but not when searching for > thread-multiplexing. Maybe I coined the term myself (don't know > any more) when reading the section here: > http://golang.org/doc/effective_go.html#goroutines > > "Goroutines are multiplexed onto multiple OS threads so if one > should block, such as while waiting for I/O, others continue to > run. Their design hides many of the complexities of thread > creation and management." The trick in D is that because statics are thread-local by default, any multiplexed app like this that expects its static data to remain consistent across calls is likely to fail. I've mentioned fiber-local storage here in the past, but it's a tricky problem. But I think it's one that we will need to sort out for things like this to work as the user expects them to.
Re: Thread-Multiplexing for D
Am 12.09.2013 19:55, schrieb Sean Kelly: On Sep 12, 2013, at 8:34 AM, Bienlein wrote: About thread-multiplexing... You find a lot in Google when searching for socket multplexing, but not when searching for thread-multiplexing. Maybe I coined the term myself (don't know any more) when reading the section here: http://golang.org/doc/effective_go.html#goroutines "Goroutines are multiplexed onto multiple OS threads so if one should block, such as while waiting for I/O, others continue to run. Their design hides many of the complexities of thread creation and management." The trick in D is that because statics are thread-local by default, any multiplexed app like this that expects its static data to remain consistent across calls is likely to fail. I've mentioned fiber-local storage here in the past, but it's a tricky problem. But I think it's one that we will need to sort out for things like this to work as the user expects them to. Until we have something built-in, I've implemented a template based solution for task local storage (same as fiber local storage, but the lifetime is different as fibers get recycled for consecutive tasks): http://vibed.org/api/vibe.core.core/TaskLocal
Re: Thread-Multiplexing for D
13-Sep-2013 13:57, Sönke Ludwig пишет: Am 12.09.2013 19:55, schrieb Sean Kelly: On Sep 12, 2013, at 8:34 AM, Bienlein wrote: About thread-multiplexing... You find a lot in Google when searching for socket multplexing, but not when searching for thread-multiplexing. Maybe I coined the term myself (don't know any more) when reading the section here: http://golang.org/doc/effective_go.html#goroutines "Goroutines are multiplexed onto multiple OS threads so if one should block, such as while waiting for I/O, others continue to run. Their design hides many of the complexities of thread creation and management." The trick in D is that because statics are thread-local by default, any multiplexed app like this that expects its static data to remain consistent across calls is likely to fail. I've mentioned fiber-local storage here in the past, but it's a tricky problem. But I think it's one that we will need to sort out for things like this to work as the user expects them to. Until we have something built-in, I've implemented a template based solution for task local storage (same as fiber local storage, but the lifetime is different as fibers get recycled for consecutive tasks): http://vibed.org/api/vibe.core.core/TaskLocal I do hope O(n) is a typo and it should be O(1). Anyhow what n stands here for? -- Dmitry Olshansky
Re: Thread-Multiplexing for D
Am 12.09.2013 16:57, schrieb Bienlein: Ah! Cool :-). They are saying that they are using libevent, see http://vibed.org/features#performance. I see... Although only shitty software comes from my country such as SAP, this is a company that seems to develop some cool stuff. So I hope that D will also pop up in job ads over here some day ;-). -- Bienlein I too hope that this will happen one day, but I fear that the company is still in its infancy :) * BTW there is at least one company (sociomantic) in Germany with an office in Berlin that uses D. * disclaimer: I might be affiliated with vibe.d and that company ;)
Re: Thread-Multiplexing for D
Am 13.09.2013 12:57, schrieb Dmitry Olshansky: 13-Sep-2013 13:57, Sönke Ludwig пишет: Am 12.09.2013 19:55, schrieb Sean Kelly: On Sep 12, 2013, at 8:34 AM, Bienlein wrote: About thread-multiplexing... You find a lot in Google when searching for socket multplexing, but not when searching for thread-multiplexing. Maybe I coined the term myself (don't know any more) when reading the section here: http://golang.org/doc/effective_go.html#goroutines "Goroutines are multiplexed onto multiple OS threads so if one should block, such as while waiting for I/O, others continue to run. Their design hides many of the complexities of thread creation and management." The trick in D is that because statics are thread-local by default, any multiplexed app like this that expects its static data to remain consistent across calls is likely to fail. I've mentioned fiber-local storage here in the past, but it's a tricky problem. But I think it's one that we will need to sort out for things like this to work as the user expects them to. Until we have something built-in, I've implemented a template based solution for task local storage (same as fiber local storage, but the lifetime is different as fibers get recycled for consecutive tasks): http://vibed.org/api/vibe.core.core/TaskLocal I do hope O(n) is a typo and it should be O(1). Anyhow what n stands here for? Indeed, stupid typo.
Re: Thread-Multiplexing for D
On Friday, 13 September 2013 at 11:06:35 UTC, Sönke Ludwig wrote: BTW there is at least one company (sociomantic) in Germany with an office in Berlin that uses D. And they are hiring ;)
Re: Thread-Multiplexing for D
Am 13.09.2013 12:57, schrieb Dmitry Olshansky: 13-Sep-2013 13:57, Sönke Ludwig пишет: Am 12.09.2013 19:55, schrieb Sean Kelly: On Sep 12, 2013, at 8:34 AM, Bienlein wrote: About thread-multiplexing... You find a lot in Google when searching for socket multplexing, but not when searching for thread-multiplexing. Maybe I coined the term myself (don't know any more) when reading the section here: http://golang.org/doc/effective_go.html#goroutines "Goroutines are multiplexed onto multiple OS threads so if one should block, such as while waiting for I/O, others continue to run. Their design hides many of the complexities of thread creation and management." The trick in D is that because statics are thread-local by default, any multiplexed app like this that expects its static data to remain consistent across calls is likely to fail. I've mentioned fiber-local storage here in the past, but it's a tricky problem. But I think it's one that we will need to sort out for things like this to work as the user expects them to. Until we have something built-in, I've implemented a template based solution for task local storage (same as fiber local storage, but the lifetime is different as fibers get recycled for consecutive tasks): http://vibed.org/api/vibe.core.core/TaskLocal I do hope O(n) is a typo and it should be O(1). Anyhow what n stands here for? n would be the number of variables declared. There is a linear initialization cost and in the naive predecessor implementation, an AA with Variants was used to store the values, which is why I've mentioned efficiency at all.
Re: Thread-Multiplexing for D
The "thread-multiplexing" in Go is described here: https://docs.google.com/document/d/1TTj4T2JO42uD5ID9e89oa0sLKhJYD0Y_kqxDv3I3XMw/edit The sources are here: http://code.google.com/p/go/source/browse/src/pkg/runtime/proc.c?r=01acf1dbe91f673f6308248b8f45ec0564b1d751 It should be possible to takes this approach from Go and bring it to D. Just an idea ...
Re: Thread-Multiplexing for D
On Wednesday, 9 October 2013 at 13:36:41 UTC, Bienlein wrote: The "thread-multiplexing" in Go is described here: https://docs.google.com/document/d/1TTj4T2JO42uD5ID9e89oa0sLKhJYD0Y_kqxDv3I3XMw/edit The sources are here: http://code.google.com/p/go/source/browse/src/pkg/runtime/proc.c?r=01acf1dbe91f673f6308248b8f45ec0564b1d751 It should be possible to takes this approach from Go and bring it to D. Just an idea ... Is this different to task parallelism as implemented in std.parallelism [0]? The downside of D is that the rest of system is not integrated. For example, if you do a blocking syscall then the threadpool is not increased to compensate for the blocked thread. [0] http://dlang.org/phobos/std_parallelism.html
Re: Thread-Multiplexing for D
On Wednesday, 9 October 2013 at 15:47:11 UTC, qznc wrote: On Wednesday, 9 October 2013 at 13:36:41 UTC, Bienlein wrote: The "thread-multiplexing" in Go is described here: https://docs.google.com/document/d/1TTj4T2JO42uD5ID9e89oa0sLKhJYD0Y_kqxDv3I3XMw/edit The sources are here: http://code.google.com/p/go/source/browse/src/pkg/runtime/proc.c?r=01acf1dbe91f673f6308248b8f45ec0564b1d751 It should be possible to takes this approach from Go and bring it to D. Just an idea ... Is this different to task parallelism as implemented in std.parallelism [0]? The downside of D is that the rest of system is not integrated. For example, if you do a blocking syscall then the threadpool is not increased to compensate for the blocked thread. [0] http://dlang.org/phobos/std_parallelism.html Yes, right. This is the big difference between CSP-style channels in Go compared to libdispatch in Apple's C/C++.
