Re: Asynchronicity in D

2011-04-05 Thread Jonas Drewsen
On 04/04/11 22.23, Max Klyga wrote: Jonas, thanks for your valuable feedback. You've expressed interest in mentoring a networking a networking project and since I couldn't find any other way to contact you directly, I'll post my message here. As was discussed later, your work on curl supersedes

Re: Asynchronicity in D

2011-04-05 Thread Robert Clipsham
On 31/03/2011 19:34, Andrej Mitrovic wrote: Are fibers really better/faster than threads? I've heard rumors that they perform exactly the same, and that there's no benefit of using fibers over threads. Is that true? I've written up a first draft of an article about this at: http://octarineparr

Re: Asynchronicity in D

2011-04-04 Thread Jose Armando Garcia
The problem with threads is the context switch not really the stack size. Threads are not the solution to increase performance. In high performance systems threads are used for fairness in the resquest-response pipeline not for performance. Obviously, this fact is not argued when talking about uni-

Re: Asynchronicity in D

2011-04-04 Thread Sean Kelly
On Apr 1, 2011, at 6:08 PM, Brad Roberts wrote: > On Fri, 1 Apr 2011, dsimcha wrote: > >> On 4/1/2011 7:27 PM, Sean Kelly wrote: >>> On Apr 1, 2011, at 2:24 PM, dsimcha wrote: >>> == Quote from Brad Roberts (bra...@puremagic.com)'s article > I've got an app that regularly runs with hund

Re: Asynchronicity in D

2011-04-04 Thread Max Klyga
Jonas, thanks for your valuable feedback. You've expressed interest in mentoring a networking a networking project and since I couldn't find any other way to contact you directly, I'll post my message here. As was discussed later, your work on curl supersedes my future effort on network clie

Re: Asynchronicity in D

2011-04-01 Thread Brad Roberts
On Fri, 1 Apr 2011, dsimcha wrote: > On 4/1/2011 7:27 PM, Sean Kelly wrote: > > On Apr 1, 2011, at 2:24 PM, dsimcha wrote: > > > > > == Quote from Brad Roberts (bra...@puremagic.com)'s article > > > > I've got an app that regularly runs with hundreds of thousands of > > > > connections (though mo

Re: Asynchronicity in D

2011-04-01 Thread dsimcha
On 4/1/2011 7:27 PM, Sean Kelly wrote: On Apr 1, 2011, at 2:24 PM, dsimcha wrote: == Quote from Brad Roberts (bra...@puremagic.com)'s article I've got an app that regularly runs with hundreds of thousands of connections (though most of them are mostly idle). I haven't seen it break 1M yet, bu

Re: Asynchronicity in D

2011-04-01 Thread Sean Kelly
On Apr 1, 2011, at 2:24 PM, dsimcha wrote: > == Quote from Brad Roberts (bra...@puremagic.com)'s article >> I've got an app that regularly runs with hundreds of thousands of >> connections (though most of them are mostly idle). I haven't seen it >> break 1M yet, but the only thing stopping it is

Re: Asynchronicity in D

2011-04-01 Thread Sean Kelly
On Apr 1, 2011, at 1:43 PM, Piotr Szturmaj wrote: > Sean Kelly wrote: >> Fair enough. Though I'd still say it's a terrible use of resources, > > given available asynchronous socket APIs. And as an aside, >> I think 32K sockets per process is not at all surprising. > > I've seen apps that use ord

Re: Asynchronicity in D

2011-04-01 Thread Sean Kelly
On Apr 1, 2011, at 12:59 PM, Jonas Drewsen wrote: > On 01/04/11 17.21, Sean Kelly wrote: >> >> On Apr 1, 2011, at 7:49 AM, Jonas Drewsen wrote: >>> >>> There doesn't have to be a thread for each socket. Actually many servers >>> have very few threads with many sockets each. 32000 sockets is not

Re: Asynchronicity in D

2011-04-01 Thread dsimcha
== Quote from Brad Roberts (bra...@puremagic.com)'s article > I've got an app that regularly runs with hundreds of thousands of > connections (though most of them are mostly idle). I haven't seen it > break 1M yet, but the only thing stopping it is file descriptor limits and > memory. It runs a v

Re: Asynchronicity in D

2011-04-01 Thread Piotr Szturmaj
Sean Kelly wrote: Fair enough. Though I'd still say it's a terrible use of resources, > given available asynchronous socket APIs. And as an aside, I think 32K sockets per process is not at all surprising. > I've seen apps that use orders of magnitude more than that, though breaking the 64K

Re: Asynchronicity in D

2011-04-01 Thread Jonas Drewsen
On 01/04/11 18.12, dsimcha wrote: == Quote from Sean Kelly (s...@invisibleduck.org)'s article On Apr 1, 2011, at 7:49 AM, Jonas Drewsen wrote: On 01/04/11 01.07, dsimcha wrote: Again forgive my naiveness, as most of my experience with concurrency is concurrency to implement parallelism, no

Re: Asynchronicity in D

2011-04-01 Thread Brad Roberts
On Fri, 1 Apr 2011, Sean Kelly wrote: > On Apr 1, 2011, at 12:59 PM, Jonas Drewsen wrote: > > > On 01/04/11 17.21, Sean Kelly wrote: > >> > >> On Apr 1, 2011, at 7:49 AM, Jonas Drewsen wrote: > >>> > >>> There doesn't have to be a thread for each socket. Actually many servers > >>> have very f

Re: Asynchronicity in D

2011-04-01 Thread Jonas Drewsen
On 01/04/11 17.21, Sean Kelly wrote: On Apr 1, 2011, at 7:49 AM, Jonas Drewsen wrote: On 01/04/11 01.07, dsimcha wrote: Again forgive my naiveness, as most of my experience with concurrency is concurrency to implement parallelism, not concurrency for its own sake. Shouldn't 32,000 threads

Re: Asynchronicity in D

2011-04-01 Thread Sean Kelly
On Apr 1, 2011, at 8:47 AM, dsimcha wrote: > == Quote from Sean Kelly (s...@invisibleduck.org)'s article >> >> It's more an issue of creating an understandable programming model. If >> someone is using statics, the result should be the same regardless of >> whether the code gets a dedicated thre

Re: Asynchronicity in D

2011-04-01 Thread Jonas Drewsen
On 31/03/11 18.43, dsimcha wrote: == Quote from Andrei Alexandrescu (seewebsiteforem...@erdani.org)'s article On 3/31/11 6:35 AM, Max Klyga wrote: I've been thinking on things I can change in my GSoC proposal to make it stronger and noticed that currently Phobos does not address asynchronous I/

Re: Asynchronicity in D

2011-04-01 Thread dsimcha
== Quote from Sean Kelly (s...@invisibleduck.org)'s article > On Apr 1, 2011, at 7:49 AM, Jonas Drewsen wrote: > > On 01/04/11 01.07, dsimcha wrote: > >> > >> > >> Again forgive my naiveness, as most of my experience with concurrency > is > >> concurrency to implement parallelism, not concurrency f

Re: Asynchronicity in D

2011-04-01 Thread Jonas Drewsen
On 01/04/11 01.07, dsimcha wrote: == Quote from Jonas Drewsen (jdrew...@nospam.com)'s article On 31/03/11 23.20, Max Klyga wrote: On 2011-03-31 22:35:43 +0300, Jonas Drewsen said: On 31/03/11 18.26, Andrei Alexandrescu wrote: snip I believe that we would need both the threaded async IO tha

Re: Asynchronicity in D

2011-04-01 Thread Sean Kelly
On Apr 1, 2011, at 7:49 AM, Jonas Drewsen wrote: > On 01/04/11 01.07, dsimcha wrote: >> >> >> Again forgive my naiveness, as most of my experience with concurrency is >> concurrency to implement parallelism, not concurrency for its own sake. >> Shouldn't >> 32,000 threads be more than enough

Re: Asynchronicity in D

2011-04-01 Thread dsimcha
== Quote from Sean Kelly (s...@invisibleduck.org)'s article > On Mar 31, 2011, at 4:03 PM, dsimcha wrote: > > == Quote from Sean Kelly (s...@invisibleduck.org)'s article > >> On Mar 31, 2011, at 11:48 AM, dsimcha wrote: > >>> == Quote from Andrej Mitrovic (andrej.mitrov...@gmail.com)'s > >> article

Re: Asynchronicity in D

2011-04-01 Thread Sean Kelly
On Mar 31, 2011, at 4:03 PM, dsimcha wrote: > == Quote from Sean Kelly (s...@invisibleduck.org)'s article >> On Mar 31, 2011, at 11:48 AM, dsimcha wrote: >>> == Quote from Andrej Mitrovic (andrej.mitrov...@gmail.com)'s >> article Are fibers really better/faster than threads? I've heard rumors

Re: Asynchronicity in D

2011-04-01 Thread Torarin
2011/3/31 Jonas Drewsen : > On 31/03/11 21.19, Torarin wrote: >> >> I'm currently working on an http and networking library that uses >> asynchronous sockets running in fibers and an event loop a la libev. >> These async sockets have the same interface as regular Berkeley >> sockets, so clients can

Re: Asynchronicity in D

2011-03-31 Thread Max Klyga
On 2011-04-01 01:45:54 +0300, Jonas Drewsen said: On 31/03/11 23.20, Max Klyga wrote: On 2011-03-31 22:35:43 +0300, Jonas Drewsen said: On 31/03/11 18.26, Andrei Alexandrescu wrote: snip I believe that we would need both the threaded async IO that you describe but also a select based one.

Re: Asynchronicity in D

2011-03-31 Thread dsimcha
== Quote from Jonas Drewsen (jdrew...@nospam.com)'s article > On 31/03/11 23.20, Max Klyga wrote: > > On 2011-03-31 22:35:43 +0300, Jonas Drewsen said: > > > >> On 31/03/11 18.26, Andrei Alexandrescu wrote: > >>> snip > >> > >> I believe that we would need both the threaded async IO that you > >> d

Re: Asynchronicity in D

2011-03-31 Thread dsimcha
== Quote from Sean Kelly (s...@invisibleduck.org)'s article > On Mar 31, 2011, at 11:48 AM, dsimcha wrote: > > == Quote from Andrej Mitrovic (andrej.mitrov...@gmail.com)'s > article > >> Are fibers really better/faster than threads? I've heard rumors that > >> they perform exactly the same, and tha

Re: Asynchronicity in D

2011-03-31 Thread Jonas Drewsen
On 31/03/11 23.20, Max Klyga wrote: On 2011-03-31 22:35:43 +0300, Jonas Drewsen said: On 31/03/11 18.26, Andrei Alexandrescu wrote: snip I believe that we would need both the threaded async IO that you describe but also a select based one. The thread based is important e.g. in order to keep

Re: Asynchronicity in D

2011-03-31 Thread Sean Kelly
On Mar 31, 2011, at 11:48 AM, dsimcha wrote: > == Quote from Andrej Mitrovic (andrej.mitrov...@gmail.com)'s article >> Are fibers really better/faster than threads? I've heard rumors that >> they perform exactly the same, and that there's no benefit of using >> fibers over threads. Is that true? >

Re: Asynchronicity in D

2011-03-31 Thread Max Klyga
On 2011-03-31 22:35:43 +0300, Jonas Drewsen said: On 31/03/11 18.26, Andrei Alexandrescu wrote: snip I believe that we would need both the threaded async IO that you describe but also a select based one. The thread based is important e.g. in order to keep buffering incoming data while proce

Re: Asynchronicity in D

2011-03-31 Thread Max Klyga
On 2011-03-31 19:26:45 +0300, Andrei Alexandrescu said: On 3/31/11 6:35 AM, Max Klyga wrote: snip I think that would be a good contribution that would complement Jonas'. You'll need to discuss cooperation with him and at best Jonas would agree to become a mentor. Jonas agreed to become a

Re: Asynchronicity in D

2011-03-31 Thread Jonas Drewsen
On 31/03/11 21.19, Torarin wrote: I'm currently working on an http and networking library that uses asynchronous sockets running in fibers and an event loop a la libev. These async sockets have the same interface as regular Berkeley sockets, so clients can choose whether to be synchronous, asynch

Re: Asynchronicity in D

2011-03-31 Thread Jonas Drewsen
On 31/03/11 20.48, dsimcha wrote: == Quote from Andrej Mitrovic (andrej.mitrov...@gmail.com)'s article Are fibers really better/faster than threads? I've heard rumors that they perform exactly the same, and that there's no benefit of using fibers over threads. Is that true? Here are some key d

Re: Asynchronicity in D

2011-03-31 Thread Jonas Drewsen
On 31/03/11 18.26, Andrei Alexandrescu wrote: On 3/31/11 6:35 AM, Max Klyga wrote: I've been thinking on things I can change in my GSoC proposal to make it stronger and noticed that currently Phobos does not address asynchronous I/O of any kind. A number of threads on thid newsgroup mentioned a

Re: Asynchronicity in D

2011-03-31 Thread Torarin
I'm currently working on an http and networking library that uses asynchronous sockets running in fibers and an event loop a la libev. These async sockets have the same interface as regular Berkeley sockets, so clients can choose whether to be synchronous, asynchronous or threaded with template arg

Re: Asynchronicity in D

2011-03-31 Thread Steven Schveighoffer
On Thu, 31 Mar 2011 14:48:13 -0400, dsimcha wrote: == Quote from Andrej Mitrovic (andrej.mitrov...@gmail.com)'s article Are fibers really better/faster than threads? I've heard rumors that they perform exactly the same, and that there's no benefit of using fibers over threads. Is that true?

Re: Asynchronicity in D

2011-03-31 Thread dsimcha
== Quote from Andrej Mitrovic (andrej.mitrov...@gmail.com)'s article > Are fibers really better/faster than threads? I've heard rumors that > they perform exactly the same, and that there's no benefit of using > fibers over threads. Is that true? Here are some key differences between fibers (as cu

Re: Asynchronicity in D

2011-03-31 Thread Andrej Mitrovic
Are fibers really better/faster than threads? I've heard rumors that they perform exactly the same, and that there's no benefit of using fibers over threads. Is that true?

Re: Asynchronicity in D

2011-03-31 Thread Robert Clipsham
On 31/03/2011 17:53, Robert Clipsham wrote: On 31/03/2011 17:26, Andrei Alexandrescu wrote: foreach (line; byLineAsync("http://d-programming-language.org";)) { ... use line ... } What would be awesome is if this was backed by fibers, then you have a really simple and easy wrapper for doing asy

Re: Asynchronicity in D

2011-03-31 Thread Robert Clipsham
On 31/03/2011 17:26, Andrei Alexandrescu wrote: foreach (line; byLineAsync("http://d-programming-language.org";)) { ... use line ... } What would be awesome is if this was backed by fibers, then you have a really simple and easy wrapper for doing async io, handling lots of connections as the

Re: Asynchronicity in D

2011-03-31 Thread dsimcha
== Quote from Andrei Alexandrescu (seewebsiteforem...@erdani.org)'s article > On 3/31/11 11:43 AM, dsimcha wrote: > > == Quote from Andrei Alexandrescu (seewebsiteforem...@erdani.org)'s article > >> On 3/31/11 6:35 AM, Max Klyga wrote: > >>> I've been thinking on things I can change in my GSoC prop

Re: Asynchronicity in D

2011-03-31 Thread Andrei Alexandrescu
On 3/31/11 11:43 AM, dsimcha wrote: == Quote from Andrei Alexandrescu (seewebsiteforem...@erdani.org)'s article On 3/31/11 6:35 AM, Max Klyga wrote: I've been thinking on things I can change in my GSoC proposal to make it stronger and noticed that currently Phobos does not address asynchronous

Re: Asynchronicity in D

2011-03-31 Thread dsimcha
== Quote from Andrei Alexandrescu (seewebsiteforem...@erdani.org)'s article > On 3/31/11 6:35 AM, Max Klyga wrote: > > I've been thinking on things I can change in my GSoC proposal to make it > > stronger and noticed that currently Phobos does not address asynchronous > > I/O of any kind. > > > > A

Re: Asynchronicity in D

2011-03-31 Thread Andrei Alexandrescu
On 3/31/11 6:35 AM, Max Klyga wrote: I've been thinking on things I can change in my GSoC proposal to make it stronger and noticed that currently Phobos does not address asynchronous I/O of any kind. A number of threads on thid newsgroup mentioned about this problem or shown ways other languages

Re: Asynchronicity in D

2011-03-31 Thread Aleksandar Ružičić
I really like design of node.js (http://nodejs.org) it's internally based on libev and everything runs in a single-threaded event loop. It's proven to be highly concurrent and memory efficient. Maybe a wrapper around libev(ent) for D ala node.js would be good solution for asynchronous API, other t

Re: Asynchronicity in D

2011-03-31 Thread Piotr Szturmaj
dsimcha wrote: == Quote from Piotr Szturmaj (bncr...@jadamspam.pl)'s article Max Klyga wrote: I've been thinking on things I can change in my GSoC proposal to make it stronger and noticed that currently Phobos does not address asynchronous I/O of any kind. A number of threads on thid newsgroup

Re: Asynchronicity in D

2011-03-31 Thread Aleksandar Ružičić
I really like design of node.js (http://nodejs.org) it's internally based on libev and everything runs in a single-threaded event loop. It's proven to be highly concurrent and memory efficient. Maybe a wrapper around libev(ent) for D ala node.js would be good solution for asynchronous API, other t

Re: Asynchronicity in D

2011-03-31 Thread dsimcha
== Quote from Piotr Szturmaj (bncr...@jadamspam.pl)'s article > Max Klyga wrote: > > I've been thinking on things I can change in my GSoC proposal to make it > > stronger and noticed that currently Phobos does not address asynchronous > > I/O of any kind. > > > > A number of threads on thid newsgro

Re: Asynchronicity in D

2011-03-31 Thread Piotr Szturmaj
Max Klyga wrote: I've been thinking on things I can change in my GSoC proposal to make it stronger and noticed that currently Phobos does not address asynchronous I/O of any kind. A number of threads on thid newsgroup mentioned about this problem or shown ways other languages address asynchronic

Asynchronicity in D

2011-03-31 Thread Max Klyga
I've been thinking on things I can change in my GSoC proposal to make it stronger and noticed that currently Phobos does not address asynchronous I/O of any kind. A number of threads on thid newsgroup mentioned about this problem or shown ways other languages address asynchronicity. I want t