Re: [GSoC Proposal draft] High-Level Networking
On 07/04/11 00.32, Max Klyga wrote: OK, I've absorbed enough informtion to transform my thoughts in the second proposal draft. I'll be posting it direcltly to GSoC website and correct it, if necessary. Proposal is situated here: http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/neku42/1 Comments are welcome. Seems good to me. Some comments: You mention that a reactor will probably be used for implementing the proactor. On windows it really doesn't make sense to do this it think. Instead overlapped IO should be used since it maps very well with the proactor pattern. I also suggest that you specify which OSes you will make support for. If that is only e.g. linux then building on reactors is the right way to go as you've planned. /Jonas
Re: [GSoC Proposal draft] High-Level Networking
2011/4/7 Jonas Drewsen jdrew...@nospam.com: On 07/04/11 00.32, Max Klyga wrote: OK, I've absorbed enough informtion to transform my thoughts in the second proposal draft. I'll be posting it direcltly to GSoC website and correct it, if necessary. Proposal is situated here: http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/neku42/1 Comments are welcome. Seems good to me. Some comments: You mention that a reactor will probably be used for implementing the proactor. On windows it really doesn't make sense to do this it think. Instead overlapped IO should be used since it maps very well with the proactor pattern. I also suggest that you specify which OSes you will make support for. If that is only e.g. linux then building on reactors is the right way to go as you've planned. /Jonas Indeed it looks good! You can use the reactor pattern on Windows as well, but several sources say that overlapped IO is more efficient, and it probably is cleaner. But it's still nice to have a reactor that wraps select and friends, and then specialize the individual proactors as needed. Was that what you were thinking, Max? Torarin
Re: [GSoC Proposal draft] High-Level Networking
On 2011-04-07 16:17:43 +0300, Torarin said: 2011/4/7 Jonas Drewsen jdrew...@nospam.com: On 07/04/11 00.32, Max Klyga wrote: snip Seems good to me. Some comments: You mention that a reactor will probably be used for implementing the proactor. On windows it really doesn't make sense to do this it think. Instead overlapped IO should be used since it maps very well with the proactor pattern. I also suggest that you specify which OSes you will make support for. If that is only e.g. linux then building on reactors is the right way to go as you've planned. /Jonas Indeed it looks good! You can use the reactor pattern on Windows as well, but several sources say that overlapped IO is more efficient, and it probably is cleaner. But it's still nice to have a reactor that wraps select and friends, and then specialize the individual proactors as needed. Was that what you were thinking, Max? Torarin I plan to start with a Mac implementation, because that's what I use most of the time, but I have a Windows box too and can install Linux on virtual machine, if that will be needed. Yes, I had an idea to implement reactor and proactor the way, user may swap implementations, e.g. portable reactor based on select and platform specific one using efficient implementation for that platform. This approach is used in Twisted python web-server (It's licence states that it needs attribution only on reuse of descent amounts of code, so it might be a good idea to peek for some platform specific details). Windows overlapped I/O allows to implement proactor on top of it without any additional layers, but reactor can be implemented on of overlapped I/O too.
Re: [GSoC Proposal draft] High-Level Networking
On 2011-04-06 02:30:44 +0300, Torarin said: 2011/4/4 Max Klyga max.kl...@gmail.com: Thanks to everyone who provided feedback! I'll post my proposal update tomorrow. I'm looking forward to it too! I've done a fair amount of research on these topics and given networking in D a lot of thought. I'd prefer the standard library's socket interface to be more low level than it is today, and to have a socket type that provides non-blocking IO transparently through fibers. This enables both generic protocol implementations that can handle several types of concurrency, and efficient parsers using buffered input ranges. I also think the library should offer it's own event loop instead of linking to libev or libevent, which have their weaknesses in such a context. For instance, you can't simply hand over garbage collected objects to them, which is unfortunate given the usefulness of closures for handling events. I also think a lower-level API than Boost ASIO is wise, because one may need to use novel and platform-specific optimizations like linux splice, a must for efficient file servers. As I mentioned, I have started implementing some of this. Would you be interested in pairing up with me on some parts of the project? Best regards, Torarin I'll be glad to cooperate if this isn't explicitly forbidden by GSoC rules (which is highly implausible). I too think that current socket interface could be improved.
Re: [GSoC Proposal draft] High-Level Networking
OK, I've absorbed enough informtion to transform my thoughts in the second proposal draft. I'll be posting it direcltly to GSoC website and correct it, if necessary. Proposal is situated here: http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/neku42/1 Comments are welcome.
Re: [GSoC Proposal draft] High-Level Networking
On 04/04/11 22.06, Max Klyga wrote: Thanks to everyone who provided feedback! I'll post my proposal update tomorrow. Looking forward to it! If possible it would be nice if the common types of async methods could be supported since each have their own forces: co-routines (fibers,scheduler), Futures (e.g. IAsyncResult) and events (select/epoll/kqueue). BTW - Personally I wouldn't like a dependency on an existing C library (e.g. libev) if this work is to go into the 'std' package of Phobos. But this is just my opinion and not anything officially stated. Best of luck Jonas
Re: [GSoC Proposal draft] High-Level Networking
2011/4/4 Max Klyga max.kl...@gmail.com: Thanks to everyone who provided feedback! I'll post my proposal update tomorrow. I'm looking forward to it too! I've done a fair amount of research on these topics and given networking in D a lot of thought. I'd prefer the standard library's socket interface to be more low level than it is today, and to have a socket type that provides non-blocking IO transparently through fibers. This enables both generic protocol implementations that can handle several types of concurrency, and efficient parsers using buffered input ranges. I also think the library should offer it's own event loop instead of linking to libev or libevent, which have their weaknesses in such a context. For instance, you can't simply hand over garbage collected objects to them, which is unfortunate given the usefulness of closures for handling events. I also think a lower-level API than Boost ASIO is wise, because one may need to use novel and platform-specific optimizations like linux splice, a must for efficient file servers. As I mentioned, I have started implementing some of this. Would you be interested in pairing up with me on some parts of the project? Best regards, Torarin
Re: [GSoC Proposal draft] High-Level Networking
Thanks to everyone who provided feedback! I'll post my proposal update tomorrow.
Re: [GSoC Proposal draft] High-Level Networking
Since my proposal theme shifted a little towards asyncronicity, I've
been eagerly consuming information that might help me with this.
I would be glad to know if I am moving in to the right direction.
Comments are welcome.
So far, here is the status report:
I looked at approaches to asyncronicity in Ruby's NeverBlock, C#, Qt,
Boost.ASIO
NeverBlock uses Fibers for managing asyncronous operations. One fiber
acts as a scheduler, polling async operations for complition and
resuming other fibers as needed.
C# uses several approaches to asyn operations. There are three common
patterns now:
- Split async functon in two (BeginFoo, EndFoo). Begin... function
returns IAsyncResult object and shedules operation in process-default
thread pool. End... function takes IAsyncResult object that we got
earlier and, if operation was complited, returns computation results or
blocks until the operation is completed. All exceptions thrown in
asyncronous operation are stored and rethrown when user calls End...
function.
Any function can be called asyncronously using built in wrapper.
- Some classes provide the following interface for time consuming operations:
class Foo {
bar(...); // blocking operation
asyncBar(...); // asyncronous operation
cancelAsync();
barCompleted;
barResult;
}
- Task-based asyncronicity, similar to std.parallelism. Async
operations (BeginFoo, EndFoo) can be wrapped as a task and scheduled to
be executed on user controlled task pools.
Qt uses it's signal/slot mechanism for asyncronous operations. It's
somewhat similar to C# events, or arrays of D delegates.
Boost.ASIO is based on callbacks/delegates. It's implementation is
based on proactor design pattern and doesn't use Fibers or Threads.
I've also spent some time digging up info on event-based servers.
This is my to-read list on this weekend:
[1] Ous96 John Ousterhout. Why threads are a bad idea (for most
purposes). In USENIX Technical Conference (Invited Talk), Austin, TX,
January 1996.
[2] Rob von Behren, Jeremy Condit, and Eric Brewer. 2003. Why events
are a bad idea (for high-concurrency servers). In Proceedings of the
9th conference on Hot Topics in Operating Systems - Volume 9
(HOTOS'03), Vol. 9. USENIX Association, Berkeley, CA, USA, 4-4.
[3] Atul Adya , Jon Howell , Marvin Theimer , William J. Bolosky ,
John R. Douceur, Cooperative Task Management Without Manual Stack
Management, Proceedings of the General Track: 2002 USENIX Annual
Technical Conference, p.289-302, June 10-15, 2002 (related
presentation: http://www.cs.nyu.edu/rgrimm/teaching/fa03-web/091603.pdf)
[4] http://www.kegel.com/c10k.html
[5] http://tomasp.net/blog/csharp-fsharp-async-intro.aspx (I should
look at how F# handles asyncronicity)
Re: [GSoC Proposal draft] High-Level Networking
Mind you that Async IO is different from Non-blocking IO (which is
what most literature talks about). I am not an expert with Async IO
but it has serious limitations in Windows like the kernel thread that
starts the IO must complete it. Not very familiar with the state of
AIO in Linux.
Good luck.
On Fri, Apr 1, 2011 at 10:51 PM, Jose Armando Garcia jsan...@gmail.com wrote:
That all sounds good. I would add Flash to your investigation:
http://www.cs.princeton.edu/~vivek/flash/. There is a must read
research paper for any distributed system person at:
http://www.cs.princeton.edu/~vivek/flash_usenix_99/
On Fri, Apr 1, 2011 at 8:17 PM, Max Klyga max.kl...@gmail.com wrote:
Since my proposal theme shifted a little towards asyncronicity, I've been
eagerly consuming information that might help me with this.
I would be glad to know if I am moving in to the right direction. Comments
are welcome.
So far, here is the status report:
I looked at approaches to asyncronicity in Ruby's NeverBlock, C#, Qt,
Boost.ASIO
NeverBlock uses Fibers for managing asyncronous operations. One fiber acts
as a scheduler, polling async operations for complition and resuming other
fibers as needed.
C# uses several approaches to asyn operations. There are three common
patterns now:
- Split async functon in two (BeginFoo, EndFoo). Begin... function returns
IAsyncResult object and shedules operation in process-default thread pool.
End... function takes IAsyncResult object that we got earlier and, if
operation was complited, returns computation results or blocks until the
operation is completed. All exceptions thrown in asyncronous operation are
stored and rethrown when user calls End... function.
Any function can be called asyncronously using built in wrapper.
- Some classes provide the following interface for time consuming
operations:
class Foo {
bar(...); // blocking operation
asyncBar(...); // asyncronous operation
cancelAsync();
barCompleted;
barResult;
}
- Task-based asyncronicity, similar to std.parallelism. Async operations
(BeginFoo, EndFoo) can be wrapped as a task and scheduled to be executed on
user controlled task pools.
Qt uses it's signal/slot mechanism for asyncronous operations. It's somewhat
similar to C# events, or arrays of D delegates.
Boost.ASIO is based on callbacks/delegates. It's implementation is based on
proactor design pattern and doesn't use Fibers or Threads.
I've also spent some time digging up info on event-based servers.
This is my to-read list on this weekend:
[1] Ous96 John Ousterhout. Why threads are a bad idea (for most
purposes). In USENIX Technical Conference (Invited Talk), Austin, TX,
January 1996.
[2] Rob von Behren, Jeremy Condit, and Eric Brewer. 2003. Why events
are a bad idea (for high-concurrency servers). In Proceedings of the 9th
conference on Hot Topics in Operating Systems - Volume 9 (HOTOS'03), Vol. 9.
USENIX Association, Berkeley, CA, USA, 4-4.
[3] Atul Adya , Jon Howell , Marvin Theimer , William J. Bolosky ,
John R. Douceur, Cooperative Task Management Without Manual Stack
Management, Proceedings of the General Track: 2002 USENIX Annual Technical
Conference, p.289-302, June 10-15, 2002 (related presentation:
http://www.cs.nyu.edu/rgrimm/teaching/fa03-web/091603.pdf)
[4] http://www.kegel.com/c10k.html
[5] http://tomasp.net/blog/csharp-fsharp-async-intro.aspx (I should
look at how F# handles asyncronicity)
Re: [GSoC Proposal draft] High-Level Networking
That all sounds good. I would add Flash to your investigation:
http://www.cs.princeton.edu/~vivek/flash/. There is a must read
research paper for any distributed system person at:
http://www.cs.princeton.edu/~vivek/flash_usenix_99/
On Fri, Apr 1, 2011 at 8:17 PM, Max Klyga max.kl...@gmail.com wrote:
Since my proposal theme shifted a little towards asyncronicity, I've been
eagerly consuming information that might help me with this.
I would be glad to know if I am moving in to the right direction. Comments
are welcome.
So far, here is the status report:
I looked at approaches to asyncronicity in Ruby's NeverBlock, C#, Qt,
Boost.ASIO
NeverBlock uses Fibers for managing asyncronous operations. One fiber acts
as a scheduler, polling async operations for complition and resuming other
fibers as needed.
C# uses several approaches to asyn operations. There are three common
patterns now:
- Split async functon in two (BeginFoo, EndFoo). Begin... function returns
IAsyncResult object and shedules operation in process-default thread pool.
End... function takes IAsyncResult object that we got earlier and, if
operation was complited, returns computation results or blocks until the
operation is completed. All exceptions thrown in asyncronous operation are
stored and rethrown when user calls End... function.
Any function can be called asyncronously using built in wrapper.
- Some classes provide the following interface for time consuming
operations:
class Foo {
bar(...); // blocking operation
asyncBar(...); // asyncronous operation
cancelAsync();
barCompleted;
barResult;
}
- Task-based asyncronicity, similar to std.parallelism. Async operations
(BeginFoo, EndFoo) can be wrapped as a task and scheduled to be executed on
user controlled task pools.
Qt uses it's signal/slot mechanism for asyncronous operations. It's somewhat
similar to C# events, or arrays of D delegates.
Boost.ASIO is based on callbacks/delegates. It's implementation is based on
proactor design pattern and doesn't use Fibers or Threads.
I've also spent some time digging up info on event-based servers.
This is my to-read list on this weekend:
[1] Ous96 John Ousterhout. Why threads are a bad idea (for most
purposes). In USENIX Technical Conference (Invited Talk), Austin, TX,
January 1996.
[2] Rob von Behren, Jeremy Condit, and Eric Brewer. 2003. Why events
are a bad idea (for high-concurrency servers). In Proceedings of the 9th
conference on Hot Topics in Operating Systems - Volume 9 (HOTOS'03), Vol. 9.
USENIX Association, Berkeley, CA, USA, 4-4.
[3] Atul Adya , Jon Howell , Marvin Theimer , William J. Bolosky ,
John R. Douceur, Cooperative Task Management Without Manual Stack
Management, Proceedings of the General Track: 2002 USENIX Annual Technical
Conference, p.289-302, June 10-15, 2002 (related presentation:
http://www.cs.nyu.edu/rgrimm/teaching/fa03-web/091603.pdf)
[4] http://www.kegel.com/c10k.html
[5] http://tomasp.net/blog/csharp-fsharp-async-intro.aspx (I should
look at how F# handles asyncronicity)
Re: [GSoC Proposal draft] High-Level Networking
On 30/03/11 23.58, Max Klyga wrote: I read your Curl RFC thread, great job. Somehow I missed it when it was originaly posted. As you stated earlier, you plan to continue working on your curl wrapper, so to boost my proposal usefullness I think it needs to address unsolved problems. Could you please be more specific about your plans on networking modules? This information might help me make a stronger proposal. I written about some of this in the other thread you've started Asynchronicity in D. Thank you. Things that phobos currently lacks, but would benefit from: - Tcp Server - a class implementing TCP-based server, handling incoming connections. (QTcpServer[1] ot Net::TcpServer[2] might act as a good reference) Also have a look at the C# .net networking library and the boost asio library. - Application level protocol servers. Especially HTTP but I guess you knew that. A framework for such servers in general would be very nice. /Jonas References 1. http://doc.qt.nokia.com/latest/qtcpserver.html#details 2. http://www.ruby-doc.org/stdlib/libdoc/socket/rdoc/classes/TCPServer.html
Re: [GSoC Proposal draft] High-Level Networking
On 30/03/11 03.04, Max Klyga wrote: Google Summer of Code 2011 Proposal Draft Abstract The D programming language standard library (Phobos) lacks modules for high-level networking using application layer protocols. This project aims to provide design and implementation of networking modules for the D programming language. Project details Networking abilities are essential for modern programming languages. Currently Phobos contains modules for low-level networking (std.socket), but lacks modules that provide implementation of application level protocols (FTP, HTTP, SMPT, etc.) The goal of this proposal is to design and implement high-level interface for interaction through application level protocols. I plan to implement TELNET-client, TELNET-server, HTTP-client and FTP-client interfaces. This modules will enable D users to easily build applications interacting with web services, applications controlled via TELNET, etc. I will familiarize myself with existing networking libraries in other programming languages to gather information about commonly used techniques and interfaces that got widely adopted. This information will help me with initial design. To ensure ease of use, I will discuss design of modules with D community, seeking for idiomatic D interfaces. I will be using libcurl as a foundation for implementing this modules. This library is portable, supports a wide range of protocols[1]. Using libcurl will provide a quick start. Benefits for D - Greatly simplify creation of network-enabled applications. About me I'm Max Klyga. I am an undergraduate software engineering student at Belarusian State University of Informatics and Radioelectronics (Minsk, Belarus). I'm mainly interested in system programming and after using C++ for some time I found D and fell in love with it. I also have great interest in data-mining and programming language design. I have good C++, C# and Python skills and experience using Qt-framework. I've been successfully using D in my class projects and enjoyed that experience a lot. Lately I've been developing ODBC wrapper for D in my spare time[2]. References 1. http://en.wikipedia.org/wiki/CURL#libcurl 2. https://bitbucket.org/nekuromento/d-odbc Great to see that you're interested in improving the Phobos library! There already is some work being done on this. I've created bindings for libcurl that was accepted into Phobos a couple of days ago (see etc.c.curl module). This is the foundation of some network client classes that I'm doing. I intent to submit this to Phobos when I've implemented the last of the suggestions that I received from the community earlier. My initial target is FTP and HTTP support. See: https://github.com/jcd/phobos/blob/curl/etc/curl.d This does have some overlap with your proposal but that doesn't have to be a bad thing. If this proposal gets accepted you have the libcurl bindings already as a kickstart. Some other options are to extend the client classes that I'm doing. Or to implement the network server classes. /Jonas
Re: [GSoC Proposal draft] High-Level Networking
On 2011-03-30 17:48:59 +0300, Jonas Drewsen said: On 30/03/11 03.04, Max Klyga wrote: snip Great to see that you're interested in improving the Phobos library! There already is some work being done on this. I've created bindings for libcurl that was accepted into Phobos a couple of days ago (see etc.c.curl module). This is the foundation of some network client classes that I'm doing. I intent to submit this to Phobos when I've implemented the last of the suggestions that I received from the community earlier. Could you give links to that suggestions? Is your work on network-clients avaliable online? My initial target is FTP and HTTP support. See: https://github.com/jcd/phobos/blob/curl/etc/curl.d This does have some overlap with your proposal but that doesn't have to be a bad thing. If this proposal gets accepted you have the libcurl bindings already as a kickstart. Some other options are to extend the client classes that I'm doing. Or to implement the network server classes. RFC compilant HTTP-server is very hard to implement, so one solution might be to implement different protocol clients (SMTP, ...) so our work will not overlap. /Jonas
Re: [GSoC Proposal draft] High-Level Networking
On 03/30/2011 03:04 AM, Max Klyga wrote: Google Summer of Code 2011 Proposal Draft Abstract The D programming language standard library (Phobos) lacks modules for high-level networking using application layer protocols. This project aims to provide design and implementation of networking modules for the D programming language. Project details Networking abilities are essential for modern programming languages. Currently Phobos contains modules for low-level networking (std.socket), but lacks modules that provide implementation of application level protocols (FTP, HTTP, SMPT, etc.) The goal of this proposal is to design and implement high-level interface for interaction through application level protocols. I plan to implement TELNET-client, TELNET-server, HTTP-client and FTP-client interfaces. This modules will enable D users to easily build applications interacting with web services, applications controlled via TELNET, etc. I will familiarize myself with existing networking libraries in other programming languages to gather information about commonly used techniques and interfaces that got widely adopted. This information will help me with initial design. To ensure ease of use, I will discuss design of modules with D community, seeking for idiomatic D interfaces. I will be using libcurl as a foundation for implementing this modules. This library is portable, supports a wide range of protocols[1]. Using libcurl will provide a quick start. Benefits for D - Greatly simplify creation of network-enabled applications. Good news, I hope this project will be accepted. Anyone to mentor it? Denis -- _ vita es estrany spir.wikidot.com
Re: [GSoC Proposal draft] High-Level Networking
On 30/03/11 19.19, BlazingWhitester wrote: On 2011-03-30 17:48:59 +0300, Jonas Drewsen said: On 30/03/11 03.04, Max Klyga wrote: snip Great to see that you're interested in improving the Phobos library! There already is some work being done on this. I've created bindings for libcurl that was accepted into Phobos a couple of days ago (see etc.c.curl module). This is the foundation of some network client classes that I'm doing. I intent to submit this to Phobos when I've implemented the last of the suggestions that I received from the community earlier. Could you give links to that suggestions? Look for threads in this newsgroup that I started and that have curl or libcurl in the title. Is your work on network-clients avaliable online? Yes. I haven't pushed my working copy for some time but you can see an older version in the link I provided (see below). My initial target is FTP and HTTP support. See: https://github.com/jcd/phobos/blob/curl/etc/curl.d This does have some overlap with your proposal but that doesn't have to be a bad thing. If this proposal gets accepted you have the libcurl bindings already as a kickstart. Some other options are to extend the client classes that I'm doing. Or to implement the network server classes. RFC compilant HTTP-server is very hard to implement, so one solution might be to implement different protocol clients (SMTP, ...) so our work will not overlap. Actually I believe that a RFC compliant HTTP-server implementation with only mandatory features is not that hard to do. That would provide a good starting point for future improvements. But my initial thoughts on server classes was not HTTP specific though. It was more like a general server network framework. This framework could in turn be used to create a HTTP server if somebody wanted that. /Jonas
Re: [GSoC Proposal draft] High-Level Networking
I read your Curl RFC thread, great job. Somehow I missed it when it was originaly posted. As you stated earlier, you plan to continue working on your curl wrapper, so to boost my proposal usefullness I think it needs to address unsolved problems. Could you please be more specific about your plans on networking modules? This information might help me make a stronger proposal. Thank you. Things that phobos currently lacks, but would benefit from: - Tcp Server - a class implementing TCP-based server, handling incoming connections. (QTcpServer[1] ot Net::TcpServer[2] might act as a good reference) - Application level protocol servers. References 1. http://doc.qt.nokia.com/latest/qtcpserver.html#details 2. http://www.ruby-doc.org/stdlib/libdoc/socket/rdoc/classes/TCPServer.html
[GSoC Proposal draft] High-Level Networking
Google Summer of Code 2011 Proposal Draft Abstract The D programming language standard library (Phobos) lacks modules for high-level networking using application layer protocols. This project aims to provide design and implementation of networking modules for the D programming language. Project details Networking abilities are essential for modern programming languages. Currently Phobos contains modules for low-level networking (std.socket), but lacks modules that provide implementation of application level protocols (FTP, HTTP, SMPT, etc.) The goal of this proposal is to design and implement high-level interface for interaction through application level protocols. I plan to implement TELNET-client, TELNET-server, HTTP-client and FTP-client interfaces. This modules will enable D users to easily build applications interacting with web services, applications controlled via TELNET, etc. I will familiarize myself with existing networking libraries in other programming languages to gather information about commonly used techniques and interfaces that got widely adopted. This information will help me with initial design. To ensure ease of use, I will discuss design of modules with D community, seeking for idiomatic D interfaces. I will be using libcurl as a foundation for implementing this modules. This library is portable, supports a wide range of protocols[1]. Using libcurl will provide a quick start. Benefits for D - Greatly simplify creation of network-enabled applications. About me I'm Max Klyga. I am an undergraduate software engineering student at Belarusian State University of Informatics and Radioelectronics (Minsk, Belarus). I'm mainly interested in system programming and after using C++ for some time I found D and fell in love with it. I also have great interest in data-mining and programming language design. I have good C++, C# and Python skills and experience using Qt-framework. I've been successfully using D in my class projects and enjoyed that experience a lot. Lately I've been developing ODBC wrapper for D in my spare time[2]. References 1. http://en.wikipedia.org/wiki/CURL#libcurl 2. https://bitbucket.org/nekuromento/d-odbc
