Re: [rust-dev] Update on I/O progress

2013-04-30 Thread Vadim
I see, thank you for clarification. Can't help but note, though, that the dynamic loading case could be worked around by having a wrapper similar to the one Rust generates around calls to foreign functions, that would ensure bigger stack segment for the dynamic linker to use (only the first time a

Re: [rust-dev] Update on I/O progress

2013-04-30 Thread Brian Anderson
On 04/27/2013 03:07 AM, Jordi Boggiano wrote: Take my input with a grain of salt since I'm so new to the language, but here goes nothing: I am particularly interested in making sure that common operations are convenient and don't require a lot of boilerplate. To that end I've been collecting [a

Re: [rust-dev] Update on I/O progress

2013-04-30 Thread Brian Anderson
On 04/26/2013 04:40 PM, Vadim wrote: I share your concern. It's likely that Rust tasks will never scale as well as Erlang and I suspect that we will ultimately want an async API that does not impose the expense of 1 task per I/O stream. I hope though that the current approach wil

Re: [rust-dev] Update on I/O progress

2013-04-30 Thread Graydon Hoare
On 30/04/2013 12:17 PM, james wrote: This is what I call blocking synchronous read, a 'pull' model. Its hardly different from a threaded rogram with a userspace switch and sync to async call translation, whether N:1 or N:M. Granted that the compiler support for task switch points and segmented

Re: [rust-dev] Update on I/O progress

2013-04-30 Thread james
On 29/04/2013 20:40, Graydon Hoare wrote: On 13-04-28 12:05 PM, james wrote: >Well, the issue is whether the program logic that will deal with data >calls (say) read and iterates over the response, or whether something >gets data and invokes a callback, in which case the code that deals with >t

Re: [rust-dev] Update on I/O progress

2013-04-29 Thread Gábor Lehel
On Fri, Apr 26, 2013 at 11:11 PM, Brian Anderson wrote: > On 04/26/2013 09:04 AM, Patrick Walton wrote: > >> On 4/26/13 6:53 AM, Gábor Lehel wrote: >> >>> Probably a crazy idea, but would it be possible to do something where if >>> the parent task launches a child task, and then does nothing else

Re: [rust-dev] Update on I/O progress

2013-04-29 Thread Graydon Hoare
On 13-04-28 12:05 PM, james wrote: > Well, the issue is whether the program logic that will deal with data > calls (say) read and iterates over the response, or whether something > gets data and invokes a callback, in which case the code that deals with > the data only has a thread (or stack if yo

Re: [rust-dev] Update on I/O progress

2013-04-28 Thread james
>It seems to me that, by default, what we'd want is a sync api which under the hood, creates one task per drive accessed, or maybe max(drives_in_use, number_of_cores). I think the OS will make them async with the drive hardware anyway, though, so number_of_cores doesn't matter. I don't think

Re: [rust-dev] Update on I/O progress

2013-04-28 Thread james
On 26/04/2013 17:06, Patrick Walton wrote: On 4/25/13 10:42 PM, james wrote: If not, I wonder if there should be more effort on the AIO layer first, and then consider a synchronous shim on top. That's exactly what's being done. Patrick OK. It wasn't clear whether the AIO layer was primaril

Re: [rust-dev] Update on I/O progress

2013-04-27 Thread David Bruant
Le 27/04/2013 19:30, Patrick Walton a écrit : On 4/27/13 4:06 AM, David Bruant wrote: I believe too this is doomed to happen. Firefox is under a huge struggle to remove as much as sync IO code as possible as part of the Snappy effort [1]. This is a completely misleading comparison. Firefox's s

Re: [rust-dev] Update on I/O progress

2013-04-27 Thread Mikhail Zabaluev
Hi Lee, 2013/4/27 Lee Braiden : > > But we're not talking about creating a new task for every io operation as > the io module's usual under-the-hood strategy, surely? > > It seems to me that, by default, what we'd want is a sync api which under > the hood, creates one task per drive accessed, or m

Re: [rust-dev] Update on I/O progress

2013-04-27 Thread Lee Braiden
Jordi Boggiano Wrote: Feels weird to have str implement something specific to another module, but if there is no other way it sounds more user friendly than a gazillion open_* methods. A few more comments on other things: - Streams are great, and the ability to register new stream protocol han

Re: [rust-dev] Update on I/O progress

2013-04-27 Thread Patrick Walton
On 4/27/13 4:06 AM, David Bruant wrote: I believe too this is doomed to happen. Firefox is under a huge struggle to remove as much as sync IO code as possible as part of the Snappy effort [1]. This is a completely misleading comparison. Firefox's sync I/O *blocks the entire main thread of the

Re: [rust-dev] Update on I/O progress

2013-04-27 Thread Daniel Micay
On Sat, Apr 27, 2013 at 12:10 PM, Lee Braiden wrote: > On 27/04/13 12:06, David Bruant wrote: > > Le 26/04/2013 23:23, Brian Anderson a écrit : > > I do think 50,000 I/O-performing tasks in a 32-bit process is within our > reach. Last year I demonstrated 500,000 simultaneous Rust tasks (that did >

Re: [rust-dev] Update on I/O progress

2013-04-27 Thread Lee Braiden
On 27/04/13 12:06, David Bruant wrote: Le 26/04/2013 23:23, Brian Anderson a écrit : I do think 50,000 I/O-performing tasks in a 32-bit process is within our reach. Last year I demonstrated 500,000 simultaneous Rust tasks (that did nothing but wait) in a 32-bit process, but there have been sev

Re: [rust-dev] Update on I/O progress

2013-04-27 Thread David Bruant
Le 26/04/2013 23:23, Brian Anderson a écrit : On 04/25/2013 10:42 PM, james wrote: On 24/04/2013 23:38, Brian Anderson wrote: What I am focused on now is the design of a synchronous API for sending and receiving streams of bytes, along with an implementation built on an internal *asynchronous*

Re: [rust-dev] Update on I/O progress

2013-04-27 Thread Jordi Boggiano
Take my input with a grain of salt since I'm so new to the language, but here goes nothing: > I am particularly interested in making sure that common operations are > convenient and don't require a lot of boilerplate. To that end I've been > collecting [aspirational examples] of succinctly writt

Re: [rust-dev] Update on I/O progress

2013-04-26 Thread Vadim
> I share your concern. It's likely that Rust tasks will never scale as well > as Erlang and I suspect that we will ultimately want an async API that does > not impose the expense of 1 task per I/O stream. I hope though that the > current approach will scale 'pretty well'. > Is there something inh

Re: [rust-dev] Update on I/O progress

2013-04-26 Thread Brian Anderson
On 04/25/2013 06:09 PM, Vadim wrote: What if Result implemented the Drop trait and fail()'ed in drop() unless it's been "disarmed" by calling some method (get_err?) which indicates that the error has been observed? I've had a few bits of code recently that called for this sort of pattern (s

Re: [rust-dev] Update on I/O progress

2013-04-26 Thread Brian Anderson
On 04/25/2013 10:42 PM, james wrote: On 24/04/2013 23:38, Brian Anderson wrote: What I am focused on now is the design of a synchronous API for sending and receiving streams of bytes, along with an implementation built on an internal *asynchronous* I/O interface attached to the scheduler event

Re: [rust-dev] Update on I/O progress

2013-04-26 Thread Brian Anderson
On 04/26/2013 09:04 AM, Patrick Walton wrote: On 4/26/13 6:53 AM, Gábor Lehel wrote: Probably a crazy idea, but would it be possible to do something where if the parent task launches a child task, and then does nothing else except wait for the child to finish, it's not actually implemented by la

Re: [rust-dev] Update on I/O progress

2013-04-26 Thread Patrick Walton
So here are my ideas. They are very worse-is-better at this point. * Failing to perform a basic I/O operation should result in a call to the `io_error` condition (default behavior: fail task) and should set an internal flag saying that this stream is dead. Stream constructors like `open` shoul

Re: [rust-dev] Update on I/O progress

2013-04-26 Thread Graydon Hoare
On 25/04/2013 10:42 PM, james wrote: This always concerns me - the design will tend towards having state on the stack etc and 'pull' processing rather than focussing on a clean design for a chain of state machines that handle 'push' of data coming from AIO completion events. I think "push" and

Re: [rust-dev] Update on I/O progress

2013-04-26 Thread Patrick Walton
On 4/25/13 10:42 PM, james wrote: If not, I wonder if there should be more effort on the AIO layer first, and then consider a synchronous shim on top. That's exactly what's being done. Patrick ___ Rust-dev mailing list Rust-dev@mozilla.org https://m

Re: [rust-dev] Update on I/O progress

2013-04-26 Thread Patrick Walton
On 4/26/13 6:53 AM, Gábor Lehel wrote: Probably a crazy idea, but would it be possible to do something where if the parent task launches a child task, and then does nothing else except wait for the child to finish, it's not actually implemented by launching a new task, but instead by calling the

Re: [rust-dev] Update on I/O progress

2013-04-26 Thread Gábor Lehel
Probably a crazy idea, but would it be possible to do something where if the parent task launches a child task, and then does nothing else except wait for the child to finish, it's not actually implemented by launching a new task, but instead by calling the closure in place? It seems semantically e

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread james
On 24/04/2013 23:38, Brian Anderson wrote: What I am focused on now is the design of a synchronous API for sending and receiving streams of bytes, along with an implementation built on an internal *asynchronous* I/O interface attached to the scheduler event loop. This includes reading and writi

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Graydon Hoare
On 25/04/2013 5:29 PM, Brian Anderson wrote: It's almost taboo here, but we could consider adding catchable exceptions. For reference sake on this taboo -- let it not be too opaque! -- I will point to the amazing powers of IRC logging, and the conversation that contains the memorable phrase:

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Gábor Lehel
On Fri, Apr 26, 2013 at 2:29 AM, Brian Anderson wrote: > It's almost taboo here, but we could consider adding catchable exceptions. > ...which would let you default to the failing/throwing versions of functions, and allow (or obviate) an efficient translation to Result/Option. (Though: the None/E

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Vadim
What if Result implemented the Drop trait and fail()'ed in drop() unless it's been "disarmed" by calling some method (get_err?) which indicates that the error has been observed? On Thu, Apr 25, 2013 at 5:30 PM, Patrick Walton wrote: > On 4/25/13 5:29 PM, Brian Anderson wrote: > >> It's almost ta

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Vadim
> > > Then nil-returning functions would need to be `Option`. >> Having two ways to perform every I/O operation will 'infect' all code that >> uses I/O indirectly. For instance, we have a `ReaderUtil` trait that >> defines maybe two dozen methods. Do those all get duplicated? That seems >> pretty d

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Patrick Walton
On 4/25/13 5:29 PM, Brian Anderson wrote: It's almost taboo here, but we could consider adding catchable exceptions. I would be in favor of catchable exceptions, for what it's worth. Patrick ___ Rust-dev mailing list Rust-dev@mozilla.org https://mai

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Brian Anderson
On 04/25/2013 05:08 PM, Gábor Lehel wrote: On Thu, Apr 25, 2013 at 10:00 PM, Brian Anderson mailto:bander...@mozilla.com>> wrote: On 04/25/2013 07:18 AM, Gábor Lehel wrote: Whether to respond to failure by failing the task or by indicating it in the return value seems l

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Gábor Lehel
On Thu, Apr 25, 2013 at 10:00 PM, Brian Anderson wrote: > On 04/25/2013 07:18 AM, Gábor Lehel wrote: > >> >> Whether to respond to failure by failing the task or by indicating it in >> the return value seems like it would be better handled by having separate >> functions for each. In the case wher

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Graydon Hoare
On 25/04/2013 4:02 PM, Brian Anderson wrote: The main TCP implementation, `TcpStream for ~RtTcpStream`, will delegate to whatever boxed RtTcpStream it is given. Because TcpStream is responsible for raising conditions, then you can configure the scheduler to provide you with either uv-based or na

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Brian Anderson
On 04/25/2013 03:46 PM, Brian Anderson wrote: On 04/25/2013 12:09 PM, Graydon Hoare wrote: and so forth on other types. Then your file conditions would look like: condition! { no_such_file : Path -> FileStream; } and condition! { write_error : (libc::fd_t, libc::errno_t) -> FileStream

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Patrick Walton
On 4/25/13 3:46 PM, Brian Anderson wrote: enum FileStream { RtFileStream(rtio::FileStream), // Not an fd_t, a boxed wrapper around uv/rt handles FileSimulation(~Stream), NullFileStream } With this factoring though you are going to be duplicating a lot of logic for FileStream, TcpS

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Brian Anderson
On 04/25/2013 12:09 PM, Graydon Hoare wrote: On 25/04/2013 7:18 AM, Gábor Lehel wrote: I'm an outsider, so apologies if I'm off the right track. But this sounds wrong to me. Isn't the point of condition handlers to /handle the condition/? Meaning resolve the problem, so that processing can cont

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Brian Anderson
On 04/25/2013 07:18 AM, Gábor Lehel wrote: You write: > So what actually happens if `new` encounters an error? To understand > that it's important to know that what `new` returns is not a `File` > but an `Option`. If the file does not open, and the condition > is handled, then `new` will simply

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Brian Anderson
On 04/25/2013 07:12 AM, Daniel Micay wrote: On Thu, Apr 25, 2013 at 8:25 AM, Nathan Myers wrote: On 04/24/2013 03:38 PM, Brian Anderson wrote: ## String encoding I have not yet put much thought about how to deal with string encoding and decoding. The existing `io` module simply has Reader and

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Graydon Hoare
On 25/04/2013 7:18 AM, Gábor Lehel wrote: I'm an outsider, so apologies if I'm off the right track. But this sounds wrong to me. Isn't the point of condition handlers to /handle the condition/? Meaning resolve the problem, so that processing can continue? Here as far as I can tell, they're only

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Brian Anderson
On 04/25/2013 05:25 AM, Nathan Myers wrote: On 04/24/2013 03:38 PM, Brian Anderson wrote: ## String encoding I have not yet put much thought about how to deal with string encoding and decoding. The existing `io` module simply has Reader and Writer extension traits that add a number of methods

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Gábor Lehel
You write: > So what actually happens if `new` encounters an error? To understand > that it's important to know that what `new` returns is not a `File` > but an `Option`. If the file does not open, and the condition > is handled, then `new` will simply return `None`. Because there is an > implemen

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Daniel Micay
On Thu, Apr 25, 2013 at 8:25 AM, Nathan Myers wrote: > On 04/24/2013 03:38 PM, Brian Anderson wrote: >> >> ## String encoding >> >> I have not yet put much thought about how to deal with string encoding and >> decoding. The existing `io` module simply has Reader and Writer extension >> traits that

Re: [rust-dev] Update on I/O progress

2013-04-25 Thread Nathan Myers
On 04/24/2013 03:38 PM, Brian Anderson wrote: ## String encoding I have not yet put much thought about how to deal with string encoding and decoding. The existing `io` module simply has Reader and Writer extension traits that add a number of methods like `read_str`, etc. I think this is the w

[rust-dev] Update on I/O progress

2013-04-24 Thread Brian Anderson
Hi. As you may know, I'm in the process of redesigning the I/O module on top of a new task scheduler. With my [latest pull request] I've built enough infrastructure that I'm about ready to start implementing TCP streams. Before I get too deep though I want to make sure that everybody who care