std.stdio overhaul by Steve Schveighoffer
Hello, There are a number of issues related to D's current handling of streams, including the existence of the imperfect etc.stream and the over-specialization of std.stdio. Steve has worked on an extensive overhaul of std.stdio which would obviate the need for etc.stream and would improve both the generality and efficiency of std.stdio. Please chime in with feedback; he's away from the Usenet but allowed me to post this on his behalf. I uploaded the docs to http://erdani.com/d/new-stdio/phobos-prerelease/std_stdio.html Thanks, Andrei
Re: std.stdio overhaul by Steve Schveighoffer
On Sat, Sep 3, 2011 at 12:54 PM, Andrei Alexandrescu wrote: > Hello, > > > There are a number of issues related to D's current handling of streams, > including the existence of the imperfect etc.stream and the > over-specialization of std.stdio. > > Steve has worked on an extensive overhaul of std.stdio which would obviate > the need for etc.stream and would improve both the generality and efficiency > of std.stdio. > > Please chime in with feedback; he's away from the Usenet but allowed me to > post this on his behalf. I uploaded the docs to > > http://erdani.com/d/new-stdio/phobos-prerelease/std_stdio.html > Interesting. How does this work with RAII? Where is the source code?
Re: std.stdio overhaul by Steve Schveighoffer
On 09/03/2011 09:54 PM, Andrei Alexandrescu wrote: Hello, There are a number of issues related to D's current handling of streams, including the existence of the imperfect etc.stream and the over-specialization of std.stdio. Steve has worked on an extensive overhaul of std.stdio which would obviate the need for etc.stream and would improve both the generality and efficiency of std.stdio. Please chime in with feedback; he's away from the Usenet but allowed me to post this on his behalf. I uploaded the docs to http://erdani.com/d/new-stdio/phobos-prerelease/std_stdio.html Thanks, Andrei File is now a class. This will break a lot of code. What is happening to the refcounted File feature? It seems that the new way of file handling, using a file class, is more error prone than the old way? But it is really great to hear that the efficiency problems of std.stdio are being sorted out!
Re: std.stdio overhaul by Steve Schveighoffer
== Quote from Andrei Alexandrescu (seewebsiteforem...@erdani.org)'s article > Hello, > There are a number of issues related to D's current handling of streams, > including the existence of the imperfect etc.stream and the > over-specialization of std.stdio. > Steve has worked on an extensive overhaul of std.stdio which would > obviate the need for etc.stream and would improve both the generality > and efficiency of std.stdio. > Please chime in with feedback; he's away from the Usenet but allowed me > to post this on his behalf. I uploaded the docs to > http://erdani.com/d/new-stdio/phobos-prerelease/std_stdio.html > Thanks, > Andrei After a quick look, I have two concerns: 1. File is a class, not a struct. This precludes using reference counting as the current std.stdio.File does, meaning you have to close all your Files manually. I loved the reference counting semantics, especially the last few releases since most of the relevant compiler bugs have been fixed. 2. File(someFileName, someMode) needs to work. Not supporting this method of instantiating a File object would break way too much code.
Re: std.stdio overhaul by Steve Schveighoffer
On 2011-09-03 19:54:05 +, Andrei Alexandrescu said: Hello, There are a number of issues related to D's current handling of streams, including the existence of the imperfect etc.stream and the over-specialization of std.stdio. Steve has worked on an extensive overhaul of std.stdio which would obviate the need for etc.stream and would improve both the generality and efficiency of std.stdio. Please chime in with feedback; he's away from the Usenet but allowed me to post this on his behalf. I uploaded the docs to http://erdani.com/d/new-stdio/phobos-prerelease/std_stdio.html Looks good… Hum, inconsistent casing of enum members. And shouldn't there be a way to do non-blocking IO? ;-) I like that File is now a class because it's cleaner that way, but non-deterministic destruction is going to be a problem. That said, it was already a problem anyway if you stored a File struct in a class, so maybe we need a more general solution for reference-counted classes. Class names DInput and DOutput sounds silly. If all classes implemented purely in D had a D prefix, it'd get redundant pretty fast (like KDE apps beginning in K). I'd suggest BufferedInput and BufferedOutput, or something else that actually describes what the class does, instead of DInput and DOutput. And I'd make them final, that way there won't be any virtual call overhead until the buffer needs to be replenished or flushed from the wrapped input or output stream. -- Michel Fortin michel.for...@michelf.com http://michelf.com/
Re: std.stdio overhaul by Steve Schveighoffer
Actually I'll generalize the comment I made before: As much as I like more efficiency, I despise the massive overhaul and code breakage and the complexity of having a zillion tiny objects to do everything that File used to do. I would like to see the native I/O under the hood plus something more like the current API for basic file I/O. I'd vote against the current design just because of the massive code breakage it would cause with no migration plan.
Re: std.stdio overhaul by Steve Schveighoffer
On 9/3/11 11:20 PM, dsimcha wrote: 2. File(someFileName, someMode) needs to work. Not supporting this method of instantiating a File object would break way too much code. This one could easily be solved by aliasing File.open to (static) opCall(). David
Re: std.stdio overhaul by Steve Schveighoffer
== Quote from David Nadlinger (s...@klickverbot.at)'s article > On 9/3/11 11:20 PM, dsimcha wrote: > > 2. File(someFileName, someMode) needs to work. Not supporting this method > > of > > instantiating a File object would break way too much code. > This one could easily be solved by aliasing File.open to (static) opCall(). > David Agreed, but in the big picture this overhaul still breaks way too much code without either a clear migration path or a clear argument about why such extensive breakage is necessary. The part about File(someFileName, someMode) is just the first thing I noticed.
Re: std.stdio overhaul by Steve Schveighoffer
I dislike naming things with a leading "D" like "DInput". Shouldn't we keep code that relies on C to be put in etc.c or somewhere?
Re: std.stdio overhaul by Steve Schveighoffer
Also, changing structs to classes is gonna *massively* break code everywhere. Why inheritance instead of a predicate like isInputStream = is(typeof(T t; t.put; t.close)), you know the drill..
Re: std.stdio overhaul by Steve Schveighoffer
What happens if I write:
printf("hello ");
writeln("world");
?
Re: std.stdio overhaul by Steve Schveighoffer
Am 04.09.2011, 00:57 Uhr, schrieb Andrej Mitrovic : Also, changing structs to classes is gonna *massively* break code everywhere. Why inheritance instead of a predicate like isInputStream = is(typeof(T t; t.put; t.close)), you know the drill.. Wasn't this overhaul _meant_ to break existing code by offering a new API? Still that's a serious issue of course, but not too surprising. I'm ambivalent on the inheritance vs predicate debate. Interfaces are the way it is meant to be done and actually ensure correct types. Predicates work with structs as well. I don't know if this would be important.
Re: std.stdio overhaul by Steve Schveighoffer
On Sunday, September 04, 2011 02:49:40 Marco Leise wrote: > Am 04.09.2011, 00:57 Uhr, schrieb Andrej Mitrovic > > : > > Also, changing structs to classes is gonna *massively* break code > > everywhere. Why inheritance instead of a predicate like isInputStream > > = is(typeof(T t; t.put; t.close)), you know the drill.. > > Wasn't this overhaul _meant_ to break existing code by offering a new API? > Still that's a serious issue of course, but not too surprising. I'm > ambivalent on the inheritance vs predicate debate. Interfaces are the way > it is meant to be done and actually ensure correct types. Predicates work > with structs as well. I don't know if this would be important. Any overhaul of existing functionality needs to improve on existing functionality. Changes just to change aren't valuable. So, changes should generally avoiding breaking backwards compatibility unless we gain something from it. So, as long as these changes are an overall improvement, then we'll just have to deal with the code breakage. However, if the code breakage doesn't actually gain us anything, then we should avoid it. So, complaints about code breakage are valid, but they aren't deal breaking. - Jonathan M Davis
Re: std.stdio overhaul by Steve Schveighoffer
On 9/3/2011 3:53 PM, dsimcha wrote: Agreed, but in the big picture this overhaul still breaks way too much code without either a clear migration path or a clear argument about why such extensive breakage is necessary. The part about File(someFileName, someMode) is just the first thing I noticed. [rant] I agree. I agree that std.stream should be replaced, but I have a lot of misgivings about replacing std.stdio. I do not want to rewrite every darn D program I've ever written. I think it is a bad idea to break everyone else's D program. Everything in dsource will break in non-trivial ways. I don't think we can afford this. I do not know of any successful system or language that breaks user code with such aplomb as D does. Not even C++ dares to break that Piece Of S*** that everyone knows iostreams is. I can compile and run unix C code from 30 years ago on Linux with no changes at all. Same with DOS code. There needs to be huge improvement to justify such breakage. [I also don't like it that all my code that uses std.path is now broken.] I would prefer to see all the energy that is going into refactoring existing, working modules go into designing new, not existing, modules that there's a crying need for. [/rant] Enough ranting for now, as for the proposed std.stdio, 1. It does look fairly straightforward, but: 2. There is only one example. Have any commonly done programming tasks been tried out with it to see how they work? 3. There is no indication of how it interacts with C stdio. A primary goal of std.stdio was interoperability with C stdio. 4. There are no benchmarks. The current std.stdio was designed/written in parallel with some benchmarks Andrei and others cooked up, as a primary goal was performance. 5. flushCheck - flushing should be done based on the file type. tty's should be \n flushed, files when the buffer is full. I question the performance of using a delegate to check for flushing. How often will it be called? 6. There is no provision for multithreaded writing, i.e. what happens when two threads write to stdout. Ideally, there should be a way to 'lock' the stream to oneself, in order to appropriately interleave the output. 7. I see nothing for 'raw' character by character input. 8. I see nothing for determining if a char is available on the input. How would one implement "press any key to continue"?
Re: std.stdio overhaul by Steve Schveighoffer
On 9/3/11 3:54 PM, Andrei Alexandrescu wrote: http://erdani.com/d/new-stdio/phobos-prerelease/std_stdio.html Here are a few points following a pass through the dox: * After thinking some more about it, I find the approach seek() plus enumerated Anchor undesirable. It's a bad case of logical coupling as one never calls seek() passing an anchor as a variable. It's really three functions - seekForward, seekBackward, and seekAbsolute. Heck, knowing what seek does, it should be just seekAbsolute. But then there are several possible designs; a logically coupled seek() is not a good turn in any case. * Seekable should document that tell() is O(1) and seek() can be considered O(1) but with a large constant factor. * Why is close() not part of Seekable, since Seekable seems to be the base of all streams? * Class File is IMHO not going to cut the mustard. It needs to be a struct with a destructor. One should be able to _get_ an InputStream or an OutputStream interface out of a File object (i.e. a file is a factory of such interfaces), but the File itself must be a struct. * I don't understand the difference between read() and readComplete(). * readUntil is a bit tenuous. I was hoping for a simpler interface to buffered streams, e.g. just expose the buffer as a ubyte[]. * readUntil(const(ubyte)[]) does not give a cheap means to figure whether the read ended because file ended or the terminator was met. * There's several readUntil but only one appendUntil. Why? * Document the difference between skip and seek. Also, skip should take a ulong. * I see encoder and decoder() in DInput, should both be decoder? * StreamWidth, TextXXX and friends are a bit sudden because they introduce a higher-level abstraction in a module so far only preoccupied to transferring bytes. I was thinking that kind of stuff would belong to a formatter/serializer module. Overall, there are interesting elements in this proposal but I don't quite feel it hit the proverbial nail on the head. Andrei
Re: std.stdio overhaul by Steve Schveighoffer
On 9/3/2011 5:58 PM, Jonathan M Davis wrote: However, if the code breakage doesn't actually gain us anything, then we should avoid it. So, complaints about code breakage are valid, but they aren't deal breaking. The larger the amount of code that is broken, the more gain there must be to justify it. Breaking std.stdio, which is used everywhere, this thoroughly needs a very high bar of justification.
Re: std.stdio overhaul by Steve Schveighoffer
On Sat, 03 Sep 2011 15:54:05 -0400, Andrei Alexandrescu wrote: Hello, There are a number of issues related to D's current handling of streams, including the existence of the imperfect etc.stream and the over-specialization of std.stdio. Steve has worked on an extensive overhaul of std.stdio which would obviate the need for etc.stream and would improve both the generality and efficiency of std.stdio. Please chime in with feedback; he's away from the Usenet but allowed me to post this on his behalf. I uploaded the docs to http://erdani.com/d/new-stdio/phobos-prerelease/std_stdio.html Thank you Andrei for posting this. Before I add some more details, let me first say, this is a very early version, but it does work (and spanks the pants off of the current stdio in the tests I've run). I'll add several very important things: 1. At the moment, this is written for Linux *ONLY*. I have very good experience with Windows i/o, and I am 100% certain I can implement this library for it. However, it's not my main OS, so I wanted to first get something working with my main working environment. 2. This is *not* currently multithread aware. But it will be. However, I think one important aspect to consider is to make a *thread-local* aware i/o library to avoid unnecessary locking when an i/o connection is only used in one thread. But please leave that part alone for now, I'm working on how to make the code reusable as shared types. Actually, if anyone has good ideas on that, please share! 3. Although I am dead-set on getting *something* into Phobos, I am not attached at all to the symbol names, or even some major design choices. I have seen so far it's one of the major concerns, and I think we can find good names. The names I came up with are not exactly arbitrary, but they are somewhat based on earlier designs that I have since abandoned, so renaming is definitely in order. 4. You can get the full source here: https://github.com/schveiguy/phobos/tree/new-io I used the 2.054 stock compiler, and a version of druntime that includes Lars' new std-process changes, also on my github account: https://github.com/schveiguy/druntime/tree/new-std-process Please use those when trying out the code. -- So let me tell you about the library design and why I did it the way I did it. Then, I'll respond to individual concerns already posted. The major problem I think the current std.stdio has is, it's buffered solution is based on C's FILE * implementation. Specifically, we have very little control and access to the buffer implementation. I think the key (or at least one of the keys) to uber-fast I/O is trying to copy as little as possible *needlessly*. Seamless and safe buffer access I think is the key to this. In addition to that, C's FILE * has several limitations: 1. On Windows, it's based on DMC's runtime, which limits 60 simultaneous open files (Windows OS limit is 10,000 I think) 2. 64-bit support is not standard in all C implementations (namely Windows) 3. All FILE * objects are inherently shared, meaning lock-free I/O is very cumbersome, especially considering we have D's shared/unshared system. 4. C supports UTF-8, and it's supposed to support UTF-16 (but I can't get UTF-16 to work). I think D ought to support all forms of UTF, since UTF is an integral part of the language. In addition to this, we have numerous D tools at our disposal -- delegates, closures, ranges, etc. In other words, limiting us to C's interfaces means either duct-taping on those features, or abandoning them. While a noble effort, and probably the best we could get, a prime example is the LockingFileReader range in std.stdio. Just reading it made me cringe. Have a look: https://github.com/D-Programming-Language/phobos/blob/master/std/stdio.d#L1282 I felt, we must be able to do something better. So I started creating what I thought would be a good i/o library. I did not start from the existing code, but just rewrote everything. The basic concept is, we implement buffering once, and implement low-level devices that can be wrapped by the buffering implementation. Almost everything that would use I/O wants to use a buffered version of it, so make the low-level aggregate minimal, and put all the useful functionality into the buffer. I also wanted to make sure it is very easy to implement *efficient* ranges. One design decision early on is that the device-level should be a class. There are a few good reasons for this: 1. an I/O device is a reference-type. Copying it does not open another handle. So even if we *wanted* structs, they would be pImpl structs. 2. One simple idea that works very well at the OS level is the file descriptor concept. The file descriptor provides an *interface* to user code for operating on a stream. And they are easily inter-changeable. This means
Re: std.stdio overhaul by Steve Schveighoffer
Seems to me like virtually every module in Phobos gets a complete rewrite sooner or later. Yikes! Afaik the upcoming ones are also std.xml, std.variant, maybe std.json too? (can't recall). Was there really so much bad code written in Phobos all along that they all require a rewrite?
Re: std.stdio overhaul by Steve Schveighoffer
== Quote from Jonathan M Davis (jmdavisp...@gmx.com)'s article > Any overhaul of existing functionality needs to improve on existing > functionality. Changes just to change aren't valuable. So, changes should > generally avoiding breaking backwards compatibility unless we gain something > from it. So, as long as these changes are an overall improvement, then we'll > just have to deal with the code breakage. However, if the code breakage > doesn't actually gain us anything, then we should avoid it. So, complaints > about code breakage are valid, but they aren't deal breaking. > - Jonathan M Davis I mostly agree with what you said, except that this proposal breaks a frequently used standard library module severely and without a clear gradual migration path.
Re: std.stdio overhaul by Steve Schveighoffer
I will come back with some more detailed feedback later on, but a few nits that caught my eye: - I don't think changing file from being a struct to a class is a good idea. First, it breaks an awful lot of D/Phobos programs already out there, both because of the struct->class change and because of the other API changes. Second, I feel we should really try to make use of RAII for things like file handles – I know we have »scope (exit) file.close()«, but forcing the user to remember to always type that needs a very good reason, imho. Couldn't File rather have some factory methods returning stream interface implementations? - CStream and DInput/Output? I don't care how it is implemented under the hood, give me something that works! ;) In this case, I guess CStream is somewhat appropriate, as C (FILE*) streams are widely known, but still I'm not too fond of the names. - bufsize -> bufSize? - Why on earth does DDoc render the enum default parameter as »(Anchor).Begin«? Is there a bug report for this? - I am sure there is a reason why the design uses decoder delegates, but without the source being available, I didn't find it immediately obvious where the advantages of using it over processing what is being read() from the stream are. Is this so data can be processed before going into the buffer? On a related note, what seems to be the decoder property getter is named »encoder()«. David On 9/3/11 9:54 PM, Andrei Alexandrescu wrote: Hello, There are a number of issues related to D's current handling of streams, including the existence of the imperfect etc.stream and the over-specialization of std.stdio. Steve has worked on an extensive overhaul of std.stdio which would obviate the need for etc.stream and would improve both the generality and efficiency of std.stdio. Please chime in with feedback; he's away from the Usenet but allowed me to post this on his behalf. I uploaded the docs to http://erdani.com/d/new-stdio/phobos-prerelease/std_stdio.html Thanks, Andrei
Re: std.stdio overhaul by Steve Schveighoffer
On Sat, 03 Sep 2011 17:20:53 -0400, dsimcha wrote: == Quote from Andrei Alexandrescu (seewebsiteforem...@erdani.org)'s article Hello, There are a number of issues related to D's current handling of streams, including the existence of the imperfect etc.stream and the over-specialization of std.stdio. Steve has worked on an extensive overhaul of std.stdio which would obviate the need for etc.stream and would improve both the generality and efficiency of std.stdio. Please chime in with feedback; he's away from the Usenet but allowed me to post this on his behalf. I uploaded the docs to http://erdani.com/d/new-stdio/phobos-prerelease/std_stdio.html Thanks, Andrei After a quick look, I have two concerns: 1. File is a class, not a struct. This precludes using reference counting as the current std.stdio.File does, meaning you have to close all your Files manually. I loved the reference counting semantics, especially the last few releases since most of the relevant compiler bugs have been fixed. As long as a class can contain a File as a member, this argument makes no sense to me. In other words, it's impossible to remove the GC from the File destructor/refcounting system. I think what may end up happening, in terms of File being a scoped entity is: File becomes a struct. File's sole member is a class that implements InputStream, OutputStream, and ref counting. This would be roughly equivalent to today's File. Except it's not buffered. I think the names need work, and you are very right to point out that we should make existing code work as much as possible. 2. File(someFileName, someMode) needs to work. Not supporting this method of instantiating a File object would break way too much code. I can change File.open to File.opCall, that will fix that. -Steve
Re: std.stdio overhaul by Steve Schveighoffer
== Quote from Andrej Mitrovic (andrej.mitrov...@gmail.com)'s article > Seems to me like virtually every module in Phobos gets a complete > rewrite sooner or later. Yikes! Afaik the upcoming ones are also > std.xml, std.variant, maybe std.json too? (can't recall). Was there > really so much bad code written in Phobos all along that they all > require a rewrite? It's really amazing how much cruft 2-3 year old D code tends to have: Workarounds for compiler bugs, workarounds for previously missing features, a generally lower standard for quality before we implemented a proper review process, etc. Heck, I've got a pull request in Github that rewrites a substantial portion of std.parallelism to take advantage of better implementations I've found for parallel foreach and amap, fix a couple bugs and get rid of tons of cruft, and this module's only been in Phobos a few months. These changes are purely under the hood, though, and there should be zero code breakage.
Re: std.stdio overhaul by Steve Schveighoffer
On 9/3/11 9:53 PM, Walter Bright wrote: On 9/3/2011 5:58 PM, Jonathan M Davis wrote: However, if the code breakage doesn't actually gain us anything, then we should avoid it. So, complaints about code breakage are valid, but they aren't deal breaking. The larger the amount of code that is broken, the more gain there must be to justify it. Breaking std.stdio, which is used everywhere, this thoroughly needs a very high bar of justification. I agree. I'm hoping the new stuff could build on top of std.stdio. Andrei
Re: std.stdio overhaul by Steve Schveighoffer
On 9/3/11 10:11 PM, Steven Schveighoffer wrote: On Sat, 03 Sep 2011 17:20:53 -0400, dsimcha wrote: == Quote from Andrei Alexandrescu (seewebsiteforem...@erdani.org)'s article Hello, There are a number of issues related to D's current handling of streams, including the existence of the imperfect etc.stream and the over-specialization of std.stdio. Steve has worked on an extensive overhaul of std.stdio which would obviate the need for etc.stream and would improve both the generality and efficiency of std.stdio. Please chime in with feedback; he's away from the Usenet but allowed me to post this on his behalf. I uploaded the docs to http://erdani.com/d/new-stdio/phobos-prerelease/std_stdio.html Thanks, Andrei After a quick look, I have two concerns: 1. File is a class, not a struct. This precludes using reference counting as the current std.stdio.File does, meaning you have to close all your Files manually. I loved the reference counting semantics, especially the last few releases since most of the relevant compiler bugs have been fixed. As long as a class can contain a File as a member, this argument makes no sense to me. In other words, it's impossible to remove the GC from the File destructor/refcounting system. The meaning of the argument is that just because there is the possibility of a File leaking, we shouldn't increase the likelihood of such a leak. Andrei
Re: std.stdio overhaul by Steve Schveighoffer
On Sat, 03 Sep 2011 21:23:26 -0400, Walter Bright wrote: On 9/3/2011 3:53 PM, dsimcha wrote: Agreed, but in the big picture this overhaul still breaks way too much code without either a clear migration path or a clear argument about why such extensive breakage is necessary. The part about File(someFileName, someMode) is just the first thing I noticed. [rant] I agree. I agree that std.stream should be replaced, but I have a lot of misgivings about replacing std.stdio. I do not want to rewrite every darn D program I've ever written. I think it is a bad idea to break everyone else's D program. Everything in dsource will break in non-trivial ways. I don't think we can afford this. I do not know of any successful system or language that breaks user code with such aplomb as D does. Not even C++ dares to break that Piece Of S*** that everyone knows iostreams is. I can compile and run unix C code from 30 years ago on Linux with no changes at all. Same with DOS code. There needs to be huge improvement to justify such breakage. [I also don't like it that all my code that uses std.path is now broken.] I would prefer to see all the energy that is going into refactoring existing, working modules go into designing new, not existing, modules that there's a crying need for. [/rant] Please, leave all pitchforks and torches at rest for the moment :) I want to stress, this is *NOT* a proposal for inclusion or generating a pull request tomorrow. It's a very very early version, almost a proof of concept, to show *why* we need to change things. Most of the library is up for debate. I agree it needs to be more compatible with current code. In hindsight, I probably should have said no when Andrei asked to post this on the NG, and did it myself when I could stress the state of it. The two most important things are: 1. the interface additions, in particular the readUntil portion (which I think provides a very powerful interface for parsing systems). 2. the performance. It's much better than current stdio. Aren't people continuously complaining at how slow i/o is in Phobos compared to other libraries? Enough ranting for now, as for the proposed std.stdio, 1. It does look fairly straightforward, but: 2. There is only one example. Have any commonly done programming tasks been tried out with it to see how they work? My main testing has been for: 1. utf input/output correctness of all formats 2. implementing readf/writef 3. testing performance. I have not written any "real world" tests. Probably the most interesting tests I've written are reading a UTF-X file and writing the data to a UTF-Y file (where X and Y are one of UTF-8, UTF-16LE, UTF-16BE, UTF-32LE, UTF-32BE). 3. There is no indication of how it interacts with C stdio. A primary goal of std.stdio was interoperability with C stdio. useCStdio(); 4. There are no benchmarks. The current std.stdio was designed/written in parallel with some benchmarks Andrei and others cooked up, as a primary goal was performance. I can include these. 5. flushCheck - flushing should be done based on the file type. tty's should be \n flushed, files when the buffer is full. I question the performance of using a delegate to check for flushing. How often will it be called? Once per write to the buffer. Data is only checked once (the delegate is never given the same data to check again). If you want, I can look at adding a means to avoid using a delegate when the trigger is a single character. And TextInput/TextOutput auto detect whether a device is a tty, and install the right flushcheck function if necessary. 6. There is no provision for multithreaded writing, i.e. what happens when two threads write to stdout. Ideally, there should be a way to 'lock' the stream to oneself, in order to appropriately interleave the output. Again, I wish I had not told Andrei to post :( Multithreaded is not supported, but will be. When that is ready, a locking mechanism (and hopefully an auto-unlock mechanism) will be provided. 7. I see nothing for 'raw' character by character input. The interface is geared to read by processing the buffer, not one character at a time. Given access to the buffer, you can process one character at a time if you want. See InputRange in TextInput to see how raw character-by-character input can be done. That being said, I think I need to add a peek function. 8. I see nothing for determining if a char is available on the input. How would one implement "press any key to continue"? I need more information. I would probably implement this as a read(ubyte[1]), so I don't see why it can't be that way. -Steve
Re: std.stdio overhaul by Steve Schveighoffer
On Sat, 03 Sep 2011 18:55:08 -0400, Andrej Mitrovic wrote: I dislike naming things with a leading "D" like "DInput". Shouldn't we keep code that relies on C to be put in etc.c or somewhere? I think the names are not great. The names are somewhat based on the metamorphosis of the entire interface structure. What about BufferedInput and BufferedOutput? Michel Fortin suggested those. -Steve
Re: std.stdio overhaul by Steve Schveighoffer
On Sat, 03 Sep 2011 20:47:05 -0400, Walter Bright
wrote:
What happens if I write:
printf("hello ");
writeln("world");
useCStdio();
This makes all the standard handles C-based.
And crap, I see I did not document it grr
See here:
https://github.com/schveiguy/phobos/blob/new-io/std/stdio.d#L3332
Sorry
-Steve
Re: std.stdio overhaul by Steve Schveighoffer
Ah, reading your post I see this is just a start of the overhaul. I assumed this was already getting ready for a review. Names can be fixed eventually. :)
Re: std.stdio overhaul by Steve Schveighoffer
On Sat, 03 Sep 2011 18:57:06 -0400, Andrej Mitrovic wrote: Also, changing structs to classes is gonna *massively* break code everywhere. Why inheritance instead of a predicate like isInputStream = is(typeof(T t; t.put; t.close)), you know the drill.. Because it breaks runtime swapping of I/O. For example, if you wanted to change stdin to a network socket, it's simple, just assign another InputStream. However, if stdin is a templated struct, you cannot do this at runtime, you have to decide at compile time what your stdin is. Believe it or not, this is not dissimilar to FILE *, except we have more flexibility. But I realize the implications now. I think I have to revisit this decision. We definitely need classes at the lower level, but I think we can wrap them with structs that are commonly used for RAII and for not breaking existing code. -Steve
Re: std.stdio overhaul by Steve Schveighoffer
On 9/3/11 10:02 PM, Andrej Mitrovic wrote: Seems to me like virtually every module in Phobos gets a complete rewrite sooner or later. Yikes! Afaik the upcoming ones are also std.xml, std.variant, maybe std.json too? (can't recall). Was there really so much bad code written in Phobos all along that they all require a rewrite? It's not that bad. First, it's understandable that now there are considerably more contributors and it's a bit easier tinkering with existing stuff than coming up with all new stuff. Second, historically we're at an all-time high of talent involved in D. I'm sure it will go up much more, but previously we've had a more accepting attitude to new functionality at the cost of scrutiny (e.g. std.xml and std.json, both written by episodic contributors). (I really regret having had that attitude, it hurt us.) So now that there are so many eyeballs focused on the code, and not just any eyeballs but eyeballs connected to good brains, there is pressure building up. There are quite a few pieces in Phobos that are withstanding scrutiny quite well: getopt, algorithm, variant (which can be, I think, safely extended to new great functionality), range, conv, random, and more. There are, unfortunately, others that didn't start off the right foot and right now are somewhat of an eyesore. I trust we will figure what to do about each on a by-case basis, though I agree with Walter that we should balance the breakage cost with correspondingly high rewards in terms of functionality improvements. Andrei
Re: std.stdio overhaul by Steve Schveighoffer
On Sat, 03 Sep 2011 17:55:12 -0400, Michel Fortin wrote: On 2011-09-03 19:54:05 +, Andrei Alexandrescu said: Hello, There are a number of issues related to D's current handling of streams, including the existence of the imperfect etc.stream and the over-specialization of std.stdio. Steve has worked on an extensive overhaul of std.stdio which would obviate the need for etc.stream and would improve both the generality and efficiency of std.stdio. Please chime in with feedback; he's away from the Usenet but allowed me to post this on his behalf. I uploaded the docs to http://erdani.com/d/new-stdio/phobos-prerelease/std_stdio.html Looks good… Well, at least someone thinks so ;) Hum, inconsistent casing of enum members. Can be fixed. And shouldn't there be a way to do non-blocking IO? ;-) Yes. I haven't gotten to that yet. This is a very early version, not ready for inclusion. It's mostly a proof-of-concept. I like that File is now a class because it's cleaner that way, but non-deterministic destruction is going to be a problem. That said, it was already a problem anyway if you stored a File struct in a class, so maybe we need a more general solution for reference-counted classes. I agree, but I think I need to revisit that aspect. As broken as the reference counting mechanism is, much code is based on it, so we can't say you have to revisit all source code in order to be compatible. And as Andrei points out, it works in cases where you *don't* store the struct on the heap, why should that be disabled? Class names DInput and DOutput sounds silly. If all classes implemented purely in D had a D prefix, it'd get redundant pretty fast (like KDE apps beginning in K). Yes, it made sense when I was going through the different iterations of my interface ideas. But you are right. BTW, these started out as DBufferedInput and DBufferedOutput, and CStream was CBufferedStream. I'd suggest BufferedInput and BufferedOutput, or something else that actually describes what the class does, instead of DInput and DOutput. And I'd make them final, that way there won't be any virtual call overhead until the buffer needs to be replenished or flushed from the wrapped input or output stream. They are final, ddoc just doesn't expose that... See my later post to the source. Things might be clearer. -Steve
Re: std.stdio overhaul by Steve Schveighoffer
On Sat, 03 Sep 2011 21:47:53 -0400, Andrei Alexandrescu wrote: On 9/3/11 3:54 PM, Andrei Alexandrescu wrote: http://erdani.com/d/new-stdio/phobos-prerelease/std_stdio.html Here are a few points following a pass through the dox: * After thinking some more about it, I find the approach seek() plus enumerated Anchor undesirable. It's a bad case of logical coupling as one never calls seek() passing an anchor as a variable. It's really three functions - seekForward, seekBackward, and seekAbsolute. Heck, knowing what seek does, it should be just seekAbsolute. But then there are several possible designs; a logically coupled seek() is not a good turn in any case. I think you need to support all three, but they could be individual functions. It just is easy to provide the same interface the OS handle provides. Let's entertain changing to three separate functions. But I think we need to support seek from front, seek from end, and seek from current. I don't know about the three you mentioned. How would you seek to the end if you didn't have seekEnd? And seeking forward or backward I think is captured much better via a positive or negative integer. I can imagine having to write code like this: if(pos < cur) seekBackward(cur - pos); else seekForward(pos - cur); * Seekable should document that tell() is O(1) and seek() can be considered O(1) but with a large constant factor. OK, docs need lots of TLC for sure. * Why is close() not part of Seekable, since Seekable seems to be the base of all streams? Hm... not really sure. I suppose it could be! But then, should the interface be called Seekable? What about just Stream? * Class File is IMHO not going to cut the mustard. It needs to be a struct with a destructor. One should be able to _get_ an InputStream or an OutputStream interface out of a File object (i.e. a file is a factory of such interfaces), but the File itself must be a struct. I'm seeing a large backlash on this decision. I'm going to revisit it. Note, however, that it was a poor choice of name for File on my part. File is *not* equivalent to the current stdio.File, in that it's not buffered, and is not text-based. * I don't understand the difference between read() and readComplete(). read() gets as much data as it can from the buffer and from the stream using at most one low-level read. readComplete() will continually read until either EOF is encountered, or the requested data is read. I started making read() do what readComplete does, but it surprisingly is a very difficult low-level thing to write. However, readComplete() is trivial to implement on top of read(), which is why I split the two functions. Please, come up with a better name, I hate readComplete :) * readUntil is a bit tenuous. I was hoping for a simpler interface to buffered streams, e.g. just expose the buffer as a ubyte[]. I think we need a const(ubyte)[] peek(size_t nbytes). Would this suffice? * readUntil(const(ubyte)[]) does not give a cheap means to figure whether the read ended because file ended or the terminator was met. You are right. I'll think about this. * There's several readUntil but only one appendUntil. Why? Didn't get around to it yet. The overloads for readUntil are trivial, so can be copied easily enough to appendUntil. * Document the difference between skip and seek. Also, skip should take a ulong. skip is buffer-only. It will never trigger a low-level call. I will fill the docs more completely. Given this, I think size_t is the right type, as a buffer cannot be more than size_t bytes in length. * I see encoder and decoder() in DInput, should both be decoder? Yes. encoder is for DOutput, copy-paste error. * StreamWidth, TextXXX and friends are a bit sudden because they introduce a higher-level abstraction in a module so far only preoccupied to transferring bytes. I was thinking that kind of stuff would belong to a formatter/serializer module. Could be moved. However, stdin stderr and stdout are traditionally text-based, and stdio contains them. I wanted to split out text-handling from the basic buffered stream, since it's very specific. For example, having to deal with an object that supports formatted text i/o for a network socket seems uncommon. I'm open to suggestions. Note, I must have had a brain-malfunction when I gave what I thought was a fairly completely-documented module. I missed some very important declarations and functions. I'll work on fixing the docs and giving you a new copy. Thanks again for hosting it. -Steve
Re: std.stdio overhaul by Steve Schveighoffer
== Quote from Andrei Alexandrescu (seewebsiteforem...@erdani.org)'s article > I'm sure it will go up much more, but previously we've had a more > accepting attitude to new functionality at the cost of scrutiny (e.g. > std.xml and std.json, both written by episodic contributors). (I really > regret having had that attitude, it hurt us.) So now that there are so > many eyeballs focused on the code, and not just any eyeballs but > eyeballs connected to good brains, there is pressure building up. > There are quite a few pieces in Phobos that are withstanding scrutiny > quite well: getopt, algorithm, variant (which can be, I think, safely > extended to new great functionality), range, conv, random, and more. > There are, unfortunately, others that didn't start off the right foot > and right now are somewhat of an eyesore. I trust we will figure what to > do about each on a by-case basis, though I agree with Walter that we > should balance the breakage cost with correspondingly high rewards in > terms of functionality improvements. > Andrei Yes, the quality standard has gone up massively. When I was prepping std.parallelism for review a few months ago, I generally used the existing Phobos documentation as a guideline for what std.parallelism's docs should resemble. Andrei, of course, ripped the documentation apart. In hindsight it led to massive improvements and was for the better. It certainly set the tone for clear, precise documentation in the future and the same high standards were applied to std.path and the std.curl. However, at the time I actually thought he just hated std.parallelism at a gut level and was looking for any excuse to keep it out of Phobos. (I apologize for having thought this and therefore taken a much more adversarial view of the review process than I should have.)
Re: std.stdio overhaul by Steve Schveighoffer
On Saturday, September 03, 2011 18:53:00 Walter Bright wrote: > On 9/3/2011 5:58 PM, Jonathan M Davis wrote: > > However, if the code breakage > > doesn't actually gain us anything, then we should avoid it. So, > > complaints about code breakage are valid, but they aren't deal > > breaking. > > The larger the amount of code that is broken, the more gain there must be to > justify it. > > Breaking std.stdio, which is used everywhere, this thoroughly needs a very > high bar of justification. Agreed. - Jonathan M Davis
Re: std.stdio overhaul by Steve Schveighoffer
On Sat, 03 Sep 2011 22:27:49 -0400, Andrei Alexandrescu wrote: On 9/3/11 9:53 PM, Walter Bright wrote: On 9/3/2011 5:58 PM, Jonathan M Davis wrote: However, if the code breakage doesn't actually gain us anything, then we should avoid it. So, complaints about code breakage are valid, but they aren't deal breaking. The larger the amount of code that is broken, the more gain there must be to justify it. Breaking std.stdio, which is used everywhere, this thoroughly needs a very high bar of justification. I agree. I'm hoping the new stuff could build on top of std.stdio. It is my plan for the eventual result to break either no code, or as little code as possible. The current library is mostly a proof-of-concept, to see what people think, and to show what might be possible. I think the interfaces in this library make for a much easier-to-write xml library for instance. It's by no means a proposal for immediate acceptance into Phobos, I'm sorry if it came across that way. We have to break something in std.stdio, because it's fixated on FILE *. We need something that allows FILE * to play the game, but is focused on a D-based solution. Otherwise, we have no room for improvement. that's what I'm striving for. And along the way, I'm trying to make it as efficient as possible. -Steve
Re: std.stdio overhaul by Steve Schveighoffer
On 9/3/11 11:33 PM, Steven Schveighoffer wrote: We have to break something in std.stdio, because it's fixated on FILE *. We need something that allows FILE * to play the game, but is focused on a D-based solution. Otherwise, we have no room for improvement. I'm not 100% convinced of that. We can achieve a good deal of improvement by resorting to platform-specific code. Clearly that's not the best way to go but it's not difficult and it does have its merit. Overall I think the design of std.stdio should be followed: 1. User opens a File (or whatever), which is a struct. The struct uses RAII. 2. Using the struct you can directly call primitives to read and write stuff. 3. You can also decide you want a polymorphic stream out of it, and you get to decide the parameters of the stream (buffering, chunking, synchronicity and whatnot). byChunk and byLine are good examples, although they aren't polymorphic. Once you have such a stream you're in polyland so you get to use all of its goodies (look ma no templates etc). 4. Once all copies of the struct is destroyed, all streams derived from it are automatically closed and will issue errors when used. That's pretty much it! It's a simple design that does all we need. Andrei
Re: std.stdio overhaul by Steve Schveighoffer
On Sunday, September 04, 2011 03:22:21 dsimcha wrote: > == Quote from Andrei Alexandrescu (seewebsiteforem...@erdani.org)'s article > > > I'm sure it will go up much more, but previously we've had a more > > accepting attitude to new functionality at the cost of scrutiny (e.g. > > std.xml and std.json, both written by episodic contributors). (I really > > regret having had that attitude, it hurt us.) So now that there are so > > many eyeballs focused on the code, and not just any eyeballs but > > eyeballs connected to good brains, there is pressure building up. > > There are quite a few pieces in Phobos that are withstanding scrutiny > > quite well: getopt, algorithm, variant (which can be, I think, safely > > extended to new great functionality), range, conv, random, and more. > > There are, unfortunately, others that didn't start off the right foot > > and right now are somewhat of an eyesore. I trust we will figure what to > > do about each on a by-case basis, though I agree with Walter that we > > should balance the breakage cost with correspondingly high rewards in > > terms of functionality improvements. > > Andrei > > Yes, the quality standard has gone up massively. When I was prepping > std.parallelism for review a few months ago, I generally used the existing > Phobos documentation as a guideline for what std.parallelism's docs should > resemble. Andrei, of course, ripped the documentation apart. In hindsight > it led to massive improvements and was for the better. It certainly set > the tone for clear, precise documentation in the future and the same high > standards were applied to std.path and the std.curl. However, at the time > I actually thought he just hated std.parallelism at a gut level and was > looking for any excuse to keep it out of Phobos. (I apologize for having > thought this and therefore taken a much more adversarial view of the review > process than I should have.) std.datetime is far better for having gone through multiple reviews as well. The resulting code isn't perfect, and reviews don't always catch everything, but thorough reviews really help improve the quality of code. Even just having other contributors look over pull requests tends to find stuff that can and should be improved. So, while there will likely always be some issues with code that make it into Phobos, the overall code quality is definitely improving. - Jonathan M Davis
Re: std.stdio overhaul by Steve Schveighoffer
On Sat, 03 Sep 2011 23:45:17 -0400, Andrei Alexandrescu wrote: On 9/3/11 11:33 PM, Steven Schveighoffer wrote: We have to break something in std.stdio, because it's fixated on FILE *. We need something that allows FILE * to play the game, but is focused on a D-based solution. Otherwise, we have no room for improvement. I'm not 100% convinced of that. We can achieve a good deal of improvement by resorting to platform-specific code. Clearly that's not the best way to go but it's not difficult and it does have its merit. Overall I think the design of std.stdio should be followed: 1. User opens a File (or whatever), which is a struct. The struct uses RAII. OK, I think that's the offer on the table I keep getting :) I'm definitely going to use this, and its name will be File. I think it has to be in order to be compatible with all current code. 2. Using the struct you can directly call primitives to read and write stuff. Buffered reads and writes? If so, don't you need to decide the items in point 3 before read/write? If not buffered, then I think I can work with this. 3. You can also decide you want a polymorphic stream out of it, and you get to decide the parameters of the stream (buffering, chunking, synchronicity and whatnot). byChunk and byLine are good examples, although they aren't polymorphic. Once you have such a stream you're in polyland so you get to use all of its goodies (look ma no templates etc). 4. Once all copies of the struct is destroyed, all streams derived from it are automatically closed and will issue errors when used. OK, I think I know how to do this. I'm assuming if you want to use exclusively the poly versions, you can do that. I.e. you don't have to keep an RAII File struct around. That's pretty much it! It's a simple design that does all we need. I'll work on that. How should text vs. non-text i/o work? C currently conflates them at the same level, but I think they are two separate layers. What do you think? -Steve
Re: std.stdio overhaul by Steve Schveighoffer
On 9/3/2011 7:27 PM, dsimcha wrote: These changes are purely under the hood, though, and there should be zero code breakage. Those are the great kind of changes, and it's also nice in that it means the API was done reasonably right.
Re: std.stdio overhaul by Steve Schveighoffer
On 9/3/2011 8:22 PM, dsimcha wrote: However, at the time I actually thought he just hated std.parallelism at a gut level and was looking for any excuse to keep it out of Phobos. (I apologize for having thought this and therefore taken a much more adversarial view of the review process than I should have.) I can vouch for Andrei's reviews appearing to be personal, but they are not. He's mercilessly ripped up some of my stuff, but I had to agree he was right and the resulting improvement was well worth it. I don't much care for blowing sunshine, flattery and false praise. Andrei sets a high bar, I'm glad he does, and we'll all be better off for it.
Re: std.stdio overhaul by Steve Schveighoffer
On Sunday, September 04, 2011 04:02:17 Andrej Mitrovic wrote: > Seems to me like virtually every module in Phobos gets a complete > rewrite sooner or later. Yikes! Afaik the upcoming ones are also > std.xml, std.variant, maybe std.json too? (can't recall). Was there > really so much bad code written in Phobos all along that they all > require a rewrite? Most of it's older stuff which has been around since D1, I believe - either that or it came fairly early in D2. - Jonathan M Davis
Re: std.stdio overhaul by Steve Schveighoffer
On 9/3/2011 7:33 PM, Steven Schveighoffer wrote: Please, leave all pitchforks and torches at rest for the moment :) I know what I wrote was a bit brutal, but this needs to be settled before we've gone so far down that path that turning away then would be horribly unfair to you. I think what you need is a marketing spiel to sell the concept of what you're trying to do. It should include: 1. The benefits over the current std.stdio 2. Why the new API is needed to achieve those benefits 3. A migration plan for existing std.stdio code Just being more flexible isn't enough, it has to be more flexible in a way that matters, i.e. a real example showing how kickass it is compared to the current way. 2. the performance. It's much better than current stdio. Aren't people continuously complaining at how slow i/o is in Phobos compared to other libraries? Why is it faster? I.e. is a wholly new interface required to make it faster, or does it just need to be better under the hood? 3. There is no indication of how it interacts with C stdio. A primary goal of std.stdio was interoperability with C stdio. useCStdio(); For some reason that just seems like a giant wart with a hair sticking out of it. Why not just use the C stdio buffers? 5. flushCheck - flushing should be done based on the file type. tty's should be \n flushed, files when the buffer is full. I question the performance of using a delegate to check for flushing. How often will it be called? Once per write to the buffer. Data is only checked once (the delegate is never given the same data to check again). If you want, I can look at adding a means to avoid using a delegate when the trigger is a single character. And TextInput/TextOutput auto detect whether a device is a tty, and install the right flushcheck function if necessary. Flushing once per write is wrong - consider the user who does a zillion putc's. I don't see a purpose to anything beyond the C stdio ones - per character, per \n, and per buffer. 7. I see nothing for 'raw' character by character input. The interface is geared to read by processing the buffer, not one character at a time. Given access to the buffer, you can process one character at a time if you want. See InputRange in TextInput to see how raw character-by-character input can be done. Raw mode is more than that - you have to set the OS to raw mode, otherwise it won't give you any characters until a \n is typed. 8. I see nothing for determining if a char is available on the input. How would one implement "press any key to continue"? I need more information. I would probably implement this as a read(ubyte[1]), so I don't see why it can't be that way. There's more to it than that. Try writing it in C and you'll see what I mean. (You have to set the io to "raw" mode, turn "echo" off, etc.)
Re: std.stdio overhaul by Steve Schveighoffer
On Sun, 04 Sep 2011 00:30:33 -0400, Walter Bright wrote: On 9/3/2011 7:33 PM, Steven Schveighoffer wrote: Please, leave all pitchforks and torches at rest for the moment :) I know what I wrote was a bit brutal, but this needs to be settled before we've gone so far down that path that turning away then would be horribly unfair to you. I appreciate feedback, but I think there was a misunderstanding of what this "review" was for. I think people thought I was proposing this as a ready-to-pull replacement for std.stdio. That is not the case. It's very much up in the air and under development. I just wanted to show people some progress and get feedback (which I've gotten a lot of!) The next version of it will look drastically different based on what's been said here. But it will still contain some of the basic designs. In essence, I am very *early* in the path, and I *want* people to turn me in the right direction before I go too far the other way. This is the first version that *actually works*, which is why I wanted to share it :) Not anyone has really commented on the new interfaces. It's my fault, for letting Andrei post the documentation as the main subject, and also not fully documenting the module. I have no excuses, so I'll just have to take this as a "ok, we'll try this again later". But I did get some very good information, and know I have a lot of work to do. I think what you need is a marketing spiel to sell the concept of what you're trying to do. It should include: 1. The benefits over the current std.stdio 2. Why the new API is needed to achieve those benefits 3. A migration plan for existing std.stdio code OK Just being more flexible isn't enough, it has to be more flexible in a way that matters, i.e. a real example showing how kickass it is compared to the current way. I'll post some numbers. 2. the performance. It's much better than current stdio. Aren't people continuously complaining at how slow i/o is in Phobos compared to other libraries? Why is it faster? I.e. is a wholly new interface required to make it faster, or does it just need to be better under the hood? Yes, a new interface is required to make it faster. You need direct buffer access, and the current stdio does not provide that. That being said, I think this proposal goes nowhere unless it's a mostly drop-in replacement to the existing std.stdio. So I have to find a way to make it fit. 3. There is no indication of how it interacts with C stdio. A primary goal of std.stdio was interoperability with C stdio. useCStdio(); For some reason that just seems like a giant wart with a hair sticking out of it. Why not just use the C stdio buffers? 1. Because most people don't care. I never ever use printf, except when I was testing my new stdio stuff, and I needed something that worked :) My opinion, if you are using this line, you are doing something weird, legacy related, or you are debugging something. 2. Because C does not provide enough access to the buffers. With my library, you can read an entire xml file, for instance, and never copy any data out of the buffer. C never gives direct access to the buffers, and while we can hack our way into it, its interface is still kludgy. If I wanted to implement, for example, readUntil using C buffers, I'd have to reimplement almost all of FILE *'s functions so I could do it properly. And even then, I'd still have to sacrifice some things -- C is still going to want to use its way of doing things, and I'd have to respect that. If you read my response to the first post in this thread, you can see my rationale. 5. flushCheck - flushing should be done based on the file type. tty's should be \n flushed, files when the buffer is full. I question the performance of using a delegate to check for flushing. How often will it be called? Once per write to the buffer. Data is only checked once (the delegate is never given the same data to check again). If you want, I can look at adding a means to avoid using a delegate when the trigger is a single character. And TextInput/TextOutput auto detect whether a device is a tty, and install the right flushcheck function if necessary. Flushing once per write is wrong - consider the user who does a zillion putc's. I don't see a purpose to anything beyond the C stdio ones - per character, per \n, and per buffer. a *check* to see if it should be flushed is done once per write. Not a flush. A flush is only done if the check says to (or the buffer is full). I think C's FILE * checks once per write as well, no? I also have thought of ways to optimize this so it's, say, once per call to writef. 7. I see nothing for 'raw' character by character input. The interface is geared to read by processing the buffer, not one character at a time. Given access to the buffer, you can process one characte
Re: std.stdio overhaul by Steve Schveighoffer
On 9/3/2011 10:09 PM, Steven Schveighoffer wrote: I appreciate feedback, but I think there was a misunderstanding of what this "review" was for. I think people thought I was proposing this as a ready-to-pull replacement for std.stdio. That is not the case. It's very much up in the air and under development. I just wanted to show people some progress and get feedback (which I've gotten a lot of!) I'm glad it's early in the process. For some reason that just seems like a giant wart with a hair sticking out of it. Why not just use the C stdio buffers? 1. Because most people don't care. I never ever use printf, except when I was testing my new stdio stuff, and I needed something that worked :) My opinion, if you are using this line, you are doing something weird, legacy related, or you are debugging something. I still use printf a lot. One reason is because it is lightweight - using writeln blows up the size of your .obj file, making it hard to track down a back end bug. This is a long standing gripe I have with writeln. D is supposed to work well with existing C code. To me, that includes working smoothly with C stdio. If you read my response to the first post in this thread, you can see my rationale. I understand the desire to do away with C stdio compatibility, but it needs to deliver a *lot* to justify that. I also don't mind if std.stdio needs to peek under the hood of C stdio to get there - yes, it'll be custom for each C library, but the user won't see that. a *check* to see if it should be flushed is done once per write. Not a flush. A flush is only done if the check says to (or the buffer is full). I think C's FILE * checks once per write as well, no? No. It checks once per char for \n, and once per buffer overflow otherwise. That is not an OS issue, that is a terminal issue. It's an OS issue. The OS does the line buffering. Note that the current std.stdio does not provide this functionality. The only raw functions are rawRead and rawWrite, which set binary mode. All binary mode does is on windows enable or disable translation of \r\n to \n. They will not do what you are asking. You're right, you have to dip under the hood to the OS protocol to do it.
Re: std.stdio overhaul by Steve Schveighoffer
On Saturday, September 03, 2011 23:49:52 Walter Bright wrote: > I still use printf a lot. One reason is because it is lightweight - using > writeln blows up the size of your .obj file, making it hard to track down a > back end bug. This is a long standing gripe I have with writeln. Well, while that may be a good reason to use printf, it really doesn't apply to very many D programmers. Your average D programmer really has no need to use printf. - Jonathan M Davis
Re: std.stdio overhaul by Steve Schveighoffer
On Sat, 03 Sep 2011 18:23:26 -0700, Walter Bright wrote: > [I also don't like it that all my code that uses std.path is now > broken.] What do you mean by "broken"? That it does not compile or work as expected, or that it spits out a bunch of annoying deprecation messages? If it is any of the former, that was not supposed to happen. The new std.path still contains all the functions of the old std.path and should therefore be backwards compatible. If the new std.path breaks existing code, I need to fix it before it is released. Please let me know what problems you are experiencing. -Lars
Re: std.stdio overhaul by Steve Schveighoffer
On 2011-09-04 04:35, Steven Schveighoffer wrote: On Sat, 03 Sep 2011 18:55:08 -0400, Andrej Mitrovic wrote: I dislike naming things with a leading "D" like "DInput". Shouldn't we keep code that relies on C to be put in etc.c or somewhere? I think the names are not great. The names are somewhat based on the metamorphosis of the entire interface structure. What about BufferedInput and BufferedOutput? Michel Fortin suggested those. -Steve These names are a lot better. -- /Jacob Carlborg
Re: std.stdio overhaul by Steve Schveighoffer
On 2011-09-03 21:54, Andrei Alexandrescu wrote: Hello, There are a number of issues related to D's current handling of streams, including the existence of the imperfect etc.stream and the over-specialization of std.stdio. Steve has worked on an extensive overhaul of std.stdio which would obviate the need for etc.stream and would improve both the generality and efficiency of std.stdio. Please chime in with feedback; he's away from the Usenet but allowed me to post this on his behalf. I uploaded the docs to http://erdani.com/d/new-stdio/phobos-prerelease/std_stdio.html Thanks, Andrei I think that openFile, File.open and CStream.open should shouldn't take a string as the mode, it should be an enum or similar. Andrei is making a big deal out of using enums instead of bools. A bool value can contain "true" or "false", a string can contain an infinite number of different values. -- /Jacob Carlborg
Re: std.stdio overhaul by Steve Schveighoffer
Hi, what is an "abstract interface" ? -- Paulo "Andrei Alexandrescu" wrote in message news:j3u0l4$1atr$1...@digitalmars.com... > Hello, > > > There are a number of issues related to D's current handling of streams, > including the existence of the imperfect etc.stream and the > over-specialization of std.stdio. > > Steve has worked on an extensive overhaul of std.stdio which would obviate > the need for etc.stream and would improve both the generality and > efficiency of std.stdio. > > Please chime in with feedback; he's away from the Usenet but allowed me to > post this on his behalf. I uploaded the docs to > > http://erdani.com/d/new-stdio/phobos-prerelease/std_stdio.html > > > Thanks, > > Andrei >
Re: std.stdio overhaul by Steve Schveighoffer
On 2011-09-04 06:53:27 +, Jonathan M Davis said: On Saturday, September 03, 2011 23:49:52 Walter Bright wrote: I still use printf a lot. One reason is because it is lightweight - using writeln blows up the size of your .obj file, making it hard to track down a back end bug. This is a long standing gripe I have with writeln. Well, while that may be a good reason to use printf, it really doesn't apply to very many D programmers. Your average D programmer really has no need to use printf. That may be true, but the average D programmer will also, directly or indirectly, call C APIs which may use printf to write things to the console. I'm not sure it's much of a problem though. For one thing, C APIs generally don't print things on their own. And also, I doubt using D IO by default will break printf that much: I mean if C IO is used to print lines, those lines will be flushed as they're emitted, with no possible weird interleaving unless the line is really too long. And if you use both D and C IO together, likely you're just logging things to the console line by line and not outputting things in a specific format where weird interleaving could cause major breakage. I'm making some assumptions here, so maybe I'm wrong, but I can't really see a use case where both IO system would be used and where the fidelity of the output is that important… please correct me if I'm wrong. So in my opinion the default should be to use D streams, as I don't expect the drawbacks to be a major inconvenience, and the performance gain of being able to access the buffer directly would certainly be welcome. -- Michel Fortin michel.for...@michelf.com http://michelf.com/
Re: std.stdio overhaul by Steve Schveighoffer
On 2011-09-04 04:48, Steven Schveighoffer wrote:
On Sat, 03 Sep 2011 18:57:06 -0400, Andrej Mitrovic
wrote:
Also, changing structs to classes is gonna *massively* break code
everywhere. Why inheritance instead of a predicate like isInputStream
= is(typeof(T t; t.put; t.close)), you know the drill..
Because it breaks runtime swapping of I/O.
For example, if you wanted to change stdin to a network socket, it's
simple, just assign another InputStream.
However, if stdin is a templated struct, you cannot do this at runtime,
you have to decide at compile time what your stdin is. Believe it or
not, this is not dissimilar to FILE *, except we have more flexibility.
But I realize the implications now. I think I have to revisit this
decision.
We definitely need classes at the lower level, but I think we can wrap
them with structs that are commonly used for RAII and for not breaking
existing code.
-Steve
Tango has added a new method to Object, "dispose". The method is called
by the runtime when a scoped class exits a scope:
void foo ()
{
scope f = new File;
}
When "foo" exits File.dispose will be called and it can close any file
handles. I think it's quite clever.
--
/Jacob Carlborg
Re: std.stdio overhaul by Steve Schveighoffer
On 9/4/11 12:11 AM, Walter Bright wrote: On 9/3/2011 8:22 PM, dsimcha wrote: However, at the time I actually thought he just hated std.parallelism at a gut level and was looking for any excuse to keep it out of Phobos. (I apologize for having thought this and therefore taken a much more adversarial view of the review process than I should have.) I can vouch for Andrei's reviews appearing to be personal, but they are not. He's mercilessly ripped up some of my stuff, but I had to agree he was right and the resulting improvement was well worth it. I don't much care for blowing sunshine, flattery and false praise. Andrei sets a high bar, I'm glad he does, and we'll all be better off for it. This is a bit of a surprise for me because I fancy (fancied...) to see myself as this emotionless, rational reviewer. Thank you all for putting up with me. Andrei
Re: std.stdio overhaul by Steve Schveighoffer
On 9/4/11 7:10 AM, Michel Fortin wrote: On 2011-09-04 06:53:27 +, Jonathan M Davis said: On Saturday, September 03, 2011 23:49:52 Walter Bright wrote: I still use printf a lot. One reason is because it is lightweight - using writeln blows up the size of your .obj file, making it hard to track down a back end bug. This is a long standing gripe I have with writeln. Well, while that may be a good reason to use printf, it really doesn't apply to very many D programmers. Your average D programmer really has no need to use printf. That may be true, but the average D programmer will also, directly or indirectly, call C APIs which may use printf to write things to the console. I'm not sure it's much of a problem though. For one thing, C APIs generally don't print things on their own. And also, I doubt using D IO by default will break printf that much: I mean if C IO is used to print lines, those lines will be flushed as they're emitted, with no possible weird interleaving unless the line is really too long. No, things are more complex; the interference will be major unless explicitly addressed. Andrei
Re: std.stdio overhaul by Steve Schveighoffer
On 9/4/11 7:11 AM, Jacob Carlborg wrote:
On 2011-09-04 04:48, Steven Schveighoffer wrote:
On Sat, 03 Sep 2011 18:57:06 -0400, Andrej Mitrovic
wrote:
Also, changing structs to classes is gonna *massively* break code
everywhere. Why inheritance instead of a predicate like isInputStream
= is(typeof(T t; t.put; t.close)), you know the drill..
Because it breaks runtime swapping of I/O.
For example, if you wanted to change stdin to a network socket, it's
simple, just assign another InputStream.
However, if stdin is a templated struct, you cannot do this at runtime,
you have to decide at compile time what your stdin is. Believe it or
not, this is not dissimilar to FILE *, except we have more flexibility.
But I realize the implications now. I think I have to revisit this
decision.
We definitely need classes at the lower level, but I think we can wrap
them with structs that are commonly used for RAII and for not breaking
existing code.
-Steve
Tango has added a new method to Object, "dispose". The method is called
by the runtime when a scoped class exits a scope:
void foo ()
{
scope f = new File;
}
When "foo" exits File.dispose will be called and it can close any file
handles. I think it's quite clever.
What happens if f is aliased beyond the existence of foo()?
Andrei
Re: std.stdio overhaul by Steve Schveighoffer
On 9/4/11, Paulo Pinto wrote: > Hi, > > what is an "abstract interface" ? I'm wondering the same thing.
Re: std.stdio overhaul by Steve Schveighoffer
On 9/4/11 4:30 PM, Andrej Mitrovic wrote: On 9/4/11, Paulo Pinto wrote: Hi, what is an "abstract interface" ? I'm wondering the same thing. A bug in ddoc. ;) David
Re: std.stdio overhaul by Steve Schveighoffer
On 2011-09-04 12:57:06 +, Andrei Alexandrescu said: On 9/4/11 7:10 AM, Michel Fortin wrote: And also, I doubt using D IO by default will break printf that much: I mean if C IO is used to print lines, those lines will be flushed as they're emitted, with no possible weird interleaving unless the line is really too long. No, things are more complex; the interference will be major unless explicitly addressed. That doesn't really help understand the issue, you're just making it more obscure. -- Michel Fortin michel.for...@michelf.com http://michelf.com/
Re: std.stdio overhaul by Steve Schveighoffer
Michel Fortin Wrote: > On 2011-09-04 12:57:06 +, Andrei Alexandrescu > said: > > > On 9/4/11 7:10 AM, Michel Fortin wrote: > >> And also, I doubt using D IO by default will break printf that much: I > >> mean if C IO is used to print lines, those lines will be flushed as > >> they're emitted, with no possible weird interleaving unless the line is > >> really too long. > > > > No, things are more complex; the interference will be major unless > > explicitly addressed. > > That doesn't really help understand the issue, you're just making it > more obscure. > > -- > Michel Fortin > michel.for...@michelf.com > http://michelf.com/ > You are assuming each write flushes the buffer. That's not always the case. -Steve
Re: std.stdio overhaul by Steve Schveighoffer
On 2011-09-04 14:59, Andrei Alexandrescu wrote:
On 9/4/11 7:11 AM, Jacob Carlborg wrote:
Tango has added a new method to Object, "dispose". The method is called
by the runtime when a scoped class exits a scope:
void foo ()
{
scope f = new File;
}
When "foo" exits File.dispose will be called and it can close any file
handles. I think it's quite clever.
What happens if f is aliased beyond the existence of foo()?
Andrei
I'm not sure if this is what you mean but:
File file;
void foo ()
{
scope f = new File;
file = f;
}
void main ()
{
foo;
// file is disposed here
}
In the above example "dispose" will be called when "foo" exits. After
the call to "foo" in the main function "file" will refer to an object
that is disposed, i.e. an object where the "dispose" method has been called.
I don't know how bad this is or if it is bad at all. I would be the same
as the following code:
File file;
void foo ()
{
auto f = new File;
f.close;
file = f;
}
void main ()
{
foo;
}
--
/Jacob Carlborg
Re: std.stdio overhaul by Steve Schveighoffer
On 2011-09-04 16:08:47 +, Steven Schveighoffer said: Michel Fortin Wrote: On 2011-09-04 12:57:06 +, Andrei Alexandrescu said: On 9/4/11 7:10 AM, Michel Fortin wrote: And also, I doubt using D IO by default will break printf that much: I mean if C IO is used to print lines, those lines will be flushed as they're emitted, with no possible weird interleaving unless the line is really too long. No, things are more complex; the interference will be major unless explicitly addressed. That doesn't really help understand the issue, you're just making it more obscure. You are assuming each write flushes the buffer. That's not always the case. Not exactly. I am assuming each write flushes the buffer __up to the last newline__, and that most writes ends with \n in a use case where you'd be intermixing the IO systems. That's what I read somewhere else in this discussion, but maybe I read it wrong. -- Michel Fortin michel.for...@michelf.com http://michelf.com/
Re: std.stdio overhaul by Steve Schveighoffer
On 9/4/11 2:58 PM, Jacob Carlborg wrote:
On 2011-09-04 14:59, Andrei Alexandrescu wrote:
On 9/4/11 7:11 AM, Jacob Carlborg wrote:
Tango has added a new method to Object, "dispose". The method is called
by the runtime when a scoped class exits a scope:
void foo ()
{
scope f = new File;
}
When "foo" exits File.dispose will be called and it can close any file
handles. I think it's quite clever.
What happens if f is aliased beyond the existence of foo()?
Andrei
I'm not sure if this is what you mean but:
File file;
void foo ()
{
scope f = new File;
file = f;
}
void main ()
{
foo;
// file is disposed here
}
In the above example "dispose" will be called when "foo" exits. After
the call to "foo" in the main function "file" will refer to an object
that is disposed, i.e. an object where the "dispose" method has been
called.
I don't know how bad this is or if it is bad at all.
Well it's not bad but a bit underwhelming. Clearly it's better than the
unsafe behavior of scope, but it's nothing to write home about. The
grand save it makes is replacing "scope(exit) f.dispose();" with "scope"
in front of the declaration. That does systematically save some typing,
but it's a feature with only local, non-modular effect, and limited
abstraction power.
Andrei
Re: std.stdio overhaul by Steve Schveighoffer
On 9/4/11 3:23 PM, Michel Fortin wrote: On 2011-09-04 16:08:47 +, Steven Schveighoffer said: Michel Fortin Wrote: On 2011-09-04 12:57:06 +, Andrei Alexandrescu said: On 9/4/11 7:10 AM, Michel Fortin wrote: And also, I doubt using D IO by default will break printf that much: I mean if C IO is used to print lines, those lines will be flushed as they're emitted, with no possible weird interleaving unless the line is really too long. No, things are more complex; the interference will be major unless explicitly addressed. That doesn't really help understand the issue, you're just making it more obscure. You are assuming each write flushes the buffer. That's not always the case. Not exactly. I am assuming each write flushes the buffer __up to the last newline__, and that most writes ends with \n in a use case where you'd be intermixing the IO systems. That's what I read somewhere else in this discussion, but maybe I read it wrong. It depends on the buffering mode of the stream, and also of the buffering mode of whatever alternative abstraction is being used. Sorry for being curt - I trusted Walter's earlier explanation would suffice. Andrei
Re: std.stdio overhaul by Steve Schveighoffer
On 2011-09-04 19:36:23 +, Andrei Alexandrescu said: It depends on the buffering mode of the stream, and also of the buffering mode of whatever alternative abstraction is being used. Sorry for being curt - I trusted Walter's earlier explanation would suffice. Actually my assumption wasn't too bad within its own boundaries. I was only thinking about stdout and it being line-buffered by default. Looking into it a little more, I'm not sure what would happen to stdin and stdout isn't always line-buffered by default anyway (if the output isn't a terminal for instance). So I have to agree with you that mixing the two won't work well in most cases. Sorry for the distraction. -- Michel Fortin michel.for...@michelf.com http://michelf.com/
Re: std.stdio overhaul by Steve Schveighoffer
On 2011-09-04 21:34, Andrei Alexandrescu wrote:
On 9/4/11 2:58 PM, Jacob Carlborg wrote:
I'm not sure if this is what you mean but:
File file;
void foo ()
{
scope f = new File;
file = f;
}
void main ()
{
foo;
// file is disposed here
}
In the above example "dispose" will be called when "foo" exits. After
the call to "foo" in the main function "file" will refer to an object
that is disposed, i.e. an object where the "dispose" method has been
called.
I don't know how bad this is or if it is bad at all.
Well it's not bad but a bit underwhelming. Clearly it's better than the
unsafe behavior of scope, but it's nothing to write home about. The
grand save it makes is replacing "scope(exit) f.dispose();" with "scope"
in front of the declaration. That does systematically save some typing,
but it's a feature with only local, non-modular effect, and limited
abstraction power.
Andrei
Yeah, a variable declared as "scope" shouldn't, preferably, exit it's
scope. The compiler will at least complain if you try to return a scoped
variable.
--
/Jacob Carlborg
Re: std.stdio overhaul by Steve Schveighoffer
Andrei Alexandrescu Wrote: > http://erdani.com/d/new-stdio/phobos-prerelease/std_stdio.html Ddoc screwed the types, right?
Re: std.stdio overhaul by Steve Schveighoffer
On 9/4/2011 2:17 AM, Lars T. Kyllingstad wrote: On Sat, 03 Sep 2011 18:23:26 -0700, Walter Bright wrote: [I also don't like it that all my code that uses std.path is now broken.] What do you mean by "broken"? That it does not compile or work as expected, or that it spits out a bunch of annoying deprecation messages? If it is any of the former, that was not supposed to happen. The new std.path still contains all the functions of the old std.path and should therefore be backwards compatible. If the new std.path breaks existing code, I need to fix it before it is released. Please let me know what problems you are experiencing. It prints out all the deprecation message. It means I'll have to go edit existing, working code to change the names. I know that the majority wants the name changes. I know the deprecation system gives people plenty of time to edit their code. But I think the cost of breaking existing code is much higher than many realize, and a lot of that cost will be hidden. It'll come in the form of people deciding not to use D because it is "not stable". It'll come in the form of invalidating existing libraries and modules unless someone is regularly maintaining them. It'll come in the form of invalidating the mass of books, articles, blog postings, and presentations about D, and those will never get updated. People will type in the code examples, they will fail to compile, and they'll get turned off about D. I'll again note that I know of know successful operating system or programming language that goes around breaking existing code unless it is really, really urgent. Camel-casing a name doesn't meet that standard. So, yes, I don't like it.
Re: std.stdio overhaul by Steve Schveighoffer
On 9/5/11, Walter Bright wrote: > It prints out all the deprecation message. It means I'll have to go edit > existing, working code to change the names. It would really help out if we had some sort of semi-automated script that can do at least partial translation of code that uses old phobos functions to new ones. Maybe this wouldn't work 100% but at least it would help out. I'm thinking of something similar to what Python 2to3 does. I know for sure I could use this, so far I've had to fix the DWinSamples for every DMD/Phobos release.
Re: std.stdio overhaul by Steve Schveighoffer
== Quote from Walter Bright (newshou...@digitalmars.com)'s article > On 9/4/2011 2:17 AM, Lars T. Kyllingstad wrote: > I'll again note that I know of know successful operating system or programming > language that goes around breaking existing code unless it is really, really > urgent. > Camel-casing a name doesn't meet that standard. So, yes, I don't like it. I agree that we've been overzealous lately in breaking code to fix small inconsistencies in style, etc. I think in a lot of cases the answer is permanent (or very long term, i.e. several years) soft deprecation, plus a real soft-deprecated language feature. This will lead to cruft accumulation but in some cases this cruft is less bad than the cruft caused by inconsistent naming conventions/style, etc. To make the docs seem less crufty to people browsing, we could even eventually remove the soft-deprecated functionality from the DDoc documentation so that people reading it can't even see the cruft, and move the code to the bottom of the source files so that people don't see it unless they go looking for it. We could also adopt a policy of zero maintenance for features that have been soft-deprecated for long periods of time, i.e. not even if they produce egregiously wrong results, security holes, etc.
Re: std.stdio overhaul by Steve Schveighoffer
Am 06.09.2011, 00:05 Uhr, schrieb Andrej Mitrovic : On 9/5/11, Walter Bright wrote: It prints out all the deprecation message. It means I'll have to go edit existing, working code to change the names. It would really help out if we had some sort of semi-automated script that can do at least partial translation of code that uses old phobos functions to new ones. Maybe this wouldn't work 100% but at least it would help out. I'm thinking of something similar to what Python 2to3 does. I know for sure I could use this, so far I've had to fix the DWinSamples for every DMD/Phobos release. It would help to have a lexical analyzer of the kind that allows for the refactorings in for example Eclipse for Java. Without clear identification of symbols it is impossible to write such a script for every new D release. And there were other changes in the past where this would have been handy although I have not writing any D code lately to feel the breakage. I think of globals (__gshared). I'm on the extreme with my urge to rewrite things if they give me the slightest feeling that they could be more elegant or effective and I've thought of such a script as well that could be distributed with every new D version while there are still breaking changes to the language. Well, you cannot write a script without a solid foundation that can reliably identify and refactor symbols. But this doesn't work well for code you copy from blogs. You would have to know what D version it was written with and run the matching chain of conversion scripts. Anyway this feels like some crazy idea that can't make it into existence. Still I have that picture of downloading a new D release and running the obligatory dmdup script to replace deprecated functionality or names with the new versions. Sure at some point 'this is it' and features of D and Phobos become set in stone. "Hello world!" console output is one of those examples that many will try first. I would understand if it breaks between major versions of a language, but not from one revision to the next. YMMV :)
Re: std.stdio overhaul by Steve Schveighoffer
Andrej Mitrovic: > It would really help out if we had some sort of semi-automated script > that can do at least partial translation of code that uses old phobos > functions to new ones. Maybe this wouldn't work 100% but at least it > would help out. I'm thinking of something similar to what Python 2to3 > does. You mean like the standard tool gofix: http://blog.golang.org/2011/04/introducing-gofix.html Bye, bearophile
Re: std.stdio overhaul by Steve Schveighoffer
Jonathan M Davis: > > Breaking std.stdio, which is used everywhere, this thoroughly needs a very > > high bar of justification. > > Agreed. The purpose of the gofix tool in the Go language library is to lower this bar significantly :-) Bye, bearophile
Re: std.stdio overhaul by Steve Schveighoffer
Count me as another who is sick and tired of the gratuitous breaking changes every damned month. The worst part is there's still some new stuff I actually want each month, so I'm not doing my usual strategy of never, ever, ever updating software. It's just pain. Trivial changes are easy enough to fix, but are a pain. More complex changes cost me time and money. (I'm still angry about the removal of std.date. But soft deprecation is even worse - I hate that so much the first thing I do when updating my dmd is to edit the source to get that useless annoying shit out of there)
Re: std.stdio overhaul by Steve Schveighoffer
On 09/05/2011 04:51 PM, Walter Bright wrote: If the new std.path breaks existing code, I need to fix it before it is released. Please let me know what problems you are experiencing. It prints out all the deprecation message. It means I'll have to go edit existing, working code to change the names. I think it means it gives you time, on your own schedule with generous deadlines, to make the changes to your code. I know that the majority wants the name changes. I know the deprecation system gives people plenty of time to edit their code. But I think the cost of breaking existing code is much higher than many realize, and a lot of that cost will be hidden. It'll come in the form of people deciding not to use D because it is "not stable". It'll come in the form of invalidating existing libraries and modules unless someone is regularly maintaining them. It'll come in the form of invalidating the mass of books, articles, blog postings, and presentations about D, and those will never get updated. People will type in the code examples, they will fail to compile, and they'll get turned off about D. I'll again note that I know of know successful operating system or programming language that goes around breaking existing code unless it is really, really urgent. Camel-casing a name doesn't meet that standard. So, yes, I don't like it. I agree with all of the above. However, as is often the case, there's more than one side to the story. Bad APIs have their costs too. We can't afford to have an XML library that offers few and badly packaged features and comes at the tail of all benchmarks. We also can't afford a JSON library that is poorly designed and badly written. Ironically, the costs mostly manifest the same way: people will decide not to use D because it "lacks good libraries" and "is quirky to use". In many ways a language's standard library is a showcase of the language, and to a newcomer an inconsistent and awkward standard library affects the perception of the language's quality. Stressing that breaking code has a cost and implying that keeping it with flaws has no cost is as mistaken as worrying in chess about the flank at the expense of the center. The reality we need to face is, we are experiencing growth pains. What we must do is NOT lament about breaking this or keeping that. We must: a) devise good language features to cope with deprecation, of which deprecation with message is one that I think we need to embrace and extend (I have a few ideas I'll discuss separately); b) supplement that with a good policy for deprecating APIs and introducing new ones - in particular decide where to draw the line when introducing a breaking change; c) possibly create programs a la gofix that help migration. Andrei
Re: std.stdio overhaul by Steve Schveighoffer
On Tue, Sep 6, 2011 at 12:48 PM, Andrei Alexandrescu wrote: > On 09/05/2011 04:51 PM, Walter Bright wrote: >>> >>> If the new std.path breaks existing code, I need to fix it before it is >>> released. Please let me know what problems you are experiencing. >> >> It prints out all the deprecation message. It means I'll have to go edit >> existing, working code to change the names. > > I think it means it gives you time, on your own schedule with generous > deadlines, to make the changes to your code. > >> I know that the majority wants the name changes. I know the deprecation >> system gives people plenty of time to edit their code. >> >> But I think the cost of breaking existing code is much higher than many >> realize, and a lot of that cost will be hidden. It'll come in the form >> of people deciding not to use D because it is "not stable". It'll come >> in the form of invalidating existing libraries and modules unless >> someone is regularly maintaining them. It'll come in the form of >> invalidating the mass of books, articles, blog postings, and >> presentations about D, and those will never get updated. People will >> type in the code examples, they will fail to compile, and they'll get >> turned off about D. >> >> I'll again note that I know of know successful operating system or >> programming language that goes around breaking existing code unless it >> is really, really urgent. >> >> Camel-casing a name doesn't meet that standard. So, yes, I don't like it. > > I agree with all of the above. However, as is often the case, there's more > than one side to the story. > > Bad APIs have their costs too. We can't afford to have an XML library that > offers few and badly packaged features and comes at the tail of all > benchmarks. We also can't afford a JSON library that is poorly designed and > badly written. Ironically, the costs mostly manifest the same way: people > will decide not to use D because it "lacks good libraries" and "is quirky to > use". In many ways a language's standard library is a showcase of the > language, and to a newcomer an inconsistent and awkward standard library > affects the perception of the language's quality. > > Stressing that breaking code has a cost and implying that keeping it with > flaws has no cost is as mistaken as worrying in chess about the flank at the > expense of the center. > > The reality we need to face is, we are experiencing growth pains. What we > must do is NOT lament about breaking this or keeping that. We must: > > a) devise good language features to cope with deprecation, of which > deprecation with message is one that I think we need to embrace and extend > (I have a few ideas I'll discuss separately); > > b) supplement that with a good policy for deprecating APIs and introducing > new ones - in particular decide where to draw the line when introducing a > breaking change; > > c) possibly create programs a la gofix that help migration. > > > Andrei > My question is why do you even need a standard API for XML and JSON. Trying to support everything out of the box to a high degree of quality and provide enough generality that it's useful for everybody is just too much work and all you achieve is to discourage alternative implementations better suited to specific needs.
Re: std.stdio overhaul by Steve Schveighoffer
== Quote from Andrei Alexandrescu (seewebsiteforem...@erdani.org)'s article > In many ways a language's standard library is a > showcase of the language, YES!!! I'm glad someone besides me finally realizes this. For example, whenever someone asks me about why D metaprogramming is so great, I just point them to a few std lib modules that showcase this, e.g.: http://stackoverflow.com/questions/7300298/metaprogramming-in-c-and-in-d/7300611#7300611
Re: std.stdio overhaul by Steve Schveighoffer
On 9/5/2011 7:48 PM, Andrei Alexandrescu wrote: I agree with all of the above. However, as is often the case, there's more than one side to the story. Bad APIs have their costs too. We can't afford to have an XML library that offers few and badly packaged features and comes at the tail of all benchmarks. We also can't afford a JSON library that is poorly designed and badly written. Ironically, the costs mostly manifest the same way: people will decide not to use D because it "lacks good libraries" and "is quirky to use". In many ways a language's standard library is a showcase of the language, and to a newcomer an inconsistent and awkward standard library affects the perception of the language's quality. I agree that the XML and JSON libraries need to be scrapped and rewritten. But simply changing the names of otherwise successful APIs is not worth while. c) possibly create programs a la gofix that help migration. gofix cannot fix books, articles, blogs, and presentations. Furthermore, in order to work successfully, gofix needs to be a complete D front end, capable of handling both the old and the new stuff. Doing a perl script would be a disaster. It's a substantial project, has a high risk of inadequacy, and I suspect our resources are better spent elsewhere. Considering also the problems people have running dmd and getting it to find their imports and libraries, add in having to run 'gofix' over their source code first, then patch up what gofix goofed up, seems a stretch.
Re: std.stdio overhaul by Steve Schveighoffer
On 2011-09-06 04:48, Andrei Alexandrescu wrote: I agree with all of the above. However, as is often the case, there's more than one side to the story. Bad APIs have their costs too. We can't afford to have an XML library that offers few and badly packaged features and comes at the tail of all benchmarks. We also can't afford a JSON library that is poorly designed and badly written. Ironically, the costs mostly manifest the same way: people will decide not to use D because it "lacks good libraries" and "is quirky to use". In many ways a language's standard library is a showcase of the language, and to a newcomer an inconsistent and awkward standard library affects the perception of the language's quality. Stressing that breaking code has a cost and implying that keeping it with flaws has no cost is as mistaken as worrying in chess about the flank at the expense of the center. The reality we need to face is, we are experiencing growth pains. What we must do is NOT lament about breaking this or keeping that. We must: a) devise good language features to cope with deprecation, of which deprecation with message is one that I think we need to embrace and extend (I have a few ideas I'll discuss separately); b) supplement that with a good policy for deprecating APIs and introducing new ones - in particular decide where to draw the line when introducing a breaking change; c) possibly create programs a la gofix that help migration. Andrei We don't want to have a standard library like the one in PHP where there seems to be no naming conventions at all. -- /Jacob Carlborg
Re: std.stdio overhaul by Steve Schveighoffer
On 2011-09-06 08:00, Walter Bright wrote: On 9/5/2011 7:48 PM, Andrei Alexandrescu wrote: I agree with all of the above. However, as is often the case, there's more than one side to the story. Bad APIs have their costs too. We can't afford to have an XML library that offers few and badly packaged features and comes at the tail of all benchmarks. We also can't afford a JSON library that is poorly designed and badly written. Ironically, the costs mostly manifest the same way: people will decide not to use D because it "lacks good libraries" and "is quirky to use". In many ways a language's standard library is a showcase of the language, and to a newcomer an inconsistent and awkward standard library affects the perception of the language's quality. I agree that the XML and JSON libraries need to be scrapped and rewritten. But simply changing the names of otherwise successful APIs is not worth while. So we have to live with these naming conventions from C forever? -- /Jacob Carlborg
Re: std.stdio overhaul by Steve Schveighoffer
On Tuesday, September 06, 2011 08:42:14 Jacob Carlborg wrote: > On 2011-09-06 08:00, Walter Bright wrote: > > On 9/5/2011 7:48 PM, Andrei Alexandrescu wrote: > >> I agree with all of the above. However, as is often the case, there's > >> more than > >> one side to the story. > >> > >> Bad APIs have their costs too. We can't afford to have an XML library > >> that > >> offers few and badly packaged features and comes at the tail of all > >> benchmarks. > >> We also can't afford a JSON library that is poorly designed and badly > >> written. > >> Ironically, the costs mostly manifest the same way: people will decide > >> not to > >> use D because it "lacks good libraries" and "is quirky to use". In > >> many ways a > >> language's standard library is a showcase of the language, and to a > >> newcomer an > >> inconsistent and awkward standard library affects the perception of > >> the > >> language's quality. > > > > I agree that the XML and JSON libraries need to be scrapped and > > rewritten. But simply changing the names of otherwise successful APIs is > > not worth while. > > So we have to live with these naming conventions from C forever? My take on it is that we need to figure out which pieces of Phobos need to be reworked or renamed and get it done as soon as possible. That way, everything follows the proper naming conventions (thus avoiding a mess like PHP) and is of an appropriately high level of quality. Then we can have an appropriately stable API which doesn't have to change often - if at all. I think that the current problem with Phobos is primarily a combination of three things: 1. Older APIs which aren't in line with how D2 and Phobos have evolved (e.g. they don't use ranges when they should). 2. Some older stuff didn't get a thorough enough peer review before making it into Phobos and is not at a high enough level of quality, so it needs to be revised or replaced. 3. Too much of what has been done in the past has been a hodgepodge of naming conventions, making it very inconsistent in some places. Once those have been sorted out (some of which can be done without breaking any existing code and some of which requires breaking changes), then we can have a stable API for Phobos which doesn't change much except where we're adding new functionality which doesn't break existing code. So ultimately, we _will_ have a stable API, but some breaking changes are required in the short term to resolve issues with Phobos which would cause problems in the long run. - Jonathan M Davis
Re: std.stdio overhaul by Steve Schveighoffer
On 9/5/2011 11:39 PM, Jacob Carlborg wrote: We don't want to have a standard library like the one in PHP where there seems to be no naming conventions at all. I don't think that is the reason PHP is such a bear to work with.
Re: std.stdio overhaul by Steve Schveighoffer
Josh Simmons: > My question is why do you even need a standard API for XML and JSON. It helps port your user code to other libs that use the same standard API. This is very useful. In D I'd like a basic standard API even for simple 2D graphics. Bye, bearophile
Re: std.stdio overhaul by Steve Schveighoffer
On Tue, Sep 6, 2011 at 6:37 PM, bearophile wrote: > Josh Simmons: > >> My question is why do you even need a standard API for XML and JSON. > > It helps port your user code to other libs that use the same standard API. > This is very useful. In D I'd like a basic standard API even for simple 2D > graphics. > > Bye, > bearophile > This would be true if there were only implementation differences between libraries doing roughly the same thing (in which case you'd not need a new library anyway). Unfortunately this is not how things work. So simple 2d graphics ey? vector or raster based? immediate rendering or scene graph representation? animation? fonts? textures? XML ey? SAX, DOM, Pull, Data Binding? XPath? XSLT? The problem with php isn't just it's awesome naming, it's the fact that anything that seemed like something somebody might use was added as opposed to limiting itself to the must-haves.
Re: std.stdio overhaul by Steve Schveighoffer
On 2011-09-06 08:56, Jonathan M Davis wrote: On Tuesday, September 06, 2011 08:42:14 Jacob Carlborg wrote: On 2011-09-06 08:00, Walter Bright wrote: On 9/5/2011 7:48 PM, Andrei Alexandrescu wrote: I agree with all of the above. However, as is often the case, there's more than one side to the story. Bad APIs have their costs too. We can't afford to have an XML library that offers few and badly packaged features and comes at the tail of all benchmarks. We also can't afford a JSON library that is poorly designed and badly written. Ironically, the costs mostly manifest the same way: people will decide not to use D because it "lacks good libraries" and "is quirky to use". In many ways a language's standard library is a showcase of the language, and to a newcomer an inconsistent and awkward standard library affects the perception of the language's quality. I agree that the XML and JSON libraries need to be scrapped and rewritten. But simply changing the names of otherwise successful APIs is not worth while. So we have to live with these naming conventions from C forever? My take on it is that we need to figure out which pieces of Phobos need to be reworked or renamed and get it done as soon as possible. That way, everything follows the proper naming conventions (thus avoiding a mess like PHP) and is of an appropriately high level of quality. Then we can have an appropriately stable API which doesn't have to change often - if at all. I think that the current problem with Phobos is primarily a combination of three things: 1. Older APIs which aren't in line with how D2 and Phobos have evolved (e.g. they don't use ranges when they should). 2. Some older stuff didn't get a thorough enough peer review before making it into Phobos and is not at a high enough level of quality, so it needs to be revised or replaced. 3. Too much of what has been done in the past has been a hodgepodge of naming conventions, making it very inconsistent in some places. Once those have been sorted out (some of which can be done without breaking any existing code and some of which requires breaking changes), then we can have a stable API for Phobos which doesn't change much except where we're adding new functionality which doesn't break existing code. So ultimately, we _will_ have a stable API, but some breaking changes are required in the short term to resolve issues with Phobos which would cause problems in the long run. - Jonathan M Davis Yes, thank you, I agree. -- /Jacob Carlborg
Re: std.stdio overhaul by Steve Schveighoffer
On 2011-09-06 09:35, Walter Bright wrote: On 9/5/2011 11:39 PM, Jacob Carlborg wrote: We don't want to have a standard library like the one in PHP where there seems to be no naming conventions at all. I don't think that is the reason PHP is such a bear to work with. I think that that is one reason, not the only one, not the biggest one, but one reason. -- /Jacob Carlborg
Re: std.stdio overhaul by Steve Schveighoffer
On Tuesday, September 06, 2011 18:54:45 Josh Simmons wrote: > On Tue, Sep 6, 2011 at 6:37 PM, bearophile wrote: > > Josh Simmons: > >> My question is why do you even need a standard API for XML and JSON. > > > > It helps port your user code to other libs that use the same standard > > API. This is very useful. In D I'd like a basic standard API even for > > simple 2D graphics. > > > > Bye, > > bearophile > > This would be true if there were only implementation differences > between libraries doing roughly the same thing (in which case you'd > not need a new library anyway). Unfortunately this is not how things > work. > > So simple 2d graphics ey? vector or raster based? immediate rendering > or scene graph representation? animation? fonts? textures? > > XML ey? SAX, DOM, Pull, Data Binding? XPath? XSLT? > > The problem with php isn't just it's awesome naming, it's the fact > that anything that seemed like something somebody might use was added > as opposed to limiting itself to the must-haves. Other major languages (such as Java and C#) have large standard libraries and have done quite well with them. In fact, I believe that the large size of their standard libraries is generally seen as major advantage of those languages. No, we can't have everything in the standard library. No, an XML parser in the standard library likely won't meet everyone's needs. However, having a large standard library can be of great benefit to the users of the language even if it doesn't solve every problem that they could possibly have. The question isn't really whether we should add stuff like XML parsing to Phobos. The question is what is the best general implementation for a such a module and whether we can get an implementation of high enough quality to be able to go in the standard library. It's a question of time, man power, and quality. Obviously, Phobos is not going to explode in size overnight, but it _is_ going to grow in size, and eventually it should be fairly large. We already have several useful additions in the review queue which will likely make it into Phobos in one form or another over the next few months. - Jonathan M Davis
Re: std.stdio overhaul by Steve Schveighoffer
On 2011-09-06 11:09, Jonathan M Davis wrote: On Tuesday, September 06, 2011 18:54:45 Josh Simmons wrote: This would be true if there were only implementation differences between libraries doing roughly the same thing (in which case you'd not need a new library anyway). Unfortunately this is not how things work. So simple 2d graphics ey? vector or raster based? immediate rendering or scene graph representation? animation? fonts? textures? XML ey? SAX, DOM, Pull, Data Binding? XPath? XSLT? The problem with php isn't just it's awesome naming, it's the fact that anything that seemed like something somebody might use was added as opposed to limiting itself to the must-haves. Other major languages (such as Java and C#) have large standard libraries and have done quite well with them. In fact, I believe that the large size of their standard libraries is generally seen as major advantage of those languages. No, we can't have everything in the standard library. No, an XML parser in the standard library likely won't meet everyone's needs. However, having a large standard library can be of great benefit to the users of the language even if it doesn't solve every problem that they could possibly have. The question isn't really whether we should add stuff like XML parsing to Phobos. The question is what is the best general implementation for a such a module and whether we can get an implementation of high enough quality to be able to go in the standard library. It's a question of time, man power, and quality. Phobos could have a low level XML parsing module and on top of that other XML APIs can be built, like SAX, DOM and so on. This is how the XML modules in Tango are built. Tango has a low level XML pull parse. Built on top of that are a SAX API and a DOM document. Obviously, Phobos is not going to explode in size overnight, but it _is_ going to grow in size, and eventually it should be fairly large. We already have several useful additions in the review queue which will likely make it into Phobos in one form or another over the next few months. - Jonathan M Davis -- /Jacob Carlborg
Re: std.stdio overhaul by Steve Schveighoffer
On Tue, Sep 6, 2011 at 7:09 PM, Jonathan M Davis wrote: > Other major languages (such as Java and C#) have large standard libraries and > have done quite well with them. In fact, I believe that the large size of > their standard libraries is generally seen as major advantage of those > languages. > > No, we can't have everything in the standard library. No, an XML parser in the > standard library likely won't meet everyone's needs. However, having a large > standard library can be of great benefit to the users of the language even if > it doesn't solve every problem that they could possibly have. The question > isn't really whether we should add stuff like XML parsing to Phobos. The > question is what is the best general implementation for a such a module and > whether we can get an implementation of high enough quality to be able to go > in the standard library. It's a question of time, man power, and quality. > > Obviously, Phobos is not going to explode in size overnight, but it _is_ going > to grow in size, and eventually it should be fairly large. We already have > several useful additions in the review queue which will likely make it into > Phobos in one form or another over the next few months. > > - Jonathan M Davis > Other languages like C# and Java have large enterprise outfits backing their massive standard libraries too. I just think the effort is better spent creating a solid language and encouraging third party libraries through better tools.
Re: std.stdio overhaul by Steve Schveighoffer
On Tuesday, September 06, 2011 19:29:13 Josh Simmons wrote: > On Tue, Sep 6, 2011 at 7:09 PM, Jonathan M Davis wrote: > > Other major languages (such as Java and C#) have large standard > > libraries and have done quite well with them. In fact, I believe that > > the large size of their standard libraries is generally seen as major > > advantage of those languages. > > > > No, we can't have everything in the standard library. No, an XML parser > > in the standard library likely won't meet everyone's needs. However, > > having a large standard library can be of great benefit to the users of > > the language even if it doesn't solve every problem that they could > > possibly have. The question isn't really whether we should add stuff > > like XML parsing to Phobos. The question is what is the best general > > implementation for a such a module and whether we can get an > > implementation of high enough quality to be able to go in the standard > > library. It's a question of time, man power, and quality. > > > > Obviously, Phobos is not going to explode in size overnight, but it _is_ > > going to grow in size, and eventually it should be fairly large. We > > already have several useful additions in the review queue which will > > likely make it into Phobos in one form or another over the next few > > months. > > > > - Jonathan M Davis > > Other languages like C# and Java have large enterprise outfits backing > their massive standard libraries too. > > I just think the effort is better spent creating a solid language and > encouraging third party libraries through better tools. For the most part, the folks working on Phobos are not the same folks who work on dmd. There's some overlap, but they're definitely not the same people. So, the fact that people are working on the standard library does _nothing_ to slow the language down. If anything, it helps, because it provides a standard code base which uses (and therefore tests) the various features of the language. Third party libraries are great, but I don't see why you would ever want to discourage development of a language's standard library in favor of third party libraries. In some cases, modules in the standard library have originated in third party libraries anyway. No, Phobos is not likely to ever rival C# or Java for volume of code. But that doesn't mean that Phobos shouldn't try and be as large is it can be while still maintaining high quality. - Jonathan M Davis
Re: std.stdio overhaul by Steve Schveighoffer
Josh Simmons: > So simple 2d graphics ey? vector or raster based? immediate rendering > or scene graph representation? animation? fonts? textures? Raster, immediate rendering, no need to specify animations, basic support for fonts and textures. Leaving most things out is not a problem here. Time ago someone has shown here a nice D module that works on Windows, Linux, that's is short. This module doesn't replace other graphics libs or GUI modules, it's something simple and small for small purposes. > XML ey? SAX, DOM, Pull, Data Binding? XPath? XSLT? > > The problem with php isn't just it's awesome naming, it's the fact > that anything that seemed like something somebody might use was added > as opposed to limiting itself to the must-haves. Some people like wide libraries like Python (batteries included), others don't like that. Both choices have their serious advantages and serious disadvantages. There are not general rules to solve this problem, each case needs to be discussed. I think a good JSON module needs to be in Phobos, while for XML maybe it just needs a standard D API (this also comes from practical size considerations: a JSON module is probably small. A good XML module will probably become large). Bye, bearophile
Re: std.stdio overhaul by Steve Schveighoffer
On Sun, 04 Sep 2011 07:07:05 -0400, Jacob Carlborg wrote: On 2011-09-03 21:54, Andrei Alexandrescu wrote: Hello, There are a number of issues related to D's current handling of streams, including the existence of the imperfect etc.stream and the over-specialization of std.stdio. Steve has worked on an extensive overhaul of std.stdio which would obviate the need for etc.stream and would improve both the generality and efficiency of std.stdio. Please chime in with feedback; he's away from the Usenet but allowed me to post this on his behalf. I uploaded the docs to http://erdani.com/d/new-stdio/phobos-prerelease/std_stdio.html Thanks, Andrei I think that openFile, File.open and CStream.open should shouldn't take a string as the mode, it should be an enum or similar. Andrei is making a big deal out of using enums instead of bools. A bool value can contain "true" or "false", a string can contain an infinite number of different values. openFile takes it as a template argument, and it will fail at compile time if the parameter is not correct (if not now, it will when the library is ready for inclusion). I agree that enum is cleaner and easier to deal with from the library's point of view, but we have 2 things going for us by using strings: 1. The string formats are backwards compatible, and well defined. In fact, CStream.open just passes the mode string without modification to fopen. 2. The brevity of and ability to comprehend a string literal vs. multiple enums. You can think of it like printf (or writef). The format string has infinitely wrong possible format strings, which must be rejected at run time. But I'll take that any day over C++'s format modifiers which are type checked at compile-time. Remember, typically, string formats are most frequently literals, and easy to read/write. While there is great potential for invalid parameters, the reality is this rarely happens, and if it does, the errors are seen immediately. -Steve
Re: std.stdio overhaul by Steve Schveighoffer
On 2011-09-05 21:51:16 +, Walter Bright said: I'll again note that I know of know successful operating system or programming language that goes around breaking existing code unless it is really, really urgent. Apple has been deprecating things a lot in Mac OS X. Deprecated APIs generally continue to work fine for a long time and only trigger warnings when you compile something that uses them, effectively making them inconvenient. Some deprecation messages that can't be compilation warnings are logged to the console when used instead (deprecated flags for instance), only once per process though. Sometime APIs are truly disabled, but they are not removed. For instance, the old API for accessing the screen's pixels has become non-functional in Mac OS X 10.7 Lion. Only the new API introduced 10.6 works now, the old one was still there but you just get a null pointer. Sometime APIs disappear when passing to a new architecture. For instance, Mac OS X still supports the old Carbon APIs, but only in 32-bit mode, those were never made available to 64-bit applications. But what works well for an operating system is not necessarily what works well for a runtime and a standard library. What Apple does is meant to keep binary compatibility. Users are not expected to have the source code of their application at hand, nor the expertise to fix them. They deprecate things so the OS can move forward and introduce new features, and using deprecated APIs generally mean that your app will have trouble using new features or move to new architectures in the future. The situation for the D standard library is a little different. If you compile D code, you do have the source code at hand. My take is that we should not remove deprecated APIs and thus break old programs unless keeping those APIs really cost too much or impede future improvements. Showing a deprecation message and marking them as deprecated in the documentation is important to incite people to use the non-deprecated APIs, but for simple things like name changes perhaps the deprecation message during compilation could be left out, as the improvement to annoyance ratio would be quite low. -- Michel Fortin michel.for...@michelf.com http://michelf.com/
Re: std.stdio overhaul by Steve Schveighoffer
On 2011-09-06 12:50, Steven Schveighoffer wrote: On Sun, 04 Sep 2011 07:07:05 -0400, Jacob Carlborg wrote: On 2011-09-03 21:54, Andrei Alexandrescu wrote: Hello, There are a number of issues related to D's current handling of streams, including the existence of the imperfect etc.stream and the over-specialization of std.stdio. Steve has worked on an extensive overhaul of std.stdio which would obviate the need for etc.stream and would improve both the generality and efficiency of std.stdio. Please chime in with feedback; he's away from the Usenet but allowed me to post this on his behalf. I uploaded the docs to http://erdani.com/d/new-stdio/phobos-prerelease/std_stdio.html Thanks, Andrei I think that openFile, File.open and CStream.open should shouldn't take a string as the mode, it should be an enum or similar. Andrei is making a big deal out of using enums instead of bools. A bool value can contain "true" or "false", a string can contain an infinite number of different values. openFile takes it as a template argument, and it will fail at compile time if the parameter is not correct (if not now, it will when the library is ready for inclusion). If it validates the string at compile time than that's great. I agree that enum is cleaner and easier to deal with from the library's point of view, but we have 2 things going for us by using strings: 1. The string formats are backwards compatible, and well defined. In fact, CStream.open just passes the mode string without modification to fopen. 2. The brevity of and ability to comprehend a string literal vs. multiple enums. You can think of it like printf (or writef). The format string has infinitely wrong possible format strings, which must be rejected at run time. But I'll take that any day over C++'s format modifiers which are type checked at compile-time. It's not very often I use the print format functions. Most of the time I use Tango and with Tango's format strings at least you don't have to specify the type. Remember, typically, string formats are most frequently literals, and easy to read/write. While there is great potential for invalid parameters, the reality is this rarely happens, and if it does, the errors are seen immediately. -Steve I would not say that they are easy to read, or at least understand/remember what a given mode means. I always have to double check the documentation when using these kind of modes. I always have to check if a given mode creates a new file or not. -- /Jacob Carlborg
Re: std.stdio overhaul by Steve Schveighoffer
On Tue, 06 Sep 2011 08:49:22 -0400, Jacob Carlborg wrote: On 2011-09-06 12:50, Steven Schveighoffer wrote: On Sun, 04 Sep 2011 07:07:05 -0400, Jacob Carlborg wrote: On 2011-09-03 21:54, Andrei Alexandrescu wrote: Hello, There are a number of issues related to D's current handling of streams, including the existence of the imperfect etc.stream and the over-specialization of std.stdio. Steve has worked on an extensive overhaul of std.stdio which would obviate the need for etc.stream and would improve both the generality and efficiency of std.stdio. Please chime in with feedback; he's away from the Usenet but allowed me to post this on his behalf. I uploaded the docs to http://erdani.com/d/new-stdio/phobos-prerelease/std_stdio.html Thanks, Andrei I think that openFile, File.open and CStream.open should shouldn't take a string as the mode, it should be an enum or similar. Andrei is making a big deal out of using enums instead of bools. A bool value can contain "true" or "false", a string can contain an infinite number of different values. openFile takes it as a template argument, and it will fail at compile time if the parameter is not correct (if not now, it will when the library is ready for inclusion). If it validates the string at compile time than that's great. I agree that enum is cleaner and easier to deal with from the library's point of view, but we have 2 things going for us by using strings: 1. The string formats are backwards compatible, and well defined. In fact, CStream.open just passes the mode string without modification to fopen. 2. The brevity of and ability to comprehend a string literal vs. multiple enums. You can think of it like printf (or writef). The format string has infinitely wrong possible format strings, which must be rejected at run time. But I'll take that any day over C++'s format modifiers which are type checked at compile-time. It's not very often I use the print format functions. Most of the time I use Tango and with Tango's format strings at least you don't have to specify the type. writef is the same, %s is equivalent to calling toString(). But the format specifiers for Tango are also strings, and not compile-time verified. My point was simply, using a string to indicate flags or formatting instructions is pretty efficient, easy to write, and easy to read. Remember, typically, string formats are most frequently literals, and easy to read/write. While there is great potential for invalid parameters, the reality is this rarely happens, and if it does, the errors are seen immediately. -Steve I would not say that they are easy to read, or at least understand/remember what a given mode means. I always have to double check the documentation when using these kind of modes. I always have to check if a given mode creates a new file or not. Yeah, creating a new file is implied by a combination of modes. The one that's confusing I think is that "a" is for append, but "+" kind of tacks on appending to any other mode. It's not the most well-designed spec for file opening. Add to that you have the "b" which is a noop on most OSes. There is the possibility that we could accept an alternative open mode string, which we could design better. But we have to keep fopen's spec, it's already used everywhere. -Steve
Re: std.stdio overhaul by Steve Schveighoffer
On 9/6/11, Steven Schveighoffer wrote: > There is the possibility that we could accept an alternative open mode > string, which we could design better. But we have to keep fopen's spec, > it's already used everywhere. > > -Steve > Or an alternative enum instead of a string. I'm another one of those people who forgets what the various read/write modes are.
Re: std.stdio overhaul by Steve Schveighoffer
Walter Bright wrote: > I don't think that is the reason PHP is such a bear to work with. It is one of the problems with PHP, but I'm not sure it applies to D the same way. Almost *every time* I write PHP, I either mess up a name or the argument order. (Sometimes, PHP functions go src, dest, and sometimes it's dest, src. Ugh! The worst part is it doesn't even catch this. A name mismatch throws an error when it's run. Reversed arguments just silently do the wrong thing.) The random argument order is a huge huge pain. Contrast with D, where I've almost never forgotten a name. Granted, it might be due to my auto-complete function so I type once and only once, but it just hasn't been a big deal. Thanks to the UFCS with arrays, the argument order is almost always the same to work with that, so the much bigger problem never occurs.
Re: std.stdio overhaul by Steve Schveighoffer
Walter Bright wrote: > I agree that the XML and JSON libraries need to be scrapped and rewritten. Ugh, I actually use the std.json. > Furthermore, in order to work successfully, gofix [...] The easiest way to do that is run the compiler. If an error occurs, go to the given line of the problem and maybe automatically replace with the spell checker's suggestion. Then in case that's wrong, ask the user to confirm it. ... which is pretty much what my editor (vim) already does! Changing names is annoying, but it's not a difficult task, with what we have now. The compiler already does 90% of the work, and even fairly simple editors will bring it to about 98%. I'll bitch about it, but it isn't a big enough deal to bother with a gofix. Trivial fixes are already trivial fixes. I'd prefer to avoid them, but let's not forget that the compiler already does most the work. The more annoying changes are where stuff changes wholesale, so the code needs to be rethought, data needs to be changed, and so on. These are just huge sinks of pain. And, no, a long deprecation time doesn't change anything. Whether I spend thousands of dollars today changing it or thousands of dollars in six months changing it, the fact is I'm still out thousands of dollars.
Re: std.stdio overhaul by Steve Schveighoffer
On 9/6/11 1:00 AM, Walter Bright wrote: On 9/5/2011 7:48 PM, Andrei Alexandrescu wrote: I agree with all of the above. However, as is often the case, there's more than one side to the story. Bad APIs have their costs too. We can't afford to have an XML library that offers few and badly packaged features and comes at the tail of all benchmarks. We also can't afford a JSON library that is poorly designed and badly written. Ironically, the costs mostly manifest the same way: people will decide not to use D because it "lacks good libraries" and "is quirky to use". In many ways a language's standard library is a showcase of the language, and to a newcomer an inconsistent and awkward standard library affects the perception of the language's quality. I agree that the XML and JSON libraries need to be scrapped and rewritten. But simply changing the names of otherwise successful APIs is not worth while. I agree we should be increasingly hawkish about such changes. c) possibly create programs a la gofix that help migration. gofix cannot fix books, articles, blogs, and presentations. Furthermore, in order to work successfully, gofix needs to be a complete D front end, capable of handling both the old and the new stuff. Doing a perl script would be a disaster. It's a substantial project, has a high risk of inadequacy, and I suspect our resources are better spent elsewhere. I'm not so sure. We're experiencing an unprecedented surge in participation to all aspects of the D programming language, and I believe you and I should start thinking differently. A certain change of phase happened to me a couple of months ago, when I commented about some fix that removed an undue limitation: "sounds good, but Walter has many other things on his plate more important than this". Within hours, the fix was available as a pull request. In this case, if one or more persons is/are determined enough to create dfix, it will happen regardless of whether you or I believe it's the optimal resource allocation. Considering also the problems people have running dmd and getting it to find their imports and libraries, add in having to run 'gofix' over their source code first, then patch up what gofix goofed up, seems a stretch. I do agree dfix must work at least as well as Apple hardware :o). Andrei
Re: std.stdio overhaul by Steve Schveighoffer
On 9/6/11 2:35 AM, Walter Bright wrote: On 9/5/2011 11:39 PM, Jacob Carlborg wrote: We don't want to have a standard library like the one in PHP where there seems to be no naming conventions at all. I don't think that is the reason PHP is such a bear to work with. Probably. At any rate, what I now think as a promising path is with new module names. Let's leave the likes of std.xml and std.json in peace, then pick a naming convention for the new ones and create whole new modules replacing them. Then people who are ready for the migration change import std.xml; with import std.some_naming_convention_involving_xml; and fix whatever code breakages that entails. If they're pleased with std.xml, nobody's holding a gun to their head. Months and years go by, and nobody uses std.xml because the new module and the migration path are copiously advertised in the documentation. At that point we can discuss excising std.xml altogether and replacing it with the new one. And so the new becomes old, just like in dialectics. There's a successful precedent in C++ - stringstream vs. strstream. The only missing thing is that C++ did not choose a naming convention because they limited themselves to only one header. So what should we use? xml2? new_xml? FWIW we use the prefix "new_" at Facebook to good effect. Or should we, au contraire, use "old_" for the old module and advise people who want to stick with the old modules to change their imports? Andrei
Re: std.stdio overhaul by Steve Schveighoffer
On 9/6/11 4:29 AM, Josh Simmons wrote: Other languages like C# and Java have large enterprise outfits backing their massive standard libraries too. I just think the effort is better spent creating a solid language and encouraging third party libraries through better tools. As always finding the right balance is key. Community-grown languages such as PHP, Python, or Ruby also enjoy large libraries, so I don't think corporate support is a prerequisite. It would probably be a mistake to stop work on Phobos now. For all I can tell I'm more annoyed by the energy spent on what I'd call "isometric churn" like changing names (even those used internally, sigh) and changing comments from /** */ to /++ +/. All that energy could go into adding value. Andrei
Re: std.stdio overhaul by Steve Schveighoffer
Andrei Alexandrescu wrote: > What should we use? xml2? That might be a good idea. If D modules were to get in the habit of writing their major version numbers as part of the name, it'd solve this as well the dget automatic library downloading thingy in one go. Going with new and old won't work if there should ever be a version 3 written.
