Re: Why is there no std.stream anymore?
On 6/18/20 11:11 PM, Jesse Phillips wrote: On Thursday, 18 June 2020 at 14:53:58 UTC, aberba wrote: On Tuesday, 12 December 2017 at 20:51:30 UTC, Steven Schveighoffer wrote: On 12/11/17 6:33 PM, Seb wrote: [...] Since iopipe was mentioned several times, I will say a couple things: [...] I should really try iopipe this time round. I think I avoided toying with it because the making conventions put me off. Don't remember about the docs and available examples though. I too was trying to utilize iopipe and asked questions earlier this year[1] and I made a file writer util[2]. I haven't really taken advantage of the power yet, though it does appear that be a really nice abstraction. 1. https://forum.dlang.org/thread/faawejguebluwodfl...@forum.dlang.org 2. https://gitlab.com/jessephillips/devarticlator/-/blob/master/source/util/file.d I reread my comment and in particular this: "I was going to write an ascii art concept to show how the pushing works, but I think I'll maybe draw an actual picture. I need some time to accomplish this, though." I still haven't done this. I need to make this happen, probably with a blog post (Mike, keep bugging me on this). -Steve
Re: Why is there no std.stream anymore?
On Thursday, 18 June 2020 at 14:53:58 UTC, aberba wrote: On Tuesday, 12 December 2017 at 20:51:30 UTC, Steven Schveighoffer wrote: On 12/11/17 6:33 PM, Seb wrote: [...] Since iopipe was mentioned several times, I will say a couple things: [...] I should really try iopipe this time round. I think I avoided toying with it because the making conventions put me off. Don't remember about the docs and available examples though. I too was trying to utilize iopipe and asked questions earlier this year[1] and I made a file writer util[2]. I haven't really taken advantage of the power yet, though it does appear that be a really nice abstraction. 1. https://forum.dlang.org/thread/faawejguebluwodfl...@forum.dlang.org 2. https://gitlab.com/jessephillips/devarticlator/-/blob/master/source/util/file.d
Re: Why is there no std.stream anymore?
On Thursday, 18 June 2020 at 15:03:38 UTC, Steven Schveighoffer wrote: On 6/18/20 10:53 AM, aberba wrote: On Tuesday, 12 December 2017 at 20:51:30 UTC, Steven Schveighoffer wrote: On 12/11/17 6:33 PM, Seb wrote: [...] Since iopipe was mentioned several times, I will say a couple things: [...] I should really try iopipe this time round. I think I avoided toying with it because the making conventions put me off. Don't remember about the docs and available examples though. I have made some updates that are probably not reflected yet in the docs. But let me know if there are any stumbling blocks. I intend to make the iopipe + std.io (from Martin) a more user-friendly library in the near future. -Steve Will discuss in the GH issue
Re: Why is there no std.stream anymore?
On 6/18/20 10:53 AM, aberba wrote: On Tuesday, 12 December 2017 at 20:51:30 UTC, Steven Schveighoffer wrote: On 12/11/17 6:33 PM, Seb wrote: [...] Since iopipe was mentioned several times, I will say a couple things: [...] I should really try iopipe this time round. I think I avoided toying with it because the making conventions put me off. Don't remember about the docs and available examples though. I have made some updates that are probably not reflected yet in the docs. But let me know if there are any stumbling blocks. I intend to make the iopipe + std.io (from Martin) a more user-friendly library in the near future. -Steve
Re: Why is there no std.stream anymore?
On Tuesday, 12 December 2017 at 20:51:30 UTC, Steven Schveighoffer wrote: On 12/11/17 6:33 PM, Seb wrote: [...] Since iopipe was mentioned several times, I will say a couple things: [...] I should really try iopipe this time round. I think I avoided toying with it because the making conventions put me off. Don't remember about the docs and available examples though.
Re: Why is there no std.stream anymore?
On Monday, 11 December 2017 at 23:33:44 UTC, Seb wrote:
---
void main(string[] args)
{
import std.conv, std.range, std.stdio;
foreach (d;
File(__FILE_FULL_PATH__).byChunk(4096).join.take(5)) {
writefln("%s", d.to!char);
}
}
---
A variant: https://run.dlang.io/is/2TUQBv
Re: Why is there no std.stream anymore?
Or this https://run.dlang.io/is/MO9Wiy
Re: Why is there no std.stream anymore?
On 12/11/17 6:33 PM, Seb wrote:
Though if you need superb performance, iopipe or similar will be faster.
Since iopipe was mentioned several times, I will say a couple things:
1. iopipe is not made for processing one element at a time, it focuses
on buffers. The reason for this is because certain tasks (i.e. parsing)
are much more efficient with buffered data than when using the range
API. Even with FILE *, using fgetc for every character is going to suck
when compared to fread, and processing the resulting array in-memory.
2. If you do want to process by element, I recommend the following chain:
// an example that uses iopipe's file stream and assumes it's UTF8 text.
// other mechanisms are available.
auto mypipe = openDev("somefile") // open a file
.bufd // buffer it
.assumeText // assume it's utf-8 text
.ensureDecodeable;// ensure there are no partial
code-points in the window
// convert to range of "chunks", and then join into one large range
foreach(c; mypipe.asInputRange.joiner) {
process(c);
}
Note, due to Phobos's auto-decoding, joiner is going to auto-decode all
of the data. This means typeof(c) is going to be dchar, and not char,
and everything needs to be proper utf-8. If you want to process the
bytes raw, you can omit the .assumeText.ensureDecodeable part, and the
data will be ubytes.
-Steve
Re: Why is there no std.stream anymore?
On Tuesday, 12 December 2017 at 02:15:13 UTC, codephantom wrote:
just playing around with this
also...in case you only want to read n bytes..
// ---
module test;
import std.stdio, std.file, std.exception;
import std.datetime.stopwatch;
void main()
{
string filename = "test.txt"; // a text file
//string filename = "test.exe"; // a binary file
enforce(filename.exists, "Umm..that file does not exist!");
auto file = File(filename, "r");
ubyte[] buf;
import std.datetime : MonoTime;
auto t2 = MonoTime.currTime;
// just read the first n bytes.
int bytesToRead = 4; // change this n
int bufCount;
while ( !file.eof() && bufCount < bytesToRead )
{
buf = file.rawRead(new ubyte[1]);
if(!file.eof())
{
process(cast(char)(buf[0]));
bufCount++;
}
}
writeln("-");
writeln("this took : ", MonoTime.currTime - t2);
writeln("-");
writeln();
return;
}
void process(char someChar)
{
import std.ascii : isPrintable;
if( isPrintable(someChar) )
writeln("found a printable character: ", someChar);
else
writeln("found a non printable character");
}
// ---
Re: Why is there no std.stream anymore?
On Monday, 11 December 2017 at 20:51:41 UTC, Jordi Gutiérrez
Hermoso wrote:
I'd like to read from a file, one byte at a time, without
loading the whole file in memory.
just playing around with this
//
module test;
import std.stdio, std.file, std.exception;
void main()
{
string filename = "test.txt";
enforce(filename.exists, "Umm..that file does not exist!");
auto file = File(filename, "r");
char[] charBuf;
while (!file.eof())
{
charBuf = file.rawRead(new char[1]);
if(!file.eof())
process(cast(char)(charBuf[0]));
}
return;
}
void process(char someChar)
{
import std.ascii : isPrintable;
if( isPrintable(someChar) )
writeln("found a printable character: ", someChar);
else
writeln("found a non printable character");
}
//
Re: Why is there no std.stream anymore?
On Monday, 11 December 2017 at 22:58:53 UTC, Jordi Gutiérrez
Hermoso wrote:
On Monday, 11 December 2017 at 21:21:51 UTC, Steven
Schveighoffer wrote:
Use the undead repository:
Wow, really? Is the removal of stream from D some kind of error
that hasn't been corrected yet?
Well of course you can use ranges for it, see e.g. this simple
example:
---
void main(string[] args)
{
import std.conv, std.range, std.stdio;
foreach (d;
File(__FILE_FULL_PATH__).byChunk(4096).join.take(5)) {
writefln("%s", d.to!char);
}
}
---
Run here: https://run.dlang.io/is/Ann9e9
Though if you need superb performance, iopipe or similar will be
faster.
Re: Why is there no std.stream anymore?
On Monday, 11 December 2017 at 20:51:41 UTC, Jordi Gutiérrez
Hermoso wrote:
I'd like to read from a file, one byte at a time, without
loading the whole file in memory.
I was hoping I could do something like
auto f = File("somefile");
foreach(c; f.byChar) {
process(c);
}
but there appears to be no such way to do it anymore. Instead,
the stdlib seems to provide several functions to do chunked
reads from the file where I have to manually manage the buffer.
I see that D1 had a stream, but it's no longer here and I
understand ranges are supposed to be used instead.
What's the explanation here? Why is there no more stream and
what am I supposed to use instead? Do I really need to be
manually managing the read buffer myself?
This should work;
scope f = new MmFile("somefile");
foreach(c; cast(string)f[]) {
process(c);
}
Re: Why is there no std.stream anymore?
On Monday, December 11, 2017 22:58:53 Jordi Gutiérrez Hermoso via Digitalmars-d-learn wrote: > On Monday, 11 December 2017 at 21:21:51 UTC, Steven Schveighoffer > > wrote: > > Use the undead repository: > Wow, really? Is the removal of stream from D some kind of error > that hasn't been corrected yet? std.stream was deemed to not be up to Phobos' current standards, and it's not in line with Phobos' current design and implementation (most notably, it doesn't support ranges at all). No one has cared enough to come up with an alternative implementation and propose it for inclusion in Phobos. However, for most needs, ranges do what you might do with a stream solution, and std.bitmanip provides useful functions for byte-level manipulation (e.g. taking the first elements from a range of ubytes and converting them to int). Depending on what you're looking for, http://code.dlang.org/packages/iopipe could also fit in quite well, though it's very much a work in progress, and there are several serialization libraries on code.dlang.org if that's more what you're looking for. - Jonathan M Davis
Re: Why is there no std.stream anymore?
On 12/11/17 5:58 PM, Jordi Gutiérrez Hermoso wrote: On Monday, 11 December 2017 at 21:21:51 UTC, Steven Schveighoffer wrote: Use the undead repository: Wow, really? Is the removal of stream from D some kind of error that hasn't been corrected yet? No, it was removed because it was considered subpar/obsolete. It doesn't jive with the rest of Phobos. But modules that are removed are put into the undead repository for those who wish to continue using it. -Steve
Re: Why is there no std.stream anymore?
On Monday, 11 December 2017 at 21:21:51 UTC, Steven Schveighoffer wrote: Use the undead repository: Wow, really? Is the removal of stream from D some kind of error that hasn't been corrected yet?
Re: Why is there no std.stream anymore?
On 12/11/17 3:51 PM, Jordi Gutiérrez Hermoso wrote:
I'd like to read from a file, one byte at a time, without loading the
whole file in memory.
I was hoping I could do something like
auto f = File("somefile");
foreach(c; f.byChar) {
process(c);
}
but there appears to be no such way to do it anymore. Instead, the
stdlib seems to provide several functions to do chunked reads from the
file where I have to manually manage the buffer. I see that D1 had a
stream, but it's no longer here and I understand ranges are supposed to
be used instead.
What's the explanation here? Why is there no more stream and what am I
supposed to use instead? Do I really need to be manually managing the
read buffer myself?
Use the undead repository:
http://code.dlang.org/packages/undead
https://github.com/dlang/undeaD
https://github.com/dlang/undeaD/blob/master/src/undead/stream.d
-Steve
Why is there no std.stream anymore?
I'd like to read from a file, one byte at a time, without loading
the whole file in memory.
I was hoping I could do something like
auto f = File("somefile");
foreach(c; f.byChar) {
process(c);
}
but there appears to be no such way to do it anymore. Instead,
the stdlib seems to provide several functions to do chunked reads
from the file where I have to manually manage the buffer. I see
that D1 had a stream, but it's no longer here and I understand
ranges are supposed to be used instead.
What's the explanation here? Why is there no more stream and what
am I supposed to use instead? Do I really need to be manually
managing the read buffer myself?
