Deprecation: module std.stream is deprecated
Deprecation: module std.stream is deprecated - It will be removed
from Phobos in October 2016.
The std.stream module documentation doesn't give any clues as to
what an alternative might be.
I have this sort of code:
this(InputStream input)
{
int line_no = 1;
foreach (char[] line; input) {
Am I right in thinking to use std.stdio with an InputRange?
- Jason
Re: Deprecation: module std.stream is deprecated
On Saturday, November 07, 2015 12:10:05 Spacen Jasset via Digitalmars-d-learn
wrote:
> Deprecation: module std.stream is deprecated - It will be removed
> from Phobos in October 2016.
>
> The std.stream module documentation doesn't give any clues as to
> what an alternative might be.
>
> I have this sort of code:
>
> this(InputStream input)
> {
>
> int line_no = 1;
> foreach (char[] line; input) {
>
> Am I right in thinking to use std.stdio with an InputRange?
At this point, you pretty much have three options:
1. Use std.stdio.File and read it in in pieces (e.g. with byLine or
byChunk).
2. Read the whole file in at once with std.file.read or std.file.readText.
3. Use std.mmfile.MmFile to mmap the file so that the parts that you access
get read into memory, and the rest don't get loaded from disk, but you get
to operate on the whole file as if you had read it all from disk.
- Jonathan M Davis
Re: Deprecation: module std.stream is deprecated
On Saturday, 7 November 2015 at 13:30:44 UTC, Jonathan M Davis
wrote:
On Saturday, November 07, 2015 12:10:05 Spacen Jasset via
Digitalmars-d-learn wrote:
Deprecation: module std.stream is deprecated - It will be
removed from Phobos in October 2016.
The std.stream module documentation doesn't give any clues as
to what an alternative might be.
I have this sort of code:
this(InputStream input)
{
int line_no = 1;
foreach (char[] line; input) {
Am I right in thinking to use std.stdio with an InputRange?
At this point, you pretty much have three options:
1. Use std.stdio.File and read it in in pieces (e.g. with
byLine or
byChunk).
2. Read the whole file in at once with std.file.read or
std.file.readText.
3. Use std.mmfile.MmFile to mmap the file so that the parts
that you access get read into memory, and the rest don't get
loaded from disk, but you get to operate on the whole file as
if you had read it all from disk.
- Jonathan M Davis
Thanks Jonathan. I don't quite see what I want to do though.
In order to abstract the file aspect of things away, and deal
with a stream of chars that could be from a file, or some some
other source would I use a range?
what I am after is a replacement for InputStream, I don't really
want to directly use File in most places.
So far I have this, and it seems to work, but it doesn't seem
right:
this(InputRange)(InputRange input)
{
int line_no = 1;
foreach (char[] line; input.byLine()) {
...
I have a used a template, because I cannot directly use the
InputRange(char) interface as a type, and auto won't work either,
so is there another parameter type I can use, such that I can
have the concept of an abstract stream of bytes.
Re: Deprecation: module std.stream is deprecated
On Saturday, November 07, 2015 13:52:26 Spacen Jasset via Digitalmars-d-learn
wrote:
> Thanks Jonathan. I don't quite see what I want to do though.
>
> In order to abstract the file aspect of things away, and deal
> with a stream of chars that could be from a file, or some some
> other source would I use a range?
>
> what I am after is a replacement for InputStream, I don't really
> want to directly use File in most places.
>
> So far I have this, and it seems to work, but it doesn't seem
> right:
>
> this(InputRange)(InputRange input)
> {
>
> int line_no = 1;
> foreach (char[] line; input.byLine()) {
> ...
>
> I have a used a template, because I cannot directly use the
> InputRange(char) interface as a type, and auto won't work either,
> so is there another parameter type I can use, such that I can
> have the concept of an abstract stream of bytes.
If your function accepts a range, then it's pretty much a guarantee that
it's going to be templated. std.range does define interfaces for the basic
types of ranges, but almost all ranges are structs, and what range types
typically look like when you start chaining them means that you pretty much
never want to type them out (or they're Voldemort types, in which case you
_can't_ type them out). typeof can be used to get around that, but it's
still ugly. So, you're not normally going to have non-templated code
accepting a range type which could come from who-knows-where. Rather, you'd
do something like
auto myFunc(R)(R range)
if(isInputRange!R && is(ElementType!R == dchar))
{
}
in which case the function will accept any range of dchar without caring
where it came from. But it does mean that the function is templated.
If you want a range of characters from a file, you have the three options
that I listed before.
1. You can use std.stdio.File. Nothing from it will give you a range of
characters directly (rather, it's going to give you a range of lines or
chunks), but you can get a range of charaters from that. e.g.
auto file = File("myfile.txt");
auto range = file.byLineCopy(KeepTerminator.yes).joiner();
The result is a range of dchar that can be passed to any function that
accepts a range of dchar, and those functions won't care about where those
characters came from - but they will need to be templated.
Note that you'd want byLineCopy, not byLine, since you're not using it in a
for loop, and byLine reuses its buffer, which doesn't usually work very well
with anything other than a for loop.
2. You can use std.file.read or std.file.readText. e.g.
string text = std.file.readText("myfile.txt");
This will read the entire file in at once (so if you were looking for
streams, this probably isn't what you want), but you do then have a string
that you can pass to any code that accepts a string or a range of dchar
without caring about where that string came from, and it does mean that you
can avoid templating your code and just have it accept string if that's what
you'd prefer.
3. You can use std.mmfile.MmFile to mmap the file. e.g.
auto mmFile = new MmFile("foo.txt");
auto text = cast(const(char)[])mmFile[];
This will only read the file into memory as you read the memory that now
refers to it, but you have an array to process like you'd get with readText.
So, it can be used as a range of dchar (though not a string, since the data
isn't actually immutable - even though the file is opened as read-only by
default - since another process could edit the file while you're reading
it).
https://en.wikipedia.org/wiki/Mmap
Whether mmap or byLineCopy would be faster probably depends on what you're
doing, whereas readText is the easiest to use but does require that you read
the whole file in at once.
- Jonathan M Davis
Re: Deprecation: module std.stream is deprecated
On Saturday, 7 November 2015 at 13:52:29 UTC, Spacen Jasset wrote:
[...]
I have a used a template, because I cannot directly use the
InputRange(char) interface as a type, and auto won't work
either, so is there another parameter type I can use, such that
I can have the concept of an abstract stream of bytes.
With the help of a template constraint you can abstract this. It
looks like your problem is more getting the whole range since
.byLine or .byChunk don't return the full stuff. Maybe use
std.alsorithm.joiner. As you noted you can carry the iterator
without knowing the type with auto, example:
---
import std.stdio;
import std.range, std.algorithm;
void main(string[] args)
{
auto inputRange = File(__FILE__).byChunk(1024).joiner;
Foo foo = Foo(inputRange);
}
struct Foo
{
this(T)(T t)
if (isInputRange!T && is(ElementType!T == ubyte))
{
foreach(byte b; t)
{
writef("0x%.2X ", b);
if (b == 0xA) writeln;
}
}
}
---
Re: Deprecation: module std.stream is deprecated
On Sunday, 8 November 2015 at 08:21:38 UTC, BBaz wrote:
On Saturday, 7 November 2015 at 13:52:29 UTC, Spacen Jasset
wrote:
[...]
I have a used a template, because I cannot directly use the
InputRange(char) interface as a type, and auto won't work
either, so is there another parameter type I can use, such
that I can have the concept of an abstract stream of bytes.
With the help of a template constraint you can abstract this.
It looks like your problem is more getting the whole range
since .byLine or .byChunk don't return the full stuff. Maybe
use std.alsorithm.joiner. As you noted you can carry the
iterator without knowing the type with auto, example:
---
import std.stdio;
import std.range, std.algorithm;
void main(string[] args)
{
auto inputRange = File(__FILE__).byChunk(1024).joiner;
Foo foo = Foo(inputRange);
}
struct Foo
{
this(T)(T t)
if (isInputRange!T && is(ElementType!T == ubyte))
{
foreach(byte b; t)
{
writef("0x%.2X ", b);
if (b == 0xA) writeln;
}
}
}
---
or even using an auto function:
---
import std.stdio;
import std.range, std.algorithm;
void main(string[] args)
{
auto inputRange0 = inputByteRange(File(__FILE__));
Foo foo0 = Foo(inputRange0);
ubyte[] arr = [1,2,3,4,5];
auto inputRange1 = inputByteRange(arr);
Foo foo1 = Foo(inputRange1);
}
auto inputByteRange(T)(auto ref T t)
{
static if (is(T == File))
return t.byChunk(1024).joiner;
else static if (is(T == ubyte[]))
return t;
else assert(0, "unsupported inputByteRange arg");
}
struct Foo
{
this(T)(T t)
if (isInputRange!T && is(ElementType!T == ubyte)){}
}
---
Re: Deprecation: module std.stream is deprecated
On Sunday, 8 November 2015 at 08:29:30 UTC, BBaz wrote:
On Sunday, 8 November 2015 at 08:21:38 UTC, BBaz wrote:
On Saturday, 7 November 2015 at 13:52:29 UTC, Spacen Jasset
wrote:
[...]
I have a used a template, because I cannot directly use the
InputRange(char) interface as a type, and auto won't work
either, so is there another parameter type I can use, such
that I can have the concept of an abstract stream of bytes.
With the help of a template constraint you can abstract this.
It looks like your problem is more getting the whole range
since .byLine or .byChunk don't return the full stuff. Maybe
use std.alsorithm.joiner. As you noted you can carry the
iterator without knowing the type with auto, example:
---
import std.stdio;
import std.range, std.algorithm;
void main(string[] args)
{
auto inputRange = File(__FILE__).byChunk(1024).joiner;
Foo foo = Foo(inputRange);
}
struct Foo
{
this(T)(T t)
if (isInputRange!T && is(ElementType!T == ubyte))
{
foreach(byte b; t)
{
writef("0x%.2X ", b);
if (b == 0xA) writeln;
}
}
}
---
or even using an auto function:
---
import std.stdio;
import std.range, std.algorithm;
void main(string[] args)
{
auto inputRange0 = inputByteRange(File(__FILE__));
Foo foo0 = Foo(inputRange0);
ubyte[] arr = [1,2,3,4,5];
auto inputRange1 = inputByteRange(arr);
Foo foo1 = Foo(inputRange1);
}
auto inputByteRange(T)(auto ref T t)
{
static if (is(T == File))
return t.byChunk(1024).joiner;
else static if (is(T == ubyte[]))
return t;
else assert(0, "unsupported inputByteRange arg");
}
struct Foo
{
this(T)(T t)
if (isInputRange!T && is(ElementType!T == ubyte)){}
}
---
Thank you for your detailed replies, I shall try them out when I
get a moment.
I would like to *winge* and say that it doesn't appear to be
straight forward, or at least not obvious and clean to get the
concept of an abstract steam working, which is to my mind a basic
thing.
This looks the simplest solution at the moment:
auto inputRange = File(__FILE__).byChunk(1024).joiner;
Foo foo = Foo(inputRange);
But it doesn't seem efficient and strays off the conceptual path.
In other words, why chunk things up, join them back, to get a
stream? Perhaps the problem is that File is missing a .range()
function?
Re: Deprecation: module std.stream is deprecated
On Sunday, 8 November 2015 at 14:41:11 UTC, Spacen Jasset wrote: But it doesn't seem efficient and strays off the conceptual path. In other words, why chunk things up, join them back, to get a stream? `.byChunk` caches and `.joiner` hides this caching mechanism. Both operations happen under the hood "incrementally" while using the final input range because of lazy evaluation, so if your file is big, you are assured that only slices of N bytes (1024 in my example) will be loaded at once in the DRAM (unless you accumulate the whole thing later). This matches well to a "file stream concept", at least to read. But as said in Jonathan M Davis's answer you can also read the whole file in a string or a ubyte[]. Perhaps the problem is that File is missing a .range() function? Yes but this is a bit like this that phobos ranges and algorithms work. You have many independant low-level blocks with which you can compose rather than big classes that wrap everything. The whole standard library is organized around this. std.stream was not compliant with this system and this is why "they" deprecated it (at least this is how I understood this).
Re: Deprecation: module std.stream is deprecated
On Sunday, November 08, 2015 14:41:05 Spacen Jasset via Digitalmars-d-learn wrote: > This looks the simplest solution at the moment: > > >> auto inputRange = File(__FILE__).byChunk(1024).joiner; > >> Foo foo = Foo(inputRange); > > But it doesn't seem efficient and strays off the conceptual path. > In other words, why chunk things up, join them back, to get a > stream? Perhaps the problem is that File is missing a .range() > function? It's actually doing what any "stream" based solution would have to do anyway. If you're not reading the entire file in at once, then you're going to be reading it in in chunks, even if that's wrapped up in some kind of Stream class/struct. std.stdio.File exposes the basic operations and higher level stuff can be done on top of them. Maybe std.stdio.File should have some sort of function on it to just give you a range of bytes or characters, but it's pretty trivial to use joiner to get that, and it would definitely be worse to _not_ provide functions like byLine or byChunk. Chaining functions like this is an incredibly common thing to do with ranges. Range-based functions tend to be treated like building blocks rather than kitchen sinks. And most range-based functions are lazy, so something like auto range = file.byChunk(1024).joiner(); is actually efficient, and if we did have a function on std.stdio.File to just return a range of bytes or characters, it would almost certainly just be wrapping something like this, since it is efficient, and it wouldn't make sense to implement that functionality again rather than just use the functions that we already have. - Jonathan M Davis
Re: Deprecation: module std.stream is deprecated
On Tuesday, 10 November 2015 at 11:57:03 UTC, Jonathan M Davis
wrote:
...
building blocks rather than kitchen sinks. And most range-based
...
I still can't really see why byChunk and byLine are part of the
File API? Especially byLine, I'll get byChunk go for now.
I think what I am trying to say is that there doesn't need to be
a byLine, if lineSplitter could work with some range that comes
out of the File.
The problem with your example using joiner is that:
(a) it is external to the calling function, and therefore messy,
since it will have to be used everywhere.
// not ideal
printFile(f.byChunk(1000).joiner());
(b) still can't use the range to split by lines afterwards,
lineSplitter etc don't work.
void printFile(Range)(Range i)
{
foreach (l; lineSplitter(i)) {
writefln("%d\n", l);
}
}
void main()
{
File f = File("/etc/passwd");
printFile(f.byChunk(1000).joiner()); // Not work,
linesplitter needs slicing and length?
}
