Re: std.range.byLine
On Wednesday, 10 September 2014 at 21:06:30 UTC, Nordlöw wrote: This is such a common feature so I believe it should have its place in std.range. Or some other Phobos module.
Re: std.range.byLine
On 09/10/2014 02:06 PM, "Nordlöw" wrote:
I'm missing a range variant of byLine that can operate on strings
instead of just File.
This is such a common feature so I believe it should have its place in
std.range.
My suggestion is to define this using
splitter!(std.uni.isNewline)
but I'm missing std.uni.isNewline.
I'm guessing the problem here is that newline separators can be 1 or 2
bytes long. that is it Separator must be of the same time as Range.
Should I add an overload in PR?
Destroy.
There is std.ascii.newline. The following works where newline is '\n'
e.g. on my Linux system. :)
import std.ascii;
import std.algorithm;
import std.range;
void main()
{
assert("foo\nbar\n"
.splitter(newline)
.filter!(a => !a.empty)
.equal([ "foo", "bar" ]));
}
Ali
Re: std.range.byLine
On Wednesday, 10 September 2014 at 22:29:55 UTC, Ali Çehreli
wrote:
assert("foo\nbar\n"
.splitter(newline)
.filter!(a => !a.empty)
.equal([ "foo", "bar" ]));
}
Ali
Ok, great.
So I got.
auto byLine(Range)(Range input) if (isForwardRange!Range)
{
import std.algorithm: splitter;
import std.ascii: newline;
static if (newline.length == 1)
{
return input.splitter(newline.front);
}
else
{
return input.splitter(newline);
}
}
unittest
{
import std.algorithm: equal;
assert(equal("a\nb".byLine, ["a", "b"]));
}
One thing still:
Is my optimization for newline.length == 1 unnecessary or perhaps
even wrong?
Re: std.range.byLine
On Wednesday, 10 September 2014 at 22:45:08 UTC, Nordlöw wrote:
auto byLine(Range)(Range input) if (isForwardRange!Range)
{
import std.algorithm: splitter;
import std.ascii: newline;
static if (newline.length == 1)
{
return input.splitter(newline.front);
}
else
{
return input.splitter(newline);
}
}
IMHO, this should be added to std.string and restricted to
isSomeString. Should I do a PR?
Re: std.range.byLine
On Wednesday, 10 September 2014 at 23:01:44 UTC, Nordlöw wrote:
On Wednesday, 10 September 2014 at 22:45:08 UTC, Nordlöw wrote:
auto byLine(Range)(Range input) if (isForwardRange!Range)
{
import std.algorithm: splitter;
import std.ascii: newline;
static if (newline.length == 1)
{
return input.splitter(newline.front);
}
else
{
return input.splitter(newline);
}
}
IMHO, this should be added to std.string and restricted to
isSomeString. Should I do a PR?
Well, the issue is that this isn't very portable for *reading*,
as even on linux, you may read files with "\r\n" line endings
(It's "standard" for csv files, for example), or read "\n"
terminated files on windows.
The issue is that (currently) we don't have any splitter that
operates on multiple needles. *That'd* be what needs to be
written (probably not too hard either, since "find" already
exists).
We also have splitLines,
"http://dlang.org/phobos/std_string.html#.splitLines";. Is that
good enough for you by any chance? Or do you need it to actually
be lazy?
Re: std.range.byLine
On Thursday, 11 September 2014 at 10:19:17 UTC, monarch_dodra wrote: Well, the issue is that this isn't very portable for *reading*, as even on linux, you may read files with "\r\n" line endings (It's "standard" for csv files, for example), or read "\n" terminated files on windows. The issue is that (currently) we don't have any splitter that operates on multiple needles. *That'd* be what needs to be written (probably not too hard either, since "find" already exists). Good idea. So its "just" a matter of extending splitter with std.algorithm.find with these three keys: - \n - \r - \r\n then? Or are there more encodings to choose from? We also have splitLines, "http://dlang.org/phobos/std_string.html#.splitLines";. Is that good enough for you by any chance? Or do you need it to actually be lazy? Lazyness is good in this case because my input files are Gigabytes in size :) I'm playing around with single-pass-parsing ConceptNet5 CSV-files at https://github.com/nordlow/justd/blob/master/conceptnet5.d
Re: std.range.byLine
On Thursday, 11 September 2014 at 20:03:26 UTC, Nordlöw wrote: On Thursday, 11 September 2014 at 10:19:17 UTC, monarch_dodra wrote: Well, the issue is that this isn't very portable for *reading*, as even on linux, you may read files with "\r\n" line endings (It's "standard" for csv files, for example), or read "\n" terminated files on windows. The issue is that (currently) we don't have any splitter that operates on multiple needles. *That'd* be what needs to be written (probably not too hard either, since "find" already exists). Good idea. So its "just" a matter of extending splitter with std.algorithm.find with these three keys: - \n - \r - \r\n then? Or are there more encodings to choose from? Hum... no, those are the correct splitting elements. However, I don't think that would actually work, as "find" will privilege the first whole element to match as a "hit", so "\r\n" never be hit (rather, it will be hit twice, in the form of two individual line breaks `\r` and '\n'). Bummer...
Re: std.range.byLine
On Thursday, 11 September 2014 at 21:29:16 UTC, monarch_dodra wrote: Bummer... Anyway, it shouldn't be too hard to express this in a new range.
Re: std.range.byLine
On Thursday, 11 September 2014 at 21:29:16 UTC, monarch_dodra wrote: Hum... no, those are the correct splitting elements. However, I don't think that would actually work, as "find" will privilege the first whole element to match as a "hit", so "\r\n" never be hit (rather, it will be hit twice, in the form of two individual line breaks `\r` and '\n'). Bummer... So why not simply change the order of the keys to - \r\n - \r - \n then?
Re: std.range.byLine
On Thursday, 11 September 2014 at 21:54:39 UTC, Nordlöw wrote: Anyway, it shouldn't be too hard to express this in a new range. I guess what we need is a variant of splitter with a more greedy alias template parameter that will digest two or one bytes.
Re: std.range.byLine
On Thu, Sep 11, 2014 at 10:31:33PM +, "Nordlöw" via Digitalmars-d-learn
wrote:
> On Thursday, 11 September 2014 at 21:54:39 UTC, Nordlöw wrote:
> >Anyway, it shouldn't be too hard to express this in a new range.
>
> I guess what we need is a variant of splitter with a more greedy alias
> template parameter that will digest two or one bytes.
Why not just use std.regex?
foreach (line; myInput.split(regex(`\n|\r\n|\r`)))
{
...
}
T
--
The problem with the world is that everybody else is stupid.
Re: std.range.byLine
On Thursday, 11 September 2014 at 22:39:40 UTC, H. S. Teoh via
Digitalmars-d-learn > Why not just use std.regex?
foreach (line; myInput.split(regex(`\n|\r\n|\r`)))
{
...
}
T
I'll try the lazy variant of std.regex
foreach (line; myInput.splitter(regex(`\n|\r\n|\r`)))
{
...
}
I wonder if this is compatible with a ctRegex aswell. I'll try
later.
See also: http://dlang.org/phobos/std_regex.html#.splitter
Thx
Re: std.range.byLine
On Thursday, 11 September 2014 at 22:39:40 UTC, H. S. Teoh via
Digitalmars-d-learn wrote:
foreach (line; myInput.split(regex(`\n|\r\n|\r`)))
Shouldn't you use
foreach (line; myInput.split(regex("\n|\r\n|\r")))
here?
Re: std.range.byLine
On Friday, 12 September 2014 at 13:25:22 UTC, Nordlöw wrote:
On Thursday, 11 September 2014 at 22:39:40 UTC, H. S. Teoh via
Digitalmars-d-learn wrote:
foreach (line; myInput.split(regex(`\n|\r\n|\r`)))
Shouldn't you use
foreach (line; myInput.split(regex("\n|\r\n|\r")))
here?
Probably not, as (AFAIK) the splitter engine *itself* will *also*
escape the passed in characters. IE: It literally needs the
characters '\' and 'n'.
Re: std.range.byLine
On Friday, 12 September 2014 at 14:16:07 UTC, monarch_dodra wrote: Probably not, as (AFAIK) the splitter engine *itself* will *also* escape the passed in characters. IE: It literally needs the characters '\' and 'n'. I ended up with this. https://github.com/nordlow/justd/blob/30806a85a5c976f3e891ca11bde3d87a16ecf5e6/algorithm_ex.d#L1858 Does it seem ok?
