Re: Using lazy code to process large files

2017-08-02 Thread Steven Schveighoffer via Digitalmars-d-learn
On 8/2/17 2:28 PM, kdevel wrote: On Wednesday, 2 August 2017 at 17:37:09 UTC, Steven Schveighoffer wrote: What is expected? What I see on the screen when I run my code is: [Ü] Upper case? Sorry, should be c3 bc, not c3 9c. I misread the table on that wikipedia entry. What I see when I

Re: Using lazy code to process large files

2017-08-02 Thread ag0aep6g via Digitalmars-d-learn
On 08/02/2017 08:28 PM, kdevel wrote: It's perfectly okay to put any value a octet can take into an octet. I did not claim that the data in the string memory is syntactically valid UTF-8. Read the comment in line 9 of my post of 15:02:22. You're claiming that the data is in UTF-8 when you use

Re: Using lazy code to process large files

2017-08-02 Thread kdevel via Digitalmars-d-learn
On Wednesday, 2 August 2017 at 17:37:09 UTC, Steven Schveighoffer wrote: What is expected? What I see on the screen when I run my code is: [Ü] Upper case? What I see when I run your "working" code is: [?] Your terminal is incapable of rendering the Latin-1 encoding. The program prints

Re: Using lazy code to process large files

2017-08-02 Thread Steven Schveighoffer via Digitalmars-d-learn
On 8/2/17 1:16 PM, kdevel wrote: On Wednesday, 2 August 2017 at 15:52:13 UTC, Steven Schveighoffer wrote: If we use the correct code unit sequence (0xc3 0x9c), then [...] If I avoid std.string.stripLeft and use std.algorithm.stripLeft(' ') instead it works as expected: What is expected? Wh

Re: Using lazy code to process large files

2017-08-02 Thread kdevel via Digitalmars-d-learn
On Wednesday, 2 August 2017 at 15:52:13 UTC, Steven Schveighoffer wrote: [...] First, as a tip, please post either a link to a paste site, or don't put the line numbers. It's much easier to copy-paste your code into an editor if you don't have the line numbers. With pleasure. [...] If we

Re: Using lazy code to process large files

2017-08-02 Thread Steven Schveighoffer via Digitalmars-d-learn
On 8/2/17 11:02 AM, kdevel wrote: On Wednesday, 2 August 2017 at 13:45:01 UTC, Steven Schveighoffer wrote: As Daniel said, using byCodeUnit will help. stripLeft seems to autodecode even when fed with CodeUnits. How do I prevent this? 1 void main () 2 { 3import std.s

Re: Using lazy code to process large files

2017-08-02 Thread kdevel via Digitalmars-d-learn
On Wednesday, 2 August 2017 at 13:45:01 UTC, Steven Schveighoffer wrote: As Daniel said, using byCodeUnit will help. stripLeft seems to autodecode even when fed with CodeUnits. How do I prevent this? 1 void main () 2 { 3import std.stdio; 4import std.string;

Re: Using lazy code to process large files

2017-08-02 Thread Steven Schveighoffer via Digitalmars-d-learn
On 8/2/17 8:59 AM, Martin Drašar via Digitalmars-d-learn wrote: Thanks Steven for the explanation. Just to clarify - what would be needed to avoid auto-decoding in this case? Process it all as an arrays, using byChunk to read it, etc? As Daniel said, using byCodeUnit will help. I don't know

Re: Using lazy code to process large files

2017-08-02 Thread Daniel Kozak via Digitalmars-d-learn
something like file.byLine.map!(a=>a.byCodeUnit) On Wed, Aug 2, 2017 at 3:01 PM, Daniel Kozak wrote: > using http://dlang.org/phobos/std_utf.html#byCodeUnit could help > > On Wed, Aug 2, 2017 at 2:59 PM, Martin Drašar via Digitalmars-d-learn < > digitalmars-d-learn@puremagic.com> wrote: > >> Dne

Re: Using lazy code to process large files

2017-08-02 Thread Daniel Kozak via Digitalmars-d-learn
using http://dlang.org/phobos/std_utf.html#byCodeUnit could help On Wed, Aug 2, 2017 at 2:59 PM, Martin Drašar via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote: > Dne 2.8.2017 v 14:45 Steven Schveighoffer via Digitalmars-d-learn > napsal(a): > > > The problem is that you are 2 r

Re: Using lazy code to process large files

2017-08-02 Thread Martin Drašar via Digitalmars-d-learn
Dne 2.8.2017 v 14:45 Steven Schveighoffer via Digitalmars-d-learn napsal(a): > The problem is that you are 2 ranges deep when you apply splitter. The > result of the map is a range of ranges. > > Then when you apply stringStripleft, you are applying to the map result, > not the splitter result. >

Re: Using lazy code to process large files

2017-08-02 Thread kdevel via Digitalmars-d-learn
On Wednesday, 2 August 2017 at 11:44:30 UTC, Martin Drašar wrote: Thank you for any hint. 1 import std.stdio; 2 import std.string; 3 import std.algorithm; 4 import std.conv; 5 6 void main () 7 { 8auto input = File("input.csv"); 9 1

Re: Using lazy code to process large files

2017-08-02 Thread Steven Schveighoffer via Digitalmars-d-learn
On 8/2/17 7:44 AM, Martin Drašar via Digitalmars-d-learn wrote: Hi, I am struggling to use a lazy range-based code to process large text files. My task is simple, i.e., I can write a non-range-based code in a really short time, but I wanted to try different approach and I am hitting a wall after

Re: Using lazy code to process large files

2017-08-02 Thread Martin Drašar via Digitalmars-d-learn
Dne 2.8.2017 v 14:11 Daniel Kozak via Digitalmars-d-learn napsal(a): > import std.stdio; > import std.algorithm; > > void main() > { > > > auto input = ["... some text, another text", "some,another","...so,an"]; > > auto result = input.filter!(a => a.startsWith("...")) > .map!(a=>a.s

Re: Using lazy code to process large files

2017-08-02 Thread Daniel Kozak via Digitalmars-d-learn
import std.stdio; import std.algorithm; void main() { auto input = ["... some text, another text", "some,another","...so,an"]; auto result = input.filter!(a => a.startsWith("...")) .map!(a=>a.splitter(",").map!(a=>a.stripLeft(' '))) .map!(a=>a.joiner(",")); writeln(result); } On

Using lazy code to process large files

2017-08-02 Thread Martin Drašar via Digitalmars-d-learn
Hi, I am struggling to use a lazy range-based code to process large text files. My task is simple, i.e., I can write a non-range-based code in a really short time, but I wanted to try different approach and I am hitting a wall after wall. Task: read a csv-like input, take only lines starting with