On Fri, Jun 9, 2017 at 9:34 AM, uncorroded via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote:
> Hi guys, > > I am a beginner in D. As a project, I converted a log-parsing script in > Python which we use at work, to D. This link was helpful - ( > https://dlang.org/blog/2017/05/24/faster-command-line-tools-in-d/ ) I > compiled it with dmd and ldc. The log file is 52 MB. With dmd (not release > build), it takes 1.1 sec and with ldc, it takes 0.3 sec. > > The Python script (run with system python, not Pypy) takes 0.75 sec. The D > and Python functions are here and on pastebin ( D - > https://pastebin.com/SeUR3wFP , Python - https://pastebin.com/F5JbfBmE ). > > Basically, i am reading a line, checking for 2 constants. If either one is > found, some processing is done on line and stored to an array for later > analysis. I tried reading the file entirely in one go using std.file : > readText and using std.algorithm : splitter for lazily splitting newline > but there is no difference in speed, so I used the byLine approach > mentioned in the linked blog. Is there a better way of doing this in D? > > There is no difference in speed because you do not process your data lazily, so you make many allocations, so this is main reason why it is so slow. I could improve that, but I will need to see some example data, which you are trying to parse. But some rules, 1.) instead of ~= you shoud use std.array.appender 2.) instead of std.string.split you could use std.algorithm.splitter or std.algorithm.findSplit 3.) instead of indexOf I would use std.algorithm.startsWith (in case it is on the begining of the line)
