Re: D is for Data Science

[Dmitry Olshansky via Digitalmars-d-announce]

25-Nov-2014 01:28, bearophile пишет:
Dmitry Olshansky:
Why is File.byLine so slow?

Seems to be mostly fixed sometime ago.

Really? I am not so sure.

Bye,
bearophile

I too has suspected it in the past and then I tested it.
Now I test it again, it's always easier to check then to argue.

Two minimal programs
//my.d:
import std.stdio;

void main(string[] args) {
    auto file = File(args[1], "r");
    size_t cnt=0;
    foreach(char[] line; file.byLine()) {
        cnt++;
    }
}
//my2.d
import core.stdc.stdio;

void main(string[] args) {
    char[] buf = new char[32768];
    size_t cnt;
    shared(FILE)* file = fopen(args[1].ptr, "r");
    while(fgets(buf.ptr, cast(int)buf.length, file) != null){
        cnt++;
    }
    fclose(file);
}

In the below console session, log file - is my dmsg log replicated many 
times (34 megs total).

dmitry@Ubu64 ~ $ wc -l log
522240 log
dmitry@Ubu64 ~ $ du -hs log
34M     log

# touch it, to have it in disk cache:
dmitry@Ubu64 ~ $ cat log > /dev/null

dmitry@Ubu64 ~ $ dmd my
dmitry@Ubu64 ~ $ dmd my2

dmitry@Ubu64 ~ $ time ./my2 log

real    0m0.062s
user    0m0.039s
sys     0m0.023s
dmitry@Ubu64 ~ $ time ./my log

real    0m0.181s
user    0m0.155s
sys     0m0.025s

~4 time in user mode, okay...
Now with full optimizations, ranges are very sensitive to optimizations:

dmitry@Ubu64 ~ $ dmd -O -release -inline  my
dmitry@Ubu64 ~ $ dmd -O -release -inline  my2
dmitry@Ubu64 ~ $ time ./my2 log

real    0m0.065s
user    0m0.042s
sys     0m0.023s
dmitry@Ubu64 ~ $ time ./my2 log

real    0m0.063s
user    0m0.040s
sys     0m0.023s

Which is 1:1 parity. Another myth busted? ;)

--
Dmitry Olshansky