On Thursday, 14 May 2015 at 10:46:53 UTC, Gerald Jansen wrote:
John Colvin's improvements to my D program seem to have
resolved the problem.
(http://forum.dlang.org/post/ydgmzhlspvvvrbeem...@forum.dlang.org
and http://dpaste.dzfl.pl/114d5a6086b7).
I have rerun my tests and now the picture is a bit different
(see tables below).
In the middle table I have used gnu parallel in combination
with a slightly modified version of the D program which runs a
single trait (specified in argv[1]). This approach runs the
jobs as completely isolated processes, but at the extra cost of
re-reading the common data for each trait. The elapsed time is
very similar with the parallel foreach in the D program or
using gnu parallel (for this particular program and these data
run on this server...). I'm guessing the program is now
essentially limited by disk I/O, so this is about as good as it
gets.
So, just to wrap up:
- there is a nice speed improvement over Python program :-)
- one needs to learn a fair bit to fully benefit from D's
potential
- thanks for all the help!
Gerald Jansen
Jobs __ time for D parallel foreach w. JC mods____
1 4.71user 0.56system 0:05.28elapsed 99%CPU
2 6.59user 0.96system 0:05.48elapsed 137%CPU
4 11.45user 1.94system 0:07.24elapsed 184%CPU
8 20.30user 5.18system 0:13.16elapsed 193%CPU
16 68.48user 13.87system 0:27.21elapsed 302%CPU
27 99.66user 18.73system 0:42.34elapsed 279%CPU
Jobs __ gnu parallel + D program for single job __
1 4.71user 0.56system 0:05.28elapsed 99%CPU as above
2 9.66user 1.28system 0:05.76elapsed 189%CPU
4 18.86user 3.85system 0:08.15elapsed 278%CPU
8 40.76user 7.53system 0:14.69elapsed 328%CPU
16 135.76user 20.68system 0:31.06elapsed 503%CPU
27 189.43user 28.26system 0:47.75elapsed 455%CPU
Jobs _____ time for python version _____________
1 45.39user 1.52system 0:46.88elapsed 100%CPU
2 77.76user 2.42system 0:47.16elapsed 170%CPU
4 141.28user 4.37system 0:48.77elapsed 298%CPU
8 280.45user 8.80system 0:56.00elapsed 516%CPU
16 926.05user 20.48system 1:31.36elapsed 1036%CPU
27 1329.09user 27.18system 2:11.79elapsed 1029%CPU
Would it be OK if I showed some parts of this code as examples in
my DConf talk in 2 weeks?
