On Tuesday, 12 May 2015 at 16:46:42 UTC, thedeemon wrote:
On Tuesday, 12 May 2015 at 14:59:38 UTC, Gerald Jansen wrote:
The output of /usr/bin/time is as follows:
Lang Jobs User System Elapsed %CPU
Py 2 79.24 2.16 0:48.90 166
D 2 19.41 10.14 0:17.96 164
Py 30 1255.17 58.38 2:39.54 823 * Pool(12)
D 30 421.61 4565.97 6:33.73 1241
The fact that most of the time is spent in System department is
quite important. I suspect there are too many system calls from
line-wise reading and writing the files. How many lines are
read and written there?
About 3.5 million lines read by main(), 0.5 to 2 million lines
read and 3.5 million lines written by runTraits (aka runJob).
I have smaller datasets that I test on my laptop with a single
quad-core I7 which sometimes show little increase in System time
and other times have a marked increase, but not nearly as
exagerated as in the large datasets on the server.
Gerald
