Hi Everyone,
I got some some undesired results when running some benchmarks with
SystemML, and I really hope to get some helps from you guys.
Say, I want to read a large matrix in csv and retrieve the first cell with
script
# simple_read.dml
data = read($RS)
print("One Cell data[1,1]: " + toString(data[1,1]))
and run it with the configuration such as
spark-submit \
--driver-memory 20G \
--executor-memory 100G \
--executor-cores 20 \
--num-executors 1 \
--conf spark.driver.maxResultSize=0 \
--conf spark.rpc.message.maxSize=128 \
$SYSTEMML_HOME/target/SystemML.jar \
-f simple_read.dml \
-stats \
-explain \
-nvargs
RS=/oasis/scratch/comet/hyoung/temp_project/scatter_data/TR20_FR4.00/RS_join.csv
When the input matrix is 2x10^7 by 103 (around 20GB), things worked fine,
as sp_rangeReIndex took 19.725s and sp_csvrblk took 11.330s. But when the
input matrix is reduced to 10^7 by 103 (around 10GB), interesting things
happened, as rangeReIndex took 92.024s and sp_csvrblk took 7.774s.
These results were obtained with SystemML v0.13.0 on Spark 2.1.0 in
standalone mode with 128GB memory and 24 cores.
>From the log, it seems that for the latter case, the control program in the
driver took the main job and led to lots of disk I/O, thus the whole
program had been slowed down.
I understand that assigning the control program some tasks is a key
component in SystemML. But this feature really brings in some chaos to the
benchmarks in my case. Any suggestion about how to choose a better
configuration or make some detours so I can obtain fair benchmarks on a
wide range of data dimensions?
If needed, I can attach the logs.
I really appreciate your help!
Regards,
Mingyang Wang
Graduate Student in UCSD CSE Dept.
