Re: The performance of group operation on SSD

[Guillaume Pitel]

Hi, I think intermediate results are stored in spark.local.dir (/tmp by default), not in hdfs. So if you see big latency in a shuffle-heavy operation, it's probably due to the device where /tmp reside. You can run iotop on your slaves to check where the operations are occuring. Check your wait% in top, to see if you have I/O problems You can activate the consolidateFiles option if you use ext4, it helps when dealing with big intermediate results Guillaume EC2 doesn't mount SSDs automatically, so I actually reformat those as ex4, mount them and point hdfs to them. I haven't tweaked any FS options and not sure what cause the problem. I will follow Christopher's suggestion and see whether I can find any cause. -chen -- Guillaume PITEL, Président +33(0)6 25 48 86 80 eXenSa S.A.S. 41, rue Périer - 92120 Montrouge - FRANCE Tel +33(0)1 84 16 36 77 / Fax +33(0)9 72 28 37 05