Shuffle file not found Exception

[Guillaume Pitel]

Hi, I've got a strange problem with 0.8.1 (we're going to make the jump to 0.9.0 in a few days, but for now I'm woring with a 0.8.1 cluster) : After a few iteration of my method, one random node of my local cluster throws an exception like that : FileNotFoundException: /sparktmp/spark-local-20140209073949-29b1/37/merged_shuffle_24_23_1 (No such file or directory) Then, either the job get stuck for hours, or it fails right away. I've got the ulimit at 131k files, and consolidateFiles=true, so I don't think it a problem related to the # of file descriptors Guillaume -- Guillaume PITEL, Président +33(0)6 25 48 86 80 / +33(0)9 70 44 67 53 eXenSa S.A.S. 41, rue Périer - 92120 Montrouge - FRANCE Tel +33(0)1 84 16 36 77 / Fax +33(0)9 72 28 37 05