Hi Larry, I don’t think current Spark’s shuffle can support HDFS as a shuffle output. Anyway, is there any specific reason to spill shuffle data to HDFS or NFS, this will severely increase the shuffle time.
Thanks Jerry From: Larry Liu [mailto:[email protected]] Sent: Sunday, January 25, 2015 4:45 PM To: [email protected] Subject: Shuffle to HDFS How to change shuffle output to HDFS or NFS?
