Is that the only kind of error you are getting. Is it possible something else dies that gets buried in other messages. Try repairing HDFS (fsck etc) to find if blocks are intact.
Few things to check 1) if you have too many small files. 2) Is your system complaining about too many inode etc.. 3) Try smaller set while increasing the data set size to make sure it is data volume related problem. 4) If you have monitoring turned on see what your driver, worker machines cpu and disk io. 5) Have you tried increasing Driver memory(more partitions means driver needs more memory to keep the metadata) ..Manas -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Help-Get-Timeout-error-and-FileNotFoundException-when-shuffling-large-files-tp25662p25675.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
