Although you need to compile it differently for different versions of HDFS / Hadoop, as far as I know Spark continues to work with Hadoop 1.x (and probably older 0.20.x as a result -- your experience is an existence proof.) And it works with the newest Hadoop 2.4.x, again with the appropriate build settings.
I think the default answer is to upgrade all the way to the newest Hadoop / HDFS unless you have a reason you can't. On Wed, May 14, 2014 at 8:17 PM, Soumya Simanta <[email protected]> wrote: > Currently I've HDFS with version hadoop0.20.2-cdh3u6 on Spark 0.9.1. I want > to upgrade to Spark 1.0.0 soon and would also like to upgrade my HDFS > version as well. > > What's the recommended version of HDFS to use with Spark 1.0.0? I don't know > much about YARN but I would just like to use the Spark standalone cluster > mode. > > Thanks > -Soumya >
