And Hadoop-3.x is not part of the release and sign off for 2.2.1. Maybe we could update the website to avoid any confusion with "later".
________________________________ From: Josh Rosen <joshro...@databricks.com> Sent: Monday, January 8, 2018 10:17:14 AM To: akshay naidu Cc: Saisai Shao; Raj Adyanthaya; spark users Subject: Re: Is Apache Spark-2.2.1 compatible with Hadoop-3.0.0 My current best guess is that Spark does not fully support Hadoop 3.x because https://issues.apache.org/jira/browse/SPARK-18673 (updates to Hive shims for Hadoop 3.x) has not been resolved. There are also likely to be transitive dependency conflicts which will need to be resolved. On Mon, Jan 8, 2018 at 8:52 AM akshay naidu <akshaynaid...@gmail.com<mailto:akshaynaid...@gmail.com>> wrote: yes , spark download page does mention that 2.2.1 is for 'hadoop-2.7 and later', but my confusion is because spark was released on 1st dec and hadoop-3 stable version released on 13th Dec. And to my similar question on stackoverflow.com<https://stackoverflow.com/questions/47920005/how-is-hadoop-3-0-0-s-compatibility-with-older-versions-of-hive-pig-sqoop-and> , Mr. jacek-laskowski<https://stackoverflow.com/users/1305344/jacek-laskowski> replied that spark-2.2.1 doesn't support hadoop-3. so I am just looking for more clarity on this doubt before moving on to upgrades. Thanks all for help. Akshay. On Mon, Jan 8, 2018 at 8:47 AM, Saisai Shao <sai.sai.s...@gmail.com<mailto:sai.sai.s...@gmail.com>> wrote: AFAIK, there's no large scale test for Hadoop 3.0 in the community. So it is not clear whether it is supported or not (or has some issues). I think in the download page "Pre-Built for Apache Hadoop 2.7 and later" mostly means that it supports Hadoop 2.7+ (2.8...), but not 3.0 (IIUC). Thanks Jerry 2018-01-08 4:50 GMT+08:00 Raj Adyanthaya <raj...@gmail.com<mailto:raj...@gmail.com>>: Hi Akshay On the Spark Download page when you select Spark 2.2.1 it gives you an option to select package type. In that, there is an option to select "Pre-Built for Apache Hadoop 2.7 and later". I am assuming it means that it does support Hadoop 3.0. http://spark.apache.org/downloads.html Thanks, Raj A. On Sat, Jan 6, 2018 at 8:23 PM, akshay naidu <akshaynaid...@gmail.com<mailto:akshaynaid...@gmail.com>> wrote: hello Users, I need to know whether we can run latest spark on latest hadoop version i.e., spark-2.2.1 released on 1st dec and hadoop-3.0.0 released on 13th dec. thanks.
