Re: spark structured streaming jobs working in HDP2.6 fail in HDP3.0

I solved this issue by using spark 2.3.1 jars copied from the HDP3.0 cluster. Thanks. On Thu, Aug 30, 2018 at 10:18 AM Lian Jiang wrote: > Per https://spark.apache.org/docs/latest/building-spark.html, spark 2.3.1 > is built with hadoop 2.6.X by default. This is why I see my fat jar > includes

Re: spark structured streaming jobs working in HDP2.6 fail in HDP3.0

Per https://spark.apache.org/docs/latest/building-spark.html, spark 2.3.1 is built with hadoop 2.6.X by default. This is why I see my fat jar includes hadoop 2.6.5 (instead of 3.1.0) jars. HftpFileSystem has been removed in hadoop 3. On https://spark.apache.org/downloads.html, I only see spark

spark structured streaming jobs working in HDP2.6 fail in HDP3.0

Hi, I am using HDP3.0 which uses HADOOP3.1.0 and Spark 2.3.1. My spark streaming jobs running fine in HDP2.6.4 (HADOOP2.7.3, spark 2.2.0) fails in HDP3: java.lang.IllegalAccessError: class org.apache.hadoop.hdfs.web.HftpFileSystem cannot access its superinterface