Greetings,
Our standalone Spark 3 cluster is trying to connect to Hadoop 2.6 cluster
running Hive server 1.2
(/usr/hdp/2.6.2.0-205/hive/lib/hive-service-1.2.1000.2.6.2.0-205.jar)
import org.apache.spark.sql.functions._
import java.sql.Timestamp
val df1 = spark.createDataFrame(
Seq(
("id1", "v2", "notshared", Timestamp.valueOf("2019-09-13
10:00:00"), false, 1, "2019-09-13"),
("id2", "v3", "notshared", Timestamp.valueOf("2019-09-13
09:00:00"), false, 2, "2019-09-13"),
("id2", "v4", "notshared", Timestamp.valueOf("2019-09-14
11:00:00"), false, 3, "2019-09-14"),
("id2", "v5", "notshared", Timestamp.valueOf("2019-09-14
13:00:00"), false, 4, "2019-09-14"),
("id3", "v4", "notshared", Timestamp.valueOf("2019-09-14
17:00:00"), false, 5, "2019-09-14"),
("id4", "v1", "notshared", Timestamp.valueOf("2019-09-15
19:00:00"), false, 6, "2019-09-15"))).toDF("user_id", "col2", "pidd",
"land_ts", "deleted","offset", "partition")
df1.write.mode("overwrite").saveAsTable("db.spark3_test")
when running above code, is throws the error :
org.apache.spark.sql.AnalysisException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table
> spark_27686. Invalid method name: 'get_table_req';
> at
> org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:110)
> at
I assume this is caused because Spark 3 ships "hive-metastore-2.3.7.jar".
To work with Hive Server 1.2 can I use "hive-metastore-1.2.1.spark2.jar"
from Spark 2.4 distribution ? Do I need any other dependencies ?
