Shaul Lahav created SPARK-24483:
-----------------------------------
Summary: enableHiveSupport doesn't work with Spark 2.3 on EMR
Key: SPARK-24483
URL: https://issues.apache.org/jira/browse/SPARK-24483
Project: Spark
Issue Type: Bug
Components: Project Infra
Affects Versions: 2.3.0
Environment: EMR v5.13 (Spark 2.3.0)
Reporter: Shaul Lahav
I run a spark job on an EMR cluster when using the SparkSession.Builder to
create a spark Session.
The job includes querying a Hive table, and when it is executed Spark throws "
org.apache.spark.sql.AnalysisException: Table or view not found".
I printed all the config options in Spark context and noticed that
"spark.sql.catalogImplementation" is missing.
If I provide this option as a "-conf" in "spark-submit" then it is added and
everything works.
This is the code I use to instantiate the SparkSession:
val spark: SparkSession = SparkSession
.builder
.config("mapreduce.fileoutputcommitter.algorithm.version", "2")
.enableHiveSupport()
.getOrCreate()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
