fengtlyer created SPARK-27826:
---------------------------------
Summary: saveAsTable() function case table have "HiveFileFormat"
"ParquetFileFormat" format issue
Key: SPARK-27826
URL: https://issues.apache.org/jira/browse/SPARK-27826
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.4.0, 2.2.0
Environment: CDH 5.13.1 - Spark version 2.2.0.cloudera2
CDH 6.1.1 - Spark version 2.4.0-cdh6.1.1
Reporter: fengtlyer
Hi Spark Dev Team,
We tested a few times and found this bug can reappearance in multi Spark version
We tested in CDH 5.13.1 - Spark version 2.2.0.cloudera2 and CDH 6.1.1 - Spark
version 2.4.0-cdh6.1.1
Both of them have this bug:
1. If one table created by Impala or Hive in the HUE, then in Spark code,
"write.format("parquet").mode("append").saveAsTable()" will case the format
issue (see the below error log)
2. Hive/Impala in the HUE created table, then
"write.format("parquet").mode("overwrite").saveAsTable()", this code still does
not work.
2.1 Hive/Impala in the HUE created table, and
"write.format("parquet").mode("overwrite").saveAsTable()", then
"write.format("parquet").mode("append").saveAsTable()" can work.
3. Hive/Impala in the HUE created table, then "insertInto()" still will work.
3.1 Hive/Impala in the HUE created a table, and used "insertInto()" insert
some new record, then try to use
"write.format("parquet").mode("append").saveAsTable()", it will get the same
format error log
4. Created parquet table and insert some data by Hive shell, then
"write.format("parquet").mode("append").saveAsTable()" can insert data, but
spark only shows data which insert by spark, and Hive only show data which hive
insert.
===========================================================================
Error Log
===========================================================================
spark.read.format("csv").option("sep",",").option("header","true").load("hdfs:///temp1/test_paquettest.csv").write.format("parquet").mode("append").saveAsTable("parquet_test_table")
org.apache.spark.sql.AnalysisException: The format of the existing table
default.parquet_test_table is `HiveFileFormat`. It doesn't match the specified
format `ParquetFileFormat`.;
at
org.apache.spark.sql.execution.datasources.PreprocessTableCreation$$anonfun$apply$2.applyOrElse(rules.scala:115)
at
org.apache.spark.sql.execution.datasources.PreprocessTableCreation$$anonfun$apply$2.applyOrElse(rules.scala:75)
at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:266)
at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:256)
at
org.apache.spark.sql.execution.datasources.PreprocessTableCreation.apply(rules.scala:75)
at
org.apache.spark.sql.execution.datasources.PreprocessTableCreation.apply(rules.scala:71)
at
org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:85)
at
org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:82)
at
scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:57)
at
scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:66)
at scala.collection.mutable.ArrayBuffer.foldLeft(ArrayBuffer.scala:48)
at
org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:82)
at
org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:74)
at scala.collection.immutable.List.foreach(List.scala:381)
at
org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:74)
at
org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:69)
at
org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:67)
at
org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:50)
at
org.apache.spark.sql.execution.QueryExecution.withCachedData$lzycompute(QueryExecution.scala:73)
at
org.apache.spark.sql.execution.QueryExecution.withCachedData(QueryExecution.scala:72)
at
org.apache.spark.sql.execution.QueryExecution.optimizedPlan$lzycompute(QueryExecution.scala:78)
at
org.apache.spark.sql.execution.QueryExecution.optimizedPlan(QueryExecution.scala:78)
at
org.apache.spark.sql.execution.QueryExecution.completeString(QueryExecution.scala:220)
at
org.apache.spark.sql.execution.QueryExecution.toString(QueryExecution.scala:203)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:74)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:609)
at org.apache.spark.sql.DataFrameWriter.createTable(DataFrameWriter.scala:419)
at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:398)
at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:354)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
