Eric Liang created SPARK-18544:
----------------------------------
Summary: Append with df.saveAsTable writes data to wrong location
Key: SPARK-18544
URL: https://issues.apache.org/jira/browse/SPARK-18544
Project: Spark
Issue Type: Bug
Components: SQL
Reporter: Eric Liang
When using saveAsTable in append mode, data will be written to the wrong
location for non-managed Datasource tables. The following example illustrates
this.
It seems somehow pass the wrong table path to InsertIntoHadoopFsRelation from
DataFrameWriter. Also, we should probably remove the repair table call at the
end of saveAsTable in DataFrameWriter. That shouldn't be needed in either the
Hive or Datasource case.
{code}
scala> spark.sqlContext.range(10000).selectExpr("id", "id as A", "id as
B").write.partitionBy("A", "B").mode("overwrite").parquet("/tmp/test_10k")
scala> sql("msck repair table test_10k")
scala> sql("select * from test_10k where A = 1").count
res6: Long = 1
scala> spark.sqlContext.range(10).selectExpr("id", "id as A", "id as
B").write.partitionBy("A", "B").mode("append").parquet("/tmp/test_10k")
scala> sql("select * from test_10k where A = 1").count
res8: Long = 1
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
