Fokko Driesprong created SPARK-16481:
----------------------------------------
Summary: Spark does not update statistics when making use of Hive
partitions
Key: SPARK-16481
URL: https://issues.apache.org/jira/browse/SPARK-16481
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.6.1
Reporter: Fokko Driesprong
Hi all,
I've had some strange behaviour using Hive partitions. Turned out, when using
Hive partitions, the statistics of the Parquet get not updated properly when
inserting new data. I've isolated the issue in the following case:
https://github.com/Fokko/spark-strange-refresh-behaviour
The fix right now is to refresh the data by hand, which is quite error prone as
it can be easily forgotten.
Cheers, Fokko Driesprong.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
