Fokko Driesprong created SPARK-16481: ----------------------------------------
Summary: Spark does not update statistics when making use of Hive partitions Key: SPARK-16481 URL: https://issues.apache.org/jira/browse/SPARK-16481 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.6.1 Reporter: Fokko Driesprong Hi all, I've had some strange behaviour using Hive partitions. Turned out, when using Hive partitions, the statistics of the Parquet get not updated properly when inserting new data. I've isolated the issue in the following case: https://github.com/Fokko/spark-strange-refresh-behaviour The fix right now is to refresh the data by hand, which is quite error prone as it can be easily forgotten. Cheers, Fokko Driesprong. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org