[
https://issues.apache.org/jira/browse/SPARK-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15008074#comment-15008074
]
Ian edited comment on SPARK-11153 at 11/17/15 5:18 AM:
-------------------------------------------------------
Hi, Cheng:
How about Filter pushdown for Timestamp type?
It appeared that the stats of Timestamp type is written as BinaryStats. is it
reason that Timestamp filter is not pushed down?
was (Author: ianlcsd):
Hi, Cheng:
Filter pushdown for Timestamp type?
It appeared that the stats of Timestamp type is written as BinaryStats. is it
reason that Timestamp filter is not pushed down?
> Turns off Parquet filter push-down for string and binary columns
> ----------------------------------------------------------------
>
> Key: SPARK-11153
> URL: https://issues.apache.org/jira/browse/SPARK-11153
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.0, 1.5.1
> Reporter: Cheng Lian
> Assignee: Cheng Lian
> Priority: Blocker
> Fix For: 1.5.2, 1.6.0
>
>
> Due to PARQUET-251, {{BINARY}} columns in existing Parquet files may be
> written with corrupted statistics information. This information is used by
> filter push-down optimization. Since Spark 1.5 turns on Parquet filter
> push-down by default, we may end up with wrong query results. PARQUET-251 has
> been fixed in parquet-mr 1.8.1, but Spark 1.5 is still using 1.7.0.
> Note that this kind of corrupted Parquet files could be produced by any
> Parquet data models.
> This affects all Spark SQL data types that can be mapped to Parquet
> {{BINARY}}, namely:
> - {{StringType}}
> - {{BinaryType}}
> - {{DecimalType}} (but Spark SQL doesn't support pushing down {{DecimalType}}
> columns for now.)
> To avoid wrong query results, we should disable filter push-down for columns
> of {{StringType}} and {{BinaryType}} until we upgrade to parquet-mr 1.8.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
