[
https://issues.apache.org/jira/browse/SPARK-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14075232#comment-14075232
]
Teng Qiu commented on SPARK-2446:
---------------------------------
Hi, thanks for the advice, i created a ticket for this: SPARK-2699
i tried to implement such a config parameter, and i find, for impala parquet
files, we can simply check the "CreatedBy" property in FileMetaData object of
parquet file. impala creates parquet file with their own "CreatedBy" string.
but for the parquet files from old spark release, it seems we can only set a
config option to allow strings without UTF8 annotation.
> Add BinaryType support to Parquet I/O.
> --------------------------------------
>
> Key: SPARK-2446
> URL: https://issues.apache.org/jira/browse/SPARK-2446
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Takuya Ueshin
> Assignee: Takuya Ueshin
> Fix For: 1.1.0
>
>
> To support {{BinaryType}}, the following changes are needed:
> - Make {{StringType}} use {{OriginalType.UTF8}}
> - Add {{BinaryType}} using {{PrimitiveTypeName.BINARY}} without
> {{OriginalType}}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
