[
https://issues.apache.org/jira/browse/SPARK-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14061382#comment-14061382
]
Michael Armbrust commented on SPARK-2446:
-----------------------------------------
Note that this commit changes the semantics when loading in data that was
created with prior versions of Spark SQL. Before, we were writing out strings
as Binary data without adding any other annotations. Thus, when data is read in
from prior versions, data that was StringType will now become BinaryType.
Users that need strings can CAST that column to a String. It was decided that
while this breaks compatibility, it does make us compatible with other systems
(Hive, Thrift, etc) and adds support for Binary data, so this is the right
decision long term.
> Add BinaryType support to Parquet I/O.
> --------------------------------------
>
> Key: SPARK-2446
> URL: https://issues.apache.org/jira/browse/SPARK-2446
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Takuya Ueshin
> Fix For: 1.1.0
>
>
> To support {{BinaryType}}, the following changes are needed:
> - Make {{StringType}} use {{OriginalType.UTF8}}
> - Add {{BinaryType}} using {{PrimitiveTypeName.BINARY}} without
> {{OriginalType}}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
