[
https://issues.apache.org/jira/browse/SPARK-24358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489507#comment-16489507
]
Joel Croteau commented on SPARK-24358:
--------------------------------------
This does mean that the current implementation has some compatibility issues
with Python 3. In Python 2, a bytes will be inferred as a StringType,
regardless of content. StringType and BinaryType are functionally identical, as
they are both just arbitrary arrays of bytes, and Python 2 will handle any
value of them just fine. In Python 3, attempting to infer the type of a bytes
is an error, and Python 3 will convert a StringType to Unicode. Since not every
byte string is valid Unicode, some errors may occur in processing StringTypes
in Python 3 that worked fine in Python 2.
> createDataFrame in Python 3 should be able to infer bytes type as Binary type
> -----------------------------------------------------------------------------
>
> Key: SPARK-24358
> URL: https://issues.apache.org/jira/browse/SPARK-24358
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Joel Croteau
> Priority: Minor
> Labels: Python3
>
> createDataFrame can infer Python 3's bytearray type as a Binary. Since bytes
> is just the immutable, hashable version of this same structure, it makes
> sense for the same thing to apply there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
