[
https://issues.apache.org/jira/browse/SPARK-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14086560#comment-14086560
]
Yin Huai commented on SPARK-2854:
---------------------------------
Since we have already do conversions for ByteType and ShortType when we have
int values, to be consistent, we can also support long values for ByteType,
ShortType and IntegerType.
For datetime.time and datetime.date values, because there are SQL Time and Date
types, datetime.time and datetime.date will not be allowed as value types for
TimestampType.
So, here will be the updated _acceptable_types
{code}
_acceptable_types = {
BooleanType: (bool,),
ByteType: (int, long),
ShortType: (int, long),
IntegerType: (int, long),
LongType: (int, long),
FloatType: (float,),
DoubleType: (float,),
DecimalType: (decimal.Decimal,),
StringType: (str, unicode),
TimestampType: (datetime.datetime,),
ArrayType: (list, tuple, array),
MapType: (dict,),
StructType: (tuple, list),
}
{code}
> Finalize _acceptable_types in pyspark.sql
> -----------------------------------------
>
> Key: SPARK-2854
> URL: https://issues.apache.org/jira/browse/SPARK-2854
> Project: Spark
> Issue Type: Task
> Components: SQL
> Reporter: Yin Huai
> Priority: Blocker
>
> In PySpark, _acceptable_types defines accepted Python data types for every
> Spark SQL data type. The list is shown below.
> {code}
> _acceptable_types = {
> BooleanType: (bool,),
> ByteType: (int, long),
> ShortType: (int, long),
> IntegerType: (int, long),
> LongType: (int, long),
> FloatType: (float,),
> DoubleType: (float,),
> DecimalType: (decimal.Decimal,),
> StringType: (str, unicode),
> TimestampType: (datetime.datetime, datetime.time, datetime.date),
> ArrayType: (list, tuple, array),
> MapType: (dict,),
> StructType: (tuple, list),
> }
> {code}
> Let's double check this mapping before 1.1 release.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
