[
https://issues.apache.org/jira/browse/SPARK-39012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Wang updated SPARK-39012:
-----------------------------
Description:
When Spark needs to infer schema, it needs to parse string to a type. Not all
data types are supported so far in this path. For example, binary is known to
not be supported.
string might be converted to all types except ARRAY, MAP, STRUCT, etc. Also
because when converting from a string, small scale type won't be identified if
there is a larger scale type. For example, short and long
Based on Spark SQL data types:
https://spark.apache.org/docs/latest/sql-ref-datatypes.html, we can support the
following types:
BINARY
BOOLEAN
And there are two types that I am not sure if SparkSQL is supporting:
YearMonthIntervalType
DayTimeIntervalType
was:
When Spark needs to infer schema, it needs to parse string to a type. Not all
data types are supported so far in this path. For example, binary is spotted to
not supported.
string might be converted to all types except ARRAY, MAP, STRUCT, etc.
Spark SQL data types:
https://spark.apache.org/docs/latest/sql-ref-datatypes.html
> SparkSQL infer schema does not support all data types
> -----------------------------------------------------
>
> Key: SPARK-39012
> URL: https://issues.apache.org/jira/browse/SPARK-39012
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: Rui Wang
> Priority: Major
>
> When Spark needs to infer schema, it needs to parse string to a type. Not all
> data types are supported so far in this path. For example, binary is known to
> not be supported.
> string might be converted to all types except ARRAY, MAP, STRUCT, etc. Also
> because when converting from a string, small scale type won't be identified
> if there is a larger scale type. For example, short and long
> Based on Spark SQL data types:
> https://spark.apache.org/docs/latest/sql-ref-datatypes.html, we can support
> the following types:
> BINARY
> BOOLEAN
> And there are two types that I am not sure if SparkSQL is supporting:
> YearMonthIntervalType
> DayTimeIntervalType
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
