[
https://issues.apache.org/jira/browse/SPARK-39012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Wang updated SPARK-39012:
-----------------------------
Summary: SparkSQL parse partition value does not support all data types
(was: SparkSQL infer schema does not support all data types)
> SparkSQL parse partition value does not support all data types
> --------------------------------------------------------------
>
> Key: SPARK-39012
> URL: https://issues.apache.org/jira/browse/SPARK-39012
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: Rui Wang
> Priority: Major
>
> When Spark needs to infer schema, it needs to parse string to a type. Not all
> data types are supported so far in this path. For example, binary is known to
> not be supported. If a user uses binary column, and if the user does not use
> a metastore, then SparkSQL could fall back to schema inference thus fail to
> execute during table scan. This should be a bug as schema inference is
> supported but some types are missing.
> string might be converted to all types except ARRAY, MAP, STRUCT, etc. Also
> because when converting from a string, small scale type won't be identified
> if there is a larger scale type. For example, short and long
> Based on Spark SQL data types:
> https://spark.apache.org/docs/latest/sql-ref-datatypes.html, we can support
> the following types:
> BINARY
> BOOLEAN
> And there are two types that I am not sure if SparkSQL is supporting:
> YearMonthIntervalType
> DayTimeIntervalType
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
