[ https://issues.apache.org/jira/browse/SPARK-39012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rui Wang updated SPARK-39012: ----------------------------- Description: When Spark needs to infer schema, it needs to parse string to a type. Not all data types are supported so far in this path. For example, binary is known to not be supported. If a user uses binary column, and if the user does not use a metastore, then SparkSQL could fall back to schema inference thus fail to execute during table scan. This should be a bug as schema inference is supported but some types are missing. string might be converted to all types except ARRAY, MAP, STRUCT, etc. Also because when converting from a string, small scale type won't be identified if there is a larger scale type. For example, short and long Based on Spark SQL data types: https://spark.apache.org/docs/latest/sql-ref-datatypes.html, we can support the following types: BINARY BOOLEAN And there are two types that I am not sure if SparkSQL is supporting: YearMonthIntervalType DayTimeIntervalType was: When Spark needs to infer schema, it needs to parse string to a type. Not all data types are supported so far in this path. For example, binary is known to not be supported. string might be converted to all types except ARRAY, MAP, STRUCT, etc. Also because when converting from a string, small scale type won't be identified if there is a larger scale type. For example, short and long Based on Spark SQL data types: https://spark.apache.org/docs/latest/sql-ref-datatypes.html, we can support the following types: BINARY BOOLEAN And there are two types that I am not sure if SparkSQL is supporting: YearMonthIntervalType DayTimeIntervalType > SparkSQL infer schema does not support all data types > ----------------------------------------------------- > > Key: SPARK-39012 > URL: https://issues.apache.org/jira/browse/SPARK-39012 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.3.0 > Reporter: Rui Wang > Priority: Major > > When Spark needs to infer schema, it needs to parse string to a type. Not all > data types are supported so far in this path. For example, binary is known to > not be supported. If a user uses binary column, and if the user does not use > a metastore, then SparkSQL could fall back to schema inference thus fail to > execute during table scan. This should be a bug as schema inference is > supported but some types are missing. > string might be converted to all types except ARRAY, MAP, STRUCT, etc. Also > because when converting from a string, small scale type won't be identified > if there is a larger scale type. For example, short and long > Based on Spark SQL data types: > https://spark.apache.org/docs/latest/sql-ref-datatypes.html, we can support > the following types: > BINARY > BOOLEAN > And there are two types that I am not sure if SparkSQL is supporting: > YearMonthIntervalType > DayTimeIntervalType -- This message was sent by Atlassian Jira (v8.20.7#820007) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org