mcdull_zhang created SPARK-36663:
------------------------------------
Summary: When the existing field name is a number, an error will
be reported when reading the orc file
Key: SPARK-36663
URL: https://issues.apache.org/jira/browse/SPARK-36663
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.1.2, 3.0.3
Reporter: mcdull_zhang
You can use the following methods to reproduce the problem:
{quote}val path = "file:///tmp/test_orc"
spark.range(1).withColumnRenamed("id", "100").repartition(1).write.orc(path)
spark.read.orc(path)
{quote}
The error message is like this:
{quote}org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input '100' expecting {'ADD', 'AFTER'....
== SQL ==
struct<100:bigint>
-------^^^
{quote}
The error is actually issued by this line of code:
{quote}CatalystSqlParser.parseDataType("100:bigint")
{quote}
The specific background is that spark calls the above code in the process of
converting the schema of the orc file into the catalyst schema.
{quote}// code in OrcUtils
private def toCatalystSchema(schema: TypeDescription): StructType = {
CharVarcharUtils.replaceCharVarcharWithStringInSchema(CatalystSqlParser.parseDataType(schema.toString).asInstanceOf[StructType])
}{quote}
There are two solutions I currently think of:
# Modify the syntax analysis of SparkSQL to identify this kind of schema
# The TypeDescription.toString method should add the quote symbol to the
numeric column name, because the following syntax is supported:
{quote}CatalystSqlParser.parseDataType("`100`:bigint"){quote}
But currently TypeDescription does not support changing the UNQUOTED_NAMES
variable, should we first submit a pr to the orc project to support the
configuration of this variable。
!image-2021-09-03-20-53-35-626.png!
How do spark members think about this issue?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
