[ https://issues.apache.org/jira/browse/SPARK-36663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409492#comment-17409492 ]
mcdull_zhang commented on SPARK-36663: -------------------------------------- cc [~hyukjin.kwon] [~cloud_fan] > When the existing field name is a number, an error will be reported when > reading the orc file > --------------------------------------------------------------------------------------------- > > Key: SPARK-36663 > URL: https://issues.apache.org/jira/browse/SPARK-36663 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.0.3, 3.1.2 > Reporter: mcdull_zhang > Priority: Critical > Attachments: image-2021-09-03-20-56-28-846.png > > > You can use the following methods to reproduce the problem: > {quote}val path = "file:///tmp/test_orc" > spark.range(1).withColumnRenamed("id", "100").repartition(1).write.orc(path) > spark.read.orc(path) > {quote} > The error message is like this: > {quote}org.apache.spark.sql.catalyst.parser.ParseException: > mismatched input '100' expecting {'ADD', 'AFTER'.... > == SQL == > struct<100:bigint> > -------^^^ > {quote} > The error is actually issued by this line of code: > {quote}CatalystSqlParser.parseDataType("100:bigint") > {quote} > > The specific background is that spark calls the above code in the process of > converting the schema of the orc file into the catalyst schema. > {quote}// code in OrcUtils > private def toCatalystSchema(schema: TypeDescription): StructType = > Unknown macro: \{ > CharVarcharUtils.replaceCharVarcharWithStringInSchema(CatalystSqlParser.parseDataType(schema.toString).asInstanceOf[StructType]) > }{quote} > There are two solutions I currently think of: > # Modify the syntax analysis of SparkSQL to identify this kind of schema > # The TypeDescription.toString method should add the quote symbol to the > numeric column name, because the following syntax is supported: > {quote}CatalystSqlParser.parseDataType("`100`:bigint") > {quote} > But currently TypeDescription does not support changing the UNQUOTED_NAMES > variable, should we first submit a pr to the orc project to support the > configuration of this variable。 > !image-2021-09-03-20-56-28-846.png! > > How do spark members think about this issue? > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org