[
https://issues.apache.org/jira/browse/SPARK-44108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jorge Machado updated SPARK-44108:
----------------------------------
Description:
Hello all,
I have a client that has a column named : bfzgtäeil
Spark cannot handle this. My test:
{code:java}
import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
import org.scalatest.funsuite.AnyFunSuite
class HiveTest extends AnyFunSuite {
test("test that Spark does not cut columns with ä") {
val data =
"bfzugtäeil:string"
CatalystSqlParser.parseDataType(data)
}
} {code}
I debugged it and I'm deep on the org.antlr.v4.runtime.Lexer class.
Any ideas ?
{code:java}
== SQL ==bfzugtäeil:string------^^^
at
org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:306)
at
org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:143)
at
org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseDataType(ParseDriver.scala:41)
at
com.deutschebahn.zod.fvdl.commons.spark.app.captured.HiveTest2.$anonfun$new$1(HiveTest2.scala:13)
{code}
was:
Hello all,
I have a client that has a column named : bfzgtäeil
Spark cannot handle this. My test:
{code:java}
import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
import org.scalatest.funsuite.AnyFunSuite
class HiveTest extends AnyFunSuite {
test("test that Spark does not cut columns with ä") {
val data =
"bfzugtäeil:string"
CatalystSqlParser.parseDataType(data)
}
} {code}
I debugged it and I'm deep on the org.antlr.v4.runtime.Lexer class.
Any ideas ?
> Cannot parse Type from german "umlaut"
> --------------------------------------
>
> Key: SPARK-44108
> URL: https://issues.apache.org/jira/browse/SPARK-44108
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: Jorge Machado
> Priority: Major
>
> Hello all,
>
> I have a client that has a column named : bfzgtäeil
> Spark cannot handle this. My test:
>
> {code:java}
> import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
> import org.scalatest.funsuite.AnyFunSuite
> class HiveTest extends AnyFunSuite {
> test("test that Spark does not cut columns with ä") {
> val data =
> "bfzugtäeil:string"
> CatalystSqlParser.parseDataType(data)
> }
> } {code}
> I debugged it and I'm deep on the org.antlr.v4.runtime.Lexer class.
> Any ideas ?
>
> {code:java}
> == SQL ==bfzugtäeil:string------^^^
> at
> org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:306)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:143)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseDataType(ParseDriver.scala:41)
> at
> com.deutschebahn.zod.fvdl.commons.spark.app.captured.HiveTest2.$anonfun$new$1(HiveTest2.scala:13)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
