[
https://issues.apache.org/jira/browse/SPARK-20399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liang-Chi Hsieh updated SPARK-20399:
------------------------------------
Description:
The new SQL parser is introduced into Spark 2.0. Seems it bring an issue
regarding the regex pattern string.
The following codes can reproduce it:
{code}
val data = Seq("\u0020\u0021\u0023", "abc")
val df = data.toDF()
// 1st usage: let parser parse pattern string: works in 1.6
val rlike1 = df.filter("value rlike '^\\x20[\\x20-\\x23]+$'")
// 2nd usage: call Column.rlike so the pattern string is a literal which
doesn't go through parser
val rlike2 = df.filter($"value".rlike("^\\x20[\\x20-\\x23]+$")) // 2: works in
1.6, 2.x
// To make 1st usage work, we need to add backslashes like this in 2.x:
val rlike3 = df.filter("value rlike '^\\\\x20[\\\\x20-\\\\x23]+$'")
{code}
Due to unescaping SQL String in parser, the first usage working in 1.6 can't
work in 2.0. To make it work, we need to add additional backslashes.
It is quite weird that we can't use the same regex pattern string the 2 usages.
was:
The new SQL parser is introduced into Spark 2.0. Seems it bring an issue
regarding the regex pattern string.
The following codes can reproduce it:
{code}
val data = Seq("\u0020\u0021\u0023", "abc")
val df = data.toDF()
// 1st usage: let parser parse pattern string: works in 1.6
val rlike1 = df.filter("value rlike '^\\x20[\\x20-\\x23]+$'")
// 2nd usage: call Column.rlike so the pattern string is a literal which
doesn't go through parser
val rlike2 = df.filter($"value".rlike("^\\x20[\\x20-\\x23]+$")) // 2: works in
1.6, 2.x
// To make 1st usage work, we need to add backslashes like this in 2.x:
val rlike3 = df.filter("value rlike '^\\\\x20[\\\\x20-\\\\x23]+$'")
{code}
Due to unescape SQL String in parser, the first usage working in 1.6 can't work
in 2.0. To make it work, we need to add additional backslashes.
It is quite weird that we can't use the same regex pattern string the 2 usages.
> Can't use same regex pattern between 1.6 and 2.x due to unescaped sql string
> in parser
> --------------------------------------------------------------------------------------
>
> Key: SPARK-20399
> URL: https://issues.apache.org/jira/browse/SPARK-20399
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Liang-Chi Hsieh
>
> The new SQL parser is introduced into Spark 2.0. Seems it bring an issue
> regarding the regex pattern string.
> The following codes can reproduce it:
> {code}
> val data = Seq("\u0020\u0021\u0023", "abc")
> val df = data.toDF()
> // 1st usage: let parser parse pattern string: works in 1.6
> val rlike1 = df.filter("value rlike '^\\x20[\\x20-\\x23]+$'")
> // 2nd usage: call Column.rlike so the pattern string is a literal which
> doesn't go through parser
> val rlike2 = df.filter($"value".rlike("^\\x20[\\x20-\\x23]+$")) // 2: works
> in 1.6, 2.x
> // To make 1st usage work, we need to add backslashes like this in 2.x:
> val rlike3 = df.filter("value rlike '^\\\\x20[\\\\x20-\\\\x23]+$'")
> {code}
> Due to unescaping SQL String in parser, the first usage working in 1.6 can't
> work in 2.0. To make it work, we need to add additional backslashes.
> It is quite weird that we can't use the same regex pattern string the 2
> usages.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
