GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/17736
[SPARK-20399][SQL][WIP] Can't use same regex pattern between 1.6 and 2.x
due to unescaped sql string in parser
## What changes were proposed in this pull request?
The new SQL parser is introduced into Spark 2.0. Seems it bring an issue
regarding the regex pattern string.
The following codes can reproduce it:
val data = Seq("\u0020\u0021\u0023", "abc")
val df = data.toDF()
// 1st usage: let parser parse pattern string: works in 1.6
val rlike1 = df.filter("value rlike '^\\x20[\\x20-\\x23]+$'")
// 2nd usage: call Column.rlike so the pattern string is a literal
which doesn't go through parser
val rlike2 = df.filter($"value".rlike("^\\x20[\\x20-\\x23]+$")) // 2:
works in 1.6, 2.x
// To make 1st usage work, we need to add backslashes like this in 2.x:
val rlike3 = df.filter("value rlike '^\\\\x20[\\\\x20-\\\\x23]+$'")
Due to unescaping SQL String in parser, the first usage working in 1.6
can't work in 2.0. To make it work, we need to add additional backslashes.
It is quite weird that we can't use the same regex pattern string in the 2
usages. We should not do unescaping on regex pattern string.
## How was this patch tested?
Jenkins tests.
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/viirya/spark-1 rlike-regex
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17736.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17736
----
commit a0f4a13763c077e57c2dcb5fff12d81f3bb2ceb9
Author: Liang-Chi Hsieh <vii...@gmail.com>
Date: 2017-04-19T01:49:47Z
Don't unescape regex pattern string.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
