[GitHub] spark pull request #19691: [SPARK-14922][SPARK-17732][SQL]ALTER TABLE DROP P...

[DazhuangSu]

GitHub user DazhuangSu opened a pull request:

    https://github.com/apache/spark/pull/19691

    [SPARK-14922][SPARK-17732][SQL]ALTER TABLE DROP PARTITION should support 
comparators

    ## What changes were proposed in this pull request?
    
    This pr is inspired by @dongjoon-hyun.
    
    quote from https://github.com/apache/spark/pull/15704 :
    
    > **What changes were proposed in this pull request?**
        This PR aims to support comparators, e.g. '<', '<=', '>', '>=', again 
in Apache Spark 2.0 for backward compatibility.
            **Spark 1.6**
            `scala> sql("CREATE TABLE sales(id INT) PARTITIONED BY (country 
STRING, quarter STRING)")
    res0: org.apache.spark.sql.DataFrame = [result: string]`
    `scala> sql("ALTER TABLE sales DROP PARTITION (country < 'KR')")
    res1: org.apache.spark.sql.DataFrame = [result: string]`
            **Spark 2.0**
            `scala> sql("CREATE TABLE sales(id INT) PARTITIONED BY (country 
STRING, quarter STRING)")
    res0: org.apache.spark.sql.DataFrame = []`
            `scala> sql("ALTER TABLE sales DROP PARTITION (country < 'KR')")`
            `org.apache.spark.sql.catalyst.parser.ParseException:`
            `mismatched input '<' expecting {')', ','}(line 1, pos 42)`
            After this PR, it's supported.
        **How was this patch tested?**
            Pass the Jenkins test with a newly added testcase.
    
    
    https://github.com/apache/spark/pull/16036 points out that if we use int 
literal in DROP PARTITION will fail after patching 
https://github.com/apache/spark/pull/15704.
    The reason of this failing in https://github.com/apache/spark/pull/15704 is 
that AlterTableDropPartitionCommand tells BinayComparison and EqualTo with 
following code:
    
    `private def isRangeComparison(expr: Expression): Boolean = {
 `
        `expr.find(e => e.isInstanceOf[BinaryComparison] && 
!e.isInstanceOf[EqualTo]).isDefined
}`
    
    This PR resolve this problem by telling a drop condition when parsing sqls.
    
    ## How was this patch tested?
    New testcase introduced from https://github.com/apache/spark/pull/15704


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/DazhuangSu/spark SPARK-17732

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/19691.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19691
    
----
commit 20f658ad8e14a94dd23bff6a8d795124d1db24e9
Author: Dylan Su <lovelo...@gmail.com>
Date:   2017-11-08T03:44:28Z

    [SPARK-14922][SPARK-17732][SQL]ALTER TABLE DROP PARTITION should support 
comparators

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org