[GitHub] spark pull request #10468: [SPARK-12409][SPARK-12387][SPARK-12391][SQL] Supp...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/10468#discussion_r151965254
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -186,8 +187,26 @@ class JDBCSuite extends SparkFunSuite
assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME =
'fred'")).collect().size == 1)
assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME >
'fred'")).collect().size == 2)
assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME !=
'fred'")).collect().size == 2)
+assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME IN
('mary', 'fred')"))
+ .collect().size == 2)
+assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME NOT IN
('fred')"))
+ .collect().size === 2)
+assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID = 1 OR
NAME = 'mary'"))
+ .collect().size == 2)
+assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID = 1 OR
NAME = 'mary' "
+ + "AND THEID = 2")).collect().size == 2)
+assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME LIKE
'fr%'")).collect().size == 1)
+assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME LIKE
'%ed'")).collect().size == 1)
+assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME LIKE
'%re%'")).collect().size == 1)
assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS
NULL")).collect().size == 1)
assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS NOT
NULL")).collect().size == 0)
+
+// This is a test to reflect discussion in SPARK-12218.
+// The older versions of spark have this kind of bugs in parquet data
source.
+val df1 = sql("SELECT * FROM foobar WHERE NOT (THEID != 2 AND NAME !=
'mary')")
--- End diff --
yea, I think ok to drop this test.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #10468: [SPARK-12409][SPARK-12387][SPARK-12391][SQL] Supp...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/10468#discussion_r151900175
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -186,8 +187,26 @@ class JDBCSuite extends SparkFunSuite
assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME =
'fred'")).collect().size == 1)
assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME >
'fred'")).collect().size == 2)
assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME !=
'fred'")).collect().size == 2)
+assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME IN
('mary', 'fred')"))
+ .collect().size == 2)
+assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME NOT IN
('fred')"))
+ .collect().size === 2)
+assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID = 1 OR
NAME = 'mary'"))
+ .collect().size == 2)
+assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID = 1 OR
NAME = 'mary' "
+ + "AND THEID = 2")).collect().size == 2)
+assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME LIKE
'fr%'")).collect().size == 1)
+assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME LIKE
'%ed'")).collect().size == 1)
+assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME LIKE
'%re%'")).collect().size == 1)
assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS
NULL")).collect().size == 1)
assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS NOT
NULL")).collect().size == 0)
+
+// This is a test to reflect discussion in SPARK-12218.
+// The older versions of spark have this kind of bugs in parquet data
source.
+val df1 = sql("SELECT * FROM foobar WHERE NOT (THEID != 2 AND NAME !=
'mary')")
--- End diff --
The two sub-conditions are both ok to be pushed down. So this doesn't
actually test against the nested AND issue in SPARK-12218. See #19776
Btw, the two sub-conditions are filtered out the same rows. This doesn't
reflect the issue too.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
