[GitHub] spark pull request #14867: [SPARK-17296][SQL] Simplify parser join processin...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14867 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14867: [SPARK-17296][SQL] Simplify parser join processin...
Github user srinathshankar commented on a diff in the pull request:
https://github.com/apache/spark/pull/14867#discussion_r77675078
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala
---
@@ -360,10 +360,54 @@ class PlanParserSuite extends PlanTest {
test("left anti join", LeftAnti, testExistence)
test("anti join", LeftAnti, testExistence)
+// Test natural cross join
+intercept("select * from a natural cross join b")
+
+// Test natural join with a condition
+intercept("select * from a natural join b on a.id = b.id")
+
// Test multiple consecutive joins
assertEqual(
"select * from a join b join c right join d",
table("a").join(table("b")).join(table("c")).join(table("d"),
RightOuter).select(star()))
+
+// SPARK-17296
+assertEqual(
+ "select * from t1 cross join t2 join t3 on t3.id = t1.id join t4 on
t4.id = t1.id",
+ table("t1")
+.join(table("t2"), Cross)
+.join(table("t3"), Inner, Option(Symbol("t3.id") ===
Symbol("t1.id")))
+.join(table("t4"), Inner, Option(Symbol("t4.id") ===
Symbol("t1.id")))
+.select(star()))
+
+// Test multiple on clauses.
+intercept("select * from t1 inner join t2 inner join t3 on col3 = col2
on col3 = col1")
+
+// Parenthesis
+assertEqual(
+ "select * from t1 inner join (t2 inner join t3 on col3 = col2) on
col3 = col1",
+ table("t1")
+.join(table("t2")
+ .join(table("t3"), Inner, Option('col3 === 'col2)), Inner,
Option('col3 === 'col1))
+.select(star()))
+assertEqual(
+ "select * from t1 inner join (t2 inner join t3) on col3 = col2",
+ table("t1")
+.join(table("t2").join(table("t3"), Inner, None), Inner,
Option('col3 === 'col2))
+.select(star()))
+assertEqual(
+ "select * from t1 inner join (t2 inner join t3 on col3 = col2)",
+ table("t1")
+.join(table("t2").join(table("t3"), Inner, Option('col3 ===
'col2)), Inner, None)
+.select(star()))
+
+// Implicit joins.
+assertEqual(
--- End diff --
Great, LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14867: [SPARK-17296][SQL] Simplify parser join processin...
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/14867#discussion_r77439638 --- Diff: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 --- @@ -374,15 +374,17 @@ setQuantifier ; relation -: left=relation - ((CROSS | joinType) JOIN right=relation joinCriteria? - | NATURAL joinType JOIN right=relation - ) #joinRelation -| relationPrimary #relationDefault +: relationPrimary joinRelation* +; + +joinRelation +: joinType JOIN right=relationPrimary joinCriteria? +| NATURAL joinType JOIN right=relationPrimary ; --- End diff -- I had to move the code around. I have added a check in the AstBuilder (spark side of the parser) to catch this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14867: [SPARK-17296][SQL] Simplify parser join processin...
Github user srinathshankar commented on a diff in the pull request:
https://github.com/apache/spark/pull/14867#discussion_r77421576
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala
---
@@ -360,10 +360,28 @@ class PlanParserSuite extends PlanTest {
test("left anti join", LeftAnti, testExistence)
test("anti join", LeftAnti, testExistence)
+// Test natural cross join
+intercept("select * from a natural cross join b")
+
+// Test natural join with a condition
+intercept("select * from a natural join b on a.id = b.id")
+
// Test multiple consecutive joins
assertEqual(
"select * from a join b join c right join d",
table("a").join(table("b")).join(table("c")).join(table("d"),
RightOuter).select(star()))
+
+// SPARK-17296
+assertEqual(
+ "select * from t1 cross join t2 join t3 on t3.id = t1.id join t4 on
t4.id = t1.id",
+ table("t1")
+.join(table("t2"))
+.join(table("t3"), Inner, Option(Symbol("t3.id") ===
Symbol("t1.id")))
+.join(table("t4"), Inner, Option(Symbol("t4.id") ===
Symbol("t1.id")))
+.select(star()))
+
+// Test multiple on clauses.
+intercept("select * from t1 inner join t2 inner join t3 on col3 = col2
on col3 = col1")
--- End diff --
As discussed, let's also add a test somewhere for
SELECT * FROM T1 INNER JOIN (T2 INNER JOIN T3 ON col3 = col2) ON col3 = col1
SELECT * FROM T1 INNER JOIN (T2 INNER JOIN T3) ON col3 = col2
SELECT * FROM T1 INNER JOIN (T2 INNER JOIN T3 ON col3 = col2)
This looks good to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14867: [SPARK-17296][SQL] Simplify parser join processin...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/14867#discussion_r77420330
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala
---
@@ -360,10 +360,25 @@ class PlanParserSuite extends PlanTest {
test("left anti join", LeftAnti, testExistence)
test("anti join", LeftAnti, testExistence)
+// Test natural cross join
+intercept("select * from a natural cross join b")
+
+// Test natural join with a condition
+intercept("select * from a natural join b on a.id = b.id")
+
// Test multiple consecutive joins
assertEqual(
"select * from a join b join c right join d",
table("a").join(table("b")).join(table("c")).join(table("d"),
RightOuter).select(star()))
+
+// SPARK-17296
+assertEqual(
+ "select * from t1 cross join t2 join t3 on t3.id = t1.id join t4 on
t4.id = t1.id",
--- End diff --
Good catch. I have added a test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14867: [SPARK-17296][SQL] Simplify parser join processin...
Github user srinathshankar commented on a diff in the pull request:
https://github.com/apache/spark/pull/14867#discussion_r77418488
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala
---
@@ -360,10 +360,25 @@ class PlanParserSuite extends PlanTest {
test("left anti join", LeftAnti, testExistence)
test("anti join", LeftAnti, testExistence)
+// Test natural cross join
+intercept("select * from a natural cross join b")
+
+// Test natural join with a condition
+intercept("select * from a natural join b on a.id = b.id")
+
// Test multiple consecutive joins
assertEqual(
"select * from a join b join c right join d",
table("a").join(table("b")).join(table("c")).join(table("d"),
RightOuter).select(star()))
+
+// SPARK-17296
+assertEqual(
+ "select * from t1 cross join t2 join t3 on t3.id = t1.id join t4 on
t4.id = t1.id",
--- End diff --
To clarify, it looks like your patch will disallow both queries at the
parser level. Could you add a test that enforces this ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14867: [SPARK-17296][SQL] Simplify parser join processin...
Github user srinathshankar commented on a diff in the pull request:
https://github.com/apache/spark/pull/14867#discussion_r77417316
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala
---
@@ -360,10 +360,25 @@ class PlanParserSuite extends PlanTest {
test("left anti join", LeftAnti, testExistence)
test("anti join", LeftAnti, testExistence)
+// Test natural cross join
+intercept("select * from a natural cross join b")
+
+// Test natural join with a condition
+intercept("select * from a natural join b on a.id = b.id")
+
// Test multiple consecutive joins
assertEqual(
"select * from a join b join c right join d",
table("a").join(table("b")).join(table("c")).join(table("d"),
RightOuter).select(star()))
+
+// SPARK-17296
+assertEqual(
+ "select * from t1 cross join t2 join t3 on t3.id = t1.id join t4 on
t4.id = t1.id",
--- End diff --
How is something like
SELECT * FROM T1 INNER JOIN T2 INNER JOIN T3 ON col3 = col2 ON col3 = col1;
supposed to parse ?
Without your change it returns the following error:
org.apache.spark.sql.AnalysisException: cannot resolve '`col3`' given input
columns: [col1, col2]; line 1 pos 63
which I don't understand. The following parses though:
SELECT * FROM T1 INNER JOIN T2 INNER JOIN T3 ON col1 = col2 ON col2 = col1
and returns a result
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
