spark git commit: [SPARK-17008][SPARK-17009][SQL] Normalization and isolation in SQLQueryTestSuite.

2016-08-11 Thread wenchen 
Repository: spark
Updated Branches:
  refs/heads/branch-2.0 8611bc205 -> 51b101668


[SPARK-17008][SPARK-17009][SQL] Normalization and isolation in 
SQLQueryTestSuite.

## What changes were proposed in this pull request?
This patch enhances SQLQueryTestSuite in two ways:

1. SPARK-17009: Use a new SparkSession for each test case to provide stronger 
isolation (e.g. config changes in one test case does not impact another). That 
said, we do not currently isolate catalog changes.
2. SPARK-17008: Normalize query output using sorting, inspired by 
HiveComparisonTest.

I also ported a few new test cases over from SQLQuerySuite.

## How was this patch tested?
This is a test harness update.

Author: petermaxlee 

Closes #14590 from petermaxlee/SPARK-17008.

(cherry picked from commit 425c7c2dbd2923094712e1215dd29272fb09cd79)
Signed-off-by: Wenchen Fan 


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/51b10166
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/51b10166
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/51b10166

Branch: refs/heads/branch-2.0
Commit: 51b1016682a805e06b857a6b1f160a877839dbd5
Parents: 8611bc2
Author: petermaxlee 
Authored: Wed Aug 10 21:05:32 2016 -0700
Committer: Wenchen Fan 
Committed: Thu Aug 11 16:46:03 2016 +0800

--
 .../resources/sql-tests/inputs/datetime.sql |  4 ++
 .../test/resources/sql-tests/inputs/having.sql  | 15 +
 .../resources/sql-tests/inputs/natural-join.sql | 20 ++
 .../sql-tests/results/datetime.sql.out  | 10 +++
 .../resources/sql-tests/results/having.sql.out  | 40 
 .../sql-tests/results/natural-join.sql.out  | 64 
 .../org/apache/spark/sql/SQLQuerySuite.scala| 62 ---
 .../apache/spark/sql/SQLQueryTestSuite.scala| 30 -
 8 files changed, 180 insertions(+), 65 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/51b10166/sql/core/src/test/resources/sql-tests/inputs/datetime.sql
--
diff --git a/sql/core/src/test/resources/sql-tests/inputs/datetime.sql 
b/sql/core/src/test/resources/sql-tests/inputs/datetime.sql
new file mode 100644
index 000..3fd1c37
--- /dev/null
+++ b/sql/core/src/test/resources/sql-tests/inputs/datetime.sql
@@ -0,0 +1,4 @@
+-- date time functions
+
+-- [SPARK-16836] current_date and current_timestamp literals
+select current_date = current_date(), current_timestamp = current_timestamp();

http://git-wip-us.apache.org/repos/asf/spark/blob/51b10166/sql/core/src/test/resources/sql-tests/inputs/having.sql
--
diff --git a/sql/core/src/test/resources/sql-tests/inputs/having.sql 
b/sql/core/src/test/resources/sql-tests/inputs/having.sql
new file mode 100644
index 000..364c022
--- /dev/null
+++ b/sql/core/src/test/resources/sql-tests/inputs/having.sql
@@ -0,0 +1,15 @@
+create temporary view hav as select * from values
+  ("one", 1),
+  ("two", 2),
+  ("three", 3),
+  ("one", 5)
+  as hav(k, v);
+
+-- having clause
+SELECT k, sum(v) FROM hav GROUP BY k HAVING sum(v) > 2;
+
+-- having condition contains grouping column
+SELECT count(k) FROM hav GROUP BY v + 1 HAVING v + 1 = 2;
+
+-- SPARK-11032: resolve having correctly
+SELECT MIN(t.v) FROM (SELECT * FROM hav WHERE v > 0) t HAVING(COUNT(1) > 0);

http://git-wip-us.apache.org/repos/asf/spark/blob/51b10166/sql/core/src/test/resources/sql-tests/inputs/natural-join.sql
--
diff --git a/sql/core/src/test/resources/sql-tests/inputs/natural-join.sql 
b/sql/core/src/test/resources/sql-tests/inputs/natural-join.sql
new file mode 100644
index 000..71a5015
--- /dev/null
+++ b/sql/core/src/test/resources/sql-tests/inputs/natural-join.sql
@@ -0,0 +1,20 @@
+create temporary view nt1 as select * from values
+  ("one", 1),
+  ("two", 2),
+  ("three", 3)
+  as nt1(k, v1);
+
+create temporary view nt2 as select * from values
+  ("one", 1),
+  ("two", 22),
+  ("one", 5)
+  as nt2(k, v2);
+
+
+SELECT * FROM nt1 natural join nt2 where k = "one";
+
+SELECT * FROM nt1 natural left join nt2 order by v1, v2;
+
+SELECT * FROM nt1 natural right join nt2 order by v1, v2;
+
+SELECT count(*) FROM nt1 natural full outer join nt2;

http://git-wip-us.apache.org/repos/asf/spark/blob/51b10166/sql/core/src/test/resources/sql-tests/results/datetime.sql.out
--
diff --git a/sql/core/src/test/resources/sql-tests/results/datetime.sql.out 
b/sql/core/src/test/resources/sql-tests/results/datetime.sql.out
new file mode 100644
index 000..5174657
---

spark git commit: [SPARK-17008][SPARK-17009][SQL] Normalization and isolation in SQLQueryTestSuite.

2016-08-10 Thread rxin 
Repository: spark
Updated Branches:
  refs/heads/master ab648c000 -> 425c7c2db


[SPARK-17008][SPARK-17009][SQL] Normalization and isolation in 
SQLQueryTestSuite.

## What changes were proposed in this pull request?
This patch enhances SQLQueryTestSuite in two ways:

1. SPARK-17009: Use a new SparkSession for each test case to provide stronger 
isolation (e.g. config changes in one test case does not impact another). That 
said, we do not currently isolate catalog changes.
2. SPARK-17008: Normalize query output using sorting, inspired by 
HiveComparisonTest.

I also ported a few new test cases over from SQLQuerySuite.

## How was this patch tested?
This is a test harness update.

Author: petermaxlee 

Closes #14590 from petermaxlee/SPARK-17008.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/425c7c2d
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/425c7c2d
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/425c7c2d

Branch: refs/heads/master
Commit: 425c7c2dbd2923094712e1215dd29272fb09cd79
Parents: ab648c0
Author: petermaxlee 
Authored: Wed Aug 10 21:05:32 2016 -0700
Committer: Reynold Xin 
Committed: Wed Aug 10 21:05:32 2016 -0700

--
 .../resources/sql-tests/inputs/datetime.sql |  4 ++
 .../test/resources/sql-tests/inputs/having.sql  | 15 +
 .../resources/sql-tests/inputs/natural-join.sql | 20 ++
 .../sql-tests/results/datetime.sql.out  | 10 +++
 .../resources/sql-tests/results/having.sql.out  | 40 
 .../sql-tests/results/natural-join.sql.out  | 64 
 .../org/apache/spark/sql/SQLQuerySuite.scala| 62 ---
 .../apache/spark/sql/SQLQueryTestSuite.scala| 30 -
 8 files changed, 180 insertions(+), 65 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/425c7c2d/sql/core/src/test/resources/sql-tests/inputs/datetime.sql
--
diff --git a/sql/core/src/test/resources/sql-tests/inputs/datetime.sql 
b/sql/core/src/test/resources/sql-tests/inputs/datetime.sql
new file mode 100644
index 000..3fd1c37
--- /dev/null
+++ b/sql/core/src/test/resources/sql-tests/inputs/datetime.sql
@@ -0,0 +1,4 @@
+-- date time functions
+
+-- [SPARK-16836] current_date and current_timestamp literals
+select current_date = current_date(), current_timestamp = current_timestamp();

http://git-wip-us.apache.org/repos/asf/spark/blob/425c7c2d/sql/core/src/test/resources/sql-tests/inputs/having.sql
--
diff --git a/sql/core/src/test/resources/sql-tests/inputs/having.sql 
b/sql/core/src/test/resources/sql-tests/inputs/having.sql
new file mode 100644
index 000..364c022
--- /dev/null
+++ b/sql/core/src/test/resources/sql-tests/inputs/having.sql
@@ -0,0 +1,15 @@
+create temporary view hav as select * from values
+  ("one", 1),
+  ("two", 2),
+  ("three", 3),
+  ("one", 5)
+  as hav(k, v);
+
+-- having clause
+SELECT k, sum(v) FROM hav GROUP BY k HAVING sum(v) > 2;
+
+-- having condition contains grouping column
+SELECT count(k) FROM hav GROUP BY v + 1 HAVING v + 1 = 2;
+
+-- SPARK-11032: resolve having correctly
+SELECT MIN(t.v) FROM (SELECT * FROM hav WHERE v > 0) t HAVING(COUNT(1) > 0);

http://git-wip-us.apache.org/repos/asf/spark/blob/425c7c2d/sql/core/src/test/resources/sql-tests/inputs/natural-join.sql
--
diff --git a/sql/core/src/test/resources/sql-tests/inputs/natural-join.sql 
b/sql/core/src/test/resources/sql-tests/inputs/natural-join.sql
new file mode 100644
index 000..71a5015
--- /dev/null
+++ b/sql/core/src/test/resources/sql-tests/inputs/natural-join.sql
@@ -0,0 +1,20 @@
+create temporary view nt1 as select * from values
+  ("one", 1),
+  ("two", 2),
+  ("three", 3)
+  as nt1(k, v1);
+
+create temporary view nt2 as select * from values
+  ("one", 1),
+  ("two", 22),
+  ("one", 5)
+  as nt2(k, v2);
+
+
+SELECT * FROM nt1 natural join nt2 where k = "one";
+
+SELECT * FROM nt1 natural left join nt2 order by v1, v2;
+
+SELECT * FROM nt1 natural right join nt2 order by v1, v2;
+
+SELECT count(*) FROM nt1 natural full outer join nt2;

http://git-wip-us.apache.org/repos/asf/spark/blob/425c7c2d/sql/core/src/test/resources/sql-tests/results/datetime.sql.out
--
diff --git a/sql/core/src/test/resources/sql-tests/results/datetime.sql.out 
b/sql/core/src/test/resources/sql-tests/results/datetime.sql.out
new file mode 100644
index 000..5174657
--- /dev/null
+++ b/sql/core/src/test/resources/sql-tests/results/datetime.sql.out
@@ -0,0 +1,10 @@
+-- Automatically generated by