[GitHub] spark pull request #21091: [SPARK-22676][FOLLOW-UP] fix code style for test.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/21091


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21091: [SPARK-22676][FOLLOW-UP] fix code style for test.

[jinxing64]

Github user jinxing64 commented on a diff in the pull request:

https://github.com/apache/spark/pull/21091#discussion_r182677298
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/QueryPartitionSuite.scala ---
@@ -34,79 +34,81 @@ class QueryPartitionSuite extends QueryTest with 
SQLTestUtils with TestHiveSingl
   import spark.implicits._
 
   test("SPARK-5068: query data when path doesn't exist") {
-withSQLConf((SQLConf.HIVE_VERIFY_PARTITION_PATH.key, "true")) {
-  val testData = sparkContext.parallelize(
-(1 to 10).map(i => TestData(i, i.toString))).toDF()
-  testData.createOrReplaceTempView("testData")
-
-  val tmpDir = Files.createTempDir()
-  // create the table for test
-  sql(s"CREATE TABLE table_with_partition(key int,value string) " +
-s"PARTITIONED by (ds string) location '${tmpDir.toURI}' ")
-  sql("INSERT OVERWRITE TABLE table_with_partition  partition (ds='1') 
" +
-"SELECT key,value FROM testData")
-  sql("INSERT OVERWRITE TABLE table_with_partition  partition (ds='2') 
" +
-"SELECT key,value FROM testData")
-  sql("INSERT OVERWRITE TABLE table_with_partition  partition (ds='3') 
" +
-"SELECT key,value FROM testData")
-  sql("INSERT OVERWRITE TABLE table_with_partition  partition (ds='4') 
" +
-"SELECT key,value FROM testData")
-
-  // test for the exist path
-  checkAnswer(sql("select key,value from table_with_partition"),
-testData.toDF.collect ++ testData.toDF.collect
-  ++ testData.toDF.collect ++ testData.toDF.collect)
-
-  // delete the path of one partition
-  tmpDir.listFiles
-.find { f => f.isDirectory && f.getName().startsWith("ds=") }
-.foreach { f => Utils.deleteRecursively(f) }
-
-  // test for after delete the path
-  checkAnswer(sql("select key,value from table_with_partition"),
-testData.toDF.collect ++ testData.toDF.collect ++ 
testData.toDF.collect)
-
-  sql("DROP TABLE IF EXISTS table_with_partition")
-  sql("DROP TABLE IF EXISTS createAndInsertTest")
+withSQLConf(SQLConf.HIVE_VERIFY_PARTITION_PATH.key -> "true") {
+  withTempView("testData") {
+withTable("table_with_partition", "createAndInsertTest") {
+  withTempDir { tmpDir =>
+val testData = sparkContext.parallelize(
+  (1 to 10).map(i => TestData(i, i.toString))).toDF()
+testData.createOrReplaceTempView("testData")
+
+// create the table for test
+sql(s"CREATE TABLE table_with_partition(key int,value string) 
" +
+s"PARTITIONED by (ds string) location '${tmpDir.toURI}' ")
+sql("INSERT OVERWRITE TABLE table_with_partition  partition 
(ds='1') " +
+"SELECT key,value FROM testData")
+sql("INSERT OVERWRITE TABLE table_with_partition  partition 
(ds='2') " +
+"SELECT key,value FROM testData")
+sql("INSERT OVERWRITE TABLE table_with_partition  partition 
(ds='3') " +
+"SELECT key,value FROM testData")
+sql("INSERT OVERWRITE TABLE table_with_partition  partition 
(ds='4') " +
+"SELECT key,value FROM testData")
+
+// test for the exist path
+checkAnswer(sql("select key,value from table_with_partition"),
+  testData.union(testData).union(testData).union(testData))
+
+// delete the path of one partition
+tmpDir.listFiles
+.find { f => f.isDirectory && 
f.getName().startsWith("ds=") }
+.foreach { f => Utils.deleteRecursively(f) }
+
+// test for after delete the path
+checkAnswer(sql("select key,value from table_with_partition"),
+  testData.union(testData).union(testData))
+  }
+}
+  }
 }
   }
 
   test("Replace spark.sql.hive.verifyPartitionPath by 
spark.files.ignoreMissingFiles") {
--- End diff --

I extracted a common method, check again.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21091: [SPARK-22676][FOLLOW-UP] fix code style for test.

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/21091#discussion_r182661319
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/QueryPartitionSuite.scala ---
@@ -34,79 +34,81 @@ class QueryPartitionSuite extends QueryTest with 
SQLTestUtils with TestHiveSingl
   import spark.implicits._
 
   test("SPARK-5068: query data when path doesn't exist") {
-withSQLConf((SQLConf.HIVE_VERIFY_PARTITION_PATH.key, "true")) {
-  val testData = sparkContext.parallelize(
-(1 to 10).map(i => TestData(i, i.toString))).toDF()
-  testData.createOrReplaceTempView("testData")
-
-  val tmpDir = Files.createTempDir()
-  // create the table for test
-  sql(s"CREATE TABLE table_with_partition(key int,value string) " +
-s"PARTITIONED by (ds string) location '${tmpDir.toURI}' ")
-  sql("INSERT OVERWRITE TABLE table_with_partition  partition (ds='1') 
" +
-"SELECT key,value FROM testData")
-  sql("INSERT OVERWRITE TABLE table_with_partition  partition (ds='2') 
" +
-"SELECT key,value FROM testData")
-  sql("INSERT OVERWRITE TABLE table_with_partition  partition (ds='3') 
" +
-"SELECT key,value FROM testData")
-  sql("INSERT OVERWRITE TABLE table_with_partition  partition (ds='4') 
" +
-"SELECT key,value FROM testData")
-
-  // test for the exist path
-  checkAnswer(sql("select key,value from table_with_partition"),
-testData.toDF.collect ++ testData.toDF.collect
-  ++ testData.toDF.collect ++ testData.toDF.collect)
-
-  // delete the path of one partition
-  tmpDir.listFiles
-.find { f => f.isDirectory && f.getName().startsWith("ds=") }
-.foreach { f => Utils.deleteRecursively(f) }
-
-  // test for after delete the path
-  checkAnswer(sql("select key,value from table_with_partition"),
-testData.toDF.collect ++ testData.toDF.collect ++ 
testData.toDF.collect)
-
-  sql("DROP TABLE IF EXISTS table_with_partition")
-  sql("DROP TABLE IF EXISTS createAndInsertTest")
+withSQLConf(SQLConf.HIVE_VERIFY_PARTITION_PATH.key -> "true") {
+  withTempView("testData") {
+withTable("table_with_partition", "createAndInsertTest") {
+  withTempDir { tmpDir =>
+val testData = sparkContext.parallelize(
+  (1 to 10).map(i => TestData(i, i.toString))).toDF()
+testData.createOrReplaceTempView("testData")
+
+// create the table for test
+sql(s"CREATE TABLE table_with_partition(key int,value string) 
" +
+s"PARTITIONED by (ds string) location '${tmpDir.toURI}' ")
+sql("INSERT OVERWRITE TABLE table_with_partition  partition 
(ds='1') " +
+"SELECT key,value FROM testData")
+sql("INSERT OVERWRITE TABLE table_with_partition  partition 
(ds='2') " +
+"SELECT key,value FROM testData")
+sql("INSERT OVERWRITE TABLE table_with_partition  partition 
(ds='3') " +
+"SELECT key,value FROM testData")
+sql("INSERT OVERWRITE TABLE table_with_partition  partition 
(ds='4') " +
+"SELECT key,value FROM testData")
+
+// test for the exist path
+checkAnswer(sql("select key,value from table_with_partition"),
+  testData.union(testData).union(testData).union(testData))
+
+// delete the path of one partition
+tmpDir.listFiles
+.find { f => f.isDirectory && 
f.getName().startsWith("ds=") }
+.foreach { f => Utils.deleteRecursively(f) }
+
+// test for after delete the path
+checkAnswer(sql("select key,value from table_with_partition"),
+  testData.union(testData).union(testData))
+  }
+}
+  }
 }
   }
 
   test("Replace spark.sql.hive.verifyPartitionPath by 
spark.files.ignoreMissingFiles") {
--- End diff --

These 2 tests are exactly same except the config, can we create a common 
method for it?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21091: [SPARK-22676][FOLLOW-UP] fix code style for test.

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/21091#discussion_r182426078
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/QueryPartitionSuite.scala ---
@@ -34,79 +34,83 @@ class QueryPartitionSuite extends QueryTest with 
SQLTestUtils with TestHiveSingl
   import spark.implicits._
 
   test("SPARK-5068: query data when path doesn't exist") {
-withSQLConf((SQLConf.HIVE_VERIFY_PARTITION_PATH.key, "true")) {
-  val testData = sparkContext.parallelize(
-(1 to 10).map(i => TestData(i, i.toString))).toDF()
-  testData.createOrReplaceTempView("testData")
-
-  val tmpDir = Files.createTempDir()
-  // create the table for test
-  sql(s"CREATE TABLE table_with_partition(key int,value string) " +
-s"PARTITIONED by (ds string) location '${tmpDir.toURI}' ")
-  sql("INSERT OVERWRITE TABLE table_with_partition  partition (ds='1') 
" +
-"SELECT key,value FROM testData")
-  sql("INSERT OVERWRITE TABLE table_with_partition  partition (ds='2') 
" +
-"SELECT key,value FROM testData")
-  sql("INSERT OVERWRITE TABLE table_with_partition  partition (ds='3') 
" +
-"SELECT key,value FROM testData")
-  sql("INSERT OVERWRITE TABLE table_with_partition  partition (ds='4') 
" +
-"SELECT key,value FROM testData")
-
-  // test for the exist path
-  checkAnswer(sql("select key,value from table_with_partition"),
-testData.toDF.collect ++ testData.toDF.collect
-  ++ testData.toDF.collect ++ testData.toDF.collect)
-
-  // delete the path of one partition
-  tmpDir.listFiles
-.find { f => f.isDirectory && f.getName().startsWith("ds=") }
-.foreach { f => Utils.deleteRecursively(f) }
-
-  // test for after delete the path
-  checkAnswer(sql("select key,value from table_with_partition"),
-testData.toDF.collect ++ testData.toDF.collect ++ 
testData.toDF.collect)
-
-  sql("DROP TABLE IF EXISTS table_with_partition")
-  sql("DROP TABLE IF EXISTS createAndInsertTest")
+withSQLConf((SQLConf.HIVE_VERIFY_PARTITION_PATH.key -> "true")) {
+  withTempView("testData") {
+withTable("table_with_partition", "createAndInsertTest") {
+  withTempPath { tmpDir =>
+tmpDir.mkdir()
+val testData = sparkContext.parallelize(
+  (1 to 10).map(i => TestData(i, i.toString))).toDF()
+testData.createOrReplaceTempView("testData")
+
+// create the table for test
+sql(s"CREATE TABLE table_with_partition(key int,value string) 
" +
+s"PARTITIONED by (ds string) location '${tmpDir.toURI}' ")
+sql("INSERT OVERWRITE TABLE table_with_partition  partition 
(ds='1') " +
+"SELECT key,value FROM testData")
+sql("INSERT OVERWRITE TABLE table_with_partition  partition 
(ds='2') " +
+"SELECT key,value FROM testData")
+sql("INSERT OVERWRITE TABLE table_with_partition  partition 
(ds='3') " +
+"SELECT key,value FROM testData")
+sql("INSERT OVERWRITE TABLE table_with_partition  partition 
(ds='4') " +
+"SELECT key,value FROM testData")
+
+// test for the exist path
+checkAnswer(sql("select key,value from table_with_partition"),
+  
testData.union(testData).union(testData).union(testData).collect)
--- End diff --

no need to call `collect`, `checkAnswer` can compare 2 dataframes


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21091: [SPARK-22676][FOLLOW-UP] fix code style for test.

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/21091#discussion_r182422457
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/QueryPartitionSuite.scala ---
@@ -34,79 +34,83 @@ class QueryPartitionSuite extends QueryTest with 
SQLTestUtils with TestHiveSingl
   import spark.implicits._
 
   test("SPARK-5068: query data when path doesn't exist") {
-withSQLConf((SQLConf.HIVE_VERIFY_PARTITION_PATH.key, "true")) {
-  val testData = sparkContext.parallelize(
-(1 to 10).map(i => TestData(i, i.toString))).toDF()
-  testData.createOrReplaceTempView("testData")
-
-  val tmpDir = Files.createTempDir()
-  // create the table for test
-  sql(s"CREATE TABLE table_with_partition(key int,value string) " +
-s"PARTITIONED by (ds string) location '${tmpDir.toURI}' ")
-  sql("INSERT OVERWRITE TABLE table_with_partition  partition (ds='1') 
" +
-"SELECT key,value FROM testData")
-  sql("INSERT OVERWRITE TABLE table_with_partition  partition (ds='2') 
" +
-"SELECT key,value FROM testData")
-  sql("INSERT OVERWRITE TABLE table_with_partition  partition (ds='3') 
" +
-"SELECT key,value FROM testData")
-  sql("INSERT OVERWRITE TABLE table_with_partition  partition (ds='4') 
" +
-"SELECT key,value FROM testData")
-
-  // test for the exist path
-  checkAnswer(sql("select key,value from table_with_partition"),
-testData.toDF.collect ++ testData.toDF.collect
-  ++ testData.toDF.collect ++ testData.toDF.collect)
-
-  // delete the path of one partition
-  tmpDir.listFiles
-.find { f => f.isDirectory && f.getName().startsWith("ds=") }
-.foreach { f => Utils.deleteRecursively(f) }
-
-  // test for after delete the path
-  checkAnswer(sql("select key,value from table_with_partition"),
-testData.toDF.collect ++ testData.toDF.collect ++ 
testData.toDF.collect)
-
-  sql("DROP TABLE IF EXISTS table_with_partition")
-  sql("DROP TABLE IF EXISTS createAndInsertTest")
+withSQLConf((SQLConf.HIVE_VERIFY_PARTITION_PATH.key -> "true")) {
+  withTempView("testData") {
+withTable("table_with_partition", "createAndInsertTest") {
+  withTempPath { tmpDir =>
+tmpDir.mkdir()
--- End diff --

`withTempDir`, then you don't need to call `mkdir` here


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21091: [SPARK-22676][FOLLOW-UP] fix code style for test.

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/21091#discussion_r182422287
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/QueryPartitionSuite.scala ---
@@ -34,79 +34,83 @@ class QueryPartitionSuite extends QueryTest with 
SQLTestUtils with TestHiveSingl
   import spark.implicits._
 
   test("SPARK-5068: query data when path doesn't exist") {
-withSQLConf((SQLConf.HIVE_VERIFY_PARTITION_PATH.key, "true")) {
-  val testData = sparkContext.parallelize(
-(1 to 10).map(i => TestData(i, i.toString))).toDF()
-  testData.createOrReplaceTempView("testData")
-
-  val tmpDir = Files.createTempDir()
-  // create the table for test
-  sql(s"CREATE TABLE table_with_partition(key int,value string) " +
-s"PARTITIONED by (ds string) location '${tmpDir.toURI}' ")
-  sql("INSERT OVERWRITE TABLE table_with_partition  partition (ds='1') 
" +
-"SELECT key,value FROM testData")
-  sql("INSERT OVERWRITE TABLE table_with_partition  partition (ds='2') 
" +
-"SELECT key,value FROM testData")
-  sql("INSERT OVERWRITE TABLE table_with_partition  partition (ds='3') 
" +
-"SELECT key,value FROM testData")
-  sql("INSERT OVERWRITE TABLE table_with_partition  partition (ds='4') 
" +
-"SELECT key,value FROM testData")
-
-  // test for the exist path
-  checkAnswer(sql("select key,value from table_with_partition"),
-testData.toDF.collect ++ testData.toDF.collect
-  ++ testData.toDF.collect ++ testData.toDF.collect)
-
-  // delete the path of one partition
-  tmpDir.listFiles
-.find { f => f.isDirectory && f.getName().startsWith("ds=") }
-.foreach { f => Utils.deleteRecursively(f) }
-
-  // test for after delete the path
-  checkAnswer(sql("select key,value from table_with_partition"),
-testData.toDF.collect ++ testData.toDF.collect ++ 
testData.toDF.collect)
-
-  sql("DROP TABLE IF EXISTS table_with_partition")
-  sql("DROP TABLE IF EXISTS createAndInsertTest")
+withSQLConf((SQLConf.HIVE_VERIFY_PARTITION_PATH.key -> "true")) {
--- End diff --

we only need one pair of `()`


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21091: [SPARK-22676][FOLLOW-UP] fix code style for test.

[jinxing64]

GitHub user jinxing64 opened a pull request:

https://github.com/apache/spark/pull/21091

[SPARK-22676][FOLLOW-UP] fix code style for test.

## What changes were proposed in this pull request?

This pr address comments in https://github.com/apache/spark/pull/19868 ;
Fix the code style for `org.apache.spark.sql.hive.QueryPartitionSuite` by 
using:
`withTempView`, `withTempDir`, `withTable`...

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jinxing64/spark SPARK-22676-FOLLOW-UP

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/21091.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #21091


commit fd099bf1e1f254c5fe8616a45b2076c41043a474
Author: jinxing 
Date:   2018-04-18T03:35:10Z

[SPARK-22676][FOLLOW-UP] fix code style for test.




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org