[GitHub] [spark] HyukjinKwon commented on a change in pull request #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
HyukjinKwon commented on a change in pull request #25684:
[SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in
Spark 2.2.0 or earlier, for Spark 3
URL: https://github.com/apache/spark/pull/25684#discussion_r321573558
##
File path: python/pyspark/ml/tests/test_image.py
##
@@ -67,47 +67,6 @@ def test_read_images(self):
lambda: ImageSchema.toImage("a"))
-class ImageFileFormatOnHiveContextTest(PySparkTestCase):
Review comment:
@srowen, this was removed (see
https://github.com/apache/spark/pull/25684#discussion_r321565529)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
HyukjinKwon commented on a change in pull request #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#discussion_r321573454 ## File path: python/pyspark/sql/tests/test_context.py ## @@ -40,15 +40,20 @@ def setUpClass(cls): ReusedPySparkTestCase.setUpClass() cls.tempdir = tempfile.NamedTemporaryFile(delete=False) cls.hive_available = True +cls.spark = None try: cls.sc._jvm.org.apache.hadoop.hive.conf.HiveConf() Review comment: @srowen, seems this code is still needed. `SparkSession.builder.enableHiveSupport().getOrCreate()` seems not able to directly check if Hive is available or not. I manually tested and just pushed the changes directly into your branch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
HyukjinKwon commented on a change in pull request #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#discussion_r321566030 ## File path: python/pyspark/sql/tests/test_context.py ## @@ -41,14 +41,13 @@ def setUpClass(cls): cls.tempdir = tempfile.NamedTemporaryFile(delete=False) cls.hive_available = True try: -cls.sc._jvm.org.apache.hadoop.hive.conf.HiveConf() +cls.spark = SparkSession.builder.enableHiveSupport().getOrCreate() Review comment: Ah, here there's another occurrence. Yes, I think it's correct. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
HyukjinKwon commented on a change in pull request #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#discussion_r321566030 ## File path: python/pyspark/sql/tests/test_context.py ## @@ -41,14 +41,13 @@ def setUpClass(cls): cls.tempdir = tempfile.NamedTemporaryFile(delete=False) cls.hive_available = True try: -cls.sc._jvm.org.apache.hadoop.hive.conf.HiveConf() +cls.spark = SparkSession.builder.enableHiveSupport().getOrCreate() Review comment: Ah, here there's another occurrence. Yes, I think it's correct. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
HyukjinKwon commented on a change in pull request #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#discussion_r321565844 ## File path: python/pyspark/sql/tests/test_context.py ## @@ -41,14 +41,13 @@ def setUpClass(cls): cls.tempdir = tempfile.NamedTemporaryFile(delete=False) cls.hive_available = True try: -cls.sc._jvm.org.apache.hadoop.hive.conf.HiveConf() +cls.spark = SparkSession.builder.enableHiveSupport().getOrCreate() except py4j.protocol.Py4JError: cls.hive_available = False except TypeError: cls.hive_available = False os.unlink(cls.tempdir.name) if cls.hive_available: -cls.spark = HiveContext._createForTesting(cls.sc) Review comment: Ah, here's another occurrence. I think we can just replace this line to: ```python cls.spark = SparkSession.builder.enableHiveSupport().getOrCreate() ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
HyukjinKwon commented on a change in pull request #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#discussion_r321565844 ## File path: python/pyspark/sql/tests/test_context.py ## @@ -41,14 +41,13 @@ def setUpClass(cls): cls.tempdir = tempfile.NamedTemporaryFile(delete=False) cls.hive_available = True try: -cls.sc._jvm.org.apache.hadoop.hive.conf.HiveConf() +cls.spark = SparkSession.builder.enableHiveSupport().getOrCreate() except py4j.protocol.Py4JError: cls.hive_available = False except TypeError: cls.hive_available = False os.unlink(cls.tempdir.name) if cls.hive_available: -cls.spark = HiveContext._createForTesting(cls.sc) Review comment: Ah, here's another occurrence. I think we can just replace this line to: ```python cls.spark = SparkSession.builder.enableHiveSupport().getOrCreate() ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
HyukjinKwon commented on a change in pull request #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#discussion_r321565529 ## File path: python/pyspark/ml/tests/test_image.py ## @@ -74,17 +74,14 @@ def setUpClass(cls): super(ImageFileFormatOnHiveContextTest, cls).setUpClass() cls.hive_available = True # Note that here we enable Hive's support. -cls.spark = None try: -cls.sc._jvm.org.apache.hadoop.hive.conf.HiveConf() +cls.spark = SparkSession.builder.enableHiveSupport().getOrCreate() Review comment: @srowen, I think actually you can remove this whole class, `ImageFileFormatOnHiveContextTest`. This test was added as of [SPARK-22651](https://jira.apache.org/jira/browse/SPARK-22651) to test `ImageSchema.readImages` but it was removed as of [SPARK-25382](https://jira.apache.org/jira/browse/SPARK-25382). So, I think it's fine to remove this whole test class. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
HyukjinKwon commented on a change in pull request #25684:
[SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in
Spark 2.2.0 or earlier, for Spark 3
URL: https://github.com/apache/spark/pull/25684#discussion_r321174033
##
File path: python/pyspark/sql/tests/test_context.py
##
@@ -66,7 +66,7 @@ def test_save_and_load_table(self):
tmpPath = tempfile.mkdtemp()
shutil.rmtree(tmpPath)
df.write.saveAsTable("savedJsonTable", "json", "append", path=tmpPath)
-actual = self.spark.createExternalTable("externalJsonTable", tmpPath,
"json")
Review comment:
Those seems can be reverted back since the tests target HiveContext.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
