Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21007#discussion_r180383755
--- Diff: python/pyspark/sql/tests.py ---
@@ -3062,6 +3062,73 @@ def
test_sparksession_with_stopped_sparkcontext(self):
sc.stop()
+class SQLTests3(unittest.TestCase):
--- End diff --
I manually tested different conditions with this test for sure:
Before:
```
test_query_execution_listener_on_collect (pyspark.sql.tests.SQLTests3) ...
FAIL
test_query_execution_listener_on_collect_with_arrow
(pyspark.sql.tests.SQLTests3) ... FAIL
======================================================================
FAIL: test_query_execution_listener_on_collect (pyspark.sql.tests.SQLTests3)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/.../spark/python/pyspark/sql/tests.py", line 3105, in
test_query_execution_listener_on_collect
"The callback from the query execution listener should be called after
'collect'")
AssertionError: The callback from the query execution listener should be
called after 'collect'
======================================================================
FAIL: test_query_execution_listener_on_collect_with_arrow
(pyspark.sql.tests.SQLTests3)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/.../spark/python/pyspark/sql/tests.py", line 3122, in
test_query_execution_listener_on_collect_with_arrow
"The callback from the query execution listener should be called after
'toPandas'")
AssertionError: The callback from the query execution listener should be
called after 'toPandas'
```
After:
```
test_query_execution_listener_on_collect (pyspark.sql.tests.SQLTests3) ...
ok
test_query_execution_listener_on_collect_with_arrow
(pyspark.sql.tests.SQLTests3) ... ok
```
Missing 'org.apache.spark.sql.TestQueryExecutionListener'
```
skipped "'org.apache.spark.sql.TestQueryExecutionListener' is not
available. Skipping the related tests."
```
Missing Pandas
```
test_query_execution_listener_on_collect (pyspark.sql.tests.SQLTests3) ...
ok
test_query_execution_listener_on_collect_with_arrow
(pyspark.sql.tests.SQLTests3) ... skipped 'Pandas >= 0.19.2 must be installed;
however, it was not found.'
```
Missing PyArrow
```
test_query_execution_listener_on_collect (pyspark.sql.tests.SQLTests3) ...
ok
test_query_execution_listener_on_collect_with_arrow
(pyspark.sql.tests.SQLTests3) ... skipped 'PyArrow >= 0.8.0 must be installed;
however, it was not found.'
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
