Repository: spark
Updated Branches:
refs/heads/master 7c3e8995f -> 56e8f48a4
[SPARK-23695][PYTHON] Fix the error message for Kinesis streaming tests
## What changes were proposed in this pull request?
This PR proposes to fix the error message for Kinesis in PySpark when its jar
is missing but explicitly enabled.
```bash
ENABLE_KINESIS_TESTS=1 SPARK_TESTING=1 bin/pyspark pyspark.streaming.tests
```
Before:
```
Skipped test_flume_stream (enable by setting environment variable
ENABLE_FLUME_TESTS=1Skipped test_kafka_stream (enable by setting environment
variable ENABLE_KAFKA_0_8_TESTS=1Traceback (most recent call last):
File
"/usr/local/Cellar/python/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py",
line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File
"/usr/local/Cellar/python/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py",
line 72, in _run_code
exec code in run_globals
File "/.../spark/python/pyspark/streaming/tests.py", line 1572, in <module>
% kinesis_asl_assembly_dir) +
NameError: name 'kinesis_asl_assembly_dir' is not defined
```
After:
```
Skipped test_flume_stream (enable by setting environment variable
ENABLE_FLUME_TESTS=1Skipped test_kafka_stream (enable by setting environment
variable ENABLE_KAFKA_0_8_TESTS=1Traceback (most recent call last):
File
"/usr/local/Cellar/python/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py",
line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File
"/usr/local/Cellar/python/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py",
line 72, in _run_code
exec code in run_globals
File "/.../spark/python/pyspark/streaming/tests.py", line 1576, in <module>
"You need to build Spark with 'build/sbt -Pkinesis-asl "
Exception: Failed to find Spark Streaming Kinesis assembly jar in
/.../spark/external/kinesis-asl-assembly. You need to build Spark with
'build/sbt -Pkinesis-asl assembly/package
streaming-kinesis-asl-assembly/assembly'or 'build/mvn -Pkinesis-asl package'
before running this test.
```
## How was this patch tested?
Manually tested.
Author: hyukjinkwon <gurwls...@gmail.com>
Closes #20834 from HyukjinKwon/minor-variable.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/56e8f48a
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/56e8f48a
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/56e8f48a
Branch: refs/heads/master
Commit: 56e8f48a43eb51e8582db2461a585b13a771a00a
Parents: 7c3e899
Author: hyukjinkwon <gurwls...@gmail.com>
Authored: Thu Mar 15 10:55:33 2018 -0700
Committer: Takuya UESHIN <ues...@databricks.com>
Committed: Thu Mar 15 10:55:33 2018 -0700
----------------------------------------------------------------------
python/pyspark/streaming/tests.py | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/56e8f48a/python/pyspark/streaming/tests.py
----------------------------------------------------------------------
diff --git a/python/pyspark/streaming/tests.py
b/python/pyspark/streaming/tests.py
index 71f8101..7dde7c0 100644
--- a/python/pyspark/streaming/tests.py
+++ b/python/pyspark/streaming/tests.py
@@ -1503,10 +1503,13 @@ def search_flume_assembly_jar():
return jars[0]
-def search_kinesis_asl_assembly_jar():
+def _kinesis_asl_assembly_dir():
SPARK_HOME = os.environ["SPARK_HOME"]
- kinesis_asl_assembly_dir = os.path.join(SPARK_HOME,
"external/kinesis-asl-assembly")
- jars = search_jar(kinesis_asl_assembly_dir,
"spark-streaming-kinesis-asl-assembly")
+ return os.path.join(SPARK_HOME, "external/kinesis-asl-assembly")
+
+
+def search_kinesis_asl_assembly_jar():
+ jars = search_jar(_kinesis_asl_assembly_dir(),
"spark-streaming-kinesis-asl-assembly")
if not jars:
return None
elif len(jars) > 1:
@@ -1569,7 +1572,7 @@ if __name__ == "__main__":
else:
raise Exception(
("Failed to find Spark Streaming Kinesis assembly jar in %s. "
- % kinesis_asl_assembly_dir) +
+ % _kinesis_asl_assembly_dir()) +
"You need to build Spark with 'build/sbt -Pkinesis-asl "
"assembly/package streaming-kinesis-asl-assembly/assembly'"
"or 'build/mvn -Pkinesis-asl package' before running this test.")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
