Repository: spark
Updated Branches:
refs/heads/branch-2.2 309c401a5 -> a86831d61
[SPARK-22043][PYTHON] Improves error message for show_profiles and dump_profiles
## What changes were proposed in this pull request?
This PR proposes to improve error message from:
```
>>> sc.show_profiles()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ".../spark/python/pyspark/context.py", line 1000, in show_profiles
self.profiler_collector.show_profiles()
AttributeError: 'NoneType' object has no attribute 'show_profiles'
>>> sc.dump_profiles("/tmp/abc")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ".../spark/python/pyspark/context.py", line 1005, in dump_profiles
self.profiler_collector.dump_profiles(path)
AttributeError: 'NoneType' object has no attribute 'dump_profiles'
```
to
```
>>> sc.show_profiles()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ".../spark/python/pyspark/context.py", line 1003, in show_profiles
raise RuntimeError("'spark.python.profile' configuration must be set "
RuntimeError: 'spark.python.profile' configuration must be set to 'true' to
enable Python profile.
>>> sc.dump_profiles("/tmp/abc")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ".../spark/python/pyspark/context.py", line 1012, in dump_profiles
raise RuntimeError("'spark.python.profile' configuration must be set "
RuntimeError: 'spark.python.profile' configuration must be set to 'true' to
enable Python profile.
```
## How was this patch tested?
Unit tests added in `python/pyspark/tests.py` and manual tests.
Author: hyukjinkwon <gurwls...@gmail.com>
Closes #19260 from HyukjinKwon/profile-errors.
(cherry picked from commit 7c7266208a3be984ac1ce53747dc0c3640f4ecac)
Signed-off-by: hyukjinkwon <gurwls...@gmail.com>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a86831d6
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a86831d6
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a86831d6
Branch: refs/heads/branch-2.2
Commit: a86831d618b05c789c2cea0afe5488c3234a14bc
Parents: 309c401
Author: hyukjinkwon <gurwls...@gmail.com>
Authored: Mon Sep 18 13:20:11 2017 +0900
Committer: hyukjinkwon <gurwls...@gmail.com>
Committed: Mon Sep 18 13:20:29 2017 +0900
----------------------------------------------------------------------
python/pyspark/context.py | 12 ++++++++++--
python/pyspark/tests.py | 16 ++++++++++++++++
2 files changed, 26 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/a86831d6/python/pyspark/context.py
----------------------------------------------------------------------
diff --git a/python/pyspark/context.py b/python/pyspark/context.py
index 49be76e..ea58b3a 100644
--- a/python/pyspark/context.py
+++ b/python/pyspark/context.py
@@ -994,12 +994,20 @@ class SparkContext(object):
def show_profiles(self):
""" Print the profile stats to stdout """
- self.profiler_collector.show_profiles()
+ if self.profiler_collector is not None:
+ self.profiler_collector.show_profiles()
+ else:
+ raise RuntimeError("'spark.python.profile' configuration must be
set "
+ "to 'true' to enable Python profile.")
def dump_profiles(self, path):
""" Dump the profile stats into directory `path`
"""
- self.profiler_collector.dump_profiles(path)
+ if self.profiler_collector is not None:
+ self.profiler_collector.dump_profiles(path)
+ else:
+ raise RuntimeError("'spark.python.profile' configuration must be
set "
+ "to 'true' to enable Python profile.")
def getConf(self):
conf = SparkConf()
http://git-wip-us.apache.org/repos/asf/spark/blob/a86831d6/python/pyspark/tests.py
----------------------------------------------------------------------
diff --git a/python/pyspark/tests.py b/python/pyspark/tests.py
index 9f47798..6a96aaf 100644
--- a/python/pyspark/tests.py
+++ b/python/pyspark/tests.py
@@ -1288,6 +1288,22 @@ class ProfilerTests(PySparkTestCase):
rdd.foreach(heavy_foo)
+class ProfilerTests2(unittest.TestCase):
+ def test_profiler_disabled(self):
+ sc = SparkContext(conf=SparkConf().set("spark.python.profile",
"false"))
+ try:
+ self.assertRaisesRegexp(
+ RuntimeError,
+ "'spark.python.profile' configuration must be set",
+ lambda: sc.show_profiles())
+ self.assertRaisesRegexp(
+ RuntimeError,
+ "'spark.python.profile' configuration must be set",
+ lambda: sc.dump_profiles("/tmp/abc"))
+ finally:
+ sc.stop()
+
+
class InputFormatTests(ReusedPySparkTestCase):
@classmethod
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
