spark git commit: [SPARK-22043][PYTHON] Improves error message for show_profiles and dump_profiles
Repository: spark Updated Branches: refs/heads/branch-2.2 309c401a5 -> a86831d61 [SPARK-22043][PYTHON] Improves error message for show_profiles and dump_profiles ## What changes were proposed in this pull request? This PR proposes to improve error message from: ``` >>> sc.show_profiles() Traceback (most recent call last): File "", line 1, in File ".../spark/python/pyspark/context.py", line 1000, in show_profiles self.profiler_collector.show_profiles() AttributeError: 'NoneType' object has no attribute 'show_profiles' >>> sc.dump_profiles("/tmp/abc") Traceback (most recent call last): File "", line 1, in File ".../spark/python/pyspark/context.py", line 1005, in dump_profiles self.profiler_collector.dump_profiles(path) AttributeError: 'NoneType' object has no attribute 'dump_profiles' ``` to ``` >>> sc.show_profiles() Traceback (most recent call last): File "", line 1, in File ".../spark/python/pyspark/context.py", line 1003, in show_profiles raise RuntimeError("'spark.python.profile' configuration must be set " RuntimeError: 'spark.python.profile' configuration must be set to 'true' to enable Python profile. >>> sc.dump_profiles("/tmp/abc") Traceback (most recent call last): File "", line 1, in File ".../spark/python/pyspark/context.py", line 1012, in dump_profiles raise RuntimeError("'spark.python.profile' configuration must be set " RuntimeError: 'spark.python.profile' configuration must be set to 'true' to enable Python profile. ``` ## How was this patch tested? Unit tests added in `python/pyspark/tests.py` and manual tests. Author: hyukjinkwon Closes #19260 from HyukjinKwon/profile-errors. (cherry picked from commit 7c7266208a3be984ac1ce53747dc0c3640f4ecac) Signed-off-by: hyukjinkwon Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a86831d6 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a86831d6 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a86831d6 Branch: refs/heads/branch-2.2 Commit: a86831d618b05c789c2cea0afe5488c3234a14bc Parents: 309c401 Author: hyukjinkwon Authored: Mon Sep 18 13:20:11 2017 +0900 Committer: hyukjinkwon Committed: Mon Sep 18 13:20:29 2017 +0900 -- python/pyspark/context.py | 12 ++-- python/pyspark/tests.py | 16 2 files changed, 26 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/a86831d6/python/pyspark/context.py -- diff --git a/python/pyspark/context.py b/python/pyspark/context.py index 49be76e..ea58b3a 100644 --- a/python/pyspark/context.py +++ b/python/pyspark/context.py @@ -994,12 +994,20 @@ class SparkContext(object): def show_profiles(self): """ Print the profile stats to stdout """ -self.profiler_collector.show_profiles() +if self.profiler_collector is not None: +self.profiler_collector.show_profiles() +else: +raise RuntimeError("'spark.python.profile' configuration must be set " + "to 'true' to enable Python profile.") def dump_profiles(self, path): """ Dump the profile stats into directory `path` """ -self.profiler_collector.dump_profiles(path) +if self.profiler_collector is not None: +self.profiler_collector.dump_profiles(path) +else: +raise RuntimeError("'spark.python.profile' configuration must be set " + "to 'true' to enable Python profile.") def getConf(self): conf = SparkConf() http://git-wip-us.apache.org/repos/asf/spark/blob/a86831d6/python/pyspark/tests.py -- diff --git a/python/pyspark/tests.py b/python/pyspark/tests.py index 9f47798..6a96aaf 100644 --- a/python/pyspark/tests.py +++ b/python/pyspark/tests.py @@ -1288,6 +1288,22 @@ class ProfilerTests(PySparkTestCase): rdd.foreach(heavy_foo) +class ProfilerTests2(unittest.TestCase): +def test_profiler_disabled(self): +sc = SparkContext(conf=SparkConf().set("spark.python.profile", "false")) +try: +self.assertRaisesRegexp( +RuntimeError, +"'spark.python.profile' configuration must be set", +lambda: sc.show_profiles()) +self.assertRaisesRegexp( +RuntimeError, +"'spark.python.profile' configuration must be set", +lambda: sc.dump_profiles("/tmp/abc")) +finally: +sc.stop() + + class InputFormatTests(ReusedPySparkTestCase): @classmethod - To unsubscribe, e-mail: commits-unsubscr...@s
spark git commit: [SPARK-22043][PYTHON] Improves error message for show_profiles and dump_profiles
Repository: spark Updated Branches: refs/heads/branch-2.1 99de4b8f5 -> b35136a9e [SPARK-22043][PYTHON] Improves error message for show_profiles and dump_profiles ## What changes were proposed in this pull request? This PR proposes to improve error message from: ``` >>> sc.show_profiles() Traceback (most recent call last): File "", line 1, in File ".../spark/python/pyspark/context.py", line 1000, in show_profiles self.profiler_collector.show_profiles() AttributeError: 'NoneType' object has no attribute 'show_profiles' >>> sc.dump_profiles("/tmp/abc") Traceback (most recent call last): File "", line 1, in File ".../spark/python/pyspark/context.py", line 1005, in dump_profiles self.profiler_collector.dump_profiles(path) AttributeError: 'NoneType' object has no attribute 'dump_profiles' ``` to ``` >>> sc.show_profiles() Traceback (most recent call last): File "", line 1, in File ".../spark/python/pyspark/context.py", line 1003, in show_profiles raise RuntimeError("'spark.python.profile' configuration must be set " RuntimeError: 'spark.python.profile' configuration must be set to 'true' to enable Python profile. >>> sc.dump_profiles("/tmp/abc") Traceback (most recent call last): File "", line 1, in File ".../spark/python/pyspark/context.py", line 1012, in dump_profiles raise RuntimeError("'spark.python.profile' configuration must be set " RuntimeError: 'spark.python.profile' configuration must be set to 'true' to enable Python profile. ``` ## How was this patch tested? Unit tests added in `python/pyspark/tests.py` and manual tests. Author: hyukjinkwon Closes #19260 from HyukjinKwon/profile-errors. (cherry picked from commit 7c7266208a3be984ac1ce53747dc0c3640f4ecac) Signed-off-by: hyukjinkwon Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b35136a9 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b35136a9 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b35136a9 Branch: refs/heads/branch-2.1 Commit: b35136a9e4b72b403434c991e111e667cfe9177d Parents: 99de4b8 Author: hyukjinkwon Authored: Mon Sep 18 13:20:11 2017 +0900 Committer: hyukjinkwon Committed: Mon Sep 18 13:20:48 2017 +0900 -- python/pyspark/context.py | 12 ++-- python/pyspark/tests.py | 16 2 files changed, 26 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/b35136a9/python/pyspark/context.py -- diff --git a/python/pyspark/context.py b/python/pyspark/context.py index 5a4c2fa..c091882 100644 --- a/python/pyspark/context.py +++ b/python/pyspark/context.py @@ -970,12 +970,20 @@ class SparkContext(object): def show_profiles(self): """ Print the profile stats to stdout """ -self.profiler_collector.show_profiles() +if self.profiler_collector is not None: +self.profiler_collector.show_profiles() +else: +raise RuntimeError("'spark.python.profile' configuration must be set " + "to 'true' to enable Python profile.") def dump_profiles(self, path): """ Dump the profile stats into directory `path` """ -self.profiler_collector.dump_profiles(path) +if self.profiler_collector is not None: +self.profiler_collector.dump_profiles(path) +else: +raise RuntimeError("'spark.python.profile' configuration must be set " + "to 'true' to enable Python profile.") def getConf(self): conf = SparkConf() http://git-wip-us.apache.org/repos/asf/spark/blob/b35136a9/python/pyspark/tests.py -- diff --git a/python/pyspark/tests.py b/python/pyspark/tests.py index bd21029..61272fe 100644 --- a/python/pyspark/tests.py +++ b/python/pyspark/tests.py @@ -1223,6 +1223,22 @@ class ProfilerTests(PySparkTestCase): rdd.foreach(heavy_foo) +class ProfilerTests2(unittest.TestCase): +def test_profiler_disabled(self): +sc = SparkContext(conf=SparkConf().set("spark.python.profile", "false")) +try: +self.assertRaisesRegexp( +RuntimeError, +"'spark.python.profile' configuration must be set", +lambda: sc.show_profiles()) +self.assertRaisesRegexp( +RuntimeError, +"'spark.python.profile' configuration must be set", +lambda: sc.dump_profiles("/tmp/abc")) +finally: +sc.stop() + + class InputFormatTests(ReusedPySparkTestCase): @classmethod - To unsubscribe, e-mail: commits-unsubscr...@s
spark git commit: [SPARK-22043][PYTHON] Improves error message for show_profiles and dump_profiles
Repository: spark Updated Branches: refs/heads/master 6308c65f0 -> 7c7266208 [SPARK-22043][PYTHON] Improves error message for show_profiles and dump_profiles ## What changes were proposed in this pull request? This PR proposes to improve error message from: ``` >>> sc.show_profiles() Traceback (most recent call last): File "", line 1, in File ".../spark/python/pyspark/context.py", line 1000, in show_profiles self.profiler_collector.show_profiles() AttributeError: 'NoneType' object has no attribute 'show_profiles' >>> sc.dump_profiles("/tmp/abc") Traceback (most recent call last): File "", line 1, in File ".../spark/python/pyspark/context.py", line 1005, in dump_profiles self.profiler_collector.dump_profiles(path) AttributeError: 'NoneType' object has no attribute 'dump_profiles' ``` to ``` >>> sc.show_profiles() Traceback (most recent call last): File "", line 1, in File ".../spark/python/pyspark/context.py", line 1003, in show_profiles raise RuntimeError("'spark.python.profile' configuration must be set " RuntimeError: 'spark.python.profile' configuration must be set to 'true' to enable Python profile. >>> sc.dump_profiles("/tmp/abc") Traceback (most recent call last): File "", line 1, in File ".../spark/python/pyspark/context.py", line 1012, in dump_profiles raise RuntimeError("'spark.python.profile' configuration must be set " RuntimeError: 'spark.python.profile' configuration must be set to 'true' to enable Python profile. ``` ## How was this patch tested? Unit tests added in `python/pyspark/tests.py` and manual tests. Author: hyukjinkwon Closes #19260 from HyukjinKwon/profile-errors. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7c726620 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7c726620 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7c726620 Branch: refs/heads/master Commit: 7c7266208a3be984ac1ce53747dc0c3640f4ecac Parents: 6308c65 Author: hyukjinkwon Authored: Mon Sep 18 13:20:11 2017 +0900 Committer: hyukjinkwon Committed: Mon Sep 18 13:20:11 2017 +0900 -- python/pyspark/context.py | 12 ++-- python/pyspark/tests.py | 16 2 files changed, 26 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/7c726620/python/pyspark/context.py -- diff --git a/python/pyspark/context.py b/python/pyspark/context.py index a704604..a33f6dc 100644 --- a/python/pyspark/context.py +++ b/python/pyspark/context.py @@ -997,12 +997,20 @@ class SparkContext(object): def show_profiles(self): """ Print the profile stats to stdout """ -self.profiler_collector.show_profiles() +if self.profiler_collector is not None: +self.profiler_collector.show_profiles() +else: +raise RuntimeError("'spark.python.profile' configuration must be set " + "to 'true' to enable Python profile.") def dump_profiles(self, path): """ Dump the profile stats into directory `path` """ -self.profiler_collector.dump_profiles(path) +if self.profiler_collector is not None: +self.profiler_collector.dump_profiles(path) +else: +raise RuntimeError("'spark.python.profile' configuration must be set " + "to 'true' to enable Python profile.") def getConf(self): conf = SparkConf() http://git-wip-us.apache.org/repos/asf/spark/blob/7c726620/python/pyspark/tests.py -- diff --git a/python/pyspark/tests.py b/python/pyspark/tests.py index 3c108ec..da99872 100644 --- a/python/pyspark/tests.py +++ b/python/pyspark/tests.py @@ -1296,6 +1296,22 @@ class ProfilerTests(PySparkTestCase): rdd.foreach(heavy_foo) +class ProfilerTests2(unittest.TestCase): +def test_profiler_disabled(self): +sc = SparkContext(conf=SparkConf().set("spark.python.profile", "false")) +try: +self.assertRaisesRegexp( +RuntimeError, +"'spark.python.profile' configuration must be set", +lambda: sc.show_profiles()) +self.assertRaisesRegexp( +RuntimeError, +"'spark.python.profile' configuration must be set", +lambda: sc.dump_profiles("/tmp/abc")) +finally: +sc.stop() + + class InputFormatTests(ReusedPySparkTestCase): @classmethod - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org