Hyukjin Kwon created SPARK-32186:
------------------------------------
Summary: User Guide - Debugging
Key: SPARK-32186
URL: https://issues.apache.org/jira/browse/SPARK-32186
Project: Spark
Issue Type: Sub-task
Components: Documentation, PySpark
Affects Versions: 3.1.0
Reporter: Hyukjin Kwon
1. Python Profiler:
https://spark.apache.org/docs/2.3.0/api/python/_modules/pyspark/profiler.html,
{code}
>>> sc._conf.set("spark.python.profile", "true")
>>> rdd = sc.parallelize(range(100)).map(str)
>>> rdd.count()
100
>>> sc.show_profiles()
{code}
2. Python Debugger, e.g.)
https://github.com/holdenk/remote-python-debugging-4-spark
3. Monitoring Python Workers: {{top}}, `{{ls}}`, etc.
4. ...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
