Hyukjin Kwon created SPARK-23517:
------------------------------------
Summary: Make pyspark.util._exception_message produce the trace
from Java side for Py4JJavaError
Key: SPARK-23517
URL: https://issues.apache.org/jira/browse/SPARK-23517
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 2.3.0
Reporter: Hyukjin Kwon
Currently {{pyspark.util._exception_message}} doesn't show its trace and
message from Py4JJavaError as below:
{code}
>>> from pyspark.util import _exception_message
>>> try:
... sc._jvm.java.lang.String(None)
... except Exception as e:
... pass
...
>>> e.message
''
{code}
This is actually a problem in some code paths we can expect this error. For
example, see
{code}
from pyspark.sql.functions import udf
spark.conf.set("spark.sql.execution.arrow.enabled", True)
spark.range(1).select(udf(lambda x: [[]])()).toPandas()
{code}
{code}
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/.../spark/python/pyspark/sql/dataframe.py", line 2009, in toPandas
raise RuntimeError("%s\n%s" % (_exception_message(e), msg))
RuntimeError:
Note: toPandas attempted Arrow optimization because
'spark.sql.execution.arrow.enabled' is set to true. Please set it to false to
disable this.
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
