Takuya Ueshin created SPARK-43323:
-------------------------------------
Summary: DataFrame.toPandas with Arrow enabled should handle
exceptions properly
Key: SPARK-43323
URL: https://issues.apache.org/jira/browse/SPARK-43323
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 3.4.0
Reporter: Takuya Ueshin
Currently {{DataFrame.toPandas}} doesn't capture exceptions happened in Spark
properly.
{code:python}
>>> spark.conf.set("spark.sql.ansi.enabled", True)
>>> spark.conf.set('spark.sql.execution.arrow.pyspark.enabled', True)
>>> spark.sql("select 1/0").toPandas()
...
An error occurred while calling o53.getResult.
: org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:322)
...
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
