Takuya Ueshin created SPARK-43323: ------------------------------------- Summary: DataFrame.toPandas with Arrow enabled should handle exceptions properly Key: SPARK-43323 URL: https://issues.apache.org/jira/browse/SPARK-43323 Project: Spark Issue Type: Improvement Components: PySpark Affects Versions: 3.4.0 Reporter: Takuya Ueshin
Currently {{DataFrame.toPandas}} doesn't capture exceptions happened in Spark properly. {code:python} >>> spark.conf.set("spark.sql.ansi.enabled", True) >>> spark.conf.set('spark.sql.execution.arrow.pyspark.enabled', True) >>> spark.sql("select 1/0").toPandas() ... An error occurred while calling o53.getResult. : org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:322) ... {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org