Craiu Constantin-Tiberiu created SPARK-61455:
------------------------------------------------
Summary: ython worker drops exception when message contains lone
surrogate
Key: SPARK-61455
URL: https://issues.apache.org/jira/browse/SPARK-61455
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 4.4.0
Reporter: Craiu Constantin-Tiberiu
When Python UDF raises exception which contains a lone surrogate this ends up
raising a {{UnicodeEncodeError}} during propagation, which is only printed to
stderr.
JVM side reports "Python worker exited unexpectedly (crashed)" instead of the
user's error.
Example:
```
def boom():
raise ValueError("bad path: " + b"/data/\xff.csv".decode("utf-8",
"surrogateescape"))
spark.udf.register("boom", boom)
spark.sql("SELECT boom()").collect()
```
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]