SHOBHIT SHUKLA created SPARK-39752:
--------------------------------------
Summary: Spark job failed with 10M rows data with Broken pipe error
Key: SPARK-39752
URL: https://issues.apache.org/jira/browse/SPARK-39752
Project: Spark
Issue Type: Bug
Components: PySpark, Spark Core
Affects Versions: 3.2.1, 3.0.3
Reporter: SHOBHIT SHUKLA
Fix For: 3.0.2
Spark job failed with 10M rows data with Broken pipe error. Same spark job was
working previously with the settings "executor_cores": 1, "executor_memory": 1,
"driver_cores": 1, "driver_memory": 1. where as the same job is failing with
spark settings in 3.0.3 and 3.2.1.
Major symptoms (slowness, timeout, out of memory as examples): Spark job is
failing with the error java.net.SocketException: Broken pipe (Write failed)
Here are the spark settings information which is working on Spark 3.0.3 and
3.2.1 : "executor_cores": 4, "executor_memory": 4, "driver_cores": 4,
"driver_memory": 4.. The spark job doesn't consistently works with the above
settings. Some times, need to increase the cores and memory.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
