[ https://issues.apache.org/jira/browse/SPARK-39752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuming Wang updated SPARK-39752: -------------------------------- Fix Version/s: (was: 3.0.2) > Spark job failed with 10M rows data with Broken pipe error > ---------------------------------------------------------- > > Key: SPARK-39752 > URL: https://issues.apache.org/jira/browse/SPARK-39752 > Project: Spark > Issue Type: Bug > Components: PySpark, Spark Core > Affects Versions: 3.0.3, 3.2.1 > Reporter: SHOBHIT SHUKLA > Priority: Major > Attachments: Failed_spark_job_3.0.3.txt, spark_job_success_3.0.2.txt > > > Spark job failed with 10M rows data with Broken pipe error. Same spark job > was working previously with the settings "executor_cores": 1, > "executor_memory": 1, "driver_cores": 1, "driver_memory": 1. where as the > same job is failing with spark settings in 3.0.3 and 3.2.1. > Major symptoms (slowness, timeout, out of memory as examples): Spark job is > failing with the error java.net.SocketException: Broken pipe (Write failed) > Here are the spark settings information which is working on Spark 3.0.3 and > 3.2.1 : "executor_cores": 4, "executor_memory": 4, "driver_cores": 4, > "driver_memory": 4.. The spark job doesn't consistently works with the above > settings. Some times, need to increase the cores and memory. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org