Re: PySpark structured streaming job throws socket exception

Thanks Ryan. Attached the whole stack trace. Let me know if you need more information. pyspark-driver-pod-exception.txt -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

Re: PySpark structured streaming job throws socket exception

As far as I know, the error log in updateAccumulators will not fail a Spark task. Did you see other error messages? Best Regards, Ryan On Thu, Oct 4, 2018 at 2:14 PM mmuru wrote: > Hi, > > Running Pyspark structured streaming job on K8S with 2 executor pods. The > driver pod failed with the

PySpark structured streaming job throws socket exception

Hi, Running Pyspark structured streaming job on K8S with 2 executor pods. The driver pod failed with the following up exception. It fails consistently after 3 to 6hrs of running. Any idea how to fix this exception. I really appreciate your help. 2018-10-04 18:48:27 ERROR DAGScheduler:91 -