PySpark: batch_df in ForeachBatch - aggregation

Hi, Using the master branch, I tried to perform SQL aggregation on batch_df in foreachBatch and only SQL API methods work but not spark sql queries on the temp table (register as a table or view createOrReplaceTempView). Is it supported? I really appreciate your help. -- Sent from:

PySpark structured streaming job throws socket exception

Hi, Running Pyspark structured streaming job on K8S with 2 executor pods. The driver pod failed with the following up exception. It fails consistently after 3 to 6hrs of running. Any idea how to fix this exception. I really appreciate your help. 2018-10-04 18:48:27 ERROR DAGScheduler:91 -

Re: PySpark structured streaming job throws socket exception

Thanks Ryan. Attached the whole stack trace. Let me know if you need more information. pyspark-driver-pod-exception.txt -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/