Hi all,
I’m curious on how foreachbatch works in spark structured streaming. So since it is taking in a micro batch dataframe, that means the code in foreachbatch is executing on spark driver? Does this mean for large batches, you could potentially have OOM issues from collecting each partition into the driver? -- Cheers, Ruijing Li
