Can you try calling batchDF.unpersist() once the work is done in loop?
On Mon, Jul 20, 2020 at 3:38 PM Yong Yuan wrote:
> It seems the following structured streaming code keeps on consuming
> usercache until all disk space are occupied.
>
> val monitoring_stream =
>
It seems the following structured streaming code keeps on consuming
usercache until all disk space are occupied.
val monitoring_stream =
monitoring_df.writeStream
.trigger(Trigger.ProcessingTime("120 seconds"))
.foreachBatch {
(batchDF: DataFrame,
