[Spark Structured Streaming, Spark 2.3.0] Calling current_timestamp() function within a streaming dataframe results in dataType error

. option("truncate", false). trigger(Trigger.ProcessingTime(10.seconds)). start(). awaitTermination() ``` Can anyone confirm the same behavior? Respectfully, Artem Moskvin

Why Spark Streaming keeps all batches in memory after processing?

Hello there! I wonder why Spark Streaming keeps all processed batches in memory? It leads to getting out of memory on executors but I really don't need them after processing. Can it be configured somewhere so that batches are not kept in memory after processing? Respectfully, Artem Moskvin