I ran it on Databricks community edition which was a local[8] cluster with
6GB of RAM. It ran fine.
That said, looking at the plan, we can definitely simplify this quite a
bit. We had a new Window physical execution node for each window
expression, when we could have collapsed all of them into a s
Hi, I posted this to users, but didn’t get any responses.
I just wanted to highlight what seems like excessive memory use when using
sliding windows.
I have attached a test case where starting with certainly less than 1MB of data
I can OOM a 10G heap.
Regards,
-JD
--
import java
