We started hitting this as well, seeing 90+ GB resident memory on a 25 GB
heap executor. After a lot of manually testing fixes, I finally figured out
the root problem: https://issues.apache.org/jira/browse/SPARK-41339
Starting to work on a PR now to fix.
On Mon, Sep 12, 2022 at 10:46 AM Artemis
The off-heap memory isn't subjected to GC. So the obvious reason is
that your have too many states to maintain in your streaming app, and
the GC couldn't keep up, and end up with resources but to die. Are you
using continues processing or microbatch in structured streaming? You
may want to
Hi Team,
We are trying to shift from HDFS State Manager to Rocks DB State Manager,
but while doing POC we realised it is using much more off-heap space than
expected. Because of this, the executors get killed with : *out of**
physical memory exception.*
Could you please help in understanding,
