I am seeing an issue with class loaders not being GCed and the metaspace
eventually OOM. Here is my setup:
- Flink 1.13.1 on EMR using JDK 8 in session mode
- Job manager is a long-running yarn session
- New jobs are submitted every 5m (and typically run for less than 5m)
I find that after a few
Hi!
As far as I remember this is a known issue a few years ago but Flink
currently has no solution to this (correct me if I'm wrong). I see that
you're running jobs on a yarn session. Could you switch to yarn-per-job
mode (where JM and TMs are created and destroyed for each job) for a
workaround?
Hi David,
If I understand the problem correctly, there is really nothing we can do
here. Soft references are garbage collected when there is a high memory
pressure and the garbage collector needs to free up more memory. The
problem here is that the GC doesn't really take high memory pressure on
Me
Thanks for the responses. I did switch to per-job mode and it is working
well of course. I suspected there wouldn't be an easy solution, but I had
to ask. Thanks!
On Fri, Jan 7, 2022 at 3:37 AM David Morávek
wrote:
> Hi David,
>
> If I understand the problem correctly, there is really nothing
