I would like to also add the same exact jobs on Flink 1.8 where running perfectly fine.
On Tue, 25 Feb 2020 at 00:20, John Smith <java.dev....@gmail.com> wrote: > Right after Job execution. Basically as soon as I deployed a 5th job. So > at 4 jobs it was ok, at 5 jobs it would take like 1-2 minutes max and the > node would just shut off. > So far with MaxMetaSpace 256m it's been stable. My task nodes are 16GB and > the memory config is done as follows... > taskmanager.memory.flink.size: 12g > taskmanager.memory.jvm-metaspace.size: 256m > > 100% of the jobs right now are ETL with checkpoints, NO state, > Kafka -----> Json Transform ----> DB > or > Kafka ----> DB lookup (to small local cache)--------> Json Transform > -----> Apache Ignite > > None of the jobs are related. > > On Mon, 24 Feb 2020 at 20:59, Xintong Song <tonysong...@gmail.com> wrote: > >> Hi John, >> >> The default metaspace size is intend for working with a major proportion >> of jobs. We are aware that for some jobs that need to load lots of classes, >> the default value might not be large enough. However, having a larger >> default value means for other jobs that do not load many classes, the >> overall memory requirements might be unnecessarily high. (Imagine you have >> a task manager with the default total memory 1.5GB, but 512m of it is >> reserved for metaspace.) >> >> Another possible problem is metaspace leak. When you say "eventually task >> nodes started shutting down with OutOfMemory Metaspace", does this problem >> happen shortly after the job execution starts, or does it happen after job >> running for a while? Does the metaspace footprint keep growing or become >> stable after the initial growth? If the metaspace keeps growing along with >> time, it's usually an indicator of metaspace memory leak. >> >> Thank you~ >> >> Xintong Song >> >> >> >> On Tue, Feb 25, 2020 at 7:50 AM John Smith <java.dev....@gmail.com> >> wrote: >> >>> Hi, I just upgraded to 1.10 and I started deploying my jobs. Eventually >>> task nodes started shutting down with OutOfMemory Metaspace. >>> >>> I look at the logs and the task managers are started with: >>> -XX:MaxMetaspaceSize=100663296 >>> >>> So I configed: taskmanager.memory.jvm-metaspace.size: 256m >>> >>> It seems to be ok for now. What are your thoughts? And should I try 512m >>> or is that too much? >>> >>
