Hi Yuriy, How do you set your TaskMananger Memory? I think 40MB is not significant high for Flink. And It’s normal to see memory increase if you have more parallelism or set another metrics on. You can try setting larger moratory for Flink as explained by following documents.
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/memory/mem_setup/ Best Yuan > On 12 Aug 2022, at 12:51 AM, Yuriy Kutlunin > <yuriy.kutlu...@glowbyteconsulting.com> wrote: > > Hi all, > > I'm running Flink Cluster in Session Mode via docker-compose as stated in > docs: > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/#session-cluster-yml > > <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/#session-cluster-yml> > > After submitting a test job with many intermediate SQL operations (~500 > select * from ...) and metrics turned on (JMX or Prometheus) I got OOM: java > heap space on initialization stage. > > Turning metrics off allows the job to get to the Running state. > Heap consumption also depends on parallelism - same job succeeds when > submitted with parallelism 1 instead of 2. > > There are Task Manager logs for 4 cases: > JMX parallelism 1 (succeeded) > JMX parallelism 2 (failed) > Prometheus parallelism 2 (failed) > No metrics parallelism 2 (succeeded) > > Post OOM heap dump (JMX parallelism 2) shows 2 main consumption points: > 1. Big value (40MB) for some task configuration > 2. Many instances (~270k) of some heavy (20KB) value in StreamConfig > > Seems like all these heavy values are related to weird task names, which > includes all the operations: > Received task Source: source1 -> SourceConversion[2001] -> mapping1 -> > SourceConversion[2003] -> mapping2 -> SourceConversion[2005] -> ... -> > mapping500 -> Sink: sink1 (1/1)#0 (1e089cf3b1581ea7c8fb1cd7b159e66b) > > Looking for some way to overcome this heap issue. > > -- > Best regards, > Yuriy Kutlunin > <many_operators_parallelism_1_with_jmx.txt><many_operators_parallelism_2_with_jmx.txt><many_operators_parallelism_2_no_jmx.txt><many_operators_parallelism_2_with_prom.txt><heap_total.png><heap_task2_conf.png><heap_many_string_instances.png><heap_task1_conf.png>
