Flink TaskManager memory configuration failed
Hi all, I'm wanting to run quite a number of PyFlink jobs on Kubernetes, where the amount of state and number of events being processed is small and therefore I'd like to use as little memory in my clusters as possible so I can bin pack most efficiently. I'm running a Flink cluster per job. I'm currently trying to see how small I can make the memory settings. I set taskmanager.memory.process.size: 512mb in my Flink config. The container is being started with requested memory of 512Mi. When my TaskManager starts up, I get an error message: IllegalConfigurationException: Sum of configured Framework Heap Memory (128mb), Framework Off-Heap Memory (128mb), Task Off-Heap Memory (0 bytes), Managed Memory (25.6mb) and Network Memory (64mb) exceed configured Total Flink Memory (64mb). My understanding of the docs (https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/memory/mem_setup_tm/) is that I should just need to set taskmanager.memory.process.size. Where is the 64mb figure coming from which makes up the Total Flink Memory? How do I change this? Many thanks, John
Re: Flink TaskManager memory configuration failed
Hi John, May I know what is your Flink version you are trying? On Thu, 23 Jun 2022 at 3:43 AM, John Tipper wrote: > Hi all, > > I'm wanting to run quite a number of PyFlink jobs on Kubernetes, where the > amount of state and number of events being processed is small and therefore > I'd like to use as little memory in my clusters as possible so I can bin > pack most efficiently. I'm running a Flink cluster per job. I'm currently > trying to see how small I can make the memory settings. > > I set taskmanager.memory.process.size: 512mb in my Flink config. The > container is being started with requested memory of 512Mi. > > When my TaskManager starts up, I get an error message: > > IllegalConfigurationException: Sum of configured Framework Heap Memory > (128mb), Framework Off-Heap Memory (128mb), Task Off-Heap Memory (0 > bytes), Managed Memory (25.6mb) and Network Memory (64mb) exceed configured > Total Flink Memory (64mb). > > > My understanding of the docs ( > https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/memory/mem_setup_tm/) > is that I should just need to set taskmanager.memory.process.size. Where > is the 64mb figure coming from which makes up the Total Flink Memory? How > do I change this? > > Many thanks, > > John >
Re: Flink TaskManager memory configuration failed
512mb is just too small for a TaskManager. You would need to either increase it, or decrease the other memory components (which currently use default values). The 64mb Total Flink Memory comes from the 512mb Total Process Memory minus 192mb minimum JVM Overhead and 256mb default JVM Metaspace. Best, Xintong On Thu, Jun 23, 2022 at 1:08 PM yu'an huang wrote: > Hi John, > > May I know what is your Flink version you are trying? > > > > On Thu, 23 Jun 2022 at 3:43 AM, John Tipper > wrote: > >> Hi all, >> >> I'm wanting to run quite a number of PyFlink jobs on Kubernetes, where >> the amount of state and number of events being processed is small and >> therefore I'd like to use as little memory in my clusters as possible so I >> can bin pack most efficiently. I'm running a Flink cluster per job. I'm >> currently trying to see how small I can make the memory settings. >> >> I set taskmanager.memory.process.size: 512mb in my Flink config. The >> container is being started with requested memory of 512Mi. >> >> When my TaskManager starts up, I get an error message: >> >> IllegalConfigurationException: Sum of configured Framework Heap Memory >> (128mb), Framework Off-Heap Memory (128mb), Task Off-Heap Memory (0 >> bytes), Managed Memory (25.6mb) and Network Memory (64mb) exceed configured >> Total Flink Memory (64mb). >> >> >> My understanding of the docs ( >> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/memory/mem_setup_tm/) >> is that I should just need to set taskmanager.memory.process.size. Where >> is the 64mb figure coming from which makes up the Total Flink Memory? How >> do I change this? >> >> Many thanks, >> >> John >> >
