Calculation of UI's maximum non-heap memory
Hello, I have configured a job manager with the following settings (Flink 1.16.1): jobmanager.memory.process.size: 1024m jobmanager.memory.jvm-metaspace.size: 150m jobmanager.memory.off-heap.size: 64m jobmanager.memory.jvm-overhead.min: 168m jobmanager.memory.jvm-overhead.max: 168m jobmanager.memory.enable-jvm-direct-memory-limit: "true" However, when I look at the job manager dashboard in the UI, I see that the value of Non-Heap Maximum is reported as 532 MB. Could someone clarify how this value is calculated? In case it's relevant, the effective configuration for JVM Heap is reported as 642 MB, with the reported maximum being 621 MB. Regards, Alexis.
Re: Calculation of UI's maximum non-heap memory
Hi, Alexis 1. With those configuration, Flink will set JVM parameters -Xms and -Xmx to 673185792(642m),-XX:MaxDirectMemorySize to 67108864(64m),-XX:MaxMetaspaceSize to 157286400(150m), you can find more information from [1] 2. As the hint in Flink UI: "The maximum heap displayed might differ from the configured values depending on the used GC algorithm for this process.", This[2] shows how JVM calculate the max heap memory from configured -Xms/-Xmx [1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/memory/mem_setup_jobmanager/ [2] https://stackoverflow.com/questions/52980629/runtime-getruntime-maxmemory-calculate-method Best, Weihua On Tue, Feb 21, 2023 at 12:15 AM Alexis Sarda-Espinosa < sarda.espin...@gmail.com> wrote: > Hello, > > I have configured a job manager with the following settings (Flink 1.16.1): > > jobmanager.memory.process.size: 1024m > jobmanager.memory.jvm-metaspace.size: 150m > jobmanager.memory.off-heap.size: 64m > jobmanager.memory.jvm-overhead.min: 168m > jobmanager.memory.jvm-overhead.max: 168m > jobmanager.memory.enable-jvm-direct-memory-limit: "true" > > However, when I look at the job manager dashboard in the UI, I see that > the value of Non-Heap Maximum is reported as 532 MB. Could someone clarify > how this value is calculated? > > In case it's relevant, the effective configuration for JVM Heap is > reported as 642 MB, with the reported maximum being 621 MB. > > Regards, > Alexis. > >
Re: Calculation of UI's maximum non-heap memory
Hi Weihua, Thanks for your response, I am familiar with those calculations, the one I don't understand is the Maximum Non-Heap value. Regards, Alexis. On Tue, 21 Feb 2023, 04:45 Weihua Hu, wrote: > Hi, Alexis > > 1. With those configuration, Flink will set JVM parameters -Xms and -Xmx > to 673185792(642m),-XX:MaxDirectMemorySize to > 67108864(64m),-XX:MaxMetaspaceSize > to 157286400(150m), you can find more information from [1] > 2. As the hint in Flink UI: "The maximum heap displayed might differ from > the configured values depending on the used GC algorithm for this > process.", This[2] shows how JVM calculate the max heap memory from > configured -Xms/-Xmx > > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/memory/mem_setup_jobmanager/ > [2] > https://stackoverflow.com/questions/52980629/runtime-getruntime-maxmemory-calculate-method > > Best, > Weihua > > > On Tue, Feb 21, 2023 at 12:15 AM Alexis Sarda-Espinosa < > sarda.espin...@gmail.com> wrote: > >> Hello, >> >> I have configured a job manager with the following settings (Flink >> 1.16.1): >> >> jobmanager.memory.process.size: 1024m >> jobmanager.memory.jvm-metaspace.size: 150m >> jobmanager.memory.off-heap.size: 64m >> jobmanager.memory.jvm-overhead.min: 168m >> jobmanager.memory.jvm-overhead.max: 168m >> jobmanager.memory.enable-jvm-direct-memory-limit: "true" >> >> However, when I look at the job manager dashboard in the UI, I see that >> the value of Non-Heap Maximum is reported as 532 MB. Could someone clarify >> how this value is calculated? >> >> In case it's relevant, the effective configuration for JVM Heap is >> reported as 642 MB, with the reported maximum being 621 MB. >> >> Regards, >> Alexis. >> >>
Re: Calculation of UI's maximum non-heap memory
Hi Alexis, The maximum Non-Heap is the sum of the memory pool (which is non-hep) max size. There are 3 memory pools(based on jdk11): 1. Metaspace, we can control the size with JVM parameter -XX:MaxMetaspaceSize or Flink configuration: jobmanager.memory.jvm-metaspace.size. For your job, this pool size is 150m 2. Compressed Class Space, this is controlled by JVM parameter -XX:CompressedClassSpaceSize. The default value is 1G or MetaspaceSize - 2 * InitialBootClassLoaderMetaspaceSize(default is 4194304). For your job, this pool size is 142m 3. CodeCache(codeHeap profiled nmethods/non-profiled nmethods/non-nmethods), this is controlled by JVM parameter -XX:ReservedCodeCacheSize. The default value is 240m So, the maximum non-heap is 150+142+240 = 532m. Best, Weihua On Tue, Feb 21, 2023 at 2:33 PM Alexis Sarda-Espinosa < sarda.espin...@gmail.com> wrote: > Hi Weihua, > > Thanks for your response, I am familiar with those calculations, the one I > don't understand is the Maximum Non-Heap value. > > Regards, > Alexis. > > On Tue, 21 Feb 2023, 04:45 Weihua Hu, wrote: > >> Hi, Alexis >> >> 1. With those configuration, Flink will set JVM parameters -Xms and -Xmx >> to 673185792(642m),-XX:MaxDirectMemorySize to >> 67108864(64m),-XX:MaxMetaspaceSize >> to 157286400(150m), you can find more information from [1] >> 2. As the hint in Flink UI: "The maximum heap displayed might differ >> from the configured values depending on the used GC algorithm for this >> process.", This[2] shows how JVM calculate the max heap memory from >> configured -Xms/-Xmx >> >> >> [1] >> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/memory/mem_setup_jobmanager/ >> [2] >> https://stackoverflow.com/questions/52980629/runtime-getruntime-maxmemory-calculate-method >> >> Best, >> Weihua >> >> >> On Tue, Feb 21, 2023 at 12:15 AM Alexis Sarda-Espinosa < >> sarda.espin...@gmail.com> wrote: >> >>> Hello, >>> >>> I have configured a job manager with the following settings (Flink >>> 1.16.1): >>> >>> jobmanager.memory.process.size: 1024m >>> jobmanager.memory.jvm-metaspace.size: 150m >>> jobmanager.memory.off-heap.size: 64m >>> jobmanager.memory.jvm-overhead.min: 168m >>> jobmanager.memory.jvm-overhead.max: 168m >>> jobmanager.memory.enable-jvm-direct-memory-limit: "true" >>> >>> However, when I look at the job manager dashboard in the UI, I see that >>> the value of Non-Heap Maximum is reported as 532 MB. Could someone clarify >>> how this value is calculated? >>> >>> In case it's relevant, the effective configuration for JVM Heap is >>> reported as 642 MB, with the reported maximum being 621 MB. >>> >>> Regards, >>> Alexis. >>> >>>
Re: Calculation of UI's maximum non-heap memory
Very useful, thanks a lot. Regards, Alexis. Am Di., 21. Feb. 2023 um 12:04 Uhr schrieb Weihua Hu : > Hi Alexis, > > The maximum Non-Heap is the sum of the memory pool (which is non-hep) max > size. There are 3 memory pools(based on jdk11): > 1. Metaspace, we can control the size with JVM parameter -XX:MaxMetaspaceSize > or Flink configuration: jobmanager.memory.jvm-metaspace.size. For your > job, this pool size is 150m > 2. Compressed Class Space, this is controlled by JVM parameter > -XX:CompressedClassSpaceSize. The default value is 1G or MetaspaceSize - 2 > * InitialBootClassLoaderMetaspaceSize(default is 4194304). For your job, > this pool size is 142m > 3. CodeCache(codeHeap profiled nmethods/non-profiled > nmethods/non-nmethods), this is controlled by JVM parameter > -XX:ReservedCodeCacheSize. The default value is 240m > > So, the maximum non-heap is 150+142+240 = 532m. > > > Best, > Weihua > > > On Tue, Feb 21, 2023 at 2:33 PM Alexis Sarda-Espinosa < > sarda.espin...@gmail.com> wrote: > >> Hi Weihua, >> >> Thanks for your response, I am familiar with those calculations, the one >> I don't understand is the Maximum Non-Heap value. >> >> Regards, >> Alexis. >> >> On Tue, 21 Feb 2023, 04:45 Weihua Hu, wrote: >> >>> Hi, Alexis >>> >>> 1. With those configuration, Flink will set JVM parameters -Xms and -Xmx >>> to 673185792(642m),-XX:MaxDirectMemorySize to >>> 67108864(64m),-XX:MaxMetaspaceSize >>> to 157286400(150m), you can find more information from [1] >>> 2. As the hint in Flink UI: "The maximum heap displayed might differ >>> from the configured values depending on the used GC algorithm for this >>> process.", This[2] shows how JVM calculate the max heap memory from >>> configured -Xms/-Xmx >>> >>> >>> [1] >>> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/memory/mem_setup_jobmanager/ >>> [2] >>> https://stackoverflow.com/questions/52980629/runtime-getruntime-maxmemory-calculate-method >>> >>> Best, >>> Weihua >>> >>> >>> On Tue, Feb 21, 2023 at 12:15 AM Alexis Sarda-Espinosa < >>> sarda.espin...@gmail.com> wrote: >>> Hello, I have configured a job manager with the following settings (Flink 1.16.1): jobmanager.memory.process.size: 1024m jobmanager.memory.jvm-metaspace.size: 150m jobmanager.memory.off-heap.size: 64m jobmanager.memory.jvm-overhead.min: 168m jobmanager.memory.jvm-overhead.max: 168m jobmanager.memory.enable-jvm-direct-memory-limit: "true" However, when I look at the job manager dashboard in the UI, I see that the value of Non-Heap Maximum is reported as 532 MB. Could someone clarify how this value is calculated? In case it's relevant, the effective configuration for JVM Heap is reported as 642 MB, with the reported maximum being 621 MB. Regards, Alexis.