Based on your description, it sounds like the pod may be too restrictive in its total system memory. The general rule-of-thumb I like to use is that the broker's max heap size should be no more than 50% of the total system memory available.
So for a POD with 9GB of memory, that would mean a max heap of 4.5GB. As for measuring the total system memory, there are ways to do so, although kubernetes complicates the picture a good amount. You would need to look at all of the processes running inside of the pod and the memory allocated by each, and understanding the difference between RSS, and other categories of process memory. And also note that part of the reason for the 50% rule is that the JVM itself uses a lot more memory than heap, so a JVM with a 6GB maximum heap size is going to use a lot more than 6GB of system memory. One way you may be able to get more insight into the POD's memory usage is to use the "kubectl exec" command to access the POD and use ps and other tools to check memory usage. BTW, even though the JVM allocates a 6GB heap initially, that doesn't necessarily mean the O/S has assigned physical pages to all of that heap (Java could force that allocation, but I don't know that it ever does). If that's correct, you could be simply running into the case that once the actual heap use grows close to its limit (which java heaps are designed to do normally all the time - hence the sawtooth memory-usage graphs), the system physical page allocations increase, eventually hitting the POD limit. So it is entirely possible that the problem is just sizing of the JVM and its heap. One way you could test this hypothesis of heap usage and physical pages would be to intentionally over-allocate a heap - for example using -Xmx12g on a POD with a limit of 9gb of memory, and see if the broker and POD fail quickly. Hope this helps. Art On Wed, Jan 14, 2026 at 5:44 AM Shiv Kumar Dixit < [email protected]> wrote: > We are hosting Artemis broker in Kubernetes using operator-based solution. > We deploy the broker as statefulset with 2 or 4 replicas. We assign for > e.g. 6 GB for heap and 9 GB for pod, 1.2 GB (1/5 of max heap) for > global-max-size. All addresses normally use -1 for max-size-bytes but some > less frequently used queues are defined with 100KB for max-size-bytes to > allow early paging. > > > > We have following observations: > > 1. As the broker pod starts, broker container immediately occupies 6 GB > for max heap. It seems expected as both min and max heap are same. > > 2. Pod memory usage starts with 6+ GB and once we have pending messages, > good producers and consumers connect to broker, invalid SSL attempts > happen, broker GUI access happens etc. during normal broker operations - > pod memory usage keeps increasing and now reaches 9 GB. > > 3. Once the pod hits limit of 9 GB, K8s kills the pod with OOMKilling > event and restarts the pod. Here we don’t see broker container getting > killed with OOM rather pod is killed and restarted. It forces the broker to > restart. > > 4. We have configured artemis.profile to capture memory dump in case of > OOM of broker but it never happens. So, we are assuming broker process is > not going out of memory, but pod is going out of memory due to increased > non-heap usage. > > 5. Only way to recover here is to increase heap and pod memory limits from > 6 GB and 9 GB to higher values and wait for next re-occurrence. > > > > 1. Is there any way to analyse what is going wrong with non-heap native > memory usage? > > 2. If non-heap native memory is expected to increase to such extent due to > pending messages, SSL errors etc.? > > 3. Is there any param we can use to restrict the non-heap native memory > usage? > > 4. If netty which handles connection aspect of broker can create such > memory consumption and cause OOM of pod? > > 5. Can we have any monitoring param that can hint that pod is potentially > in danger of getting killed? > > > > Thanks > > Shiv >
