Hi Oli, Maybe a compromise here would be to not filter out the filler but to replace it in the heap dump as something recognizable. I’d be happier with that than having to use heuristics to sort things out. Right now FillerElement[] and FillerObject show up in the heap dump.
Kind regards, Kirk > On Aug 28, 2026, at 11:44 AM, Gillespie, Oli <[email protected]> wrote: > > Fair enough. I don't consider confusion and surprise disqualifying, but they > should be weighed against value. Re flag - if there were a flag, I of course > would propose that not including filler be the default - the defaults ought > to be suitable for the majority, sophisticated users that can make use of it > can set the flag. And by the time you're setting a flag to see filler > objects, why not just monitor them through logs and metrics? If we start from > the goal of monitoring TLAB waste, region tail waste etc. then I think the > best solution would be direct metrics. > > I do get that we'd be losing information (and gaining some, since now we know > that all the remaining int[]s are user objects), I just like the tradeoff. I > don't know the history, but it seems to me that hprof is not intended to > represent VM layout internals. It does not encode object header (even header > size) nor field layout. With Valhalla's flat fields/arrays, the gap between > hprof and memory representation could be huge. It also does not encode GC > specifics like region boundaries, generation info, object age, GC roots. In a > simple list of objects, I don't think VM filler conceptually or practically > fits, and as discussed has real downsides due to the current implementation. > I would for sure like a richer format where it could fit. > > Oli > From: Kirk Pepperdine > Sent: Friday, August 28, 2026 09:29 > To: Gillespie, Oli > Cc: [email protected] > Subject: RE: [EXTERNAL] [External] : Re: Omit GC filler objects from heap dump > > > Hi Oli, > I'd like to push back on framing confusion and surprise as a reason for this > information being removed. My experience with workshops that include reading > heap dumps is that many developers are confused by many aspects of a heap > dump. Moreover, we are often surprised at what is in a heap dump (or produced > by any profiler for that matter). I don’t believe these are valid reasons for > removing the data from the heap. Instead this should be motivation to improve > the tooling. A heap dump tooling should help make issues visible and help and > guide users to a solution. Filtering out a problem doesn’t help with > visibility. Prescribed filtering creates blind spots for those that know what > to do with the data. Maybe the filtering could be turned on with a feature > flag? > > Kind regards, > > Kirk > > On Aug 28, 2026, at 9:57 AM, Gillespie, Oli <[email protected]> wrote: > > It has both value and cost, so I'm trying to weigh that up. So far it seems > that the value it provides is smaller (only a small portion of heap dump > users can usefully distinguish filler int[] from real int[], and make use of > that information) than the cost (confusion, leak/disclosure of old memory, > heap dump time and size, parsing time for a wider swath of users), so I'm > still in favour. > > I'll wait a few more days for any more opinions. > > Oli > > From: Kirk Pepperdine > Sent: Thursday, August 27, 2026 14:55 > To: Gillespie, Oli > Cc: [email protected] > Subject: RE: [EXTERNAL] [External] : Re: Omit GC filler objects from heap dump > > Hi Oli, > > I’m aware of the GC log source and I’m happy to work from that if it’s > available. However, I’m also working from the heap dump in cases where a GC > log may not be available. Again, this is information in the heap dump that > has value and it would be disruptive for it to be stripped out. > > Kind regards, > Kirk > > > On Aug 27, 2026, at 2:31 PM, Gillespie, Oli <[email protected]> wrote: > > For humongous region waste, you can calculate the waste without observing the > fillers by using the non-filler object size and region size, as you said (an > explicit counter for this would be nice, and should be very easy to add). > > For TLAB waste, it's logged directly: "[0.365s][debug][gc,tlab] GC(0) TLAB > totals: thrds: 5 alloc-frac: 54.6% refills: 104 max: 100 slow allocs: 60 max > 60 waste: 0.9% gc: 65832B(0.9%) max: 16416B refill: 3096B(0.0%) max: 3096B", > and these counters are also available through hsPerfData. > > So it seems like the main causes will still be observable. > > Oli > > From: Kirk Pepperdine > Sent: Thursday, August 27, 2026 13:12 > To: Gillespie, Oli > Cc: [email protected] > Subject: RE: [EXTERNAL] [External] : Re: Omit GC filler objects from heap dump > > My first heuristic was finding humongous allocations and knowing the region > size one can calculate the waste. From there one can find int[] arrays that > are ~ the size of the tlab. It’s not a perfect heuristic but it’s been > helpful. > > Kind regards, > Kirk > > > On Aug 27, 2026, at 10:56 AM, Gillespie, Oli <[email protected]> wrote: > > Thanks. Could you share how you find and distinguish them? I suppose, 'large > unreachable int[]' is one heuristic. Perhaps the fact that the contents are > unzeroed heap memory makes them look quite distinctive too? What sources do > you look for and what are the remedies? I'm wondering if there's any more > direct measurement that's either already available or could be made > available. For example, for the filler at the end of humongous G1 regions, it > would be easy for G1 to track and report directly. > > Do you always operate on hprof files? The filler objects will still be > present in GC.class_histogram and JVMTI heap iteration, do those cover your > needs? > > "I would leave it up to the tooling" - my problem is that the hprof doesn't > retain the information to (perfectly) distinguish filler arrays from real int > arrays. If they were clearly marked, then I might agree. But even then, the > increased dump size and the fact that leaks random unused heap memory remain. > Those could possibly be addressed by a flag to opt-in to dumping fillers, and > to dump them zeroed, but I think that's too much. > > Oli > > > From: Kirk Pepperdine > Sent: Thursday, August 27, 2026 08:26 > To: Gillespie, Oli > Cc: [email protected] > Subject: RE: [EXTERNAL] [External] : Re: Omit GC filler objects from heap dump > > Hi Ali, > I share your frustration with filler objects, they are confusing. That said, > I have tooling that finds these filler objects and works to sort out if > anything can be done to reduce this type of heap “wastage”. I have some > heuristics that help me distinguish source and propose possible remedies. So > I would like to object to having this information hidden from me. I would > leave it up to the tooling to decide if these filler classes should filtered > out. > > Kind regards, > Kirk Pepperdine > > > On Aug 26, 2026, at 5:50 PM, Gillespie, Oli <[email protected]> wrote: > > Hearing no objections, I created https://github.com/openjdk/jdk/pull/32542 to > implement this. I found the > existinghttps://bugs.openjdk.org/browse/JDK-8372389 which had the same idea. > From: Gillespie, Oli <[email protected] <mailto:[email protected]>> > Sent: 24 August 2026 10:20 > To: [email protected] <mailto:[email protected]> > <[email protected] <mailto:[email protected]>> > Subject: Re: Omit GC filler objects from heap dump > > Bumping this - does anyone have any objection to excluding filler objects > from heap dumps? > > Oli > From: Gillespie, Oli > Sent: 18 August 2026 15:29 > To: [email protected] <mailto:[email protected]> > <[email protected] <mailto:[email protected]>> > Subject: Omit GC filler objects from heap dump > > Proposal: exclude filler objects from heap dumps. They confuse more than they > help. > > I've had a few cases where filler arrays in heap dumps confused me for quite > a while. It's okay in things like GC.class_histogram as they have the > identifying name of "[Ljdk.internal.vm.FillerArray", but in other tools like > Eclipse MAT they still show up as plain int arrays. Normally unreachable > objects would not be looked at much, I get that, but here are the cases I > heard about recently. 1 - HeapDumpOnOutOfMemoryError produced a heap dump, > which only had around 20% space filled with reachable objects, the rest > unreachable. I figured the unreachable stuff must be relevant to the OOM. > Actually, the 'OutOfMemoryError: java heap space' was triggered by GCLocker > retry count exceeded, the JVM was not out of heap space at all. 2 - a service > had an issue with periodic allocation spikes causing G1 full GC. They enabled > HeapDumpBeforeFullGC to understand the garbage that was triggering the full > gc. Many large int arrays seemed suspicious, but in the end they were mostly > fillers. > > I propose the simple option of excluding filler objects from heap dumps, but > if there's a way to instead make these identifiable in MAT and other tools, > that would also be fine. I wonder if Valhalla might provide any options here > to represent the filler elements not as primitives but as values? > > Feedback appreciated! > > Oli > > (Filler object background - > https://tschatzl.github.io/2022/09/26/jdk-vm-internal-fillerarray.html) > > > > Amazon Development Centre (London) Ltd.Registered in England and Wales with > registration number 04543232 with its registered office at 1 Principal Place, > Worship Street, London EC2A 2FA, United Kingdom. > > > > > Amazon Development Centre (London) Ltd.Registered in England and Wales with > registration number 04543232 with its registered office at 1 Principal Place, > Worship Street, London EC2A 2FA, United Kingdom. > > > > > Amazon Development Centre (London) Ltd.Registered in England and Wales with > registration number 04543232 with its registered office at 1 Principal Place, > Worship Street, London EC2A 2FA, United Kingdom. > > > > > Amazon Development Centre (London) Ltd.Registered in England and Wales with > registration number 04543232 with its registered office at 1 Principal Place, > Worship Street, London EC2A 2FA, United Kingdom. > > > > > Amazon Development Centre (London) Ltd.Registered in England and Wales with > registration number 04543232 with its registered office at 1 Principal Place, > Worship Street, London EC2A 2FA, United Kingdom.
smime.p7s
Description: S/MIME cryptographic signature
