Re: RFR(L) 8237354: Add option to jcmd to write a gzipped heap dump

[Laurence Cable]

On 2/21/20 5:19 PM, Ioi Lam wrote:

Ralf and Christoph,

I agree that making it easy for the user is important, so dependency on an external program like pgzip will be a hassle.

How about implementing the compression in a Java program? Will something like this be too much of a hassle?

    jcmd $PID GC.dump -stdout | java -jar HeapDumpZipper.jar > heap.gz

we could integrate the compression into cmd itself?

This way, we can implement the exact compression algorithm as Ralf described, without making it part of the VM. Writing it in Java probably would be easier to maintain.

If it makes sense, we can include the Java code as part of the JDK, so there's no need to ship a separate JAR file to the user.

jcmd $PID GC.dump -stdout | java jdk.internal.heapdump.Zipper > heap.gz

Thanks
- Ioi

On 2/21/20 8:35 AM, Langer, Christoph wrote:

Hi all,

let me share my thoughts after going through this mail thread and interrogating Ralf quite a bit about the feature 😉.

First of all, I very much value the discussion and the points brought up here. When deciding about the introduction of an enhancement or a new feature, it's always wise to thoroughly discuss it and value benefits against maintenance cost incurred. However, in this case I'm at a point where I would really like to see this going in. Let me elaborate on this.

In the mail cited below, I think Ralf enumerates all the benefits quite comprehensively. With the gzip feature built into the heapdumper, we'll get the option to easily have the VM dump its heap in a space saving format in the same time (or even a bit quicker) than we currently can get fully exploded hprofs. There's no need for additional configuration steps and arrangements, just a simple additional option in the existing jcmd. And with the slightly updated dump format, tool builders will get options to improve handling of compressed heap dumps.

Speaking as somebody who has to do customer support once in a while, I can't tell you how valuable it is to be able to give the customer simple instructions that just work when it comes to directing them to provide diagnosis data. And that's clearly a point here. Also, given the loads of different deployment scenarios of JVM applications, e.g. cloud, containers, monolith servers... it's really good to have simple options.

On the other hand, that's true, the change introduces a bit of additional complexity. But, without looking into the new code in all details, I think the amount is acceptable. Most of the code really only touches a distinct module for dumping the heap (heapdumper.cpp). Some additional 600 lines of code (the file already had 2000 before). But the code actually is not messing too deep with hotspot internals, so it should be quite maintainable. The rest of the code is a few lines about enhancing the dcmd and some additional access points into zlib. Furthermore, it brings a bit of testing code, but that is a good thing. So, this should really be acceptable - given that Ralf is around to support this once it's checked in and there's also the rest of the SAP team which will be able to help out here.

The ideas collected in this thread that go beyond this change, e.g. the possibility to dump the heap out to the network, the option to get heapdumps out to the jcmd and also the potential enhancements to the -XX: HeapDumpBeforeFullGC, -XX: HeapDumpAfterFullGC and -XX:HeapDumpOnOutOfMemoryError are partly orthogonal and are probably worth pursuing on their own.

So I really think we should allow this enhancement in and start focusing on a good code review 😊.

Best regards
Christoph

-----Original Message-----
From: hotspot-runtime-dev <hotspot-runtime-dev-
boun...@openjdk.java.net> On Behalf Of Schmelter, Ralf
Sent: Donnerstag, 20. Februar 2020 14:21
To: Yasumasa Suenaga <suen...@oss.nttdata.com>; Ioi Lam
<ioi....@oracle.com>; serguei.spit...@oracle.com; hotspot-runtime-
d...@openjdk.java.net runtime <hotspot-runtime-...@openjdk.java.net>
Cc: serviceability-dev@openjdk.java.net
Subject: [CAUTION] RE: RFR(L) 8237354: Add option to jcmd to write a
gzipped heap dump

Hi Yasumasa,

I think it would be great if we could redirect larger chunks data to jcmd.

But you have to differentiate between binary data (for the heap dump) and

text data (for the e.g. codelist).

Currently jcmd assumes all bytes to be UTF-8 encoded, converts them to

Unicode and then uses the platform encoding to write characters. This is not

suitable for binary data.

And of course you cannot use the bufferedStream to get the output to jcmd.

You would have to implement an outputStream which can directly write to
the AttachListener connection.

But even with this change, I would still like the gzip compression to be done

in the VM. Let me try to list all the advantages I see for doing this:

1. It is by far the easiest to use. You just have to specify -gz for the jcmd. While your command line (jcmd .... | gzip -c > file) is easy enough, it assumes you have gzip (not by default on Windows) and it would be painfully slow (~ 10 x and more), since it is not parallel. You could use pigz, but it is not as ubiquitous as gzip. I know it is sometimes hard to image this could be a

problem for anyone, but it is.

It is easy to tell a customer to execute jcmd <pid> GC.heap_dump -gz

test.hprof.gz. Adding additional requirements, especially if it is external

programs, and your chance of success diminish fast.


2. The -XX:HeapDumpOnOutOfMemoryError, -XX: HeapDumpBeforeFullGC
and -XX: HeapDumpAfterFullGC options can easily create gzipped heap
dumps directly when the compression is in the VM. And especially if you

create more than one dump (with the before/after gc flags), compression is

very useful. Or if you want to support compressed heap dumps it in the
HotSpotDiagnosticMXBean. Just add a flag and/or compression level.

3. The created gz-file is not a simple gz-file you would get when simply using

gzip.

It is created in a way that makes it possible to treat it like a random access file

without decompressing it.

Currently for example the Eclipse Memory Analyzer (MAT) has the option to directly open a gzipped hprof file and use it without decompression. And for the initial parsing, they can just read the file sequentially, so this is not too

slow.

But when accessing the values of objects or arrays, they have to seek to specific positions in the gzipped hprof file. This is currently implemented by having a Java implementation of a InflaterInputStream which is capable to completely copy its state. This copy is then used to start decompressing at the specific offset for which is was created. As you can imagine, the state of the inflater is not small (MAT assumes about 64Kb, 32kB is needed at least for the dictionary), so it limits the number of starting positions you can use for

large files. But it works for all kinds of gzip compressed streams.

The gzip implementation used to write the heap dump in the VM creates
many small gzip compressed chunks. At the start of each chunk you can

create a fresh GZIPInputStream without having to store any internal state. You only need to remember the physical offset and the logical offset (so 2 long values) for each chunk. If you then want to read data at a specific logical

offset, you binary search the nearest preceding chunk and create a
GZIPInputStream reading from the physical offset of that chunk. So on

average you have to decompress about half a chunk to get to the data you

need.

If you look in the in webrev, you can see

http://cr.openjdk.java.net/~rschmelter/webrevs/8237354/webrev.0/test/lib

/jdk/test/lib/hprof/parser/GzipRandomAccess.java.html. This implements

the needed logic to treat the gzipped hprof file as a random access file. I have used it to add support for gzipped files in the jhat library (which is only used in tests). In jhat hat for example, the resolution of references is done via random access. And the file also contains all the functionality MAT would

need.

You can generate a more or less equivalent file if you use pigz with the -- independent option. But to make it easier to detect that the gzip file is chunked (without decompressing it first), I've added a comment marking it as a hprof file with a given chunk size. This would be missing from the pigz file, but they instead adding 9 bytes when --independent is specified (00 00 ff ff

00 00 00 ff ff), so you could detect it too.

To summarize, the gzipped hprof file created by the VM makes it much

easier for tools to access them efficiently at random positions. You can do

something equivalent with pigz, but not with gzip.

And getting support for this type of gzipped hprof file by the heap dump tools will be much easier, if this is the format the openjdk produces, so it will

be widespread.

Best regards,
Ralf

-----Original Message-----
From: Yasumasa Suenaga <suen...@oss.nttdata.com>
Sent: Donnerstag, 20. Februar 2020 00:59
To: Ioi Lam <ioi....@oracle.com>; Schmelter, Ralf
<ralf.schmel...@sap.com>; serguei.spit...@oracle.com; hotspot-runtime-
d...@openjdk.java.net runtime <hotspot-runtime-...@openjdk.java.net>
Cc: serviceability-dev@openjdk.java.net
Subject: Re: RFR(L) 8237354: Add option to jcmd to write a gzipped heap
dump

Hi,

Generally I agree with Ioi, but I think it is not a problem only for gzipped heap

dump.

For example, Compiler.codelist and Compiler.CodeHeap_Analytics might be
large text.
In addition, some users want to redirect the result from jcmd to other
command or log collector.

So I think it would be better if jcmd provides stdout redurect option to all

subocmmands. E.g.

    $ jcmd <PID> GC.heap_dump -stdout | gzip -c - > heapdump.hprof.gz


Thanks,

Yasumasa