Re: RFR: JDK-8296360: Track native memory used by zlib via NMT
On Fri, 4 Nov 2022 14:35:00 GMT, Thomas Stuefe wrote: > This patch adds NMT tracking to the zlib. > > *Please note: we currently discuss whether NMT can be expanded across the JDK > in this ML discussion [1]. This PR depends on the outcome of that discussion > and won't proceed unless greenlighted. But since [1] is stalled, I post the > PR for RFR to get some feedback on the code itself and for people to try it > out.* > > NMT tracks hotspot native allocations but does not cover the JDK (apart from > small exceptions). But the native memory > footprint of JDK libraries can be very significant. Recently we had a > customer whose zlib footprint went into the ~40GB range. We analyzed the > problem with an in-house memory tracker, but things would have been a lot > simpler using NMT. > > This patch instruments the zlib via zalloc hooks, which is minimally > invasive. It does not touch zlib sources, so it works with both the bundled > zlib and system zlib. All the instrumentation happens in the JVM zlib wrapper. > > > - j.u.zip (deflate) (reserved=624, committed=624) > (malloc=624 #3) > > - j.u.zip (inflate) (reserved=221377904, committed=221377904) > (malloc=221377904 #60877) > > - Zip (other) (reserved=8163446896, committed=8163446896) > (malloc=8163446896 #182622) > > > [1] https://mail.openjdk.org/pipermail/core-libs-dev/2022-November/096197.html Withdrawing for now; let's see how our discussion goes. - PR: https://git.openjdk.org/jdk/pull/10988
Re: RFR: JDK-8296360: Track native memory used by zlib via NMT
On Fri, 4 Nov 2022 14:35:00 GMT, Thomas Stuefe wrote: > This patch adds NMT tracking to the zlib. > > *Please note: we currently discuss whether NMT can be expanded across the JDK > in this ML discussion [1]. This PR depends on the outcome of that discussion > and won't proceed unless greenlighted. But since [1] is stalled, I post the > PR for RFR to get some feedback on the code itself and for people to try it > out.* > > NMT tracks hotspot native allocations but does not cover the JDK (apart from > small exceptions). But the native memory > footprint of JDK libraries can be very significant. Recently we had a > customer whose zlib footprint went into the ~40GB range. We analyzed the > problem with an in-house memory tracker, but things would have been a lot > simpler using NMT. > > This patch instruments the zlib via zalloc hooks, which is minimally > invasive. It does not touch zlib sources, so it works with both the bundled > zlib and system zlib. All the instrumentation happens in the JVM zlib wrapper. > > > - j.u.zip (deflate) (reserved=624, committed=624) > (malloc=624 #3) > > - j.u.zip (inflate) (reserved=221377904, committed=221377904) > (malloc=221377904 #60877) > > - Zip (other) (reserved=8163446896, committed=8163446896) > (malloc=8163446896 #182622) > > > [1] https://mail.openjdk.org/pipermail/core-libs-dev/2022-November/096197.html I think it's a bit premature to mark this review. The discussion thread "Extend Native Memory Tracking over the JDK ?" needs more discussion and input from others. We need to be forward looking in those discussions as it is very likely that the JNI functions and memory allocation will move to Panama in the future. Also we need to think about the maintainable and be able to manage catagories in the libs code without needing to add to allocation.hpp - I suspect this is probably something you've thought about already. - PR: https://git.openjdk.org/jdk/pull/10988
RFR: JDK-8296360: Track native memory used by zlib via NMT
This patch adds NMT tracking to the zlib. *Please note: we currently discuss whether NMT can be expanded across the JDK in this ML discussion [1]. This PR depends on the outcome of that discussion and won't proceed unless greenlighted. But since [1] is stalled, I post the PR for RFR to get some feedback on the code itself and for people to try it out.* NMT tracks hotspot native allocations but does not cover the JDK (apart from small exceptions). But the native memory footprint of JDK libraries can be very significant. Recently we had a customer whose zlib footprint went into the ~40GB range. We analyzed the problem with an in-house memory tracker, but things would have been a lot simpler using NMT. This patch instruments the zlib via zalloc hooks, which is minimally invasive. It does not touch zlib sources, so it works with both the bundled zlib and system zlib. All the instrumentation happens in the JVM zlib wrapper. - j.u.zip (deflate) (reserved=624, committed=624) (malloc=624 #3) - j.u.zip (inflate) (reserved=221377904, committed=221377904) (malloc=221377904 #60877) - Zip (other) (reserved=8163446896, committed=8163446896) (malloc=8163446896 #182622) [1] https://mail.openjdk.org/pipermail/core-libs-dev/2022-November/096197.html - Commit messages: - JDK-8296360-Track-native-memory-used-by-zlib-via-NMT Changes: https://git.openjdk.org/jdk/pull/10988/files Webrev: https://webrevs.openjdk.org/?repo=jdk=10988=00 Issue: https://bugs.openjdk.org/browse/JDK-8296360 Stats: 192 lines in 9 files changed: 179 ins; 1 del; 12 mod Patch: https://git.openjdk.org/jdk/pull/10988.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10988/head:pull/10988 PR: https://git.openjdk.org/jdk/pull/10988
