Package: tomcat9 Version: 9.0.43-2~deb11u6 Severity: normal X-Debbugs-Cc: sebastian.lovd...@hibox.tv
Hi, We noticed while rolling out JDK 17 support for our in-house application that the following command is "broken" (moral-martin is an LXD container in my examples below, PID 4108 is the tomcat9 java process): root@moral-martin:~# lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 11 (bullseye) Release: 11 Codename: bullseye root@moral-martin:~# sudo -u tomcat jstack 4108 4108: Unable to open socket file /proc/4108/root/tmp/.java_pid4108: target process 4108 doesn't respond within 10500ms or HotSpot VM not loaded ...when all following conditions are met: * tomcat9 is running from systemd, _and_ * the JDK is of version 11 or greater, _and_ * the systemd unit (/lib/systemd/system/tomcat9.service) sets AmbientCapabilities=CAP_NET_BIND_SERVICE (which is done by the Debian package) We have spent a significant amount of time debugging this and I'll try to do my best to summarize our findings here: The problem is that the way jstack and similar tools work have changed from JDK8 to JDK11. In JDK8, it simply uses /tmp to try and communicate with the target process: https://github.com/AdoptOpenJDK/openjdk-jdk8u/blob/master/jdk/src/solaris/classes/sun/tools/attach/LinuxVirtualMachine.java#L40-L45 and https://github.com/AdoptOpenJDK/openjdk-jdk8u/blob/master/jdk/src/solaris/classes/sun/tools/attach/LinuxVirtualMachine.java#L293 In newer JDK versions (JDK 17 as an example), the code has been made "smarter" to support mount namespaces: https://github.com/openjdk/jdk17u/blob/master/src/jdk.attach/linux/classes/sun/tools/attach/VirtualMachineImpl.java#L299-L302 _However_... bear with me, this is where it gets interesting: this presumes that the calling process can access /proc/<pid>/root/tmp. When AmbientCapabilities=CAP_NET_BIND_SERVICE is set in the systemd unit, this is not the case: root@moral-martin:~# sudo -u tomcat ls -l /proc/4108/root ls: cannot read symbolic link '/proc/4108/root': Permission denied lrwxrwxrwx 1 tomcat tomcat 0 Apr 13 12:55 /proc/4108/root We have tested this and concluded that: 1. This happens whever _any_ capability is set in the systemd unit; it's not limited to CAP_NET_BIND_SERVICE. (Note: I haven't tested adding all possible capabilities yet; I believed I had but when writing this bug report I realize that my attempt at setting all of them didn't actually list all of them in `getpcaps pid`; will test this a bit more and see if it makes any difference) 2. When you remove AmbientCapabilities or set it to AmbientCapabilities= (empty string), it also works correctly. I honestly don't know if tomcat9 is the correct package to report this to; it can also be seen as a bug in the JDK. (We will work with the JDK maintainers to get this reported to them as well.) Feel free to reassign the bug report to another package. With JDK 8, this works correctly. Some of our tooling/monitoring is dependent on being able to connect to Tomcat (running on JDK 8 or 17) at runtime. That's why this is imporant for us. Workaround What we have seen that on JDK 17, running `jstack` as root works; this will connect to the target process correctly. However, this does _not_ work on JDK 8 and doesn't seem to work properly on JDK 11 either (I think this has been fixed upstream in JDK for more recent JDK versions, which is why it behaves differently on JDK 17). Our application supports both JDK 8 and 17 for now, and running `jstack` as root *does not* work on JDK 8. Hence, having to run it as root with our JDK 17-based installations (only) makes things unnecessarily complex. Conclusions It puzzles me why setting the ambient capabilities for a process breaks this. It's uncertain whether this is a "feature" by the kernel or elsewhere. We have tried to find more details about this by studying the systemd and dbus code to a certain extent, but have yet been unable to find anything. If anyone reading this knows the prctl and cap_set_proc semantics by heart, your help would be greatly appreciated. Best regards, Per -- System Information: Debian Release: bookworm/sid APT prefers testing-security APT policy: (500, 'testing-security'), (500, 'testing') Architecture: amd64 (x86_64) Kernel: Linux 6.1.0-6-amd64 (SMP w/20 CPU threads; PREEMPT) Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages tomcat9 depends on: ii lsb-base 11.6 ii systemd [systemd-tmpfiles] 252.6-1 ii sysvinit-utils [lsb-base] 3.06-2 pn tomcat9-common <none> ii ucf 3.0043+nmu1 Versions of packages tomcat9 recommends: ii libtcnative-1 1.2.35-1 Versions of packages tomcat9 suggests: pn tomcat9-admin <none> pn tomcat9-docs <none> pn tomcat9-examples <none> pn tomcat9-user <none>