[Bug 2055280] Re: openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper
Not in Jenkins, but we saw the same error in another application when unattended upgrades upgraded from 17.0.9 to 17.0.10 back in February. Now with the 17.0.10 to 17.0.11 upgrade we are seeing the same problem again. Was there another change in the protocol or what's going on here? I found https://bugs.openjdk.org/browse/JDK-8307990 that could possibly be related at least. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2055280 Title: openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openjdk-17/+bug/2055280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2055280] Re: openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: unattended-upgrades (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2055280 Title: openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openjdk-17/+bug/2055280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2055280] Re: openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper
** Also affects: unattended-upgrades (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2055280 Title: openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openjdk-17/+bug/2055280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2055280] Re: openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: openjdk-17 (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2055280 Title: openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openjdk-17/+bug/2055280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2055280] Re: openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper
Security team advised that standard USN says --- "This update uses a new upstream release, which includes additional bug fixes. After a standard system update you need to restart any Java applications to make all the necessary changes." [1] I will submit MR upstream to print a warning rather than sigsegv in this case. [1] https://ubuntu.com/security/notices/USN-6660-1 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2055280 Title: openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openjdk-17/+bug/2055280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2055280] Re: openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper
Thank you for investigating this I think we need to discuss the best way to fix it, e.g. maybe offer a compatibility patch that will check if argc == 1 and try to check contents of argv[0] in this case. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2055280 Title: openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openjdk-17/+bug/2055280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2055280] Re: openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper
Okay, I think the mystery might be solved.
The root cause is that unattended-upgrades (or some other apt upgrade)
does a openjdk-17 package update, while at the same time a java process
is running. After this minor upgrade, the protocol between the JRE's
forkAndExec JNI function and the jspawnhelper tool is changed! The
jspawnhelper tool now expects argv[0] to be the executable name of
itself, argv[1] to be a "%d:%d" format string with two file descriptors,
and argv[2] to be NULL.
However, the any already-running java process will still use the old
protocol, which invoked jspawnhelper with the "%d:%d" format string in
argv[0], and argv[1] set to NULL. This is what makes the new
jspawnhelper executable segfault.
Therefore, with this particular openjdk-17 upgrade, even it is a minor
'patch' upgrade, it is vital that _ALL_ java processes that intend to
spawn external processes are immediately terminated, and restarted.
I would suggest a BIG PROMINENT note in the upgrade message for this
particular update, since it is likely to bite a lot of people...
Some references:
https://bugs.openjdk.org/browse/JDK-8310265 ("(process) jspawnhelper
should not use argv[0]") is the bug that eventually changed to the JRE
<--> jspawnhelper protocol
https://github.com/openjdk/jdk17u-dev/commit/cd6cb730c934d8e16d4bd8e3342e59e806f158f9
is the corresponding commit for OpenJDK 17.
https://bugs.openjdk.org/browse/JDK-8325567 ("jspawnhelper without args
fails with segfault") is a related upstream bug. I also noticed the same
after the Ubuntu 17.0.10+7-1~22.04.1 package upgrade, because I tried
running jspawnhelper myself, and the very first invocation (without
arguments) segfaulted. :)
In that bug, Aleksey Shipilev notes:
> So this would only affect whoever is invoking jspawnhelper directly. But that
> would also run into problems when jspawnhelper protocol changes like in
> JDK-8310265.
E.g. it is clear that the jspawnhelper protocol was changed without
taking into account that any "old" JRE process would now run the helper
tool in a way that makes it segfault. I don't think they thought this
through correctly, even though it is an internal JRE implementation
detail...
Bottom line, this is not really an Ubuntu bug in the package, so feel free to
close this ticket, but I would still suggest adding a visible notice that any
running OpenJDK processes should be restarted!
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2055280
Title:
openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openjdk-17/+bug/2055280/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2055280] Re: openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper
There isn't anything special as far as I know, it's just a plain Ubuntu 22.04 VM which is accessed by Jenkins over SSH. Note that we're using the stable branch Jenkins, which is at 2.440.1, so it may be possible that only that version is buggy. I noticed a `_usr_lib_jvm_java-17-openjdk- amd64_lib_jspawnhelper.1007.crash` file in `/var/crash`, so I could unpack that and throw it in gdb: ``` Core was generated by `41:44'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __rawmemchr_evex () at ../sysdeps/x86_64/multiarch/memchr-evex.S:111 Download failed: Invalid argument. Continuing without source file ./string/../sysdeps/x86_64/multiarch/memchr-evex.S. 111 ../sysdeps/x86_64/multiarch/memchr-evex.S: No such file or directory. (gdb) bt #0 __rawmemchr_evex () at ../sysdeps/x86_64/multiarch/memchr-evex.S:111 #1 0x7f21c298d9e8 in _IO_str_init_static_internal (sf=sf@entry=0x7fffde73e550, ptr=ptr@entry=0x0, size=size@entry=0, pstart=pstart@entry=0x0) at ./libio/strops.c:41 #2 0x7f21c2960323 in _IO_strfile_read (string=0x0, sf=0x7fffde73e550) at ../libio/strfile.h:95 #3 __GI___isoc99_sscanf (s=0x0, format=format@entry=0x55f74431f0a1 "%d:%d") at ./stdio-common/isoc99_sscanf.c:28 #4 0x55f74431d391 in main (argc=, argv=) at src/java.base/unix/native/jspawnhelper/jspawnhelper.c:140 ``` So for some reason, it looks like `argv[0]` is actually "41:44", which results in sscanf() being called on argv[1] which is NULL. I have no idea yet whether this is a Jenkins bug or a Java bug. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2055280 Title: openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openjdk-17/+bug/2055280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2055280] Re: openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper
I agree - they do not check argc there, causing the startup segfault. I have done the test in lxc container: $lxc launch ubuntu-daily:noble There i have installed jenkins weekly release: --- sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \ https://pkg.jenkins.io/debian/jenkins.io-2023.key echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \ https://pkg.jenkins.io/debian binary/ | sudo tee \ /etc/apt/sources.list.d/jenkins.list > /dev/null sudo apt-get update sudo apt-get install jenkins -- and configured a pipeline to build a maven project[1] I have configured an agent to connect and ran it as following: JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64/ java -jar agent.jar -url http://:8080/ -secret @secret-file -name "self-server" -workDir "/home/test/work" The build successfully clones the project, starts maven and builds it. I wonder if there is something specific in your setup (e.g. a docker container used as an agent) that may contribute to the issue? [1] https://github.com/vpa1977/spring-petclinic/tree/spring-boot-2.7.3 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2055280 Title: openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openjdk-17/+bug/2055280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2055280] Re: openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper
Possibly related: https://issues.jenkins.io/browse/JENKINS-72665 So the question is what the Jenkins agent does: I'm not sure it invokes jspawnhelper directly, I assume it is going via the Java API, but there could be some other bug that causes it to pass incorrect arguments to jspawnhelper. It looks like the command line interface of jspawnhelper is not very bullet proof, which is understandable since it's not meant to be run directly. But still, segfaulting is bad :) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2055280 Title: openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openjdk-17/+bug/2055280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2055280] Re: openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper
Hi,
process spawning works (both jtreg tests and a small reproducer below):
---
public class Test {
public static void main(String[] args) throws Throwable {
Process p = new ProcessBuilder("ls", "-alrt", "/tmp").start();
p.waitFor();
}
}
---
but there were changes in jspawnhelper that might be triggering the crash.
Now it expects the child data to be passed through argv[1] - see [1].
I will try to set up jenkins environment to see if I can reproduce it.
[1]
https://github.com/openjdk/jdk17u/commit/cd6cb730c934d8e16d4bd8e3342e59e806f158f9
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2055280
Title:
openjdk-17-jre-headless 17.0.10+7-1~22.04.1: segfault in jspawnhelper
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openjdk-17/+bug/2055280/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
