On Tue, Apr 16, 2024 at 12:39 PM Jiangli Zhou <jiangliz...@google.com> wrote: > > Magnus, thanks for the meeting summary! Please see clarifications > below, to avoid any confusion. > > On Tue, Apr 16, 2024 at 11:27 AM Magnus Ihse Bursie > <magnus.ihse.bur...@oracle.com> wrote: > > > > This is my summary of what was said in today's meeting: > > > > * Jiangli reported that her team had done extensive testing and not seen > > any problems, both with just the static launcher as generated by the > > leyden branch, and with bundled user applications using the One Jar (?) > > tool. They tested JDK tier 1, and an suite of Google's internal tests. > > When testing JTReg tests with native libraries, these were dynamically > > loaded. > > Clarifications, as discussed in the meeting: > > We have done multiple levels of testing for static/hermetic Java > prototype with our internal codebase on JDK 11, JDK 21 and JDK head > (mainline based). We run into bugs/failures and have addressed the > issues found along the way. The layden branch contains most of the > static/hermetic Java work from the prototype. In general > static/hermetic Java support looks robust in current state on JDK 21 > in our prototype based on testing (already communicated some of the > the testing outcome on JDK 11 in last years JVMLS), with some of the > remaining areas (e.g. how to handle user code accessing java.home, and > https://github.com/openjdk/leyden/blob/hermetic-java-runtime/src/java.base/share/classes/jdk/internal/misc/JavaHome.java > is still a internal class in the prototpye) that require broad > discussions involving potential spec updates, and a small number of > remaining failures to be looked into. > > Testing we have done includes: > - jtreg with tier1 using `javastatic` ("fully" statically linked Java > launcher) on JDK 11 and JDK 21 > - Explicit functional and integration tests (most of them are > developed base on existing jtreg tests) to test the final hermetic > Java image. Image is built using singlejar, and the test and JDK are > in a single image. > - Scattered hermetic Java testing using our internal tests. The > scattered hermetic image is a special mode for testing to emulate > hermetic Java image execution without building the final hermetic > Java. This requires some additional launcher changes that are not in > the lenden branch currently. > - Some production testing on JDK 11 and JDK 21 > > > > > * Alan asked about Hotspot JTReg tests that launched "java". Jiangli > > reported that they had not seen any problems, > > Also clarification: > > As mentioned during the meeting, most of the issues that we found with > jtreg tier1 testing were due to the assumption of using dynamic > libraries in the tests. We have addressed those. > > Alan had some questions about launching sub-processes. In the > prototype, we had done work to support POSIX_SPAWN launch mechanism > for ProcessBuilder.start() on hermetic Java, e.g. > https://github.com/openjdk/leyden/commit/931b71b0845d24b1949a23333ef1cdb3d6622596. > We need to look into tier1 testing to verify if they cover > sub-processing testing (Alan mentioned there are some in tier1). > > > but my understanding was > > that there was some confusion if any such tests were actually run. I > > think this is something that will need further attention, but if someone > > said they would look into it, I missed it. > > Megnas, can you please elaborate the above about "if any such tests > were actually run"? > > > > > * Jiangli will get numbers on how much time is added to the GHA > > testing if we add building and linking of static libraries, without > > fixing so we can compile to a single set of object files. > > Will follow up on this.
I did some measurements using the https://github.com/openjdk/leyden/tree/hermetic-java-runtime branch on my Linux desktop today. Here's the detail: - Build machine: Debian 6.6.* - gcc version 13.2.0 - `Time` command is used to do the measurements - Same configuration is used: --with-boot-jdk=<jdk_path> --with-debug-level=slowdebug --with-static-java=yes - `make clean` is done before both of the measurements # # Building JDK image with just dynamic libraries # Build command: make images # real 5m27.582s user 75m11.272s sys 8m2.783s # # Building JDK image with both dynamic libraries and static libraries, also link bin/javastatic # Build command: make static-java-image # real 6m5.958s user 115m59.353s sys 15m11.664s There is some overhead with the overall build time, but it's not as significant as doubling the time. I think we should consider prioritizing the launcher static linking work for now. I also want to reiterate the importance of compiling just the .o files only once and use them for both the dynamic libraries and static libraries. In our prototype on JDK 11, we did that. When moving the work to JDK head for the leyden branch and mainline, we adopted to use the existing `static-libs-image` for building the full set of static libraries with separately compiled .o files (https://bugs.openjdk.org/browse/JDK-8307858). As mentioned in some of the earlier meetings with Alan and Ron last year, we ran into memory issues during JDK build time on our system due to the duplication of the .o files. I did some workaround to mitigate the memory issue. So we should also prioritize the work although it is a non-blocker for integrating some of the leyden branch work into the mainline at this point. I'd recommend addressing that immediately after the launcher static linking work. As you are already aware of, it would require some of the runtime work to clean up the STATIC_BUILD macro usages. Best, Jiangli > > > > > * We did not fully come to a conclusion if compiling on a single set of > > object files for static and dynamic linking was a hard requirement or > > not, but at a minimum it is a desirable goal. (My personal opinion is > > that is a hard requirement if the GHA build times are seriously affected > > otherwise.) > > > > There are basically two problems prohibiting single object file > > compilation: > > > > 1) using dynamic checks instead of #ifdefs for code that differs between > > static and dynamic. > > > > 2) Handling the difference between JNI_OnLoad (as required for dynamic > > libraries) and JNI_OnLoad_<libname> (as required for static libraries). > > > > * The leyden branch has basically solved both these problems. The first > > one could more or less be integrated already (given perhaps some > > discussion on exactly *how* the JDK should discover in runtime if it is > > static or dynamic), but the latter requires a spec change to be integrated. > > > > * I think everyone agreed that moving on with a spec change was a good > > idea, regardless of if this is blocker or not, but I don't recall that > > there were any concrete next steps decided. Ron and Alan said that we do > > spec changes all the time so it will not add as much bureaucracy as one > > might fear. > > There was also a question raised by Dave during the meeting on the > timeline for the spec/JSR related work. > > > > > * Regarding which native libraries to include, I think we agreed on the > > following: > > - Static linking will only support headless-only builds (in which the > > build system excludes the AWT library that does "headful" stuff -- > > otherwise there would be duplicate symbols) > > Yes. > > > - As a first delivery, the build system will just create a static > > version of the "java" launcher (not jar, javac, etc). This will include > > all native libraries from all modules that are included in the build. > > Yes. It would be headless based. > > > - Going forward, the correct solution is to make jlink create a > > launcher that includes just the native libraries from the modules that > > is included in the jlink command. This will require jlink to understand > > how to call the native linker. > > Yes. That would be one of the N-step for supporting hermetic Java. > > > - Somewhere in there we probably also needs to have jlink know about > > headless-only vs normal (headless or "headful" determined on runtime), > > so it can create a java.desktop output that includes only the headless > > library. > > Alan has described an idea of dealing with java.desktop using jlink. > > > > > * Magnus reported that the refactoring and fixing of technical debt that > > was required for doing static builds properly has just been finished, > > and that his attention is now turning into creating a properly > > integrated system for generating static builds alongside dynamic builds. > > Thank you, Magnus! > > > > > * Jiangli and Magnus will work outside the meeting to resolve the build > > issues Magnus faced with the hermetic java branch in the Leyden repo. > > Yes. > > > > > * Just before the meeting unfortunately had to be aborted, Jiangli > > mentioned that they had discovered issues with some JDK native libraries > > when using objcopy to localize all non-visible symbols. It was at the > > time of writing not clear what those issues were. Jiangli will report > > back with what they found. (And while I had not time to mention it on > > the meeting, I will also look into this.) > > Best, > Jiangli > > > > > /Magnus > >
