Jiangli,
First of all: I tried building the leyden branch "hermetic-java-runtime"
but failed. :-( I tried some various attempts to work around the
failures, but the javastatic file I ended up with were unable to start.
What kind of environment are you using to build this branch? (OS,
compiler, libc version, etc) Do you have any special configure flags
that are required?
I would very much like to test it out myself to check this static java
launcher that you are creating.
On 2024-04-16 03:01, Jiangli Zhou wrote:
The main purpose of StaticLink.gmk is to support the static-java-image
make target, which can be used to perform the actual static linking
step using libjvm.a and JDK static libraries. That currently doesn't
exist in the JDK mainline. Creating a "fully" statically linked Java
launcher is the first step (out of many) towards supporting
static/hermetic Java.
I am still not sure it is the best design to have this in a separate
file. You are in some way "just" creating a new launcher.
As part of cleaning/refactoring/integrating for the static linking
step, we want to agree and decide/accept on the following:
- Support the "fully" statically linked java launcher for testing and
demoing the capability of static JDK support, e.g.
- Support running jtreg testing using the "fully" statically linked
Java launcher
So how do you want to test native jtreg tests? By compiling the jtreg
tests to native .a files and statically link with those as well, or to
make this a hybrid mode where a statically linked launcher loads the
jtreg tests dynamically?
- Set up tests in github workflow to help detect any breaking
changes for static support, e.g. new symbol issues introduced by any
changes. There were some earlier discussions on this with Ron and Alan
during the zoom meetings.
If we do this correct, we should not have to worry about symbol
conflicts. (Local symbols should all be hidden before creation of the
static libraries.) That said, creating static libraries needs to be
regularly tested or it will break.
- Which JDK native libraries to be statically linked with the new
launcher target? E.g. StaticLink.gmk currently excludes libjsound.a,
libawt_xawt.a, etc from statically linked with the launcher.
That sounds rather arbitrarily. I would assume that a statically linked
launcher should include native libraries from all included modules. If
you are not interested in java.desktop native libraries, then you'd need
to exclude java.desktop from the build. But my assumption here is that
we need to build a solution that works with all modules included in the
JDK, and that is what must be tested in GHA.
- Do we want more than one statically linked launcher target, based on
the set of linked native libraries?
Do you mean that you want like "static-java-java.base-only",
"static-java-eveything-but-java.desktop" etc? That does not sound like
something that belongs in the makefiles. If you want to build a JDK that
can only support a certain subset of modules, you need to specify those
module in your build scripts as input to configure.
(II) What missing:
- Static linking step as mentioned above
(III) What needs to be improved (require cleanups and refactoring, and
you mentioned some of those in your response as well):
- Support building both the static libraries and dynamic libraries
using the same set of .o files, instead of separately compiled .o
files. That helps improve build speed and reduce memory overhead for
building JDK. Your current refactoring work aims to help that.
- Clean up the usages of STATIC_BUILD macro. Most of the usages are in
test code.
- Other runtime fixes/enhancements in the leyden
https://github.com/openjdk/leyden/tree/hermetic-java-runtime branch
I think most work mentioned in III has dependencies on II. We need a
workable base to be able to build the "fully" statically linked
launcher for building and testing the work mentioned in III, when
integrating any of those to the JDK mainline. The makefile refactoring
work can be done in parallel but does not need to be completed before
we add the static linking step in JDK mainline.
I'm not sure I understand why you consider (II) to be a prerequisite for
the (III) issues. From my point of view, they are all mutually
independent changes that needs to be done, and if anything, I think
getting in code to create a statically linked launcher will probably be
easier if we get further along on the other issues.
In particular, I'd like to see changes to the STATIC_BUILD macro come in
first. Ideally, we should not send in such a macro at all. The
JNI_OnLoad stuff is bugging me mostly.
As of today, the leyden
https://github.com/openjdk/leyden/tree/hermetic-java-runtime branch
can build a "fully" statically linked Java launcher. The issue of
compiling the dynamic and static libraries .o files separately is not
a blocker. It's good to have it resolved at some point of time.
Weeeell, yes and no... It is not *technically* a blocker, but unless we
can reuse the .o files, including static builds in the standard testing
on GHA will effectively double the build time, and that is definitely
not acceptable. And the alternative is to bring in static builds without
having it regularly tested, which I don't think is acceptable either.
So the conclusion is that getting static and dynamic libraries built
from the same .o files is in effect a blocker for Hermetic Java.
This, in turn, require several changes:
1) The linking code needs to be cleaned up, and all technical debt needs to be
resolved. This is what I have been doing since I started working on static
builds for Hermetic Java. JDK-8329704 (which was integrated yesterday) was the
first major milestone of this cleanup. Now, the path were to find a library
created by the JDK (static or dynamic) is encapsulated in ResolveLibPath. This
is currently a monster, but at least all knowledge is collected in a single
location, instead of spread over the code base. Getting this simplified is the
next step.
2) We need to stop passing the STATIC_BUILD define when compiling. This is
partially addressed in your PR, where you have replaced #ifdef STATIC_BUILD
with a dynamic lookup. But there is also the problem of JNI/JVMTI entry points.
I have been pondering how we can compile the code in a way so we support both
dynamic and static name resolution, and I think I have a solution.
This is unfortunately quite complex, and I have started a discussion with Alan if it is
possible to update the JNI spec so that both static and dynamic entry points can have the form
"JNI_OnLoad_<library-name>". Ideally, I'd like to see us push for this with as
much effort as possible. If we got this in place, static builds would be much easier, and the
changes required for Hermetic Java even smaller.
Thumbs up! That seems to be a good direction. Currently in the leyden
branch, it first looks up the unique
JNI_OnLoad<_lib_name>|Agent_OnLoad<_lib_name> etc for built-in
libraries, then search for the dynamic libraries using the
conventional naming when necessary. e.g.:
https://github.com/openjdk/leyden/commit/a5c886d2e85a0ff0c3712a5488ae61d8c9d7ba1a
https://github.com/openjdk/leyden/commit/1da8e3240e0bd27366d19f2e7dde386e46015135
When spec supports JNI_OnLoad_<library-name> and etc. for dynamic
libraries, we may still need to support the conventional naming
without the <_lib_name> part for existing libraries out there.
That looks interesting. Basically, this is very similar to what I
imagine needs to be done in the mainline. However, I don't think we can
just change this code like that without a CSR? Or are we allowed to
treat JDK-internal libraries different than the specification states?
I would say getting this part into mainline should be the main focus
right now, not the StaticLink.gmk file. And that includes getting any
CSR approvals etc.
Thank you for taking this on! Potentially we could consider taking the
objcopy to localizing hotspot symbols on unix-like platforms, based on
https://github.com/openjdk/jdk/pull/17456 discussions. Additional
testing is still needed to verify the solution.
We don't need any additional testing on that; proof of concept shows it
works. It just needs to be implemented. With JDK-8330261 (pushed today),
the ground is now prepared to get it done. It is next on my todo-list.
It might be a good idea to follow up on the static linking discussion
in tomorrow's zoom meeting (hope you'll be able to join tomorrow).
I'll join but might be somewhat late due to family commitments.
/Magnus
