Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v21]
On Mon, 18 Mar 2024 20:18:09 GMT, Mandy Chung wrote: > This is what I understand from your implementation: > > 1. create a regular JDK image under > `support/images/runtime-link-initial-jdk` using jlink from the exploded JDK > build > > 2. create a linkable JDK image under `images/jdk` using jlink from the > exploded JDK build with jlink command-line options used in step 1 plus > > > ``` > -create-linkable-runtime > jimage=support/images/runtime-link-initial-jdk/lib/modules:module-path=support/images/jmods > ``` > > jlink in step 2 has to run through the plugin pipeline as in step 1 and also > the `create-linkable-runtime` plugin to generate the diffs and record the > metadata for reconstituting the modules from the images (without the packaged > modules). > > Step 2 re-creates a new jimage file from the input packaged modules while the > diffs are generated from `runtime-link-initial-jdk/lib/modules`. In this > approach, step 1 and step 2 assume that the same set of modules are linked > and set set of plugin transformation is applied except > `--create-linkable-runtime` as in step 1, i.e. it relies on the makefile to > pass proper commands to step 1 and step 2 to result in the same `lib/modules` > minus the new metadata and diffs added by `--create-linkable-runtime`. That's correct. > The original idea we discussed is that step 1 creates a normal JDK image (as > step 1 above) and then step 2 solely creates a new JDK linkable image without > involving plugin pipeline that would be a simpler model. The idea was > something like this: > > ``` > $ runtime-link-initial-jdk/bin/jlink -create-linkable-runtime > support/images/jmods --output images/jdk > ``` > > jlink `--create-linkable-runtime` will copy everything under > `runtime-link-initial-jdk` (the runtime executing jlink) except `lib/modules` > to `images/jdk` and create a new jimage `lib/modules` from > `runtime-link-initial-jdk/lib/modules` (include diffs and metadata files). > > Have you seen any issues with this approach over your current approach which > involves the plugin transformation in both step 1 and 2? Yes, a couple of issues. In no particular order: - `jlink` is the only tool capable of generating the `jimage` file (`lib/modules`) as of yet. See class `ImageFileCreator`. - `jlink` is currently designed in a way to work on a) some form of an `Archive` (packaged modules, really) and b) a plugin pipeline. a) changes with this patch due to introduction of `JRTArchive`, but this archive relies on the "diff parts" as well as the "non-class-resource" parts. So in order to "use" `lib/modules` from `runtime-link-initial-jdk` we have a chicken and egg problem: Use `JRTArchive` before it's ready to be used. Or we'd have to distinguish between `JRTArchive` usage at JDK build time and `JRTArchive` usage at jlink runtime (after the JDK build has completed). - Let's assume the `JRTArchive` issue as described above is solved then we wouldn't have a way to **not** run any plugins at jlink time, unless we specify `--disable-plugin ` for each. Even if we had, then there is no guarantee to present the resources in the `jimage` file in the desired order. - Let's assume we don't use `JRTArchive` and use an `Archive` implementation from packaged modules instead. We cannot use that approach either, since we'd miss generated resources from plugins (like `--system-modules`, `--release-info` and `--generate-jli-classes`) or change the order of the resources in the `jimage` (plugin `--order-resources`) or have a different compression level (plugin `--compress`). There is more, which affects generation of the actual `jimage` file. - The way I understand the `jimage` [format](https://cr.openjdk.org/~sgehwolf/leyden/jimage_visualization_1.png) and how it's generated, it doesn't allow for post-factum addition of resources for a module. Those need to be known at jlink pipeline run time so that this information is available when the actual image format is being serialized. So unless, I'm missing something, the approach that I've taken would be the one with less compatibility impact. Thoughts? - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-2006724776
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v21]
On Fri, 15 Mar 2024 15:19:17 GMT, Severin Gehwolf wrote: >> Please review this patch which adds a jlink mode to the JDK which doesn't >> need the packaged modules being present. A.k.a run-time image based jlink. >> Fundamentally this patch adds an option to use `jlink` even though your JDK >> install might not come with the packaged modules (directory `jmods`). This >> is particularly useful to further reduce the size of a jlinked runtime. >> After the removal of the concept of a JRE, a common distribution mechanism >> is still the full JDK with all modules and packaged modules. However, >> packaged modules can incur an additional size tax. For example in a >> container scenario it could be useful to have a base JDK container including >> all modules, but without also delivering the packaged modules. This comes at >> a size advantage of `~25%`. Such a base JDK container could then be used to >> `jlink` application specific runtimes, further reducing the size of the >> application runtime image (App + JDK runtime; as a single image *or* >> separate bundles, depending on the app being modularized). >> >> The basic design of this approach is to add a jlink plugin for tracking >> non-class and non-resource files of a JDK install. I.e. files which aren't >> present in the jimage (`lib/modules`). This enables producing a `JRTArchive` >> class which has all the info of what constitutes the final jlinked runtime. >> >> Basic usage example: >> >> $ diff -u <(./bin/java --list-modules --limit-modules java.se) >> <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules >> --limit-modules java.se) >> $ diff -u <(./bin/java --list-modules --limit-modules jdk.jlink) >> <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules >> --limit-modules jdk.jlink) >> $ ls ../linux-x86_64-server-release/images/jdk/jmods >> java.base.jmodjava.net.http.jmod java.sql.rowset.jmod >> jdk.crypto.ec.jmod jdk.internal.opt.jmod >> jdk.jdi.jmod jdk.management.agent.jmod jdk.security.auth.jmod >> java.compiler.jmodjava.prefs.jmod java.transaction.xa.jmod >> jdk.dynalink.jmod jdk.internal.vm.ci.jmod >> jdk.jdwp.agent.jmod jdk.management.jfr.jmodjdk.security.jgss.jmod >> java.datatransfer.jmodjava.rmi.jmodjava.xml.crypto.jmod >> jdk.editpad.jmod jdk.internal.vm.compiler.jmod >> jdk.jfr.jmod jdk.management.jmodjdk.unsupported.desktop.jmod >> java.desktop.jmod java.scripting.jmod java.xml.jmod >> jdk.hotspot.agent.jmod jdk.i... > > Severin Gehwolf has updated the pull request incrementally with two > additional commits since the last revision: > > - Fix comment. > - Review feedback and Windows build fix (FixPath) This is what I understand from your implementation: 1. create a regular JDK image under `support/images/runtime-link-initial-jdk` using jlink from the exploded JDK build 2. create a linkable JDK image under `images/jdk` using jlink from the exploded JDK build with jlink command-line options used in step 1 plus -create-linkable-runtime jimage=support/images/runtime-link-initial-jdk/lib/modules:module-path=support/images/jmods jlink in step 2 has to run through the plugin pipeline as in step 1 and also the `create-linkable-runtime` plugin to generate the diffs and record the metadata for reconstituting the modules from the images (without the packaged modules). Step 2 re-creates a new jimage file from the input packaged modules while the diffs are generated from `runtime-link-initial-jdk/lib/modules`. In this approach, step 1 and step 2 assume that the same set of modules are linked and set set of plugin transformation is applied except `--create-linkable-runtime` as in step 1, i.e. it relies on the makefile to pass proper commands to step 1 and step 2 to result in the same `lib/modules` minus the new metadata and diffs added by `--create-linkable-runtime`. The original idea we discussed is that step 1 creates a normal JDK image (as step 1 above) and then step 2 solely creates a new JDK linkable image without involving plugin pipeline that would be a simpler model.The idea was something like this: $ runtime-link-initial-jdk/bin/jlink -create-linkable-runtime support/images/jmods --output images/jdk jlink `--create-linkable-runtime` will copy everything under `runtime-link-initial-jdk` (the runtime executing jlink) except `lib/modules` to `images/jdk` and create a new jimage `lib/modules` from `runtime-link-initial-jdk/lib/modules` (include diffs and metadata files). Have you seen any issues with this approach over your current approach which involves the plugin transformation in both step 1 and 2? - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-2004888430
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v21]
> Please review this patch which adds a jlink mode to the JDK which doesn't > need the packaged modules being present. A.k.a run-time image based jlink. > Fundamentally this patch adds an option to use `jlink` even though your JDK > install might not come with the packaged modules (directory `jmods`). This is > particularly useful to further reduce the size of a jlinked runtime. After > the removal of the concept of a JRE, a common distribution mechanism is still > the full JDK with all modules and packaged modules. However, packaged modules > can incur an additional size tax. For example in a container scenario it > could be useful to have a base JDK container including all modules, but > without also delivering the packaged modules. This comes at a size advantage > of `~25%`. Such a base JDK container could then be used to `jlink` > application specific runtimes, further reducing the size of the application > runtime image (App + JDK runtime; as a single image *or* separate bundles, > depending on the app b eing modularized). > > The basic design of this approach is to add a jlink plugin for tracking > non-class and non-resource files of a JDK install. I.e. files which aren't > present in the jimage (`lib/modules`). This enables producing a `JRTArchive` > class which has all the info of what constitutes the final jlinked runtime. > > Basic usage example: > > $ diff -u <(./bin/java --list-modules --limit-modules java.se) > <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules > --limit-modules java.se) > $ diff -u <(./bin/java --list-modules --limit-modules jdk.jlink) > <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules > --limit-modules jdk.jlink) > $ ls ../linux-x86_64-server-release/images/jdk/jmods > java.base.jmodjava.net.http.jmod java.sql.rowset.jmod > jdk.crypto.ec.jmod jdk.internal.opt.jmod > jdk.jdi.jmod jdk.management.agent.jmod jdk.security.auth.jmod > java.compiler.jmodjava.prefs.jmod java.transaction.xa.jmod > jdk.dynalink.jmod jdk.internal.vm.ci.jmod > jdk.jdwp.agent.jmod jdk.management.jfr.jmodjdk.security.jgss.jmod > java.datatransfer.jmodjava.rmi.jmodjava.xml.crypto.jmod > jdk.editpad.jmod jdk.internal.vm.compiler.jmod > jdk.jfr.jmod jdk.management.jmodjdk.unsupported.desktop.jmod > java.desktop.jmod java.scripting.jmod java.xml.jmod > jdk.hotspot.agent.jmod jdk.internal.vm.compiler.manage... Severin Gehwolf has updated the pull request incrementally with two additional commits since the last revision: - Fix comment. - Review feedback and Windows build fix (FixPath) - Changes: - all: https://git.openjdk.org/jdk/pull/14787/files - new: https://git.openjdk.org/jdk/pull/14787/files/1fe6161b..2ea9cab8 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk=14787=20 - incr: https://webrevs.openjdk.org/?repo=jdk=14787=19-20 Stats: 16 lines in 1 file changed: 6 ins; 0 del; 10 mod Patch: https://git.openjdk.org/jdk/pull/14787.diff Fetch: git fetch https://git.openjdk.org/jdk.git pull/14787/head:pull/14787 PR: https://git.openjdk.org/jdk/pull/14787
