Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v3]
On Fri, 29 Sep 2023 16:42:37 GMT, Severin Gehwolf wrote: > The command line options to override and change the error to a warning or > silently ignore seems to be "-run-time-ignore-single-hop" and > "--run-image-only-warnings". Maybe it should be reduced to just one option > and maybe it should contain the word "clone" to something that makes it more > obvious that it's just copying or cloning the contents of the modules in the > run-time image After some more thought I've changed my mind and I've come to the conclusion that it makes sense to combine them. The latest patch uses `--unlock-run-image` as the "force" option which demotes the error to a warning. Perhaps it makes sense to get some consensus on the naming parts at this point. The CSR seems a good vehicle for that and I've fleshed it out a bit now. - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-1764514749
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v3]
On Tue, 19 Sep 2023 17:39:04 GMT, Alan Bateman wrote:
> I did a pass over this to see where this proposal is currently at. At a
> high-level I think good progress since the discussion on leyden-dev some time
> ago. A few comments this from this pass:
Thanks, Alan!
> If I read it correctly, the default behavior is now to fail if jlink is
> executed from a run-time image that has been modified from the original and
> the packaged modules are not observable on the module path (this includes the
> default module path effectively appended by jlink). That seems the right
> policy as it avoids modifications by plugins or conf changes silently
> propagating.
Correct.
> If I read the code correctly, the error when a hash doesn't match is a very
> terse "Run image links only allow single-hop" so that probably needs to be
> looked to make sure the message says that the run-time image has been
> modified and maybe include one of the files/resources that has changed so
> there is something real to go with the error.
Not quite. See the example I've added in the [previous
comment](https://github.com/openjdk/jdk/pull/14787#issuecomment-1673592876). It
already includes that info. It looks like this:
Error: java.lang.IllegalArgumentException:
/disk/openjdk/upstream-sources/git/jdk-jdk/build/jmodless-copy-jmods-removed/conf/net.properties
has been modified. Please double check!
The message can of course be modified what we think is best.
> The command line options to override and change the error to a warning or
> silently ignore seems to be "-run-time-ignore-single-hop" and
> "--run-image-only-warnings". Maybe it should be reduced to just one option
> and maybe it should contain the word "clone" to something that makes it more
> obvious that it's just copying or cloning the contents of the modules in the
> run-time image (I suspect "single hop" won't be understood).
I believe keeping them both makes sense. Or at least make it so that it takes
an argument to be able to distinguish between the two. The reason why I think
this is useful is in the container case, where say - the JDK got installed by
an installer - which didn't include packaged modules, but is otherwise a full
JDK (with all modules). The installer already does integrity checking of the
files and it might be desirable to do multi-hop evolutions, while still wanting
to get warnings/errors on modified *configuration* files, for example. Happy to
change the name of that flag/flags to
`--clone-run-image=ignore-multi,warn-only` or something similar.
> Adding a new jdk.tools.jlink.internal.Archive implementation where the
> resources come from the run-time image seems a sensible approach. I don't
> particularly like the name "JmodLessArchive" because the original module may
> not have been packaged as a JMOD file, it may have been a modular JAR. Same
> comment on the BOM file "jmod_resources" added to the top-level directory of
> each module.
OK. How about `RunimageArchive` and `runimage_resources` or `clone_resources`?
> I think it's clever to add this to each module in the container image but
> part of me wonders if it should be hidden in some way to avoid tools
> depending on it. I don't have a concrete suggestion except to wonder if jrtfs
> should filter it out, same thing in the SystemModuleReader code as
> ModuleReader.find("jmod_resources") will find the resource, could be an
> attractive nuisance.
That seems fine. I can add code to filter those resources out. On the other
hand, no special handling is being made for `jdk/internal/vm/options`. Perhaps
that should be handled uniformly?
-
PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-1741195090
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v3]
On Thu, 10 Aug 2023 17:21:31 GMT, Severin Gehwolf wrote: >> Please review this patch which adds a "jmodless" jlink mode to the JDK. >> Fundamentally this patch adds an option to use `jlink` even though your JDK >> install might not come with the packaged modules (directory `jmods`). This >> is particularly useful to further reduce the size of a jlinked runtime. >> After the removal of the concept of a JRE, a common distribution mechanism >> is still the full JDK with all modules and packaged modules. However, >> packaged modules can incur an additional size tax. For example in a >> container scenario it could be useful to have a base JDK container including >> all modules, but without also delivering the packaged modules. This comes at >> a size advantage of `~25%`. Such a base JDK container could then be used to >> `jlink` application specific runtimes, further reducing the size of the >> application runtime image (App + JDK runtime; as a single image *or* >> separate bundles, depending on the app being modularized). >> >> The basic design of this approach is to add a jlink plugin for tracking >> non-class and non-resource files of a JDK install. I.e. files which aren't >> present in the jimage (`lib/modules`). This enables producing a >> `JmodLessArchive` class which has all the info of what constitutes the final >> jlinked runtime. >> >> Basic usage example: >> >> $ diff -u <(./bin/java --list-modules --limit-modules java.se) >> <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules >> --limit-modules java.se) >> $ diff -u <(./bin/java --list-modules --limit-modules java.se) >> <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules >> --limit-modules jdk.jlink) >> $ ls ../linux-x86_64-server-release/images/jdk/jmods >> java.base.jmodjava.net.http.jmod java.sql.rowset.jmod >> jdk.crypto.ec.jmod jdk.internal.opt.jmod >> jdk.jdi.jmod jdk.management.agent.jmod jdk.security.auth.jmod >> java.compiler.jmodjava.prefs.jmod java.transaction.xa.jmod >> jdk.dynalink.jmod jdk.internal.vm.ci.jmod >> jdk.jdwp.agent.jmod jdk.management.jfr.jmodjdk.security.jgss.jmod >> java.datatransfer.jmodjava.rmi.jmodjava.xml.crypto.jmod >> jdk.editpad.jmod jdk.internal.vm.compiler.jmod >> jdk.jfr.jmod jdk.management.jmodjdk.unsupported.desktop.jmod >> java.desktop.jmod java.scripting.jmod java.xml.jmod >> jdk.hotspot.agent.jmod jdk.internal.vm.compiler.management.jmod >> jdk.jlink.jmod jdk.naming.dns.j... > > Severin Gehwolf has updated the pull request incrementally with two > additional commits since the last revision: > > - Implementation for run-image link and single-hop > >This uses a stamp file in 'lib/modules' jimage in order to recognize >multi-hop links. However, this has the caveat of no longer producing >reproducible builds as compared to a packaged module-based link. > >Add an option to allow for multi-hop, which disables the stamp file >addition and makes it reproducible again. > - Exit the jlink on modified files by default > >This is configurable so add tests for both scenarios. I did a pass over this to see where this proposal is currently at. At a high-level I think good progress since the discussion on leyden-dev some time ago. A few comments this from this pass: If I read it correctly, the default behavior is now to fail if jlink is executed from a run-time image that has been modified from the original and the packaged modules are not observable on the module path (this includes the default module path effectively appended by jlink). That seems the right policy as it avoids modifications by plugins or conf changes silently propagating. If I read the code correctly, the error when a hash doesn't match is a very terse "Run image links only allow single-hop" so that probably needs to be looked to make sure the message says that the run-time image has been modified and maybe include one of the files/resources that has changed so there is something real to go with the error. The command line options to override and change the error to a warning or silently ignore seems to be "-run-time-ignore-single-hop" and "--run-image-only-warnings". Maybe it should be reduced to just one option and maybe it should contain the word "clone" to something that makes it more obvious that it's just copying or cloning the contents of the modules in the run-time image (I suspect "single hop" won't be understood). Adding a new jdk.tools.jlink.internal.Archive implementation where the resources come from the run-time image seems a sensible approach. I don't particularly like the name "JmodLessArchive" because the original module may not have been packaged as a JMOD file, it may have been a modular JAR. Same comment
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v3]
On Tue, 29 Aug 2023 17:31:00 GMT, Alan Bateman wrote: > > @AlanBateman Gentle ping. > > On my list, it's a lot to get through and a number of aspects to this that I > think will require refinement and discussion. Thanks for the heads-up! Your input is much appreciated. - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-1698743940
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v3]
On Tue, 29 Aug 2023 12:35:19 GMT, Severin Gehwolf wrote: > @AlanBateman Gentle ping. On my list, it's a lot to get through and a number of aspects to this that I think will require refinement and discussion. - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-1697864131
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v3]
On Thu, 10 Aug 2023 17:21:31 GMT, Severin Gehwolf wrote: >> Please review this patch which adds a "jmodless" jlink mode to the JDK. >> Fundamentally this patch adds an option to use `jlink` even though your JDK >> install might not come with the packaged modules (directory `jmods`). This >> is particularly useful to further reduce the size of a jlinked runtime. >> After the removal of the concept of a JRE, a common distribution mechanism >> is still the full JDK with all modules and packaged modules. However, >> packaged modules can incur an additional size tax. For example in a >> container scenario it could be useful to have a base JDK container including >> all modules, but without also delivering the packaged modules. This comes at >> a size advantage of `~25%`. Such a base JDK container could then be used to >> `jlink` application specific runtimes, further reducing the size of the >> application runtime image (App + JDK runtime; as a single image *or* >> separate bundles, depending on the app being modularized). >> >> The basic design of this approach is to add a jlink plugin for tracking >> non-class and non-resource files of a JDK install. I.e. files which aren't >> present in the jimage (`lib/modules`). This enables producing a >> `JmodLessArchive` class which has all the info of what constitutes the final >> jlinked runtime. >> >> Basic usage example: >> >> $ diff -u <(./bin/java --list-modules --limit-modules java.se) >> <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules >> --limit-modules java.se) >> $ diff -u <(./bin/java --list-modules --limit-modules java.se) >> <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules >> --limit-modules jdk.jlink) >> $ ls ../linux-x86_64-server-release/images/jdk/jmods >> java.base.jmodjava.net.http.jmod java.sql.rowset.jmod >> jdk.crypto.ec.jmod jdk.internal.opt.jmod >> jdk.jdi.jmod jdk.management.agent.jmod jdk.security.auth.jmod >> java.compiler.jmodjava.prefs.jmod java.transaction.xa.jmod >> jdk.dynalink.jmod jdk.internal.vm.ci.jmod >> jdk.jdwp.agent.jmod jdk.management.jfr.jmodjdk.security.jgss.jmod >> java.datatransfer.jmodjava.rmi.jmodjava.xml.crypto.jmod >> jdk.editpad.jmod jdk.internal.vm.compiler.jmod >> jdk.jfr.jmod jdk.management.jmodjdk.unsupported.desktop.jmod >> java.desktop.jmod java.scripting.jmod java.xml.jmod >> jdk.hotspot.agent.jmod jdk.internal.vm.compiler.management.jmod >> jdk.jlink.jmod jdk.naming.dns.j... > > Severin Gehwolf has updated the pull request incrementally with two > additional commits since the last revision: > > - Implementation for run-image link and single-hop > >This uses a stamp file in 'lib/modules' jimage in order to recognize >multi-hop links. However, this has the caveat of no longer producing >reproducible builds as compared to a packaged module-based link. > >Add an option to allow for multi-hop, which disables the stamp file >addition and makes it reproducible again. > - Exit the jlink on modified files by default > >This is configurable so add tests for both scenarios. @AlanBateman Gentle ping. - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-1697359809
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v3]
> Please review this patch which adds a "jmodless" jlink mode to the JDK. > Fundamentally this patch adds an option to use `jlink` even though your JDK > install might not come with the packaged modules (directory `jmods`). This is > particularly useful to further reduce the size of a jlinked runtime. After > the removal of the concept of a JRE, a common distribution mechanism is still > the full JDK with all modules and packaged modules. However, packaged modules > can incur an additional size tax. For example in a container scenario it > could be useful to have a base JDK container including all modules, but > without also delivering the packaged modules. This comes at a size advantage > of `~25%`. Such a base JDK container could then be used to `jlink` > application specific runtimes, further reducing the size of the application > runtime image (App + JDK runtime; as a single image *or* separate bundles, > depending on the app being modularized). > > The basic design of this approach is to add a jlink plugin for tracking > non-class and non-resource files of a JDK install. I.e. files which aren't > present in the jimage (`lib/modules`). This enables producing a > `JmodLessArchive` class which has all the info of what constitutes the final > jlinked runtime. > > Basic usage example: > > $ diff -u <(./bin/java --list-modules --limit-modules java.se) > <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules > --limit-modules java.se) > $ diff -u <(./bin/java --list-modules --limit-modules java.se) > <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules > --limit-modules jdk.jlink) > $ ls ../linux-x86_64-server-release/images/jdk/jmods > java.base.jmodjava.net.http.jmod java.sql.rowset.jmod > jdk.crypto.ec.jmod jdk.internal.opt.jmod > jdk.jdi.jmod jdk.management.agent.jmod jdk.security.auth.jmod > java.compiler.jmodjava.prefs.jmod java.transaction.xa.jmod > jdk.dynalink.jmod jdk.internal.vm.ci.jmod > jdk.jdwp.agent.jmod jdk.management.jfr.jmodjdk.security.jgss.jmod > java.datatransfer.jmodjava.rmi.jmodjava.xml.crypto.jmod > jdk.editpad.jmod jdk.internal.vm.compiler.jmod > jdk.jfr.jmod jdk.management.jmodjdk.unsupported.desktop.jmod > java.desktop.jmod java.scripting.jmod java.xml.jmod > jdk.hotspot.agent.jmod jdk.internal.vm.compiler.management.jmod > jdk.jlink.jmod jdk.naming.dns.jmodjdk.unsupported... Severin Gehwolf has updated the pull request incrementally with two additional commits since the last revision: - Implementation for run-image link and single-hop This uses a stamp file in 'lib/modules' jimage in order to recognize multi-hop links. However, this has the caveat of no longer producing reproducible builds as compared to a packaged module-based link. Add an option to allow for multi-hop, which disables the stamp file addition and makes it reproducible again. - Exit the jlink on modified files by default This is configurable so add tests for both scenarios. - Changes: - all: https://git.openjdk.org/jdk/pull/14787/files - new: https://git.openjdk.org/jdk/pull/14787/files/baeaaf5d..738b8a31 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk=14787=02 - incr: https://webrevs.openjdk.org/?repo=jdk=14787=01-02 Stats: 529 lines in 15 files changed: 450 ins; 45 del; 34 mod Patch: https://git.openjdk.org/jdk/pull/14787.diff Fetch: git fetch https://git.openjdk.org/jdk.git pull/14787/head:pull/14787 PR: https://git.openjdk.org/jdk/pull/14787
