Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v12]
On Fri, 8 Mar 2024 19:49:19 GMT, Mandy Chung wrote: >>> > @AlanBateman @mlchung I've now pushed an update of this patch which now >>> > uses a build-time approach as discussed elsewhere. In order to produce a >>> > linkable runtime JDK image, one needs to set --enable-runtime-link-image >>> > at configure time. >>> >>> What is the rationale for introducing a special configure flag for this >>> functionality? I've tried to look though all comments in this PR, and read >>> the JBS issue and CSR, but I have not found any motivation for this. I'm >>> sorry if it's there and I missed it, but this is a huge PR with a lot of >>> discussion. >> >> Sorry, yes this was part of a meeting discussion we had outside this PR. My >> understanding is that by default the produced jimage isn't runtime-link >> enabled. We (Red Hat) would turn it on in our builds, though. @AlanBateman >> or @mlchung might have more details. I think it was a requirement to get >> this patch in. At least for the initial contribution. >> >>> In general, it is better not to introduce variants of the build like this. >>> The testing matrix just explodes a bit more. And my understanding from the >>> discussion is that this functionality is likely to be turned on anyway, >>> otherwise you'll build a crippled jlink without full functionality. >> >> I would be happy if this could be on by default. For now, I think though we >> need to work on the premise that whether or not the resulting JDK image is >> suitable for runtime linking (without jmods) is a build-time config >> decision. Therefore we need the configure option. > > @jerboaa thanks for the update. First to recap the revised proposal (based > on Alan's notes shared with me earlier): > > The JDK build is capable of producing a JDK run-time image that does not > include packaged modules and the new JDK image is capable to create custom > run-time images (call it "linkable" JDK image for now). To reconstitute to > the original module content, the new JDK image conceptually needs to contain > the "diffs" from the original packaged packaged. This makes it possible for > the jlink plugins to run "as if" the resources for each module were coming > from the original packaged module. The new image also has the checksums of > at least the user-editable configuration files so that edits can be detected. > > The revised proposal has a few limitations: > > 1. The "linkable" JDK image can only be created by the JDK build. > 2. The "linkable" JDK image is created from the JDK image produced by the JDK > build today. It contains the same set of modules, there is no possibility to > combine its generation with other jlink options or code transformations. > 3. The "linkable" JDK image cannot create a run-time image that contains the > "jdk.jlink" module. > 4. The "linkable" JDK image only reconstitutes classes/resources to the > original module bits. Other plugins such as man pages and dedup legal files > are not reconstituted. > > These limitations are trade-off to make for a much simpler approach. It > removes the issues of creating a linkable JDK, or creating another image > using a linkable JDK, while at the same time executing a pipeline of plugins > that do other transformations. > > Here is my feedback to the prototype: > > - I think the reader and writer code for generating the diffs should be in > `jdk.jlink` so that they are always kept in sync. > - The diffs generation can be done as a tool or `jlink` option. If it's a > tool, it should be run on the interim image (like the tool generating > classlist - see `GenerateLinkOptData.gmk`). > > I have no strong opinion in which approach. It seems worth exploring the > tool approach residing in `jdk.jlink` to generate the diffs as well as create > the linkable image could remove the need of a jlink internal > `--create-linkable-runtime` option. Hi @mlchung! > Here is my feedback to the prototype: > > * I think the reader and writer code for generating the diffs should be > in `jdk.jlink` so that they are always kept in sync. Done in the update. The duplication of `ResourceDiff` is now gone. > * The diffs generation can be done as a tool or `jlink` option. If it's > a tool, it should be run on the interim image (like the tool generating > classlist - see `GenerateLinkOptData.gmk`). > > I have no strong opinion in which approach. It seems worth exploring the tool > approach residing in `jdk.jlink` to generate the diffs as well as create the > linkable image could remove the need of a jlink internal > `--create-linkable-runtime` option. The updated patch uses a **build-only** `jlink` plugin, still called `--create-linkable-runtime` which is *a)* added only at build time with `--add-modules` and *b)* now generates the diff and prepares the image for runtime linking as before in one step. The code for this now lives in `src/jdk.jlink/build/classes`. - PR
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v12]
On Fri, 8 Mar 2024 16:47:33 GMT, Magnus Ihse Bursie wrote: > What is the rationale for introducing a special configure flag for this > functionality? I've tried to look though all comments in this PR, and read > the JBS issue and CSR, but I have not found any motivation for this. I'm > sorry if it's there and I missed it, but this is a huge PR with a lot of > discussion. > > In general, it is better not to introduce variants of the build like this. > The testing matrix just explodes a bit more. And my understanding from the > discussion is that this functionality is likely to be turned on anyway, > otherwise you'll build a crippled jlink without full functionality. The JDK image (images/jdk) includes the packaged modules (as .jmod files) that the jlink tool can use to create other run-time images. The proposal here isn't meant to change this. "make images" should create the JDK image as before and that image will include the packaged modules. The inclusion of the packaged modules has been problematic in some environments, esp. when there are debug symbols. libjvm.so can be huge, which begs the question as to why there is a copy in java.base.jmod. There are of course options to build the JDK image without --keep-packaged-modules and host the packaged modules as a separate download. Back in the JDK 9 we decided not to do this for Oracle downloads. Severin has implemented an approach that allows "observable modules" be found in the current run-time image when using jlink. This requires some heretics and computation of diffs between the bits in the original packaged modules and the transformed bits in the run-time image. This exploration has gone through many iterations. Recently, Severin, Mandy and I have met to try to simplify things and tame the goals in order to get to something that is maintainable, and to allow time to get experience with this direction. So at a high-level, the intention is that the build be capable of producing an alternative JDK image that doesn't have a "jmods" directory. The jlink tool in that image has a limitation, a compromise to keep the complexity at a manageable level. I have no opinion on whether the opt-in is at configure time or its just make target (like "make legacy-jre-image"). In the discussions it wasn't meant to be built by default. If distributions choose to distribute this image then this will likely influence what they test. Once experience builds up with using these run-time images then it may be that there is a proposal (and JEP) to make it the default, meaning images/jdk would not include .jmod files and multi-hop restriction is removed from jlink. That may be something for much further down the road. - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-1986796401
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v12]
On Fri, 8 Mar 2024 17:19:41 GMT, Severin Gehwolf wrote: >> I tried out the latest commit (a797ea69). >> >> The output "The default module path, '$java.home/jmods' not present. Use >> --verbose to show the full list of plugin options applied" is bit confusing >> as it looks like jlink failed but it actually succeeded. Blowing away the >> generated image and retrying with --verbose tripped this assert >> >> java.lang.AssertionError: handling of scratch options failed >> at >> jdk.jlink/jdk.tools.jlink.internal.JlinkTask.logPackagedModuleEquivalent(JlinkTask.java:675) >> at >> jdk.jlink/jdk.tools.jlink.internal.JlinkTask.createImageProvider(JlinkTask.java:581) >> at >> jdk.jlink/jdk.tools.jlink.internal.JlinkTask.createImage(JlinkTask.java:430) >> at jdk.jlink/jdk.tools.jlink.internal.JlinkTask.run(JlinkTask.java:302) >> at jdk.jlink/jdk.tools.jlink.internal.Main.run(Main.java:56) >> at jdk.jlink/jdk.tools.jlink.internal.Main.main(Main.java:34) >> Caused by: jdk.tools.jlink.internal.TaskHelper$BadArgs: >> (...my-original-jdk-directory..)/build/linux-x64/images/jdk/jmods already >> exists >> at >> jdk.jlink/jdk.tools.jlink.internal.TaskHelper.newBadArgs(TaskHelper.java:730) >> at >> jdk.jlink/jdk.tools.jlink.internal.JlinkTask.lambda$static$12(JlinkTask.java:183) >> at >> jdk.jlink/jdk.tools.jlink.internal.TaskHelper$Option.process(TaskHelper.java:177) >> at >> jdk.jlink/jdk.tools.jlink.internal.TaskHelper$OptionsHelper.handleOptions(TaskHelper.java:600) >> at >> jdk.jlink/jdk.tools.jlink.internal.JlinkTask.logPackagedModuleEquivalent(JlinkTask.java:672) >> ... 5 more >> >> I haven't dug into this yet but I'm puzzled that the file path to where the >> original build was created is in the exception messages, is that recorded? > >> > @AlanBateman @mlchung I've now pushed an update of this patch which now >> > uses a build-time approach as discussed elsewhere. In order to produce a >> > linkable runtime JDK image, one needs to set --enable-runtime-link-image >> > at configure time. >> >> What is the rationale for introducing a special configure flag for this >> functionality? I've tried to look though all comments in this PR, and read >> the JBS issue and CSR, but I have not found any motivation for this. I'm >> sorry if it's there and I missed it, but this is a huge PR with a lot of >> discussion. > > Sorry, yes this was part of a meeting discussion we had outside this PR. My > understanding is that by default the produced jimage isn't runtime-link > enabled. We (Red Hat) would turn it on in our builds, though. @AlanBateman or > @mlchung might have more details. I think it was a requirement to get this > patch in. At least for the initial contribution. > >> In general, it is better not to introduce variants of the build like this. >> The testing matrix just explodes a bit more. And my understanding from the >> discussion is that this functionality is likely to be turned on anyway, >> otherwise you'll build a crippled jlink without full functionality. > > I would be happy if this could be on by default. For now, I think though we > need to work on the premise that whether or not the resulting JDK image is > suitable for runtime linking (without jmods) is a build-time config decision. > Therefore we need the configure option. @jerboaa thanks for the update. First to recap the revised proposal (based on Alan's notes shared with me earlier): The JDK build is capable of producing a JDK run-time image that does not include packaged modules and the new JDK image is capable to create custom run-time images (call it "linkable" JDK image for now). To reconstitute to the original module content, the new JDK image conceptually needs to contain the "diffs" from the original packaged packaged. This makes it possible for the jlink plugins to run "as if" the resources for each module were coming from the original packaged module. The new image also has the checksums of at least the user-editable configuration files so that edits can be detected. The revised proposal has a few limitations: 1. The "linkable" JDK image can only be created by the JDK build. 2. The "linkable" JDK image is created from the JDK image produced by the JDK build today. It contains the same set of modules, there is no possibility to combine its generation with other jlink options or code transformations. 3. The "linkable" JDK image cannot create a run-time image that contains the "jdk.jlink" module. 4. The "linkable" JDK image only reconstitutes classes/resources to the original module bits. Other plugins such as man pages and dedup legal files are not reconstituted. These limitations are trade-off to make for a much simpler approach. It removes the issues of creating a linkable JDK, or creating another image using a linkable JDK, while at the same time executing a pipeline of plugins that do other transformations. Here i
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v12]
On Fri, 8 Dec 2023 15:44:07 GMT, Alan Bateman wrote: >> Severin Gehwolf has updated the pull request with a new target base due to a >> merge or a rebase. The pull request now contains 46 commits: >> >> - Add @enablePreview for JImageValidator that uses classfile API >> - Fix SystemModulesPlugin after merge >> - Merge branch 'master' into jdk-8311302-jmodless-link >> - Don't show the verbose hint when already verbose >> - Use '_files' over '_resources' as the suffix for listing resources >> - Remove the hidden option hint. >> >>Also adjust the messages being printed when performing >>a run-time image link. >> - Localize messages, switch expression >> - Rename RunImageArchive => JRTArchive and RunImageLinkException => >> RuntimeImageLinkException >> >>Also moved the stamp file to jdk.jlink module. The resources files per >>module now get unconditionally created (empty if no resources not in the >>jimage). >> - First round of addressing review feedback. >> >>- Share resource names (JlinkTask and JlinkResourcesListPlugin) >>- Exclude resources in JlinkResourcesListPlugin the same way >> as done for other plugins. >> - Rename AddRunImageResourcesPlugin => JlinkResourcesListPlugin >> - ... and 36 more: https://git.openjdk.org/jdk/compare/87516e29...a797ea69 > > I tried out the latest commit (a797ea69). > > The output "The default module path, '$java.home/jmods' not present. Use > --verbose to show the full list of plugin options applied" is bit confusing > as it looks like jlink failed but it actually succeeded. Blowing away the > generated image and retrying with --verbose tripped this assert > > java.lang.AssertionError: handling of scratch options failed > at > jdk.jlink/jdk.tools.jlink.internal.JlinkTask.logPackagedModuleEquivalent(JlinkTask.java:675) > at > jdk.jlink/jdk.tools.jlink.internal.JlinkTask.createImageProvider(JlinkTask.java:581) > at > jdk.jlink/jdk.tools.jlink.internal.JlinkTask.createImage(JlinkTask.java:430) > at jdk.jlink/jdk.tools.jlink.internal.JlinkTask.run(JlinkTask.java:302) > at jdk.jlink/jdk.tools.jlink.internal.Main.run(Main.java:56) > at jdk.jlink/jdk.tools.jlink.internal.Main.main(Main.java:34) > Caused by: jdk.tools.jlink.internal.TaskHelper$BadArgs: > (...my-original-jdk-directory..)/build/linux-x64/images/jdk/jmods already > exists > at > jdk.jlink/jdk.tools.jlink.internal.TaskHelper.newBadArgs(TaskHelper.java:730) > at > jdk.jlink/jdk.tools.jlink.internal.JlinkTask.lambda$static$12(JlinkTask.java:183) > at > jdk.jlink/jdk.tools.jlink.internal.TaskHelper$Option.process(TaskHelper.java:177) > at > jdk.jlink/jdk.tools.jlink.internal.TaskHelper$OptionsHelper.handleOptions(TaskHelper.java:600) > at > jdk.jlink/jdk.tools.jlink.internal.JlinkTask.logPackagedModuleEquivalent(JlinkTask.java:672) > ... 5 more > > I haven't dug into this yet but I'm puzzled that the file path to where the > original build was created is in the exception messages, is that recorded? > > @AlanBateman @mlchung I've now pushed an update of this patch which now > > uses a build-time approach as discussed elsewhere. In order to produce a > > linkable runtime JDK image, one needs to set --enable-runtime-link-image at > > configure time. > > What is the rationale for introducing a special configure flag for this > functionality? I've tried to look though all comments in this PR, and read > the JBS issue and CSR, but I have not found any motivation for this. I'm > sorry if it's there and I missed it, but this is a huge PR with a lot of > discussion. Sorry, yes this was part of a meeting discussion we had outside this PR. My understanding is that by default the produced jimage isn't runtime-link enabled. We (Red Hat) would turn it on in our builds, though. @AlanBateman or @mlchung might have more details. I think it was a requirement to get this patch in. At least for the initial contribution. > In general, it is better not to introduce variants of the build like this. > The testing matrix just explodes a bit more. And my understanding from the > discussion is that this functionality is likely to be turned on anyway, > otherwise you'll build a crippled jlink without full functionality. I would be happy if this could be on by default. For now, I think though we need to work on the premise that whether or not the resulting JDK image is suitable for runtime linking (without jmods) is a build-time config decision. Therefore we need the configure option. - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-1986098525
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v12]
On Fri, 8 Dec 2023 15:44:07 GMT, Alan Bateman wrote: >> Severin Gehwolf has updated the pull request with a new target base due to a >> merge or a rebase. The pull request now contains 46 commits: >> >> - Add @enablePreview for JImageValidator that uses classfile API >> - Fix SystemModulesPlugin after merge >> - Merge branch 'master' into jdk-8311302-jmodless-link >> - Don't show the verbose hint when already verbose >> - Use '_files' over '_resources' as the suffix for listing resources >> - Remove the hidden option hint. >> >>Also adjust the messages being printed when performing >>a run-time image link. >> - Localize messages, switch expression >> - Rename RunImageArchive => JRTArchive and RunImageLinkException => >> RuntimeImageLinkException >> >>Also moved the stamp file to jdk.jlink module. The resources files per >>module now get unconditionally created (empty if no resources not in the >>jimage). >> - First round of addressing review feedback. >> >>- Share resource names (JlinkTask and JlinkResourcesListPlugin) >>- Exclude resources in JlinkResourcesListPlugin the same way >> as done for other plugins. >> - Rename AddRunImageResourcesPlugin => JlinkResourcesListPlugin >> - ... and 36 more: https://git.openjdk.org/jdk/compare/87516e29...a797ea69 > > I tried out the latest commit (a797ea69). > > The output "The default module path, '$java.home/jmods' not present. Use > --verbose to show the full list of plugin options applied" is bit confusing > as it looks like jlink failed but it actually succeeded. Blowing away the > generated image and retrying with --verbose tripped this assert > > java.lang.AssertionError: handling of scratch options failed > at > jdk.jlink/jdk.tools.jlink.internal.JlinkTask.logPackagedModuleEquivalent(JlinkTask.java:675) > at > jdk.jlink/jdk.tools.jlink.internal.JlinkTask.createImageProvider(JlinkTask.java:581) > at > jdk.jlink/jdk.tools.jlink.internal.JlinkTask.createImage(JlinkTask.java:430) > at jdk.jlink/jdk.tools.jlink.internal.JlinkTask.run(JlinkTask.java:302) > at jdk.jlink/jdk.tools.jlink.internal.Main.run(Main.java:56) > at jdk.jlink/jdk.tools.jlink.internal.Main.main(Main.java:34) > Caused by: jdk.tools.jlink.internal.TaskHelper$BadArgs: > (...my-original-jdk-directory..)/build/linux-x64/images/jdk/jmods already > exists > at > jdk.jlink/jdk.tools.jlink.internal.TaskHelper.newBadArgs(TaskHelper.java:730) > at > jdk.jlink/jdk.tools.jlink.internal.JlinkTask.lambda$static$12(JlinkTask.java:183) > at > jdk.jlink/jdk.tools.jlink.internal.TaskHelper$Option.process(TaskHelper.java:177) > at > jdk.jlink/jdk.tools.jlink.internal.TaskHelper$OptionsHelper.handleOptions(TaskHelper.java:600) > at > jdk.jlink/jdk.tools.jlink.internal.JlinkTask.logPackagedModuleEquivalent(JlinkTask.java:672) > ... 5 more > > I haven't dug into this yet but I'm puzzled that the file path to where the > original build was created is in the exception messages, is that recorded? > @AlanBateman @mlchung I've now pushed an update of this patch which now uses > a build-time approach as discussed elsewhere. In order to produce a linkable > runtime JDK image, one needs to set --enable-runtime-link-image at configure > time. What is the rationale for introducing a special configure flag for this functionality? I've tried to look though all comments in this PR, and read the JBS issue and CSR, but I have not found any motivation for this. I'm sorry if it's there and I missed it, but this is a huge PR with a lot of discussion. In general, it is better not to introduce variants of the build like this. The testing matrix just explodes a bit more. And my understanding from the discussion is that this functionality is likely to be turned on anyway, otherwise you'll build a crippled jlink without full functionality. - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-1986042463
