Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Thu, 23 May 2024 18:52:53 GMT, Alan Bateman wrote: > Yes, I want to help you get this one over the line. Thanks! Appreciate that. - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-2133375454
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Thu, 23 May 2024 16:42:39 GMT, Severin Gehwolf wrote: > Do you think you'll be able to review this next week? Yes, I want to help you get this one over the line. - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-2127828050
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Thu, 16 May 2024 13:47:20 GMT, Alan Bateman wrote: >>> If I understand you correctly, this would be no longer a build-time only >>> approach to produce the "linkable runtime"? It would be some-kind of >>> jlink-option driven approach (as it would run some code that should only >>> run when producing a linkable runtime is requested)? Other than that, it >>> should be fine as the previous iteration basically did that but at a >>> different phase. Also note that the previous iteration used a build-only >>> jlink plugin so as to satisfy the build-time only approach, yet it ran as >>> part of the plugin-pipeline which wasn't desired either. But it was >>> something similar to what you seem to be describing now. Did I get >>> something wrong? >> >> I think it continues to build time, it's just using some hidden jlink >> option. So yes, it similar to a previous iteration except that it doesn't >> run as a plugin the pipeline and the delta goes to the lib directory. >> >> Let's see what @mlchung says. You've put a lot of work into this so let's >> see if we can converge to avoid too many more rounds. > >> @AlanBateman @mlchung The latest update now uses the `jlink` build time >> option `--generate-linkable-runtime` to add needed resources to the `jimage` >> when a runtime linkable JDK image is being asked for using the configure >> option. This now runs outside the plugin-pipeline. I think this is what you >> meant. Sorry it took longer to get back to this... > > I think you've got this to a good place and I think the overall solution is > good. It may be that JDK should move to this by default in the future, and at > the same time re-visit the restriction on generating an image containing > jdk.jlink, but let's see if any issues come up. > > I've added myself as Reviewer to the CSR and I'm working through the code > changes. @AlanBateman > I'm working through the code changes. Do you think you'll be able to review this next week? - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-2127614784
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Tue, 16 Apr 2024 13:54:53 GMT, Alan Bateman wrote: >>> > @mlchung @AlanBateman Any thoughts on this latest version? Is this going >>> > into the direction you had envisioned? Any more blockers? The CSR should >>> > be up-to-date and is open for review as well. If no more blockers I'll go >>> > over this patch once more and clean it up to prepare for integration. >>> > Thanks! >>> >>> Thanks for all the efforts on this. >> >> Thanks for looking at this, Alan! >> >>> I've looked through the latest version. I'm a little bit comfortable with >>> LinkDeltaProducer. One reason it's build tool that is tied tied to code in >>> the jdk.jlink module, and secondly is that it copies >>> runtime-image-link.delta into the run-time image. Our long standing >>> position is that the run-time image is created by jlink rather than a >>> combination of jlink and extra stuff copied in by the build. >>> >>> The part that I like with the current proposal is >>> lib/runtime-image-link.delta as it's less complicated that previous >>> iterations that added a resource into the jimage container. >>> >>> What would you (and @mlchung) think of having jlink generate >>> lib/runtime-image-link.delta as a step between the pipeline and image >>> generation? >> >> If I understand you correctly, this would be no longer a build-time only >> approach to produce the "linkable runtime"? It would be some-kind of >> jlink-option driven approach (as it would run some code that should only run >> when producing a linkable runtime is requested)? Other than that, it should >> be fine as the previous iteration basically did that but at a different >> phase. Also note that the previous iteration used a build-only jlink plugin >> so as to satisfy the build-time only approach, yet it ran as part of the >> plugin-pipeline which wasn't desired either. But it was something similar to >> what you seem to be describing now. Did I get something wrong? > >> If I understand you correctly, this would be no longer a build-time only >> approach to produce the "linkable runtime"? It would be some-kind of >> jlink-option driven approach (as it would run some code that should only run >> when producing a linkable runtime is requested)? Other than that, it should >> be fine as the previous iteration basically did that but at a different >> phase. Also note that the previous iteration used a build-only jlink plugin >> so as to satisfy the build-time only approach, yet it ran as part of the >> plugin-pipeline which wasn't desired either. But it was something similar to >> what you seem to be describing now. Did I get something wrong? > > I think it continues to build time, it's just using some hidden jlink option. > So yes, it similar to a previous iteration except that it doesn't run as a > plugin the pipeline and the delta goes to the lib directory. > > Let's see what @mlchung says. You've put a lot of work into this so let's see > if we can converge to avoid too many more rounds. > @AlanBateman @mlchung The latest update now uses the `jlink` build time > option `--generate-linkable-runtime` to add needed resources to the `jimage` > when a runtime linkable JDK image is being asked for using the configure > option. This now runs outside the plugin-pipeline. I think this is what you > meant. Sorry it took longer to get back to this... I think you've got this to a good place and I think the overall solution is good. It may be that JDK should move to this by default in the future, and at the same time re-visit the restriction on generating an image containing jdk.jlink, but let's see if any issues come up. I've added myself as Reviewer to the CSR and I'm working through the code changes. - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-2115298661
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Tue, 16 Apr 2024 13:54:53 GMT, Alan Bateman wrote: >>> > @mlchung @AlanBateman Any thoughts on this latest version? Is this going >>> > into the direction you had envisioned? Any more blockers? The CSR should >>> > be up-to-date and is open for review as well. If no more blockers I'll go >>> > over this patch once more and clean it up to prepare for integration. >>> > Thanks! >>> >>> Thanks for all the efforts on this. >> >> Thanks for looking at this, Alan! >> >>> I've looked through the latest version. I'm a little bit comfortable with >>> LinkDeltaProducer. One reason it's build tool that is tied tied to code in >>> the jdk.jlink module, and secondly is that it copies >>> runtime-image-link.delta into the run-time image. Our long standing >>> position is that the run-time image is created by jlink rather than a >>> combination of jlink and extra stuff copied in by the build. >>> >>> The part that I like with the current proposal is >>> lib/runtime-image-link.delta as it's less complicated that previous >>> iterations that added a resource into the jimage container. >>> >>> What would you (and @mlchung) think of having jlink generate >>> lib/runtime-image-link.delta as a step between the pipeline and image >>> generation? >> >> If I understand you correctly, this would be no longer a build-time only >> approach to produce the "linkable runtime"? It would be some-kind of >> jlink-option driven approach (as it would run some code that should only run >> when producing a linkable runtime is requested)? Other than that, it should >> be fine as the previous iteration basically did that but at a different >> phase. Also note that the previous iteration used a build-only jlink plugin >> so as to satisfy the build-time only approach, yet it ran as part of the >> plugin-pipeline which wasn't desired either. But it was something similar to >> what you seem to be describing now. Did I get something wrong? > >> If I understand you correctly, this would be no longer a build-time only >> approach to produce the "linkable runtime"? It would be some-kind of >> jlink-option driven approach (as it would run some code that should only run >> when producing a linkable runtime is requested)? Other than that, it should >> be fine as the previous iteration basically did that but at a different >> phase. Also note that the previous iteration used a build-only jlink plugin >> so as to satisfy the build-time only approach, yet it ran as part of the >> plugin-pipeline which wasn't desired either. But it was something similar to >> what you seem to be describing now. Did I get something wrong? > > I think it continues to build time, it's just using some hidden jlink option. > So yes, it similar to a previous iteration except that it doesn't run as a > plugin the pipeline and the delta goes to the lib directory. > > Let's see what @mlchung says. You've put a lot of work into this so let's see > if we can converge to avoid too many more rounds. @AlanBateman @mlchung The latest update now uses the `jlink` build time option `--generate-linkable-runtime` to add needed resources to the `jimage` when a runtime linkable JDK image is being asked for using the configure option. This now runs outside the plugin-pipeline. I think this is what you meant. Sorry it took longer to get back to this... - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-2098895722
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Tue, 16 Apr 2024 13:54:53 GMT, Alan Bateman wrote: > I think it continues to build time, it's just using some hidden jlink option. > So yes, it similar to a previous iteration except that it doesn't run as a > plugin the pipeline and the delta goes to the lib directory. OK. If a hidden jlink option is good enough, then this works for me. I hope to have time to update the patch in the coming days. - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-2060691122
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Wed, 3 Apr 2024 22:31:39 GMT, Mandy Chung wrote: >> Severin Gehwolf has updated the pull request incrementally with one >> additional commit since the last revision: >> >> Move CreateLinkableRuntimePlugin to build folder >> >> Keep runtime link supporting classes in package >> jdk.tools.jlink.internal.runtimelink > > Thanks for the investigation w.r.t. extending jimage. It does not seem worth > the effort in pursuing the support of adding resources to an existing jimage > file. To me, putting the diff data under `lib` directory as a private file > is a simpler and acceptable solution. It allows the second jlink invocation > to avoid the plugin pipeline if the per-module metadata is generated in > normal jlink invocation. > > (An observation: The current implementation requires the diffs to be > generated as a plugin. For this approach, it may be possible to implement > with a separate main class that calls JLink API and generates the diffs.) > What would you (and @mlchung) think of having jlink generate > lib/runtime-image-link.delta as a step between the pipeline and image > generation? I have similar thought before I went on vacation last week and also think that this would allow this be further simplified to one single jlink invocation to produce a linkable image. In `ImageFileCreator::generateJImage`, after the plugin pipeline, it already adds a step to record the per-module meta information as additional resources. The delta file can be generated after that comparing the original resource pool with the transformed resource pool. I believe `allContent` has the original content. If it is a single jlink step to produce a linkable image (generate the delta file after the plugin pipeline before generating the image), we might be able to get rid of `--ignore-modified-runtime` by storing a copy of conf files in the delta file and the original conf files will be used when the linkable image is used to create a new runtime image. [An observation from the above, it might be possible to generate the delta file and insert in the jimage :-) This might be a bonus. ] - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-2059631414
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Tue, 16 Apr 2024 11:59:12 GMT, Severin Gehwolf wrote: > If I understand you correctly, this would be no longer a build-time only > approach to produce the "linkable runtime"? It would be some-kind of > jlink-option driven approach (as it would run some code that should only run > when producing a linkable runtime is requested)? Other than that, it should > be fine as the previous iteration basically did that but at a different > phase. Also note that the previous iteration used a build-only jlink plugin > so as to satisfy the build-time only approach, yet it ran as part of the > plugin-pipeline which wasn't desired either. But it was something similar to > what you seem to be describing now. Did I get something wrong? I think it continues to build time, it's just using some hidden jlink option. So yes, it similar to a previous iteration except that it doesn't run as a plugin the pipeline and the delta goes to the lib directory. Let's see what @mlchung says. You've put a lot of work into this so let's see if we can converge to avoid too many more rounds. - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-2059157584
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Wed, 3 Apr 2024 22:31:39 GMT, Mandy Chung wrote: >> Severin Gehwolf has updated the pull request incrementally with one >> additional commit since the last revision: >> >> Move CreateLinkableRuntimePlugin to build folder >> >> Keep runtime link supporting classes in package >> jdk.tools.jlink.internal.runtimelink > > Thanks for the investigation w.r.t. extending jimage. It does not seem worth > the effort in pursuing the support of adding resources to an existing jimage > file. To me, putting the diff data under `lib` directory as a private file > is a simpler and acceptable solution. It allows the second jlink invocation > to avoid the plugin pipeline if the per-module metadata is generated in > normal jlink invocation. > > (An observation: The current implementation requires the diffs to be > generated as a plugin. For this approach, it may be possible to implement > with a separate main class that calls JLink API and generates the diffs.) > > @mlchung @AlanBateman Any thoughts on this latest version? Is this going > > into the direction you had envisioned? Any more blockers? The CSR should be > > up-to-date and is open for review as well. If no more blockers I'll go over > > this patch once more and clean it up to prepare for integration. Thanks! > > Thanks for all the efforts on this. Thanks for looking at this, Alan! > I've looked through the latest version. I'm a little bit comfortable with > LinkDeltaProducer. One reason it's build tool that is tied tied to code in > the jdk.jlink module, and secondly is that it copies runtime-image-link.delta > into the run-time image. Our long standing position is that the run-time > image is created by jlink rather than a combination of jlink and extra stuff > copied in by the build. > > The part that I like with the current proposal is > lib/runtime-image-link.delta as it's less complicated that previous > iterations that added a resource into the jimage container. > > What would you (and @mlchung) think of having jlink generate > lib/runtime-image-link.delta as a step between the pipeline and image > generation? If I understand you correctly, this would be no longer a build-time only approach to produce the "linkable runtime"? It would be some-kind of jlink-option driven approach (as it would run some code that should only run when producing a linkable runtime is requested)? Other than that, it should be fine as the previous iteration basically did that but at a different phase. Also note that the previous iteration used a build-only jlink plugin so as to satisfy the build-time only approach, yet it ran as part of the plugin-pipeline which wasn't desired either. But it was something similar to what you seem to be describing now. Did I get something wrong? - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-2058919838
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Wed, 3 Apr 2024 22:31:39 GMT, Mandy Chung wrote: >> Severin Gehwolf has updated the pull request incrementally with one >> additional commit since the last revision: >> >> Move CreateLinkableRuntimePlugin to build folder >> >> Keep runtime link supporting classes in package >> jdk.tools.jlink.internal.runtimelink > > Thanks for the investigation w.r.t. extending jimage. It does not seem worth > the effort in pursuing the support of adding resources to an existing jimage > file. To me, putting the diff data under `lib` directory as a private file > is a simpler and acceptable solution. It allows the second jlink invocation > to avoid the plugin pipeline if the per-module metadata is generated in > normal jlink invocation. > > (An observation: The current implementation requires the diffs to be > generated as a plugin. For this approach, it may be possible to implement > with a separate main class that calls JLink API and generates the diffs.) > @mlchung @AlanBateman Any thoughts on this latest version? Is this going into > the direction you had envisioned? Any more blockers? The CSR should be > up-to-date and is open for review as well. If no more blockers I'll go over > this patch once more and clean it up to prepare for integration. Thanks! Thanks for all the efforts on this. I've looked through the latest version. I'm a little bit comfortable with LinkDeltaProducer. One reason it's build tool that is tied tied to code in the jdk.jlink module, and secondly is that it copies runtime-image-link.delta into the run-time image. Our long standing position is that the run-time image is created by jlink rather than a combination of jlink and extra stuff copied in by the build. The part that I like with the current proposal is lib/runtime-image-link.delta as it's less complicated that previous iterations that added a resource into the jimage container. What would you (and @mlchung) think of having jlink generate lib/runtime-image-link.delta as a step between the pipeline and image generation? - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-2058876568
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Wed, 3 Apr 2024 22:31:39 GMT, Mandy Chung wrote: >> Severin Gehwolf has updated the pull request incrementally with one >> additional commit since the last revision: >> >> Move CreateLinkableRuntimePlugin to build folder >> >> Keep runtime link supporting classes in package >> jdk.tools.jlink.internal.runtimelink > > Thanks for the investigation w.r.t. extending jimage. It does not seem worth > the effort in pursuing the support of adding resources to an existing jimage > file. To me, putting the diff data under `lib` directory as a private file > is a simpler and acceptable solution. It allows the second jlink invocation > to avoid the plugin pipeline if the per-module metadata is generated in > normal jlink invocation. > > (An observation: The current implementation requires the diffs to be > generated as a plugin. For this approach, it may be possible to implement > with a separate main class that calls JLink API and generates the diffs.) @mlchung @AlanBateman Any thoughts on this latest version? Is this going into the direction you had envisioned? Any more blockers? The CSR should be up-to-date and is open for review as well. If no more blockers I'll go over this patch once more and clean it up to prepare for integration. Thanks! - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-2044593102
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Thu, 21 Mar 2024 15:29:45 GMT, Severin Gehwolf wrote:
>> make/Images.gmk line 145:
>>
>>> 143: $(eval $(call SetupJavaCompilation, BUILD_JDK_RUNLINK_CLASSES, \
>>> 144: COMPILER := buildjdk, \
>>> 145: DISABLED_WARNINGS := try, \
>>
>> Why do we get warnings in the java code?
>
> That's not needed anymore. There are some `try` warnings in the `JmodsReader`
> and `JimageDiffGenerator` classes which used to get compiled with this. It'll
> probably change again...
No longer there.
>> src/jdk.jlink/share/classes/jdk/tools/jlink/internal/runtimelink/JimageDiffGenerator.java
>> line 40:
>>
>>> 38: public class JimageDiffGenerator {
>>> 39:
>>> 40: private static final boolean DEBUG = false;
>>
>> This seems like left-over debug code. If this should go into product code I
>> suggest you either remove it, or alternatively make it possible to change at
>> runtime, if the debug functionality will be needed.
>
> OK.
Removed in the latest version. The debug static in the build tools class can be
enabled with a property.
-
PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1551544541
PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1551547273
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Thu, 21 Mar 2024 15:30:01 GMT, Magnus Ihse Bursie wrote: >> Thanks! This will also likely change. I'm thinking of just generating the >> diff file and putting it into `lib/` of the JDK image. That avoids needing >> to call this build-time only jlink plugin and this `FixPath` magic. > > I see. I'm sorry if I did not fully understand how the ongoing discussions > affected build source changes. > > Maybe I should just put the build part review of this PR on the backburner, > and then you can ping me when you think you have a solution that will stick, > and I can check how it holds up from a build perspective? I hope the latest version is where we're headed now. I'll ping you for a review once the dust settles. Thanks! - PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1551544078
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Thu, 21 Mar 2024 15:35:06 GMT, Severin Gehwolf wrote: >> Ok, fine. Will the new solution still include a build-time only tool? > > Yes, but I'll likely go with the interim solution and that would only require > the a single "driver" class in the build tree with a `main` method. Now it's no longer a module, since no jlink plugin is being used any more. - PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1551545310
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Wed, 3 Apr 2024 22:31:39 GMT, Mandy Chung wrote: > Thanks for the investigation w.r.t. extending jimage. It does not seem worth > the effort in pursuing the support of adding resources to an existing jimage > file. To me, putting the diff data under `lib` directory as a private file is > a simpler and acceptable solution. It allows the second jlink invocation to > avoid the plugin pipeline if the per-module metadata is generated in normal > jlink invocation. Agreed. I'm pushing an update of the patch later today which does this. There is only one jlink call producing the final `jdk` image. That image is then amended to add the delta file under `lib`. The file is generated as a build step. - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-2036428963
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Tue, 19 Mar 2024 16:55:14 GMT, Severin Gehwolf wrote: >> Please review this patch which adds a jlink mode to the JDK which doesn't >> need the packaged modules being present. A.k.a run-time image based jlink. >> Fundamentally this patch adds an option to use `jlink` even though your JDK >> install might not come with the packaged modules (directory `jmods`). This >> is particularly useful to further reduce the size of a jlinked runtime. >> After the removal of the concept of a JRE, a common distribution mechanism >> is still the full JDK with all modules and packaged modules. However, >> packaged modules can incur an additional size tax. For example in a >> container scenario it could be useful to have a base JDK container including >> all modules, but without also delivering the packaged modules. This comes at >> a size advantage of `~25%`. Such a base JDK container could then be used to >> `jlink` application specific runtimes, further reducing the size of the >> application runtime image (App + JDK runtime; as a single image *or* >> separate bundles, depending on the app being modularized). >> >> The basic design of this approach is to add a jlink plugin for tracking >> non-class and non-resource files of a JDK install. I.e. files which aren't >> present in the jimage (`lib/modules`). This enables producing a `JRTArchive` >> class which has all the info of what constitutes the final jlinked runtime. >> >> Basic usage example: >> >> $ diff -u <(./bin/java --list-modules --limit-modules java.se) >> <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules >> --limit-modules java.se) >> $ diff -u <(./bin/java --list-modules --limit-modules jdk.jlink) >> <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules >> --limit-modules jdk.jlink) >> $ ls ../linux-x86_64-server-release/images/jdk/jmods >> java.base.jmodjava.net.http.jmod java.sql.rowset.jmod >> jdk.crypto.ec.jmod jdk.internal.opt.jmod >> jdk.jdi.jmod jdk.management.agent.jmod jdk.security.auth.jmod >> java.compiler.jmodjava.prefs.jmod java.transaction.xa.jmod >> jdk.dynalink.jmod jdk.internal.vm.ci.jmod >> jdk.jdwp.agent.jmod jdk.management.jfr.jmodjdk.security.jgss.jmod >> java.datatransfer.jmodjava.rmi.jmodjava.xml.crypto.jmod >> jdk.editpad.jmod jdk.internal.vm.compiler.jmod >> jdk.jfr.jmod jdk.management.jmodjdk.unsupported.desktop.jmod >> java.desktop.jmod java.scripting.jmod java.xml.jmod >> jdk.hotspot.agent.jmod jdk.i... > > Severin Gehwolf has updated the pull request incrementally with one > additional commit since the last revision: > > Move CreateLinkableRuntimePlugin to build folder > > Keep runtime link supporting classes in package > jdk.tools.jlink.internal.runtimelink Thanks for the investigation w.r.t. extending jimage. It does not seem worth the effort in pursuing the support of adding resources to an existing jimage file. To me, putting the diff data under `lib` directory as a private file is a simpler and acceptable solution. It allows the second jlink invocation to avoid the plugin pipeline if the per-module metadata is generated in normal jlink invocation. (An observation: The current implementation requires the diffs to be generated as a plugin. For this approach, it may be possible to implement with a separate main class that calls JLink API and generates the diffs.) - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-2035719255
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Tue, 19 Mar 2024 16:55:14 GMT, Severin Gehwolf wrote: >> Please review this patch which adds a jlink mode to the JDK which doesn't >> need the packaged modules being present. A.k.a run-time image based jlink. >> Fundamentally this patch adds an option to use `jlink` even though your JDK >> install might not come with the packaged modules (directory `jmods`). This >> is particularly useful to further reduce the size of a jlinked runtime. >> After the removal of the concept of a JRE, a common distribution mechanism >> is still the full JDK with all modules and packaged modules. However, >> packaged modules can incur an additional size tax. For example in a >> container scenario it could be useful to have a base JDK container including >> all modules, but without also delivering the packaged modules. This comes at >> a size advantage of `~25%`. Such a base JDK container could then be used to >> `jlink` application specific runtimes, further reducing the size of the >> application runtime image (App + JDK runtime; as a single image *or* >> separate bundles, depending on the app being modularized). >> >> The basic design of this approach is to add a jlink plugin for tracking >> non-class and non-resource files of a JDK install. I.e. files which aren't >> present in the jimage (`lib/modules`). This enables producing a `JRTArchive` >> class which has all the info of what constitutes the final jlinked runtime. >> >> Basic usage example: >> >> $ diff -u <(./bin/java --list-modules --limit-modules java.se) >> <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules >> --limit-modules java.se) >> $ diff -u <(./bin/java --list-modules --limit-modules jdk.jlink) >> <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules >> --limit-modules jdk.jlink) >> $ ls ../linux-x86_64-server-release/images/jdk/jmods >> java.base.jmodjava.net.http.jmod java.sql.rowset.jmod >> jdk.crypto.ec.jmod jdk.internal.opt.jmod >> jdk.jdi.jmod jdk.management.agent.jmod jdk.security.auth.jmod >> java.compiler.jmodjava.prefs.jmod java.transaction.xa.jmod >> jdk.dynalink.jmod jdk.internal.vm.ci.jmod >> jdk.jdwp.agent.jmod jdk.management.jfr.jmodjdk.security.jgss.jmod >> java.datatransfer.jmodjava.rmi.jmodjava.xml.crypto.jmod >> jdk.editpad.jmod jdk.internal.vm.compiler.jmod >> jdk.jfr.jmod jdk.management.jmodjdk.unsupported.desktop.jmod >> java.desktop.jmod java.scripting.jmod java.xml.jmod >> jdk.hotspot.agent.jmod jdk.i... > > Severin Gehwolf has updated the pull request incrementally with one > additional commit since the last revision: > > Move CreateLinkableRuntimePlugin to build folder > > Keep runtime link supporting classes in package > jdk.tools.jlink.internal.runtimelink I'm posting this for posterity, since I did some research on the `jimage` format in light of being able to re-create an existing `jimage` file with potentially a few resources being added. One use-case for this would be to add the "diff" data to an pre-existing, optimized `jimage` in `images/jdk/lib/modules`. The way that the `jimage` write algorithm works is based on the fact that it knows the resources and bytes at `jlink` time in **full**. Therefore, it can iterate over all resources, generate an index (and header) for them and then can serialize resource bytes as well as container (folder) information for them so as to support various iteration entry points. Since it's inherently relying on the header, index and container information at `jimage` read-time the format doesn't lend itself nicely for "appending". By adding just a few resources, the header, index, and container iteration information get invalidated. Therefore, a new `jimage` would need to be created, based on the old `jimage`'s resources inferring `ResourcePoolEntry`s and then working with them. The additional resources would then get added and header/index/container info generated afresh. In order to support this, the existing `jimage` file would need to have sufficient information in it, to re-constitute a "similar" jimage from it (at the infer `ResourcePoolEntry` step). There are two specific issues that need to be overcome to support this: resource ordering and resource compression. There might be more, that I'm missing. ## Ordering of Resources `jlink` supports ordering of resources. In other words, the `--order-resources` plugin instructs `jlink` to produce a `jimage` where resources will end up in the target `jimage` in a specified order. This can be observed by listing `jimage` contents sorted by content byte offsets. For example, the default JDK build for Linux adds this expression:
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Thu, 21 Mar 2024 15:28:23 GMT, Magnus Ihse Bursie wrote: >>> First question, do this class really need to be in a separate module? (I'm >>> afraid the answer is "yes" but I need to ask it anyway). >> >> Yes, because it uses the `Plugin` ServiceLoader extension using the boot >> ModuleLayer. But it's going to be a moot point because this'll get reworked >> due to concerns raised by @mlchung (having the plugin pipeline running when >> producing a linkable runtime jimage). > > Ok, fine. Will the new solution still include a build-time only tool? Yes, but I'll likely go with the interim solution and that would only require the a single "driver" class in the build tree with a `main` method. - PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1534158026
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Thu, 21 Mar 2024 15:27:06 GMT, Severin Gehwolf wrote: >> make/Images.gmk line 131: >> >>> 129: # in FixPath call in order to avoid needing to use strip. >>> 130: RL_JIMAGE_PATH_ARG := $(call >>> FixPath,$(JDK_LINK_OUTPUT_DIR)/lib/modules) >>> 131: RL_MOD_PATH_ARG := $(call FixPath,$(IMAGES_OUTPUTDIR)/jmods) >> >> I'd much rather prefer to use strip and have proper space after commas. This >> is the approach taken elsewhere in the build system. >> >> Suggestion: >> >> RL_JIMAGE_PATH_ARG := $(strip $(call FixPath, >> $(JDK_LINK_OUTPUT_DIR)/lib/modules)) >> RL_MOD_PATH_ARG := $(strip $(call FixPath, $(IMAGES_OUTPUTDIR)/jmods)) > > Thanks! This will also likely change. I'm thinking of just generating the > diff file and putting it into `lib/` of the JDK image. That avoids needing to > call this build-time only jlink plugin and this `FixPath` magic. I see. I'm sorry if I did not fully understand how the ongoing discussions affected build source changes. Maybe I should just put the build part review of this PR on the backburner, and then you can ping me when you think you have a solution that will stick, and I can check how it holds up from a build perspective? >> make/jdk/src/jdk.unsupported_jlink_runtime/share/classes/build/tools/runtimelink/CreateLinkableRuntimePlugin.java >> line 1: >> >>> 1: /* >> >> This file does not at all fit in with the pattern of other build tools, >> which are not module-based but instead live in packages `build.tools.*` in >> the `make/jdk/src/classes` directory. We will need to find a better >> arrangement for this. >> >> First question, do this class really need to be in a separate module? (I'm >> afraid the answer is "yes" but I need to ask it anyway). > >> First question, do this class really need to be in a separate module? (I'm >> afraid the answer is "yes" but I need to ask it anyway). > > Yes, because it uses the `Plugin` ServiceLoader extension using the boot > ModuleLayer. But it's going to be a moot point because this'll get reworked > due to concerns raised by @mlchung (having the plugin pipeline running when > producing a linkable runtime jimage). Ok, fine. Will the new solution still include a build-time only tool? - PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1534149409 PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1534146496
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Thu, 21 Mar 2024 14:54:15 GMT, Magnus Ihse Bursie wrote:
>> Severin Gehwolf has updated the pull request incrementally with one
>> additional commit since the last revision:
>>
>> Move CreateLinkableRuntimePlugin to build folder
>>
>> Keep runtime link supporting classes in package
>> jdk.tools.jlink.internal.runtimelink
>
> make/Images.gmk line 124:
>
>> 122:
>> 123: ifeq ($(JLINK_PRODUCE_RUNTIME_LINK_JDK), true)
>> 124:
>
> Please avoid newlines after if statements.
>
> Suggestion:
OK.
> make/Images.gmk line 131:
>
>> 129: # in FixPath call in order to avoid needing to use strip.
>> 130: RL_JIMAGE_PATH_ARG := $(call
>> FixPath,$(JDK_LINK_OUTPUT_DIR)/lib/modules)
>> 131: RL_MOD_PATH_ARG := $(call FixPath,$(IMAGES_OUTPUTDIR)/jmods)
>
> I'd much rather prefer to use strip and have proper space after commas. This
> is the approach taken elsewhere in the build system.
>
> Suggestion:
>
> RL_JIMAGE_PATH_ARG := $(strip $(call FixPath,
> $(JDK_LINK_OUTPUT_DIR)/lib/modules))
> RL_MOD_PATH_ARG := $(strip $(call FixPath, $(IMAGES_OUTPUTDIR)/jmods))
Thanks! This will also likely change. I'm thinking of just generating the diff
file and putting it into `lib/` of the JDK image. That avoids needing to call
this build-time only jlink plugin and this `FixPath` magic.
> make/Images.gmk line 145:
>
>> 143: $(eval $(call SetupJavaCompilation, BUILD_JDK_RUNLINK_CLASSES, \
>> 144: COMPILER := buildjdk, \
>> 145: DISABLED_WARNINGS := try, \
>
> Why do we get warnings in the java code?
That's not needed anymore. There are some `try` warnings in the `JmodsReader`
and `JimageDiffGenerator` classes which used to get compiled with this. It'll
probably change again...
> make/Images.gmk line 171:
>
>> 169:
>> 170: JLINK_JDK_TARGETS += $(jlink_runtime_jdk)
>> 171:
>
> Please avoid newlines before endif statements.
>
> Suggestion:
OK. Will fix.
> First question, do this class really need to be in a separate module? (I'm
> afraid the answer is "yes" but I need to ask it anyway).
Yes, because it uses the `Plugin` ServiceLoader extension using the boot
ModuleLayer. But it's going to be a moot point because this'll get reworked due
to concerns raised by @mlchung (having the plugin pipeline running when
producing a linkable runtime jimage).
> src/jdk.jlink/share/classes/jdk/tools/jlink/internal/runtimelink/JimageDiffGenerator.java
> line 36:
>
>> 34:
>> 35: /**
>> 36: * Used by the build-only jlink plugin CreateLinkableRuntimePlugin.
>
> Why does this file not live next to the jlink plugin then? Does it have to be
> part of the jdk.jlink module?
The idea was to have the "supporting" classes inside jdk.jlink to keep
code-bases in sync. But the actual invocation of them should be in the build
code. This will likely change again.
> src/jdk.jlink/share/classes/jdk/tools/jlink/internal/runtimelink/JimageDiffGenerator.java
> line 40:
>
>> 38: public class JimageDiffGenerator {
>> 39:
>> 40: private static final boolean DEBUG = false;
>
> This seems like left-over debug code. If this should go into product code I
> suggest you either remove it, or alternatively make it possible to change at
> runtime, if the debug functionality will be needed.
OK.
-
PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1534137541
PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1534141577
PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1534148956
PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1534137294
PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1534136264
PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1534154082
PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1534151066
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Tue, 19 Mar 2024 16:55:14 GMT, Severin Gehwolf wrote:
>> Please review this patch which adds a jlink mode to the JDK which doesn't
>> need the packaged modules being present. A.k.a run-time image based jlink.
>> Fundamentally this patch adds an option to use `jlink` even though your JDK
>> install might not come with the packaged modules (directory `jmods`). This
>> is particularly useful to further reduce the size of a jlinked runtime.
>> After the removal of the concept of a JRE, a common distribution mechanism
>> is still the full JDK with all modules and packaged modules. However,
>> packaged modules can incur an additional size tax. For example in a
>> container scenario it could be useful to have a base JDK container including
>> all modules, but without also delivering the packaged modules. This comes at
>> a size advantage of `~25%`. Such a base JDK container could then be used to
>> `jlink` application specific runtimes, further reducing the size of the
>> application runtime image (App + JDK runtime; as a single image *or*
>> separate bundles, depending on the app
being modularized).
>>
>> The basic design of this approach is to add a jlink plugin for tracking
>> non-class and non-resource files of a JDK install. I.e. files which aren't
>> present in the jimage (`lib/modules`). This enables producing a `JRTArchive`
>> class which has all the info of what constitutes the final jlinked runtime.
>>
>> Basic usage example:
>>
>> $ diff -u <(./bin/java --list-modules --limit-modules java.se)
>> <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules
>> --limit-modules java.se)
>> $ diff -u <(./bin/java --list-modules --limit-modules jdk.jlink)
>> <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules
>> --limit-modules jdk.jlink)
>> $ ls ../linux-x86_64-server-release/images/jdk/jmods
>> java.base.jmodjava.net.http.jmod java.sql.rowset.jmod
>> jdk.crypto.ec.jmod jdk.internal.opt.jmod
>> jdk.jdi.jmod jdk.management.agent.jmod jdk.security.auth.jmod
>> java.compiler.jmodjava.prefs.jmod java.transaction.xa.jmod
>> jdk.dynalink.jmod jdk.internal.vm.ci.jmod
>> jdk.jdwp.agent.jmod jdk.management.jfr.jmodjdk.security.jgss.jmod
>> java.datatransfer.jmodjava.rmi.jmodjava.xml.crypto.jmod
>> jdk.editpad.jmod jdk.internal.vm.compiler.jmod
>> jdk.jfr.jmod jdk.management.jmodjdk.unsupported.desktop.jmod
>> java.desktop.jmod java.scripting.jmod java.xml.jmod
>> jdk.hotspot.agent.jmod jdk.i...
>
> Severin Gehwolf has updated the pull request incrementally with one
> additional commit since the last revision:
>
> Move CreateLinkableRuntimePlugin to build folder
>
> Keep runtime link supporting classes in package
> jdk.tools.jlink.internal.runtimelink
src/jdk.jlink/share/classes/jdk/tools/jlink/internal/runtimelink/JimageDiffGenerator.java
line 36:
> 34:
> 35: /**
> 36: * Used by the build-only jlink plugin CreateLinkableRuntimePlugin.
Why does this file not live next to the jlink plugin then? Does it have to be
part of the jdk.jlink module?
src/jdk.jlink/share/classes/jdk/tools/jlink/internal/runtimelink/JimageDiffGenerator.java
line 40:
> 38: public class JimageDiffGenerator {
> 39:
> 40: private static final boolean DEBUG = false;
This seems like left-over debug code. If this should go into product code I
suggest you either remove it, or alternatively make it possible to change at
runtime, if the debug functionality will be needed.
-
PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1534102351
PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1534103809
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Tue, 19 Mar 2024 16:55:14 GMT, Severin Gehwolf wrote: >> Please review this patch which adds a jlink mode to the JDK which doesn't >> need the packaged modules being present. A.k.a run-time image based jlink. >> Fundamentally this patch adds an option to use `jlink` even though your JDK >> install might not come with the packaged modules (directory `jmods`). This >> is particularly useful to further reduce the size of a jlinked runtime. >> After the removal of the concept of a JRE, a common distribution mechanism >> is still the full JDK with all modules and packaged modules. However, >> packaged modules can incur an additional size tax. For example in a >> container scenario it could be useful to have a base JDK container including >> all modules, but without also delivering the packaged modules. This comes at >> a size advantage of `~25%`. Such a base JDK container could then be used to >> `jlink` application specific runtimes, further reducing the size of the >> application runtime image (App + JDK runtime; as a single image *or* >> separate bundles, depending on the app being modularized). >> >> The basic design of this approach is to add a jlink plugin for tracking >> non-class and non-resource files of a JDK install. I.e. files which aren't >> present in the jimage (`lib/modules`). This enables producing a `JRTArchive` >> class which has all the info of what constitutes the final jlinked runtime. >> >> Basic usage example: >> >> $ diff -u <(./bin/java --list-modules --limit-modules java.se) >> <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules >> --limit-modules java.se) >> $ diff -u <(./bin/java --list-modules --limit-modules jdk.jlink) >> <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules >> --limit-modules jdk.jlink) >> $ ls ../linux-x86_64-server-release/images/jdk/jmods >> java.base.jmodjava.net.http.jmod java.sql.rowset.jmod >> jdk.crypto.ec.jmod jdk.internal.opt.jmod >> jdk.jdi.jmod jdk.management.agent.jmod jdk.security.auth.jmod >> java.compiler.jmodjava.prefs.jmod java.transaction.xa.jmod >> jdk.dynalink.jmod jdk.internal.vm.ci.jmod >> jdk.jdwp.agent.jmod jdk.management.jfr.jmodjdk.security.jgss.jmod >> java.datatransfer.jmodjava.rmi.jmodjava.xml.crypto.jmod >> jdk.editpad.jmod jdk.internal.vm.compiler.jmod >> jdk.jfr.jmod jdk.management.jmodjdk.unsupported.desktop.jmod >> java.desktop.jmod java.scripting.jmod java.xml.jmod >> jdk.hotspot.agent.jmod jdk.i... > > Severin Gehwolf has updated the pull request incrementally with one > additional commit since the last revision: > > Move CreateLinkableRuntimePlugin to build folder > > Keep runtime link supporting classes in package > jdk.tools.jlink.internal.runtimelink make/jdk/src/jdk.unsupported_jlink_runtime/share/classes/build/tools/runtimelink/CreateLinkableRuntimePlugin.java line 1: > 1: /* This file does not at all fit in with the pattern of other build tools, which are not module-based but instead live in packages `build.tools.*` in the `make/jdk/src/classes` directory. We will need to find a better arrangement for this. First question, do this class really need to be in a separate module? (I'm afraid the answer is "yes" but I need to ask it anyway). - PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1534094775
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Tue, 19 Mar 2024 16:55:14 GMT, Severin Gehwolf wrote: >> Please review this patch which adds a jlink mode to the JDK which doesn't >> need the packaged modules being present. A.k.a run-time image based jlink. >> Fundamentally this patch adds an option to use `jlink` even though your JDK >> install might not come with the packaged modules (directory `jmods`). This >> is particularly useful to further reduce the size of a jlinked runtime. >> After the removal of the concept of a JRE, a common distribution mechanism >> is still the full JDK with all modules and packaged modules. However, >> packaged modules can incur an additional size tax. For example in a >> container scenario it could be useful to have a base JDK container including >> all modules, but without also delivering the packaged modules. This comes at >> a size advantage of `~25%`. Such a base JDK container could then be used to >> `jlink` application specific runtimes, further reducing the size of the >> application runtime image (App + JDK runtime; as a single image *or* >> separate bundles, depending on the app being modularized). >> >> The basic design of this approach is to add a jlink plugin for tracking >> non-class and non-resource files of a JDK install. I.e. files which aren't >> present in the jimage (`lib/modules`). This enables producing a `JRTArchive` >> class which has all the info of what constitutes the final jlinked runtime. >> >> Basic usage example: >> >> $ diff -u <(./bin/java --list-modules --limit-modules java.se) >> <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules >> --limit-modules java.se) >> $ diff -u <(./bin/java --list-modules --limit-modules jdk.jlink) >> <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules >> --limit-modules jdk.jlink) >> $ ls ../linux-x86_64-server-release/images/jdk/jmods >> java.base.jmodjava.net.http.jmod java.sql.rowset.jmod >> jdk.crypto.ec.jmod jdk.internal.opt.jmod >> jdk.jdi.jmod jdk.management.agent.jmod jdk.security.auth.jmod >> java.compiler.jmodjava.prefs.jmod java.transaction.xa.jmod >> jdk.dynalink.jmod jdk.internal.vm.ci.jmod >> jdk.jdwp.agent.jmod jdk.management.jfr.jmodjdk.security.jgss.jmod >> java.datatransfer.jmodjava.rmi.jmodjava.xml.crypto.jmod >> jdk.editpad.jmod jdk.internal.vm.compiler.jmod >> jdk.jfr.jmod jdk.management.jmodjdk.unsupported.desktop.jmod >> java.desktop.jmod java.scripting.jmod java.xml.jmod >> jdk.hotspot.agent.jmod jdk.i... > > Severin Gehwolf has updated the pull request incrementally with one > additional commit since the last revision: > > Move CreateLinkableRuntimePlugin to build folder > > Keep runtime link supporting classes in package > jdk.tools.jlink.internal.runtimelink make/Images.gmk line 124: > 122: > 123: ifeq ($(JLINK_PRODUCE_RUNTIME_LINK_JDK), true) > 124: Please avoid newlines after if statements. Suggestion: make/Images.gmk line 131: > 129: # in FixPath call in order to avoid needing to use strip. > 130: RL_JIMAGE_PATH_ARG := $(call > FixPath,$(JDK_LINK_OUTPUT_DIR)/lib/modules) > 131: RL_MOD_PATH_ARG := $(call FixPath,$(IMAGES_OUTPUTDIR)/jmods) I'd much rather prefer to use strip and have proper space after commas. This is the approach taken elsewhere in the build system. Suggestion: RL_JIMAGE_PATH_ARG := $(strip $(call FixPath, $(JDK_LINK_OUTPUT_DIR)/lib/modules)) RL_MOD_PATH_ARG := $(strip $(call FixPath, $(IMAGES_OUTPUTDIR)/jmods)) make/Images.gmk line 145: > 143: $(eval $(call SetupJavaCompilation, BUILD_JDK_RUNLINK_CLASSES, \ > 144: COMPILER := buildjdk, \ > 145: DISABLED_WARNINGS := try, \ Why do we get warnings in the java code? make/Images.gmk line 171: > 169: > 170: JLINK_JDK_TARGETS += $(jlink_runtime_jdk) > 171: Please avoid newlines before endif statements. Suggestion: - PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1534080047 PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1534079021 PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1534082359 PR Review Comment: https://git.openjdk.org/jdk/pull/14787#discussion_r1534080744
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Wed, 20 Mar 2024 10:24:23 GMT, Severin Gehwolf wrote: > What we really want is some form of API to extend/patch an existing jimage > preserving everything else. Perhaps I should look into that. Would that be > worth doing? I think avoiding the plugin pipeline in creating a linkable image is simpler and more reliable design. I think it's worth understanding how big effort to support that. The reason why I proposed the alternative putting the diffs data as a side file under `lib` is an interim solution that would allow this work to continue while the API for extending jimage can be done as a follow up. > it seems almost too easy to change if it was a file outside the image. Only a few files in the `lib` directory such as libjvm.so or libjvm.dylib, src.zip and a few other files are intended for external use. All other files and directories under it must be treated are private implementation details and their names, format, and content are subject to change without notice. See JEP 220 for details. > Thirdly, it would go against where the hermetic java work is going (one > effort of this project is to try to include most of the conf/resource files > outside the jimage inside the jimage, next to the classes that use them). I agree that putting the diffs file in the jimage is a good direction. I am also ok with the interim solution. - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-2010349483
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Tue, 19 Mar 2024 18:05:31 GMT, Mandy Chung wrote: > Thanks for the details. I feel the pain in extending jlink for this work as > the current jlink implementation is not easily understandable and has been > yearning for rewrite in my perspective (looking forward to Project Leyden's > condenser model). +1 > Really appreciate your effort for this. Sure. > IIUC, the main issue is regarding extending an jimage with additional entries. Yes, it's really extending the jimage with additional entries which isn't supported currently (outside the jlink + archive + plugin pipeline). > The other option is to add the diffs as a separate file in the JDK image > under `lib` instead of part of the jimage. Sure that would work. On the other hand, it's another additional artefact that downstream distributors need to account for. What's more, it seems almost too easy to change if it was a file outside the image. Thirdly, it would go against where the hermetic java work is going (one effort of this project is to try to include most of the conf/resource files outside the jimage inside the jimage, next to the classes that use them). > The per-module mapping metadata files can be generated as part of the normal > linking as the footprint is small. Sure. > Would that simplify it? I'm not sure. What we really want is some form of API to extend/patch an existing jimage preserving everything else. Perhaps I should look into that. Would that be worth doing? > I think the other issues may be solvable. It's not a question of being solvable. All the issues are solvable in some way. Question is how exactly we want to address them. - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-2009217508
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
On Tue, 19 Mar 2024 16:55:14 GMT, Severin Gehwolf wrote: >> Please review this patch which adds a jlink mode to the JDK which doesn't >> need the packaged modules being present. A.k.a run-time image based jlink. >> Fundamentally this patch adds an option to use `jlink` even though your JDK >> install might not come with the packaged modules (directory `jmods`). This >> is particularly useful to further reduce the size of a jlinked runtime. >> After the removal of the concept of a JRE, a common distribution mechanism >> is still the full JDK with all modules and packaged modules. However, >> packaged modules can incur an additional size tax. For example in a >> container scenario it could be useful to have a base JDK container including >> all modules, but without also delivering the packaged modules. This comes at >> a size advantage of `~25%`. Such a base JDK container could then be used to >> `jlink` application specific runtimes, further reducing the size of the >> application runtime image (App + JDK runtime; as a single image *or* >> separate bundles, depending on the app being modularized). >> >> The basic design of this approach is to add a jlink plugin for tracking >> non-class and non-resource files of a JDK install. I.e. files which aren't >> present in the jimage (`lib/modules`). This enables producing a `JRTArchive` >> class which has all the info of what constitutes the final jlinked runtime. >> >> Basic usage example: >> >> $ diff -u <(./bin/java --list-modules --limit-modules java.se) >> <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules >> --limit-modules java.se) >> $ diff -u <(./bin/java --list-modules --limit-modules jdk.jlink) >> <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules >> --limit-modules jdk.jlink) >> $ ls ../linux-x86_64-server-release/images/jdk/jmods >> java.base.jmodjava.net.http.jmod java.sql.rowset.jmod >> jdk.crypto.ec.jmod jdk.internal.opt.jmod >> jdk.jdi.jmod jdk.management.agent.jmod jdk.security.auth.jmod >> java.compiler.jmodjava.prefs.jmod java.transaction.xa.jmod >> jdk.dynalink.jmod jdk.internal.vm.ci.jmod >> jdk.jdwp.agent.jmod jdk.management.jfr.jmodjdk.security.jgss.jmod >> java.datatransfer.jmodjava.rmi.jmodjava.xml.crypto.jmod >> jdk.editpad.jmod jdk.internal.vm.compiler.jmod >> jdk.jfr.jmod jdk.management.jmodjdk.unsupported.desktop.jmod >> java.desktop.jmod java.scripting.jmod java.xml.jmod >> jdk.hotspot.agent.jmod jdk.i... > > Severin Gehwolf has updated the pull request incrementally with one > additional commit since the last revision: > > Move CreateLinkableRuntimePlugin to build folder > > Keep runtime link supporting classes in package > jdk.tools.jlink.internal.runtimelink Thanks for the details.I feel the pain in extending jlink for this work as the current jlink implementation is not easily understandable and has been yearning for rewrite in my perspective (looking forward to Project Leyden's condenser model). Really appreciate your effort for this. IIUC, the main issue is regarding extending an jimage with additional entries in the same compression level. The other option is to add the diffs as a separate file in the JDK image under `lib` instead of part of the jimage. The per-module mapping metadata files can be generated as part of the normal linking as the footprint is small. Would that simplify it? I think the other issues may be solvable. - PR Comment: https://git.openjdk.org/jdk/pull/14787#issuecomment-2007823095
Re: RFR: 8311302: Allow for jlinking a custom runtime without packaged modules being present [v23]
> Please review this patch which adds a jlink mode to the JDK which doesn't > need the packaged modules being present. A.k.a run-time image based jlink. > Fundamentally this patch adds an option to use `jlink` even though your JDK > install might not come with the packaged modules (directory `jmods`). This is > particularly useful to further reduce the size of a jlinked runtime. After > the removal of the concept of a JRE, a common distribution mechanism is still > the full JDK with all modules and packaged modules. However, packaged modules > can incur an additional size tax. For example in a container scenario it > could be useful to have a base JDK container including all modules, but > without also delivering the packaged modules. This comes at a size advantage > of `~25%`. Such a base JDK container could then be used to `jlink` > application specific runtimes, further reducing the size of the application > runtime image (App + JDK runtime; as a single image *or* separate bundles, > depending on the app b eing modularized). > > The basic design of this approach is to add a jlink plugin for tracking > non-class and non-resource files of a JDK install. I.e. files which aren't > present in the jimage (`lib/modules`). This enables producing a `JRTArchive` > class which has all the info of what constitutes the final jlinked runtime. > > Basic usage example: > > $ diff -u <(./bin/java --list-modules --limit-modules java.se) > <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules > --limit-modules java.se) > $ diff -u <(./bin/java --list-modules --limit-modules jdk.jlink) > <(../linux-x86_64-server-release/images/jdk/bin/java --list-modules > --limit-modules jdk.jlink) > $ ls ../linux-x86_64-server-release/images/jdk/jmods > java.base.jmodjava.net.http.jmod java.sql.rowset.jmod > jdk.crypto.ec.jmod jdk.internal.opt.jmod > jdk.jdi.jmod jdk.management.agent.jmod jdk.security.auth.jmod > java.compiler.jmodjava.prefs.jmod java.transaction.xa.jmod > jdk.dynalink.jmod jdk.internal.vm.ci.jmod > jdk.jdwp.agent.jmod jdk.management.jfr.jmodjdk.security.jgss.jmod > java.datatransfer.jmodjava.rmi.jmodjava.xml.crypto.jmod > jdk.editpad.jmod jdk.internal.vm.compiler.jmod > jdk.jfr.jmod jdk.management.jmodjdk.unsupported.desktop.jmod > java.desktop.jmod java.scripting.jmod java.xml.jmod > jdk.hotspot.agent.jmod jdk.internal.vm.compiler.manage... Severin Gehwolf has updated the pull request incrementally with one additional commit since the last revision: Move CreateLinkableRuntimePlugin to build folder Keep runtime link supporting classes in package jdk.tools.jlink.internal.runtimelink - Changes: - all: https://git.openjdk.org/jdk/pull/14787/files - new: https://git.openjdk.org/jdk/pull/14787/files/e25dd440..5a6d2e4f Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk=14787=22 - incr: https://webrevs.openjdk.org/?repo=jdk=14787=21-22 Stats: 29 lines in 12 files changed: 14 ins; 1 del; 14 mod Patch: https://git.openjdk.org/jdk/pull/14787.diff Fetch: git fetch https://git.openjdk.org/jdk.git pull/14787/head:pull/14787 PR: https://git.openjdk.org/jdk/pull/14787
