Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Wed, 6 Mar 2024 11:57:27 GMT, Andrew John Hughes wrote: > The HotSpot code has inserted the value of `HOTSPOT_BUILD_USER` into the > internal version string since before the JDK was open-sourced, so the > reasoning for this is not in the public history. See > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/abstract_vm_version.cpp#L284 > > ~~~ > $ /usr/lib/jvm/java-21-openjdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (21.0.2+13) for linux-amd64 JRE (21.0.2+13), built > on 2024-01-19T16:39:52Z by "mockbuild" with gcc 8.5.0 20210514 (Red Hat > 8.5.0-20) > ~~~ > > It is not clear what value this provides. By default, the value is set to the > system username, but it can be set to a static value using > `--with-build-user`. > > In trying to compare builds, this is a source of difference if the builds are > produced under different usernames and `--with-build-user` is not set to the > same value. > > It may also be a source of information leakage from the host system. It is > not clear what value it provides to the end user of the build. > > This patch proposes removing it altogether, but at least knowing why it needs > to be there would be progress from where we are now. We could also consider a > middle ground of only setting it for "adhoc" builds, those without set > version information, as is the case with the overall version output. > > With this patch: > > ~~~ > $ ~/builder/23/images/jdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) for > linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on 2024-03-06T00:23:37Z > with gcc 13.2.1 20230826 > > $ ~/builder/23/images/jdk/bin/java -XX:ErrorHandlerTest=1 > > $ grep '^vm_info' > /localhome/andrew/projects/openjdk/upstream/jdk/hs_err_pid1119824.log > vm_info: OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) > for linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on > 2024-03-06T00:23:37Z with gcc 13.2.1 20230826 > ~~~ > > The above are from a fastdebug build but I also built a product build with no > issues. Thanks all for the interesting discussion. Integrating this change now. - PR Comment: https://git.openjdk.org/jdk/pull/18136#issuecomment-1986857857
Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Wed, 6 Mar 2024 11:57:27 GMT, Andrew John Hughes wrote: > The HotSpot code has inserted the value of `HOTSPOT_BUILD_USER` into the > internal version string since before the JDK was open-sourced, so the > reasoning for this is not in the public history. See > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/abstract_vm_version.cpp#L284 > > ~~~ > $ /usr/lib/jvm/java-21-openjdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (21.0.2+13) for linux-amd64 JRE (21.0.2+13), built > on 2024-01-19T16:39:52Z by "mockbuild" with gcc 8.5.0 20210514 (Red Hat > 8.5.0-20) > ~~~ > > It is not clear what value this provides. By default, the value is set to the > system username, but it can be set to a static value using > `--with-build-user`. > > In trying to compare builds, this is a source of difference if the builds are > produced under different usernames and `--with-build-user` is not set to the > same value. > > It may also be a source of information leakage from the host system. It is > not clear what value it provides to the end user of the build. > > This patch proposes removing it altogether, but at least knowing why it needs > to be there would be progress from where we are now. We could also consider a > middle ground of only setting it for "adhoc" builds, those without set > version information, as is the case with the overall version output. > > With this patch: > > ~~~ > $ ~/builder/23/images/jdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) for > linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on 2024-03-06T00:23:37Z > with gcc 13.2.1 20230826 > > $ ~/builder/23/images/jdk/bin/java -XX:ErrorHandlerTest=1 > > $ grep '^vm_info' > /localhome/andrew/projects/openjdk/upstream/jdk/hs_err_pid1119824.log > vm_info: OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) > for linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on > 2024-03-06T00:23:37Z with gcc 13.2.1 20230826 > ~~~ > > The above are from a fastdebug build but I also built a product build with no > issues. This bug spurred several interesting discussions: 1) Should we store the version string in a text file instead of compiling it into a binary library? 2) Should we store `git describe` information in the version string? I think both are worth discussing (and likely worth implementing, too), but it also seems out of scope for this PR. My main concern here was that tests were looking at the internal string format, but if that is not the case then I see no reason to keep a dummy "unknown" value -- better to get rid of it entirely. So I'd say, just go ahead and integrate this. - PR Comment: https://git.openjdk.org/jdk/pull/18136#issuecomment-1985965596
Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Wed, 6 Mar 2024 11:57:27 GMT, Andrew John Hughes wrote: > The HotSpot code has inserted the value of `HOTSPOT_BUILD_USER` into the > internal version string since before the JDK was open-sourced, so the > reasoning for this is not in the public history. See > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/abstract_vm_version.cpp#L284 > > ~~~ > $ /usr/lib/jvm/java-21-openjdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (21.0.2+13) for linux-amd64 JRE (21.0.2+13), built > on 2024-01-19T16:39:52Z by "mockbuild" with gcc 8.5.0 20210514 (Red Hat > 8.5.0-20) > ~~~ > > It is not clear what value this provides. By default, the value is set to the > system username, but it can be set to a static value using > `--with-build-user`. > > In trying to compare builds, this is a source of difference if the builds are > produced under different usernames and `--with-build-user` is not set to the > same value. > > It may also be a source of information leakage from the host system. It is > not clear what value it provides to the end user of the build. > > This patch proposes removing it altogether, but at least knowing why it needs > to be there would be progress from where we are now. We could also consider a > middle ground of only setting it for "adhoc" builds, those without set > version information, as is the case with the overall version output. > > With this patch: > > ~~~ > $ ~/builder/23/images/jdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) for > linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on 2024-03-06T00:23:37Z > with gcc 13.2.1 20230826 > > $ ~/builder/23/images/jdk/bin/java -XX:ErrorHandlerTest=1 > > $ grep '^vm_info' > /localhome/andrew/projects/openjdk/upstream/jdk/hs_err_pid1119824.log > vm_info: OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) > for linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on > 2024-03-06T00:23:37Z with gcc 13.2.1 20230826 > ~~~ > > The above are from a fastdebug build but I also built a product build with no > issues. Marked as reviewed by ihse (Reviewer). - PR Review: https://git.openjdk.org/jdk/pull/18136#pullrequestreview-1925213033
Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Thu, 7 Mar 2024 19:53:35 GMT, Andrew John Hughes wrote: >>> Also, it might be worth repeating one of my long-standing wishes: that the >>> version string should not be hard-coded into the build, but e.g. stored as >>> a string in the `release` file, and read from there. If we did that, the >>> cost of changing the version string would be negligible, and we wouldn't >>> need to worry as much about it. It would also be simple to compare >>> different builds which end up with the same bits since they are built from >>> the same sources, but by different version flags (e.g. -ea vs GA). (In >>> fact, we'd turn a -ea build into a GA just by updating the version string, >>> so we'd know for sure we are publishing what we tested.) >> >> +1 >> This would be a great step in making comparability (beyond reproducibility) >> of builds of OpenJDK at lot simpler. > >> Also, it might be worth repeating one of my long-standing wishes: that the >> version string should not be hard-coded into the build, but e.g. stored as a >> string in the `release` file, and read from there. If we did that, the cost >> of changing the version string would be negligible, and we wouldn't need to >> worry as much about it. It would also be simple to compare different builds >> which end up with the same bits since they are built from the same sources, >> but by different version flags (e.g. -ea vs GA). (In fact, we'd turn a -ea >> build into a GA just by updating the version string, so we'd know for sure >> we are publishing what we tested.) > > This certainly sounds like it has the potential to solve a lot of these kind > of problems. I would point out that, if you can flip the EA status in the > text file, someone could also easily masquerade a build as something > completely different from what it is. However, it is already possible to > create a build like this via the `configure` options so it's really only a > slight change in accessibility. > > For example, if I specify `--with-version-feature=11`, I can produce: > > ~~~ > $ /home/andrew/builder/fake11/images/jdk/bin/java -version > openjdk version "11-internal" 2024-09-17 > OpenJDK Runtime Environment (fastdebug build 11-internal-adhoc.andrew.jdk) > OpenJDK 64-Bit Server VM (fastdebug build 11-internal-adhoc.andrew.jdk, mixed > mode, sharing) > > $ /home/andrew/builder/fake11/images/jdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (fastdebug 11-internal-adhoc.andrew.jdk) for > linux-amd64 JRE (11-internal-adhoc.andrew.jdk), built on 2024-03-07T19:41:08Z > with gcc 13.2.1 20230826 > ~~~ > > despite the fact that the source code I've built is actually an > in-development JDK 23. All we'd be doing is moving that into a text file. I agree with @gnu-andrew : the version string reported by the JVM, as it stands today, is only really useful if taken in good faith. If we wanted - or needed - to make it temper-proof from an adversarial threat model standpoint, then we couldn't spare the need for a cryptographically strong solution (which by design would render any direct comparison impossible). Right now, it feels like we are kind of stuck somewhere in-between, incurring all of the drawbacks and none of the benefits. - PR Comment: https://git.openjdk.org/jdk/pull/18136#issuecomment-1985459372
Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Thu, 7 Mar 2024 19:53:35 GMT, Andrew John Hughes wrote: > Also, it might be worth repeating one of my long-standing wishes: that the > version string should not be hard-coded into the build, but e.g. stored as a > string in the `release` file, and read from there. If we did that, the cost > of changing the version string would be negligible, and we wouldn't need to > worry as much about it. It would also be simple to compare different builds > which end up with the same bits since they are built from the same sources, > but by different version flags (e.g. -ea vs GA). (In fact, we'd turn a -ea > build into a GA just by updating the version string, so we'd know for sure we > are publishing what we tested.) Why not store the version string inside HotSpot, and have it as the one source of truth for the version string so it doesn't need to be hardcoded in other places? A text file seems too easy to modify to set the version string to a rubbish value - PR Comment: https://git.openjdk.org/jdk/pull/18136#issuecomment-1985087876
Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Wed, 6 Mar 2024 11:57:27 GMT, Andrew John Hughes wrote: > The HotSpot code has inserted the value of `HOTSPOT_BUILD_USER` into the > internal version string since before the JDK was open-sourced, so the > reasoning for this is not in the public history. See > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/abstract_vm_version.cpp#L284 > > ~~~ > $ /usr/lib/jvm/java-21-openjdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (21.0.2+13) for linux-amd64 JRE (21.0.2+13), built > on 2024-01-19T16:39:52Z by "mockbuild" with gcc 8.5.0 20210514 (Red Hat > 8.5.0-20) > ~~~ > > It is not clear what value this provides. By default, the value is set to the > system username, but it can be set to a static value using > `--with-build-user`. > > In trying to compare builds, this is a source of difference if the builds are > produced under different usernames and `--with-build-user` is not set to the > same value. > > It may also be a source of information leakage from the host system. It is > not clear what value it provides to the end user of the build. > > This patch proposes removing it altogether, but at least knowing why it needs > to be there would be progress from where we are now. We could also consider a > middle ground of only setting it for "adhoc" builds, those without set > version information, as is the case with the overall version output. > > With this patch: > > ~~~ > $ ~/builder/23/images/jdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) for > linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on 2024-03-06T00:23:37Z > with gcc 13.2.1 20230826 > > $ ~/builder/23/images/jdk/bin/java -XX:ErrorHandlerTest=1 > > $ grep '^vm_info' > /localhome/andrew/projects/openjdk/upstream/jdk/hs_err_pid1119824.log > vm_info: OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) > for linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on > 2024-03-06T00:23:37Z with gcc 13.2.1 20230826 > ~~~ > > The above are from a fastdebug build but I also built a product build with no > issues. I agree with Andrew. We aren't removing the username from adhoc builds, just the copy in the hotspot internal version string, so the concern about that part seems baseless to me. I don't see the connection to the `git describe` discussion, I think that should move to a separate thread. I definitely don't think we need to add a fake HOTSPOT_BUILD_USER to the _internal_ hotspot version string. Lets just get rid of it by accepting this PR. - PR Comment: https://git.openjdk.org/jdk/pull/18136#issuecomment-1984415570
Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Thu, 7 Mar 2024 17:32:02 GMT, Frederic Thevenet wrote: > Also, it might be worth repeating one of my long-standing wishes: that the > version string should not be hard-coded into the build, but e.g. stored as a > string in the `release` file, and read from there. If we did that, the cost > of changing the version string would be negligible, and we wouldn't need to > worry as much about it. It would also be simple to compare different builds > which end up with the same bits since they are built from the same sources, > but by different version flags (e.g. -ea vs GA). (In fact, we'd turn a -ea > build into a GA just by updating the version string, so we'd know for sure we > are publishing what we tested.) This certainly sounds like it has the potential to solve a lot of these kind of problems. I would point out that, if you can flip the EA status in the text file, someone could also easily masquerade a build as something completely different from what it is. However, it is already possible to create a build like this via the `configure` options so it's really only a slight change in accessibility. For example, if I specify `--with-version-feature=11`, I can produce: ~~~ $ /home/andrew/builder/fake11/images/jdk/bin/java -version openjdk version "11-internal" 2024-09-17 OpenJDK Runtime Environment (fastdebug build 11-internal-adhoc.andrew.jdk) OpenJDK 64-Bit Server VM (fastdebug build 11-internal-adhoc.andrew.jdk, mixed mode, sharing) $ /home/andrew/builder/fake11/images/jdk/bin/java -Xinternalversion OpenJDK 64-Bit Server VM (fastdebug 11-internal-adhoc.andrew.jdk) for linux-amd64 JRE (11-internal-adhoc.andrew.jdk), built on 2024-03-07T19:41:08Z with gcc 13.2.1 20230826 ~~~ despite the fact that the source code I've built is actually an in-development JDK 23. All we'd be doing is moving that into a text file. - PR Comment: https://git.openjdk.org/jdk/pull/18136#issuecomment-1984317603
Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Thu, 7 Mar 2024 17:07:05 GMT, Magnus Ihse Bursie wrote: > There is an inherent conflict with creating a version string that is very > much up-to-date and includes ephemeral build data, and creating a robustly > reproducible build. If this patch were to remove HOTSPOT_BUILD_USER, and we > would later on store `git describe` in the version string, I'm not sure we > actually gained anything in terms of reproducability. :-( It's worth remembering that there are essentially two potential forms of version string at play; those for "adhoc" builds and those for release builds, which depends on whether `NO_DEFAULT_VERSION_PARTS` ends up being set to `true` or not. This change does nothing to remove the username used in the "adhoc" version of `--with-version-opt` i.e. `adhoc.$USERNAME.$basedirname`: >From the patched build: ~~~ $ /home/andrew/builder/23/images/jdk/bin/java -version openjdk version "23-internal" 2024-09-17 OpenJDK Runtime Environment (fastdebug build 23-internal-adhoc.andrew.jdk) OpenJDK 64-Bit Server VM (fastdebug build 23-internal-adhoc.andrew.jdk, mixed mode, sharing) $ /home/andrew/builder/23/images/jdk/bin/java -Xinternalversion OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) for linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on 2024-03-06T00:23:37Z with gcc 13.2.1 20230826 ~~~ What we aim to get rid of is the duplicate username copy which trickles into the `-Xinternalversion` via `HOTSPOT_BUILD_USER`. From an unpatched build of 22: ~~~ $ /home/andrew/build/openjdk22/bin/java -Xinternalversion OpenJDK 64-Bit Server VM (22.0.1-internal-adhoc.andrew.jdk) for linux-amd64 JRE (22.0.1-internal-adhoc.andrew.jdk), built on 2024-03-05T19:52:35Z by "andrew" with gcc 13.2.1 20230826 ~~~ The only difference between the two is my username is there twice, instead of thrice. The problem with the `HOTSPOT_BUILD_USER` one is that it also turns up when you do a build for release: ~~~ $ /usr/lib/jvm/java-21-openjdk/bin/java -Xinternalversion OpenJDK 64-Bit Server VM (21.0.2+13) for linux-amd64 JRE (21.0.2+13), built on 2024-01-19T16:39:52Z by "mockbuild" with gcc 8.5.0 20210514 (Red Hat 8.5.0-20) ~~~ Because `NO_DEFAULT_VERSION_PARTS` has been set, the two instances of the username from the adhoc build string are gone. However, the one from `HOTSPOT_BUILD_USER` being set to `mockbuild` still remains. Regarding the conflict between very accurate version information and reproducibility, it is possible to support both scenarios by having "adhoc" builds provide one and release builds the other. The problem with `HOTSPOT_BUILD_USER` as it stands is it ends up in both. I don't see an issue with including `git describe` output in adhoc builds. It may well not even be a problem for reproducibility in release builds as they are likely built from a set tag without local changes or even from a bare source tree without repository information at all. It does seem like a separate issue from this PR though. > 3. We can agree to disagree about reproducibility limits, but accept that > HOTSPOT_BUILD_USER is silly, and accept this PR, but open a separate JBS > issue to discuss if adding `git describe` is desirable, and if it can be done > in an opt-in manner, with the understanding that it might not be added if we > can't agree on it. > > > My personal preference would be 3) above, but I am willing to accept any of > the paths. > > @gnu-andrew and @shipilev, what do you think? I recon you are the one most > opinionated about this. I would go for 3. As I say, I don't see why this change would depend on a `git describe` change. Would you still prefer to hardcode a value for `HOTSPOT_BUILD_USER` or remove it altogether? I have a slight preference for the cleanliness of removing it altogether, especially as this is early in the 23 development cycle, so there is some time to see how it pans out. Alternatively, we can hardcode it to `unknown` (the current default if `HOTSPOT_BUILD_USER is undefined). Not only is this accurate (we don't know who the user was as we haven't recorded the information) but it is already a possible value for compatibility. Changing to this would simply be a matter of removing only the `#ifndef` in `abstract_vm_version.cpp` and leaving the actual version string alone. Looking further ahead, it would be preferable for us to backport this to 21, as that is where we are trying to achieve comparability between Red Hat & Temurin builds, but there is no rush for that and we could even hardcode it only in the backport. As regards testing, I have built this in release and fastdebug builds and checked the output where it is used in `-Xinternalversion` and the `vm info` line of a crash dump. Like David, I can't see any tests that pertain to this. The only ones that mention `-Xinternalversion` are `test/hotspot/jtreg/runtime/CommandLine/TestNullTerminatedFlags.java` and `test/hotspot/jtreg/sanity/BasicVMTest.java`
Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Thu, 7 Mar 2024 17:11:18 GMT, Magnus Ihse Bursie wrote: > Also, it might be worth repeating one of my long-standing wishes: that the > version string should not be hard-coded into the build, but e.g. stored as a > string in the `release` file, and read from there. If we did that, the cost > of changing the version string would be negligible, and we wouldn't need to > worry as much about it. It would also be simple to compare different builds > which end up with the same bits since they are built from the same sources, > but by different version flags (e.g. -ea vs GA). (In fact, we'd turn a -ea > build into a GA just by updating the version string, so we'd know for sure we > are publishing what we tested.) +1 This would be a great step in making comparability (beyond reproducibility) of builds of OpenJDK at lot simpler. - PR Comment: https://git.openjdk.org/jdk/pull/18136#issuecomment-1984065312
Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Wed, 6 Mar 2024 11:57:27 GMT, Andrew John Hughes wrote: > The HotSpot code has inserted the value of `HOTSPOT_BUILD_USER` into the > internal version string since before the JDK was open-sourced, so the > reasoning for this is not in the public history. See > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/abstract_vm_version.cpp#L284 > > ~~~ > $ /usr/lib/jvm/java-21-openjdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (21.0.2+13) for linux-amd64 JRE (21.0.2+13), built > on 2024-01-19T16:39:52Z by "mockbuild" with gcc 8.5.0 20210514 (Red Hat > 8.5.0-20) > ~~~ > > It is not clear what value this provides. By default, the value is set to the > system username, but it can be set to a static value using > `--with-build-user`. > > In trying to compare builds, this is a source of difference if the builds are > produced under different usernames and `--with-build-user` is not set to the > same value. > > It may also be a source of information leakage from the host system. It is > not clear what value it provides to the end user of the build. > > This patch proposes removing it altogether, but at least knowing why it needs > to be there would be progress from where we are now. We could also consider a > middle ground of only setting it for "adhoc" builds, those without set > version information, as is the case with the overall version output. > > With this patch: > > ~~~ > $ ~/builder/23/images/jdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) for > linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on 2024-03-06T00:23:37Z > with gcc 13.2.1 20230826 > > $ ~/builder/23/images/jdk/bin/java -XX:ErrorHandlerTest=1 > > $ grep '^vm_info' > /localhome/andrew/projects/openjdk/upstream/jdk/hs_err_pid1119824.log > vm_info: OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) > for linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on > 2024-03-06T00:23:37Z with gcc 13.2.1 20230826 > ~~~ > > The above are from a fastdebug build but I also built a product build with no > issues. Also, it might be worth repeating one of my long-standing wishes: that the version string should not be hard-coded into the build, but e.g. stored as a string in the `release` file, and read from there. If we did that, the cost of changing the version string would be negligible, and we wouldn't need to worry as much about it. It would also be simple to compare different builds which end up with the same bits since they are built from the same sources, but by different version flags (e.g. -ea vs GA). (In fact, we'd turn a -ea build into a GA just by updating the version string, so we'd know for sure we are publishing what we tested.) - PR Comment: https://git.openjdk.org/jdk/pull/18136#issuecomment-1984028842
Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Wed, 6 Mar 2024 11:57:27 GMT, Andrew John Hughes wrote: > The HotSpot code has inserted the value of `HOTSPOT_BUILD_USER` into the > internal version string since before the JDK was open-sourced, so the > reasoning for this is not in the public history. See > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/abstract_vm_version.cpp#L284 > > ~~~ > $ /usr/lib/jvm/java-21-openjdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (21.0.2+13) for linux-amd64 JRE (21.0.2+13), built > on 2024-01-19T16:39:52Z by "mockbuild" with gcc 8.5.0 20210514 (Red Hat > 8.5.0-20) > ~~~ > > It is not clear what value this provides. By default, the value is set to the > system username, but it can be set to a static value using > `--with-build-user`. > > In trying to compare builds, this is a source of difference if the builds are > produced under different usernames and `--with-build-user` is not set to the > same value. > > It may also be a source of information leakage from the host system. It is > not clear what value it provides to the end user of the build. > > This patch proposes removing it altogether, but at least knowing why it needs > to be there would be progress from where we are now. We could also consider a > middle ground of only setting it for "adhoc" builds, those without set > version information, as is the case with the overall version output. > > With this patch: > > ~~~ > $ ~/builder/23/images/jdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) for > linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on 2024-03-06T00:23:37Z > with gcc 13.2.1 20230826 > > $ ~/builder/23/images/jdk/bin/java -XX:ErrorHandlerTest=1 > > $ grep '^vm_info' > /localhome/andrew/projects/openjdk/upstream/jdk/hs_err_pid1119824.log > vm_info: OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) > for linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on > 2024-03-06T00:23:37Z with gcc 13.2.1 20230826 > ~~~ > > The above are from a fastdebug build but I also built a product build with no > issues. I am okay with either (3) or (4). That is, I agree `HOTSPOT_BUILD_USER` is silly. I think we can survive without `git describe` thing for a while too. - PR Comment: https://git.openjdk.org/jdk/pull/18136#issuecomment-1984028130
Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Wed, 6 Mar 2024 18:01:00 GMT, Aleksey Shipilev wrote: > > I agree that it is good to get rid of this. However, the reason it has > > stuck around for so long is, eh, that it has been around for so long, so it > > is a bit unclear what or who could be relying on this. > > The example I know is to get a signal that test infrastructures actually > picked up my ad-hoc binary build, not some other build from somewhere else. > Maybe we should revisit the `git describe` hash idea we kicked along for a > while now: https://bugs.openjdk.org/browse/JDK-8274980?focusedId=14452017 -- > which would cover that case too. That bug was resolved with a different patch; presumably it conflated several issues. There is an inherent conflict with creating a version string that is very much up-to-date and includes ephemeral build data, and creating a robustly reproducible build. If this patch were to remove HOTSPOT_BUILD_USER, and we would later on store `git describe` in the version string, I'm not sure we actually gained anything in terms of reproducability. :-( As I see it, given this PR, we have a few different options: 1) We can agree that stable and robust reproducibility trumps any automatic inclusion of build situation metadata, and accept this PR and drop the idea of using `git describe` 2) We can agree that reproducibility has its limits, and we need to include (technically irrelevant) metadata in builds to facilitate development. This means dropping this PR since it is used to verify that the correct build was tested, and possibly also adding `git describe` later on. 3) We can agree to disagree about reproducibility limits, but accept that HOTSPOT_BUILD_USER is silly, and accept this PR, but open a separate JBS issue to discuss if adding `git describe` is desirable, and if it can be done in an opt-in manner, with the understanding that it might not be added if we can't agree on it. 4) We can agree to disagree about reproducibility limits, but accept that HOTSPOT_BUILD_USER is silly, and accept this PR, but require that we first implement a substitute functionality using `git describe`, before we can get rid of HOTSPOT_BUILD_USER. My personal preference would be 3) above, but I am willing to accept any of the paths. @gnu-andrew and @shipilev, what do you think? I recon you are the one most opinionated about this. - PR Comment: https://git.openjdk.org/jdk/pull/18136#issuecomment-1984021748
Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Wed, 6 Mar 2024 11:57:27 GMT, Andrew John Hughes wrote: > The HotSpot code has inserted the value of `HOTSPOT_BUILD_USER` into the > internal version string since before the JDK was open-sourced, so the > reasoning for this is not in the public history. See > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/abstract_vm_version.cpp#L284 > > ~~~ > $ /usr/lib/jvm/java-21-openjdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (21.0.2+13) for linux-amd64 JRE (21.0.2+13), built > on 2024-01-19T16:39:52Z by "mockbuild" with gcc 8.5.0 20210514 (Red Hat > 8.5.0-20) > ~~~ > > It is not clear what value this provides. By default, the value is set to the > system username, but it can be set to a static value using > `--with-build-user`. > > In trying to compare builds, this is a source of difference if the builds are > produced under different usernames and `--with-build-user` is not set to the > same value. > > It may also be a source of information leakage from the host system. It is > not clear what value it provides to the end user of the build. > > This patch proposes removing it altogether, but at least knowing why it needs > to be there would be progress from where we are now. We could also consider a > middle ground of only setting it for "adhoc" builds, those without set > version information, as is the case with the overall version output. > > With this patch: > > ~~~ > $ ~/builder/23/images/jdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) for > linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on 2024-03-06T00:23:37Z > with gcc 13.2.1 20230826 > > $ ~/builder/23/images/jdk/bin/java -XX:ErrorHandlerTest=1 > > $ grep '^vm_info' > /localhome/andrew/projects/openjdk/upstream/jdk/hs_err_pid1119824.log > vm_info: OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) > for linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on > 2024-03-06T00:23:37Z with gcc 13.2.1 20230826 > ~~~ > > The above are from a fastdebug build but I also built a product build with no > issues. The internal version is reported in crash reports (hs_err file) as well as via `-Xinternalversion`. My recollection is that the username was needed/used to identify whether a hs_err report came from an official build, back when official builds were created via a very specific build process and a specific user name. These days it seems redundant given the way the rest of the version string is constructed both for "official" builds and personal builds. I can't see any tests that look at the output from `-Xinternalversion`. - PR Comment: https://git.openjdk.org/jdk/pull/18136#issuecomment-1982679336
Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Wed, 6 Mar 2024 17:17:28 GMT, Magnus Ihse Bursie wrote: > I agree that it is good to get rid of this. However, the reason it has stuck > around for so long is, eh, that it has been around for so long, so it is a > bit unclear what or who could be relying on this. The example I know is to get a signal that test infrastructures actually picked up my ad-hoc binary build, not some other build from somewhere else. Maybe we should revisit the `git describe` hash idea we kicked along for a while now: https://bugs.openjdk.org/browse/JDK-8274980?focusedId=14452017 -- which would cover that case too. - PR Comment: https://git.openjdk.org/jdk/pull/18136#issuecomment-1981482991
Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Wed, 6 Mar 2024 15:24:59 GMT, Andrew John Hughes wrote: >> I'm fine with this change. I wasn't around when this was introduced, but my >> guess is that it was relevant back when Hotspot and the rest of the JDK were >> often built separately. We have the username in the default $OPT string for >> personal builds, so the HOTSPOT_BUILD_USER is just redundant information. >> There is no reason to have the build user recorded in non personal builds. > >> I'm fine with this change. I wasn't around when this was introduced, but my >> guess is that it was relevant back when Hotspot and the rest of the JDK were >> often built separately. We have the username in the default $OPT string for >> personal builds, so the HOTSPOT_BUILD_USER is just redundant information. >> There is no reason to have the build user recorded in non personal builds. > > Thanks Erik. I'll leave this open a little longer for others to comment, but > I also can't see a reason why this would be needed, especially in > non-personal builds. @gnu-andrew You might also want to draw specific attention to this change on hotspot-dev, to reach all developers who might know or care about the user name. It is worth making sure to point out that the big change here is that the hotspot version string format changes; that worries me more than the removal of the user name. (In fact, a safer change might be to hardcode the user name to a dummy value like `openjdk`). - PR Comment: https://git.openjdk.org/jdk/pull/18136#issuecomment-1981393909
Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Wed, 6 Mar 2024 11:57:27 GMT, Andrew John Hughes wrote: > The HotSpot code has inserted the value of `HOTSPOT_BUILD_USER` into the > internal version string since before the JDK was open-sourced, so the > reasoning for this is not in the public history. See > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/abstract_vm_version.cpp#L284 > > ~~~ > $ /usr/lib/jvm/java-21-openjdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (21.0.2+13) for linux-amd64 JRE (21.0.2+13), built > on 2024-01-19T16:39:52Z by "mockbuild" with gcc 8.5.0 20210514 (Red Hat > 8.5.0-20) > ~~~ > > It is not clear what value this provides. By default, the value is set to the > system username, but it can be set to a static value using > `--with-build-user`. > > In trying to compare builds, this is a source of difference if the builds are > produced under different usernames and `--with-build-user` is not set to the > same value. > > It may also be a source of information leakage from the host system. It is > not clear what value it provides to the end user of the build. > > This patch proposes removing it altogether, but at least knowing why it needs > to be there would be progress from where we are now. We could also consider a > middle ground of only setting it for "adhoc" builds, those without set > version information, as is the case with the overall version output. > > With this patch: > > ~~~ > $ ~/builder/23/images/jdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) for > linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on 2024-03-06T00:23:37Z > with gcc 13.2.1 20230826 > > $ ~/builder/23/images/jdk/bin/java -XX:ErrorHandlerTest=1 > > $ grep '^vm_info' > /localhome/andrew/projects/openjdk/upstream/jdk/hs_err_pid1119824.log > vm_info: OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) > for linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on > 2024-03-06T00:23:37Z with gcc 13.2.1 20230826 > ~~~ > > The above are from a fastdebug build but I also built a product build with no > issues. What testing have you run? I imagine there are tests that check/read the Hotspot version string. In fact, the more I think about it, the more I believe a better approach is to replace the variable user name with a fixed dummy replacement as a first step. That would only need for verification that nobody really cares about the user name. Then we can, as a follow up, change the version string (if we really care) -- this has a much higher chance of breakage, and it is not by far as important to solve, so it can easily be dropped if it causes problems. - PR Comment: https://git.openjdk.org/jdk/pull/18136#issuecomment-1981394987
Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Wed, 6 Mar 2024 11:57:27 GMT, Andrew John Hughes wrote: > The HotSpot code has inserted the value of `HOTSPOT_BUILD_USER` into the > internal version string since before the JDK was open-sourced, so the > reasoning for this is not in the public history. See > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/abstract_vm_version.cpp#L284 > > ~~~ > $ /usr/lib/jvm/java-21-openjdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (21.0.2+13) for linux-amd64 JRE (21.0.2+13), built > on 2024-01-19T16:39:52Z by "mockbuild" with gcc 8.5.0 20210514 (Red Hat > 8.5.0-20) > ~~~ > > It is not clear what value this provides. By default, the value is set to the > system username, but it can be set to a static value using > `--with-build-user`. > > In trying to compare builds, this is a source of difference if the builds are > produced under different usernames and `--with-build-user` is not set to the > same value. > > It may also be a source of information leakage from the host system. It is > not clear what value it provides to the end user of the build. > > This patch proposes removing it altogether, but at least knowing why it needs > to be there would be progress from where we are now. We could also consider a > middle ground of only setting it for "adhoc" builds, those without set > version information, as is the case with the overall version output. > > With this patch: > > ~~~ > $ ~/builder/23/images/jdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) for > linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on 2024-03-06T00:23:37Z > with gcc 13.2.1 20230826 > > $ ~/builder/23/images/jdk/bin/java -XX:ErrorHandlerTest=1 > > $ grep '^vm_info' > /localhome/andrew/projects/openjdk/upstream/jdk/hs_err_pid1119824.log > vm_info: OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) > for linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on > 2024-03-06T00:23:37Z with gcc 13.2.1 20230826 > ~~~ > > The above are from a fastdebug build but I also built a product build with no > issues. I agree that it is good to get rid of this. However, the reason it has stuck around for so long is, eh, that it has been around for so long, so it is a bit unclear what or who could be relying on this. Let's not be too eager to push this, and collect a bit more information about this. - Marked as reviewed by ihse (Reviewer). PR Review: https://git.openjdk.org/jdk/pull/18136#pullrequestreview-1920369274
Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Wed, 6 Mar 2024 13:57:37 GMT, Erik Joelsson wrote: > I'm fine with this change. I wasn't around when this was introduced, but my > guess is that it was relevant back when Hotspot and the rest of the JDK were > often built separately. We have the username in the default $OPT string for > personal builds, so the HOTSPOT_BUILD_USER is just redundant information. > There is no reason to have the build user recorded in non personal builds. Thanks Erik. I'll leave this open a little longer for others to comment, but I also can't see a reason why this would be needed, especially in non-personal builds. - PR Comment: https://git.openjdk.org/jdk/pull/18136#issuecomment-1981126634
Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Wed, 6 Mar 2024 11:57:27 GMT, Andrew John Hughes wrote: > The HotSpot code has inserted the value of `HOTSPOT_BUILD_USER` into the > internal version string since before the JDK was open-sourced, so the > reasoning for this is not in the public history. See > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/abstract_vm_version.cpp#L284 > > ~~~ > $ /usr/lib/jvm/java-21-openjdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (21.0.2+13) for linux-amd64 JRE (21.0.2+13), built > on 2024-01-19T16:39:52Z by "mockbuild" with gcc 8.5.0 20210514 (Red Hat > 8.5.0-20) > ~~~ > > It is not clear what value this provides. By default, the value is set to the > system username, but it can be set to a static value using > `--with-build-user`. > > In trying to compare builds, this is a source of difference if the builds are > produced under different usernames and `--with-build-user` is not set to the > same value. > > It may also be a source of information leakage from the host system. It is > not clear what value it provides to the end user of the build. > > This patch proposes removing it altogether, but at least knowing why it needs > to be there would be progress from where we are now. We could also consider a > middle ground of only setting it for "adhoc" builds, those without set > version information, as is the case with the overall version output. > > With this patch: > > ~~~ > $ ~/builder/23/images/jdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) for > linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on 2024-03-06T00:23:37Z > with gcc 13.2.1 20230826 > > $ ~/builder/23/images/jdk/bin/java -XX:ErrorHandlerTest=1 > > $ grep '^vm_info' > /localhome/andrew/projects/openjdk/upstream/jdk/hs_err_pid1119824.log > vm_info: OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) > for linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on > 2024-03-06T00:23:37Z with gcc 13.2.1 20230826 > ~~~ > > The above are from a fastdebug build but I also built a product build with no > issues. I'm fine with this change. I wasn't around when this was introduced, but my guess is that it was relevant back when Hotspot and the rest of the JDK were often built separately. We have the username in the default $OPT string for personal builds, so the HOTSPOT_BUILD_USER is just redundant information. There is no reason to have the build user recorded in non personal builds. - Marked as reviewed by erikj (Reviewer). PR Review: https://git.openjdk.org/jdk/pull/18136#pullrequestreview-1919834395
Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Wed, 6 Mar 2024 11:57:27 GMT, Andrew John Hughes wrote: > The HotSpot code has inserted the value of `HOTSPOT_BUILD_USER` into the > internal version string since before the JDK was open-sourced, so the > reasoning for this is not in the public history. See > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/abstract_vm_version.cpp#L284 > > ~~~ > $ /usr/lib/jvm/java-21-openjdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (21.0.2+13) for linux-amd64 JRE (21.0.2+13), built > on 2024-01-19T16:39:52Z by "mockbuild" with gcc 8.5.0 20210514 (Red Hat > 8.5.0-20) > ~~~ > > It is not clear what value this provides. By default, the value is set to the > system username, but it can be set to a static value using > `--with-build-user`. > > In trying to compare builds, this is a source of difference if the builds are > produced under different usernames and `--with-build-user` is not set to the > same value. > > It may also be a source of information leakage from the host system. It is > not clear what value it provides to the end user of the build. > > This patch proposes removing it altogether, but at least knowing why it needs > to be there would be progress from where we are now. We could also consider a > middle ground of only setting it for "adhoc" builds, those without set > version information, as is the case with the overall version output. > > With this patch: > > ~~~ > $ ~/builder/23/images/jdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) for > linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on 2024-03-06T00:23:37Z > with gcc 13.2.1 20230826 > > $ ~/builder/23/images/jdk/bin/java -XX:ErrorHandlerTest=1 > > $ grep '^vm_info' > /localhome/andrew/projects/openjdk/upstream/jdk/hs_err_pid1119824.log > vm_info: OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) > for linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on > 2024-03-06T00:23:37Z with gcc 13.2.1 20230826 > ~~~ > > The above are from a fastdebug build but I also built a product build with no > issues. cross-compile failure is an unrelated risc download issue: ~~~ W: Failure while installing base packages. This will be re-attempted up to five times. W: See /home/runner/work/jdk/jdk/sysroot/debootstrap/debootstrap.log for details (possibly the package /var/cache/apt/archives/libssl3t64_3.1.5-1.1_riscv64.deb is at fault) ~~~ - PR Comment: https://git.openjdk.org/jdk/pull/18136#issuecomment-198085
Re: RFR: 8327389: Remove use of HOTSPOT_BUILD_USER
On Wed, 6 Mar 2024 11:57:27 GMT, Andrew John Hughes wrote: > The HotSpot code has inserted the value of `HOTSPOT_BUILD_USER` into the > internal version string since before the JDK was open-sourced, so the > reasoning for this is not in the public history. See > https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/abstract_vm_version.cpp#L284 > > ~~~ > $ /usr/lib/jvm/java-21-openjdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (21.0.2+13) for linux-amd64 JRE (21.0.2+13), built > on 2024-01-19T16:39:52Z by "mockbuild" with gcc 8.5.0 20210514 (Red Hat > 8.5.0-20) > ~~~ > > It is not clear what value this provides. By default, the value is set to the > system username, but it can be set to a static value using > `--with-build-user`. > > In trying to compare builds, this is a source of difference if the builds are > produced under different usernames and `--with-build-user` is not set to the > same value. > > It may also be a source of information leakage from the host system. It is > not clear what value it provides to the end user of the build. > > This patch proposes removing it altogether, but at least knowing why it needs > to be there would be progress from where we are now. We could also consider a > middle ground of only setting it for "adhoc" builds, those without set > version information, as is the case with the overall version output. > > With this patch: > > ~~~ > $ ~/builder/23/images/jdk/bin/java -Xinternalversion > OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) for > linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on 2024-03-06T00:23:37Z > with gcc 13.2.1 20230826 > > $ ~/builder/23/images/jdk/bin/java -XX:ErrorHandlerTest=1 > > $ grep '^vm_info' > /localhome/andrew/projects/openjdk/upstream/jdk/hs_err_pid1119824.log > vm_info: OpenJDK 64-Bit Server VM (fastdebug 23-internal-adhoc.andrew.jdk) > for linux-amd64 JRE (23-internal-adhoc.andrew.jdk), built on > 2024-03-06T00:23:37Z with gcc 13.2.1 20230826 > ~~~ > > The above are from a fastdebug build but I also built a product build with no > issues. Leaving a comment here as a reminder for myself to come back to this, this intrigues me - PR Comment: https://git.openjdk.org/jdk/pull/18136#issuecomment-1980739523
