Hello Magnus, Sorry for the late reply! This is awesome work and I don't want to see that to bit rot :-)
I rebased your changes on top of current master: https://github.com/openjdk/jdk/compare/master...lewurm:winenv-testing This branch is by no means ready to be reviewed, it's just whatever I've used to do my experiments. Also note that it was a bit painful to merge the changes in toolchain_windows.m4 due to the renaming. While I agree that toolchain_microsoft.m4 is the better name, I suggest to defer that change to a later point. I've tested the following scenarios: 1. cygwin on x86_64 to build x86_64 2. cygwin on x86_64 to build arm64 3. WSL1 on x86_64 to build x86_64 4. WSL1 on x86_64 to build arm64 5. WSL2 on x86_64 to build x86_64 6. WSL1 on arm64 to build arm64 Some context: Microsoft doesn't ship native Arm64 binaries for the VS toolchain (yet). Instead one can run x86 binaries via xtajit (a binary translator), but as one can imagine this comes with some performance penalty. Cygwin is also not available on Arm64. Cygwin already being slow, it made sense to avoid running Cygwin x86 on Arm64, and instead go the cross-compilation route in the OpenJDK build for Windows+Arm64. Previously the OpenJDK build did not support cross-compiling on Windows. Even today, on openjdk/jdk:master a dirty workaround for fixpath.exe is needed to build Windows+Arm64 ( https://github.com/openjdk/jdk/pull/212#issuecomment-695024586 ). So we have been very excited to see this WINENV patch :-) Here are comments for each scenario. ====================================== ## 1. cygwin on x86_64 to build x86_64 ====================================== ``` $ bash configure --with-devkit=/cygdrive/c/work/VS2019-16.6.1-devkit --with-build-devkit=/cygdrive/c/work/VS2019-16.6.1-devkit --with-boot-jdk=/cygdrive/c/work/jdk-16+22 --openjdk-target=aarch64-unknown-cygwin configure: Configuration created at Mon Nov 9 21:33:42 CET 2020. checking for basename... /usr/bin/basename checking for dirname... /usr/bin/dirname checking for file... /usr/bin/file checking for ldd... /usr/bin/ldd checking for bash... /usr/bin/bash checking for cat... /usr/bin/cat checking for chmod... /usr/bin/chmod checking for cp... /usr/bin/cp checking for cut... /usr/bin/cut checking for date... /usr/bin/date checking for gdiff... /cygdrive/c/work/winenv-cygwin/build/.configure-support/generated-configure.sh: line 10043: test: too many arguments /cygdrive/c/work/winenv-cygwin/build/.configure-support/generated-configure.sh: line 10047: test: too many arguments /cygdrive/c/work/winenv-cygwin/build/.configure-support/generated-configure.sh: line 10043: test: too many arguments /cygdrive/c/work/winenv-cygwin/build/.configure-support/generated-configure.sh: line 10047: test: too many arguments /cygdrive/c/work/winenv-cygwin/build/.configure-support/generated-configure.sh: line 10043: test: too many arguments /cygdrive/c/work/winenv-cygwin/build/.configure-support/generated-configure.sh: line 10047: test: too many arguments /cygdrive/c/work/winenv-cygwin/build/.configure-support/generated-configure.sh: line 10043: test: /cygdrive/c/Program: binary operator expected /cygdrive/c/work/winenv-cygwin/build/.configure-support/generated-configure.sh: line 10047: test: /cygdrive/c/Program: binary operator expected /cygdrive/c/work/winenv-cygwin/build/.configure-support/generated-configure.sh: line 10043: test: too many arguments /cygdrive/c/work/winenv-cygwin/build/.configure-support/generated-configure.sh: line 10047: test: too many arguments [not found] [...] ``` I didn't investigate it any further, as this has already been reported on the mailing list. ===================================== ## 2. cygwin on x86_64 to build arm64 ===================================== ``` $ bash configure --with-devkit=/cygdrive/c/work/VS2019-16.6.1-devkit --with-build-devkit=/cygdrive/c/work/VS2019-16.6.1-devkit --with-boot-jdk=/cygdrive/c/work/jdk-16+22 --openjdk-target=aarch64-unknown-cygwin ``` Same as scenario 1) ==================================== ## 3. WSL1 on x86_64 to build x86_64 ==================================== I do get some weird permission errors, so I run everything with sudo (chmod -R $user doesn't help): ``` $ sudo bash configure --with-build-devkit="/mnt/c/work/VS2019-16.6.1-devkit" --with-devkit="/mnt/c/work/VS2019-16.6.1-devkit" --with-boot-jdk="/mnt/c/work/jdk-16+22" [...] Configuration summary: * Debug level: release * HS debug level: product * JVM variants: server * JVM features: server: 'aot cds compiler1 compiler2 epsilongc g1gc graal jfr jni-check jvmci jvmti management nmt parallelgc serialgc services shenandoahgc vm-structs zgc' * OpenJDK target: OS: windows, CPU architecture: x86, address length: 64 * Version string: 16-internal+0-adhoc.root.winenv-wsl1 (16-internal) Tools summary: * Environment: wsl1 version 4.4.0-19041-Microsoft, #488-Microsoft Mon Sep 01 13:43:00 PST 2020 (Ubuntu 20.04.1 LTS); windows version 10.0.19041.572; prefix "/mnt"; root "[unavailable]" * Boot JDK: openjdk version "16" 2021-03-16 OpenJDK Runtime Environment AdoptOpenJDK (build 16+22-202011040337) OpenJDK 64-Bit Server VM AdoptOpenJDK (build 16+22-202011040337, mixed mode, sharing) (at /mnt/c/work/jdk-16~1) * Toolchain: microsoft (Microsoft Visual Studio 2019 16.6.1 (devkit)) * C Compiler: Version 19.26.28806 (at /mnt/c/work/vs2019-16.6.1-devkit/vc/bin/x64/cl.exe) * C++ Compiler: Version 19.26.28806 (at /mnt/c/work/vs2019-16.6.1-devkit/vc/bin/x64/cl.exe) Build performance summary: * Cores to use: 12 * Memory limit: 32605 MB ``` The build fails eventually: ``` $ sudo make jdk-image JOBS=12 LOG=info [...] /mnt/c/work/winenv-wsl1/make/scripts/fixpath.sh: line 405: /mnt/c/work/winenv-wsl1/build/windows-x86_64-server-release/jdk/bin/java: No such file or directory make[3]: *** [ExplodedImageOptimize.gmk:41: /mnt/c/work/winenv-wsl1/build/windows-x86_64-server-release/jdk/_optimize_image_exec.marker] Error 127 make[2]: *** [make/Main.gmk:437: exploded-image-optimize] Error 2 make[2]: *** Waiting for unfinished jobs.... ``` Which looks like a path issue, that should be relatively easy to fix (I didn't investigate). HotSpot and parts of the JDK built successfully though: ``` $ ./build/windows-x86_64-server-release/jdk/bin/java.exe -version openjdk version "16-internal" 2021-03-16 OpenJDK Runtime Environment (build 16-internal+0-adhoc.root.winenv-wsl1) OpenJDK 64-Bit Server VM (build 16-internal+0-adhoc.root.winenv-wsl1, mixed mode) ``` So I declare success on this one :-) =================================== ## 4. WSL1 on x86_64 to build arm64 =================================== ``` $ bash configure --with-build-devkit="/mnt/c/work/VS2019-16.6.1-devkit" --with-devkit="/mnt/c/work/VS2019-16.6.1-devkit" --with-boot-jdk="/mnt/c/work/jdk-16+22" --openjdk-target=aarch64-unknown-cygwin [...] checking for link... $FIXPATH /mnt/c/work/vs2019-16.6.1-devkit/vc/bin/arm64/link.exe checking if the found link.exe is actually the Visual Studio linker... yes configure: Using microsoft linker version 14.26.28806.0 [Microsoft (R) Incremental Linker Version 14.26.28806.0] checking for aarch64-unknown-cygwin-ml... [not found] checking for ml... [not found] configure: WARNING: using cross tools not prefixed with host triplet configure: WARNING: Ignoring value of AS from the environment. Use command line variables instead. checking for ml... [not found] checking for aarch64-unknown-cygwin-lib... [not found] checking for lib... $FIXPATH /mnt/c/work/vs2019-16.6.1-devkit/vc/bin/arm64/lib.exe checking for aarch64-unknown-cygwin-mt... [not found] checking for mt... $FIXPATH /mnt/c/work/vs2019-16.6.1-devkit/10/bin/x64/mt.exe checking for aarch64-unknown-cygwin-rc... [not found] checking for rc... $FIXPATH /mnt/c/work/vs2019-16.6.1-devkit/10/bin/x64/rc.exe checking for aarch64-unknown-cygwin-dumpbin... [not found] checking for dumpbin... $FIXPATH /mnt/c/work/vs2019-16.6.1-devkit/vc/bin/arm64/dumpbin.exe checking for aarch64-unknown-cygwin-gobjdump... [not found] checking for gobjdump... [not found] checking for aarch64-unknown-cygwin-objdump... [not found] checking for objdump... /usr/bin/objdump checking for build platform devkit... Microsoft Visual Studio 2019 16.6.1 (devkit) in /mnt/c/work/vs2019-16.6.1-devkit checking for cl... fixpath: failure: Cannot locate cygpath or wslpath configure: error: Could not find required tool for BUILD_CC /mnt/c/work/winenv-wsl1/build/.configure-support/generated-configure.sh: line 12: sed: command not found /mnt/c/work/winenv-wsl1/build/.configure-support/generated-configure.sh: line 38: sort: command not found /mnt/c/work/winenv-wsl1/build/.configure-support/generated-configure.sh: line 35: sed: command not found /mnt/c/work/winenv-wsl1/build/.configure-support/generated-configure.sh: line 53: sort: command not found /mnt/c/work/winenv-wsl1/build/.configure-support/generated-configure.sh: line 77: cat: command not found /mnt/c/work/winenv-wsl1/build/.configure-support/generated-configure.sh: line 40: rm: command not found ``` It looks like it's still not suitable for cross compilation. I might have messed up resolving some merge conflicts. I haven't investigated that further, but that would probably be a scenario that would be nice to have eventually. ==================================== ## 5. WSL2 on x86_64 to build x86_64 ==================================== I second the clock skew observations :-( I had a problem like that on WSL2 in a different context before https://github.com/microsoft/WSL/issues/4898 Apart from that, crossing the "file system boundary" got worse with WSL2, and as long checking out the sources on a Windows visible path is a requirment, I'm not sure if it's worth going down this road. ================================== ## 6. WSL1 on arm64 to build arm64 ================================== Same problem with permissions. I must be doing something wrong. ``` $ sudo bash configure --with-devkit="/mnt/c/lewurm/VS2019-16.6.1-devkit" --with-boot-jdk="/mnt/c/lewurm/jdk-16-ea+19-windows-aarch64" --disable-precompiled-headers [...] Configuration summary: * Debug level: release * HS debug level: product * JVM variants: server * JVM features: server: 'cds compiler1 compiler2 epsilongc g1gc jfr jni-check jvmti management nmt parallelgc serialgc services shenandoahgc vm-structs zgc' * OpenJDK target: OS: windows, CPU architecture: aarch64, address length: 64 * Version string: 16-internal+0-adhoc.root.openjdk-jdk (16-internal) Tools summary: * Environment: wsl1 version 4.4.0-19607-Microsoft, #1000-Microsoft Wed Apr 09 15:02:00 PST 2020 (Ubuntu 20.04 LTS); windows version 10.0.19607.1000; prefix "/mnt"; root "[unavailable]" * Boot JDK: openjdk version "16-ea" 2020-10-09 OpenJDK Runtime Environment Microsoft (build 16-ea+19-windows-aarch64) OpenJDK 64-Bit Server VM Microsoft (build 16-ea+19-windows-aarch64, mixed mode) (at /mnt/c/lewurm/jdk-16~1) * Toolchain: microsoft (Microsoft Visual Studio 2019 16.6.1 (devkit)) * C Compiler: Version 19.27.29112 (at /mnt/c/lewurm/vs2019-16.6.1-devkit/vc/bin/arm64/cl.exe) * C++ Compiler: Version 19.27.29112 (at /mnt/c/lewurm/vs2019-16.6.1-devkit/vc/bin/arm64/cl.exe) Build performance summary: * Cores to use: 64 * Memory limit: 524204 MB ``` So a couple of things that I hacked up: * I manually copied x86 MSVC binaries into /mnt/c/lewurm/vs2019-16.6.1-devkit/vc/bin/arm64 * I manually copied arm64 binaries from the Windows SDK into /mnt/c/lewurm/vs2019-16.6.1-devkit/10/bin/arm64 Eventually I need to update the devkit script to create a proper archive and/or update the build script to be aware that x86 MSVC binaries must be used on arm64 (the hope is that this eventually won't be true anymore). * add detection for WSL on aarch64: https://github.com/lewurm/openjdk/commit/4da4b6df16d05c97b7f48bef166e6d2080d37511 * The boot-jdk is from https://github.com/microsoft/openjdk-aarch64/releases * `--disable-precompiled-headers` is needed because otherwise a bug in the binary translator (xtajit) is triggered when running x86 cl.exe with it. Note to myself: File a bug report. Let's build a jdk image: ``` $ sudo make jdk-image JOBS=64 LOG=info [...] /mnt/c/lewurm/openjdk-jdk/make/scripts/fixpath.sh: line 405: /mnt/c/lewurm/openjdk-jdk/build/windows-aarch64-server-release/jdk/bin/java: No such file or directory make[3]: *** [ExplodedImageOptimize.gmk:41: /mnt/c/lewurm/openjdk-jdk/build/windows-aarch64-server-release/jdk/_optimize_image_exec.marker] Error 127 make[2]: *** [make/Main.gmk:437: exploded-image-optimize] Error 2 $ ./build/windows-aarch64-server-release/jdk/bin/java.exe -version openjdk version "16-internal" 2021-03-16 OpenJDK Runtime Environment (build 16-internal+0-adhoc.root.openjdk-jdk) OpenJDK 64-Bit Server VM (build 16-internal+0-adhoc.root.openjdk-jdk, mixed mode) ``` So same issue as in scenario 3 (WSL1 on x86_64 building x86_64), which is _great_ news :-) ============== ## Performance ============== I did a comparision run on WSL1 on x86_64 to build x86_64 (scenario 3), once with the WINENV patches applied and once without it. Overall it seems there is some performance regression, but not too bad. The profile logs aren't very meaningful to me, but if I would need to guess the difference is due to having a native `fixpath.exe` vs. `fixpath.sh`. Logs are here: https://gist.github.com/lewurm/2fbf2971a55a3a4a4f5933b105925976 ========= ## Epilog ========= I hope this has met your bar of exotic setup. Unfortunately I can't provide much feedback on the automagic. I would love to see your work landing eventually! Thank you, -Bernhard ________________________________________ From: build-dev <build-dev-r...@openjdk.java.net> on behalf of Magnus Ihse Bursie <magnus.ihse.bur...@oracle.com> Sent: Friday, August 28, 2020 11:09 To: Ludovic Henry; Yasumasa Suenaga; Monica Beckwith; build-dev Subject: Re: Preliminary review for new WINENV support On 2020-08-24 19:21, Ludovic Henry wrote: > Hi Magnus, > > I want to follow up on this work of yours, as we've particular interest in it > for the Windows-AArch64 port. > > Let me know how I could assist you in this effort. The big blockers for this fix is basically these two issues: >> I have also temporarily disabled the javac server, and building without >> absolute paths. I believe a better way forward is to address these with >> improved functionality in separate patches, instead of trying to work around >> the issues introduced by them in this patch. Until that is fixed, any integration of this patch is a no-go. I have been working on those, but a lot of other stuff has kept me from gaining much progress; everything from long vacations to preparations to the switch to github. Unfortunately, I don't think there's an easy way for you to help on those two blockers. However, there are two more things that I'd like to look into, if possible, before integrating my patch, and there you might be able to help. One is performance. I got a somewhat different performance profile with my patch than without -- some things were faster, and some were slower. I'd like to understand why some things got slower, and if it is possible to do anything about. (Why things got faster I don't care about, I'm just happy :)) So maybe you can help me with some performance analysis? With the limited availability of tooling for makefiles, this is not really trivial. What I usually do is I run with JOBS=1 (to be able to track each task individually without effects of concurrency), and LOG=info,profile. This will give you time stamps for each individual execution in build.log. And see if there is any interesting differences with and without the patch. The other thing is that WSL2 works awfully bad. :( Maybe there's nothing to be done about it, if the thing being shipped is just too buggy, but maybe we do things incorrectly, or that unnecessarily triggers bugs. Or maybe there are workarounds we could use. But I think it could be worth some more effort to try to get it working. Maybe you can help with this? /Magnus
