Re: [JENKINS] Lucene-main-MacOSX (64bit/hotspot/jdk-21.0.1) - Build # 11448 - Still Failing!
Uwe - something has been failing fairly regularly on the Mac VM (and on openj9). I again wonder how much sense these builds have. We can run a MacOS build on github and openj9 has been... unreliable to say the least. I'm afraid people will stop looking at errors if we have too many false positives. Dawid On Tue, Jun 11, 2024 at 2:13 AM Policeman Jenkins Server < jenk...@thetaphi.de> wrote: > Build: https://jenkins.thetaphi.de/job/Lucene-main-MacOSX/11448/ > Java: 64bit/hotspot/jdk-21.0.1 -XX:+UseCompressedOops -XX:+UseParallelGC > > No tests ran. > > - > To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org > For additional commands, e-mail: builds-h...@lucene.apache.org
Re: Intellij build/test times
I've taken a look at intellij source code and I don't think the default annotation preprocessor location can be easily redirected in "intellij compilation" mode. This personally doesn't bother me much but maybe you're right that the official developer docs shouldn't make people think too hard and assume gradle is used both from command line and the IDE - I honestly don't know... IDEs are a moving target, it's really difficult to keep up with the changes, eh. D. On Mon, Jun 10, 2024 at 5:07 PM Dawid Weiss wrote: > > > If I set IJ build/test to "gradle" and then right click on "core" in >> the Project tab -- it gives an option like "run tests in >> lucene-root.lucene.core" which works. > > > It works because you're running all tests in a module. > > >> At the very top (lucene >> [lucene-root]) of the hierarchy you can right-click and select "run >> all tests", but this fails with "Error running 'All in lucene-root': >> No junit.jar". I thought this had once worked, but maybe I was only >> running tests in core? >> > > It can't work in the current setup - the topmost project isn't a Java > module. Maybe it worked at some point when we had ant-generated intellij > files (which aggregated everything on the same classpath)? I honestly can't > remember. > > Honestly, if you want to run everything, add a gradle configuration and > run it there - it'll be faster than a sequential run from the IDE. > > Dawid >
Re: Intellij build/test times
If I set IJ build/test to "gradle" and then right click on "core" in > the Project tab -- it gives an option like "run tests in > lucene-root.lucene.core" which works. It works because you're running all tests in a module. > At the very top (lucene > [lucene-root]) of the hierarchy you can right-click and select "run > all tests", but this fails with "Error running 'All in lucene-root': > No junit.jar". I thought this had once worked, but maybe I was only > running tests in core? > It can't work in the current setup - the topmost project isn't a Java module. Maybe it worked at some point when we had ant-generated intellij files (which aggregated everything on the same classpath)? I honestly can't remember. Honestly, if you want to run everything, add a gradle configuration and run it there - it'll be faster than a sequential run from the IDE. Dawid
Re: Intellij build/test times
If I set IJ build/test to "gradle" and then right click on "core" in the Project tab -- it gives an option like "run tests in lucene-root.lucene.core" which works. At the very top (lucene [lucene-root]) of the hierarchy you can right-click and select "run all tests", but this fails with "Error running 'All in lucene-root': No junit.jar". I thought this had once worked, but maybe I was only running tests in core? On Mon, Jun 10, 2024 at 9:37 AM Dawid Weiss wrote: >> >> When I say "run in IJ" I mean I right clicked a button somewhere and said >> "run all tests" :) I expect it was with the gradle runner selected. > > > When you find that button, let me know. It's probably right next to the Holy > Grail. ;) > > Dawid - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Intellij build/test times
> > When I say "run in IJ" I mean I right clicked a button somewhere and said > "run all tests" :) I expect it was with the gradle runner selected. > When you find that button, let me know. It's probably right next to the Holy Grail. ;) Dawid >
Re: Intellij build/test times
> > Yet I feel certain I have been able to run all tests in IJ before. > > > > I don't think this was ever the case with intellij. Or maybe you ran those > tests via gradle? When I say "run in IJ" I mean I right clicked a button somewhere and said "run all tests" :) I expect it was with the gradle runner selected. On Mon, Jun 10, 2024 at 6:38 AM Dawid Weiss wrote: > > Yet I feel certain I have been able to run all tests in IJ before. >> > > I don't think this was ever the case with intellij. Or maybe you ran those > tests via gradle? > > >> There are a few oddities that happen in intellij that require you to >> fiddle with the build in odd ways, but I wonder if these will be >> reproducible or if they maybe happen because there is some bad state: >> > > Intellij changes from version to version so there is no "one" version, > unfortunately. Also, sometimes > some of the settings intellij sets up on the initial import persist and > 'reloading' the gradle plugin does not > help to update them. An occasional import from scratch is a good way to > check if something like this happens. > > 1. Building branch_9x with intellij builder selected in the gradle >> settings failed to build the benchmark module due to some modules not >> being visible to it (e.g. icu). So I "unload module benchmark" >> effectively skipping building that, and then I am able to build the >> rest of lucene. YMMV >> > > I can't reproduce this on main, haven't tried 9x. > > >> 2. After switching back to main branch, I got a build failure "error: >> Annotation generator had thrown the exception. >> javax.annotation.processing.FilerException: Attempt to recreate a file >> for type >> org.apache.lucene.benchmark.jmh.jmh_generated.ExpressionsBenchmark_expression_jmhTest". >> I see there are some generated classes in >> lucene/benchmark_jmh/src/java/generated, that show up in git status, >> so I remove that folder and then everything is fine - some cruft left >> from a previous build? >> > > This is intellij's compiler emitting annotation processor output (jmh) to > an incorrect location. > It's javac's '-s' option. Not sure how to configure this option so that > intellij picks it up from the gradle build model. > > >> Side note: when running all tests in "intellij" mode you cannot do it >> by selecting the "core" module - you have to navigate down to the >> "tests" folder. > > > Correct, This is the classpath-container unit which contains tests. > > >> Also I observed that when running tests in "gradle" >> mode I no longer observed the slow startup times? Really unsure what >> that means. Maybe some networking thing? >> > > Or the daemon starting for the first time - this is relatively expensive. > Once the daemon is up, launch times should > be faster. > > >> But the main thing I learned is that while running tests using >> intellij builder mostly works, MemorySegmentIndexInputProvider fails >> to get loaded and any test using MMapDirectory will fail, regardless >> of whether I run a single test or a whole suite. This is true on both >> 9x and main branches and causes 1/3-1/2 of tests to fail in core. >> > > This class is in src/java21 - it's not picked up as a source folder by > intellij. And if you add it manually, you'll get errors related to > compilation because of the way the gradle build "cheats" javac and bypasses > explicitly importing > jdk.incubator.vector and not declaring --enable-preview... > > In short: yes, it'll be difficult to work around that, especially with an > automatic project import in intellij (perhaps you could hand-craft > configuration files so that it works, I'm not sure). > > >> At this point I'm reluctant to recommend using the intellij build >> mode. Maybe it will become viable again if we can figure out how to >> get MMapDirectory tests to work with it? >> > > I use it because it's much faster. Whenever I need something more > complex, I set up a dedicated gradle launch configuration for that - like > so: > > [image: image.png] > > I'm neither gradle nor intellij expert though, it's mostly a > trial-and-error of what works and what doesn't... > > D. >
Re: Intellij build/test times
> Yet I feel certain I have been able to run all tests in IJ before. > I don't think this was ever the case with intellij. Or maybe you ran those tests via gradle? > There are a few oddities that happen in intellij that require you to > fiddle with the build in odd ways, but I wonder if these will be > reproducible or if they maybe happen because there is some bad state: > Intellij changes from version to version so there is no "one" version, unfortunately. Also, sometimes some of the settings intellij sets up on the initial import persist and 'reloading' the gradle plugin does not help to update them. An occasional import from scratch is a good way to check if something like this happens. 1. Building branch_9x with intellij builder selected in the gradle > settings failed to build the benchmark module due to some modules not > being visible to it (e.g. icu). So I "unload module benchmark" > effectively skipping building that, and then I am able to build the > rest of lucene. YMMV > I can't reproduce this on main, haven't tried 9x. > 2. After switching back to main branch, I got a build failure "error: > Annotation generator had thrown the exception. > javax.annotation.processing.FilerException: Attempt to recreate a file > for type > org.apache.lucene.benchmark.jmh.jmh_generated.ExpressionsBenchmark_expression_jmhTest". > I see there are some generated classes in > lucene/benchmark_jmh/src/java/generated, that show up in git status, > so I remove that folder and then everything is fine - some cruft left > from a previous build? > This is intellij's compiler emitting annotation processor output (jmh) to an incorrect location. It's javac's '-s' option. Not sure how to configure this option so that intellij picks it up from the gradle build model. > Side note: when running all tests in "intellij" mode you cannot do it > by selecting the "core" module - you have to navigate down to the > "tests" folder. Correct, This is the classpath-container unit which contains tests. > Also I observed that when running tests in "gradle" > mode I no longer observed the slow startup times? Really unsure what > that means. Maybe some networking thing? > Or the daemon starting for the first time - this is relatively expensive. Once the daemon is up, launch times should be faster. > But the main thing I learned is that while running tests using > intellij builder mostly works, MemorySegmentIndexInputProvider fails > to get loaded and any test using MMapDirectory will fail, regardless > of whether I run a single test or a whole suite. This is true on both > 9x and main branches and causes 1/3-1/2 of tests to fail in core. > This class is in src/java21 - it's not picked up as a source folder by intellij. And if you add it manually, you'll get errors related to compilation because of the way the gradle build "cheats" javac and bypasses explicitly importing jdk.incubator.vector and not declaring --enable-preview... In short: yes, it'll be difficult to work around that, especially with an automatic project import in intellij (perhaps you could hand-craft configuration files so that it works, I'm not sure). > At this point I'm reluctant to recommend using the intellij build > mode. Maybe it will become viable again if we can figure out how to > get MMapDirectory tests to work with it? > I use it because it's much faster. Whenever I need something more complex, I set up a dedicated gradle launch configuration for that - like so: [image: image.png] I'm neither gradle nor intellij expert though, it's mostly a trial-and-error of what works and what doesn't... D.
JDK 23 Feature Freeze / New Loom EA builds
Welcome to the latest OpenJDK Quality Outreach update! JDK 23, scheduled for General Availability on September 17, 2024, is now in Rampdown Phase One (RDP1) [1]. At this point, the overall JDK 23 feature set is frozen (see the final list of JEPs integrated into JDK 23 below) and only low-risk enhancements might still be considered. The coming weeks should be leveraged to identify and resolve as many issues as possible, i.e. before JDK 23 enters the Release Candidates phase in early August [2]. We count on you to test your projects and help us make JDK 23 another solid release! This time, we are covering several heads-up related to JDK 23 : Deprecate the Memory-Access Methods in sun.misc.Unsafe for Removal and default annotation processing policy change. Also, make sure to check the new Loom early-access builds which have an improved Java monitors implementation to work better with virtual threads. [1] https://mail.openjdk.org/pipermail/jdk-dev/2024-June/009053.html [2] https://openjdk.org/projects/jdk/23/ ## Heads-Up - JDK 23: Deprecate the Memory-Access Methods in sun.misc.Unsafe for Removal As mentioned in a previous communication [3], there’s a plan to ultimately remove the sun.misc.Unsafe memory-access methods as the platform offers safer alternatives. JEP 471 (Deprecate the Memory-Access Methods in sun.misc.Unsafe for Removal) [4] outlines in more detail this plan including the initial step which is happening in JDK 23, i.e., all of the sun.misc unsafe memory-access methods are now marked as deprecated for removal. This will cause, in JDK 23, compile-time deprecation warnings for code that refers to these methods, alerting library developers to their forthcoming removal. A new command-line option also enables application developers and users to receive runtime warnings when those methods are used. Developers relying on those sun.misc.Unsafe APIs for access memory are strongly encouraged to start, if they haven't done so yet, the migration from the sun.misc.Unsafe APIs to supported replacements. For more details, make sure to read JEP 471 (Deprecate the Memory-Access Methods in sun.misc.Unsafe for Removal). [3] https://mail.openjdk.org/pipermail/quality-discuss/2024-January/001132.html [4] https://openjdk.org/jeps/471 ## Heads-Up - JDK 23: Changes Default Annotation Processing Policy Annotation processing is a compile-time feature, where javac scans the to-be-compiled source files for annotations and then the class path for matching annotation processors, so they can generate source code. Up to JDK 22, this feature is enabled by default, which may have been reasonable when it was introduced in JDK 6 circa 2006, but from a current perspective, in the interest of making build output more robust against annotation processors being placed on the class path unintentionally, this is much less reasonable. Hence, starting with JDK 23, javac requires an additional command-line option to enable annotation processing. ### New `-proc` Value To that end, the pre-existing option `-proc:$policy` was extended, where `$policy` can now have the following values: - `none`: compilation _without_ annotation processing, this policy exists since JDK 6 - `only`: annotation processing _without_ compilation, this policy exists since JDK 6 - `full`: annotation processing followed by compilation, this policy is the default in JDK ≤22 but the value itself is new (see next section for versions that support it) Up to and including JDK 22, code bases that require annotation processing before compilation could rely on javac's default behavior to process annotations but that is no longer the case. Starting with JDK 23, at least one annotation-processing command line option needs to be present. If neither `-processor`, `--processor-path`, now `--processor-module-path` is used, `-proc:only` or `-proc:full` has to be provided. In other words, absent other command line options, `-proc:none` is the default on JDK 23. ### Migration to `-proc:full` Several measures were undertaken to help projects prepare for the switch to `-proc:full`: - As of the April 2024 JDK security updates, support for `-proc:full` has been backported to 17u (17.0.11) and 11u (11.0.23) for both Oracle JDK and OpenJDK distributions. Additionally, Oracle's 8u release (8u411) also supports `-proc:full`. - Starting in JDK 21, javac prints an informative message if implicit usage of annotation processing under the default policy is detected. With `-proc:full` backported, it is possible to configure a build that will work the same before and after the change in javac's default policy. Additional details can be found in the original proposal [5]. [5] https://mail.openjdk.org/pipermail/jdk-dev/2024-May/009028.html ## Heads-up - Loom: New EA builds with improved Java monitors implementation to work better with virtual threads Project Loom published new early-access builds [6]. These builds have an improved
