Re: Java 17 and Lucene

[Jigar Shah]

Michael,

Is this recommended "-XX:+UseZGC options to enable ZGC." as it claims very
low pauses.

For "*DY* (2021-10-19 08:14:33): Upgrade to JDK17+35" execution for "Indexing
throughput "
is ZGC used for the "Indexing throughput
" test?


On Wed, Oct 20, 2021 at 8:27 AM Michael McCandless <
luc...@mikemccandless.com> wrote:

> Nightly benchmarks managed to succeed (once, so far) on JDK 17:
> https://home.apache.org/~mikemccand/lucenebench/
>
> No obvious performance changes on quick look.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Tue, Oct 19, 2021 at 8:42 PM Gautam Worah 
> wrote:
>
> > Thanks for the note of caution Uwe.
> >
> > > On our Jenkins server running with AMD Ryzen CPU it happens quite often
> > that JDK 16, JDK 17 and JDK 18 hang during tests and stay unkillable
> (only
> > a hard kill with" kill -9")
> >
> > Scary stuff.
> > I'll try to reproduce the hang first and then try to get the JVM logs.
> I'll
> > respond back here if I find something useful.
> >
> > > Do you get this error in lucene:core:ecjLintMain and not during
> compile?
> > Then this is https://issues.apache.org/jira/browse/LUCENE-10185, solved
> > already.
> >
> > Ahh. I should've been clearer with my comment. The error we see is
> because
> > we have forked the class and have modified it a bit.
> > I just assumed that the upstream Lucene package would've also gotten
> errors
> > on the JDK17 build because it was untouched.
> >
> > -
> > Gautam Worah.
> >
> >
> > On Tue, Oct 19, 2021 at 5:07 AM Michael Sokolov 
> > wrote:
> >
> > > > I would a bit careful: On our Jenkins server running with AMD Ryzen
> CPU
> > > it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests
> > and
> > > stay unkillable (only a hard kill with" kill -9"). Previous Java
> versions
> > > don't hang. It happens not all the time (about 1/4th of all builds) and
> > due
> > > to the fact that the JVM is unresponsible it is not possible to get a
> > stack
> > > trace with "jstack". If you know a way to get the stack trace, I'd
> happy
> > to
> > > get help.
> > >
> > > ooh that sounds scary. I suppose one could maybe get core dumps using
> > > the right signal and debug that way? Oh wait you said only 9 works,
> > > darn! How about attaching using gdb? Do we maintain GC logs for these
> > > Jenkins builds? Maybe something suspicious would show up there.
> > >
> > > By the way the JDK is absolutely "responsible" in this situation! Not
> > > responsive maybe ...
> > >
> > > On Tue, Oct 19, 2021 at 4:46 AM Uwe Schindler  wrote:
> > > >
> > > > Hi,
> > > >
> > > > > Hey,
> > > > >
> > > > > Our team at Amazon Product Search recently ran our internal
> > benchmarks
> > > with
> > > > > JDK 17.
> > > > > We saw a ~5% increase in throughput and are in the process of
> > > > > experimenting/enabling it in production.
> > > > > We also plan to test the new Corretto Generational Shenandoah GC.
> > > >
> > > > I would a bit careful: On our Jenkins server running with AMD Ryzen
> CPU
> > > it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests
> > and
> > > stay unkillable (only a hard kill with" kill -9"). Previous Java
> versions
> > > don't hang. It happens not all the time (about 1/4th of all builds) and
> > due
> > > to the fact that the JVM is unresponsible it is not possible to get a
> > stack
> > > trace with "jstack". If you know a way to get the stack trace, I'd
> happy
> > to
> > > get help.
> > > >
> > > > Once I figured out what makes it hang, I will open issues in OpenJDK
> (I
> > > am OpenJDK member/editor). I have now many stuck JVMs running to
> analyze
> > on
> > > the server, so you're invited to help! At the moment, I have no time to
> > > take care, so any help is useful.
> > > >
> > > > > On a side note, the Lucene codebase still uses the deprecated (as
> of
> > > > > JDK17) AccessController
> > > > > in the RamUsageEstimator class.
> > > > > We suppressed the warning for now (based on recommendations
> > > > >  > > > > dev/202106.mbox/%3CJIRA.13369440.1617476525000.615331.16239514800
> > > > > 5...@atlassian.jira%3E>
> > > > > from the Apache Derby mailing list).
> > > >
> > > > This should not be an issue, because we compile Lucene with javac
> > > parameter "--release 11", so it won't show any warning that you need to
> > > suppress. Looks like your build system at Amazon is not the original
> one
> > by
> > > Lucene's Gradle, which shows no warnings at all.
> > > >
> > > > Uwe
> > > >
> > > > > Gautam Worah.
> > > > >
> > > > >
> > > > > On Mon, Oct 18, 2021 at 3:02 PM Michael McCandless <
> > > > > luc...@mikemccandless.com> wrote:
> > > > >
> > > > > > Also, I try to semi-aggressively upgrade Lucene's nightly
> > benchmarks
> > > to new
> > > > > > JDK releases and leave an annotation on the nightly charts:
> > > 

Re: Java 17 and Lucene

[Michael Sokolov]

The "System Requirements" page for each release lists the JDK it was
built with and tested most extensively with; eg
https://lucene.apache.org/core/8_10_1/SYSTEM_REQUIREMENTS.html (JDK8
there; 9.0 will target JDK11)

That is pretty conservative but safe. Generally speaking we are always
testing cutting-edge JDKs, so you can be pretty confident about say
JDK11, but it's best to run your own tests, of course.

On Tue, Oct 19, 2021 at 8:19 PM Kevin Rosendahl
 wrote:
>
> Thank you all for the information, it's very useful. Seems like it's best
> to hold off on upgrading for now, but great to know that different JDK
> versions are at least being exercised in CI.
>
> I'm wondering, is there a better way to assess the production readiness of
> a Lucene/JDK combination than just emailing the user group, or is this our
> best bet in the future as well?
>
> Thanks again!
> Kevin
>
> On Tue, Oct 19, 2021 at 5:07 AM Michael Sokolov  wrote:
>
> > > I would a bit careful: On our Jenkins server running with AMD Ryzen CPU
> > it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests and
> > stay unkillable (only a hard kill with" kill -9"). Previous Java versions
> > don't hang. It happens not all the time (about 1/4th of all builds) and due
> > to the fact that the JVM is unresponsible it is not possible to get a stack
> > trace with "jstack". If you know a way to get the stack trace, I'd happy to
> > get help.
> >
> > ooh that sounds scary. I suppose one could maybe get core dumps using
> > the right signal and debug that way? Oh wait you said only 9 works,
> > darn! How about attaching using gdb? Do we maintain GC logs for these
> > Jenkins builds? Maybe something suspicious would show up there.
> >
> > By the way the JDK is absolutely "responsible" in this situation! Not
> > responsive maybe ...
> >
> > On Tue, Oct 19, 2021 at 4:46 AM Uwe Schindler  wrote:
> > >
> > > Hi,
> > >
> > > > Hey,
> > > >
> > > > Our team at Amazon Product Search recently ran our internal benchmarks
> > with
> > > > JDK 17.
> > > > We saw a ~5% increase in throughput and are in the process of
> > > > experimenting/enabling it in production.
> > > > We also plan to test the new Corretto Generational Shenandoah GC.
> > >
> > > I would a bit careful: On our Jenkins server running with AMD Ryzen CPU
> > it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests and
> > stay unkillable (only a hard kill with" kill -9"). Previous Java versions
> > don't hang. It happens not all the time (about 1/4th of all builds) and due
> > to the fact that the JVM is unresponsible it is not possible to get a stack
> > trace with "jstack". If you know a way to get the stack trace, I'd happy to
> > get help.
> > >
> > > Once I figured out what makes it hang, I will open issues in OpenJDK (I
> > am OpenJDK member/editor). I have now many stuck JVMs running to analyze on
> > the server, so you're invited to help! At the moment, I have no time to
> > take care, so any help is useful.
> > >
> > > > On a side note, the Lucene codebase still uses the deprecated (as of
> > > > JDK17) AccessController
> > > > in the RamUsageEstimator class.
> > > > We suppressed the warning for now (based on recommendations
> > > >  > > > dev/202106.mbox/%3CJIRA.13369440.1617476525000.615331.16239514800
> > > > 5...@atlassian.jira%3E>
> > > > from the Apache Derby mailing list).
> > >
> > > This should not be an issue, because we compile Lucene with javac
> > parameter "--release 11", so it won't show any warning that you need to
> > suppress. Looks like your build system at Amazon is not the original one by
> > Lucene's Gradle, which shows no warnings at all.
> > >
> > > Uwe
> > >
> > > > Gautam Worah.
> > > >
> > > >
> > > > On Mon, Oct 18, 2021 at 3:02 PM Michael McCandless <
> > > > luc...@mikemccandless.com> wrote:
> > > >
> > > > > Also, I try to semi-aggressively upgrade Lucene's nightly benchmarks
> > to new
> > > > > JDK releases and leave an annotation on the nightly charts:
> > > > > https://home.apache.org/~mikemccand/lucenebench/
> > > > >
> > > > > I just now upgraded to JDK 17 and kicked off a new benchmark run ...
> > in a
> > > > > few hours it should show the new data points and then I'll try to
> > remember
> > > > > to annotate it tomorrow.
> > > > >
> > > > > So let's see whether nightly benchmarks uncover any performance
> > changes
> > > > > from JDK17 :)
> > > > >
> > > > > Mike McCandless
> > > > >
> > > > > http://blog.mikemccandless.com
> > > > >
> > > > >
> > > > > On Mon, Oct 18, 2021 at 5:36 PM Robert Muir 
> > wrote:
> > > > >
> > > > > > We test different releases on different platforms (e.g. Linux,
> > Windows,
> > > > > > Mac).
> > > > > > We also test EA (Early Access) releases of openjdk versions during
> > the
> > > > > > development process.
> > > > > > This finds bugs before they get released.
> > > > > >
> > > > > > More information about versions/EA testing:
> > 

Re: Java 17 and Lucene

[Michael McCandless]

Nightly benchmarks managed to succeed (once, so far) on JDK 17:
https://home.apache.org/~mikemccand/lucenebench/

No obvious performance changes on quick look.

Mike McCandless

http://blog.mikemccandless.com


On Tue, Oct 19, 2021 at 8:42 PM Gautam Worah  wrote:

> Thanks for the note of caution Uwe.
>
> > On our Jenkins server running with AMD Ryzen CPU it happens quite often
> that JDK 16, JDK 17 and JDK 18 hang during tests and stay unkillable (only
> a hard kill with" kill -9")
>
> Scary stuff.
> I'll try to reproduce the hang first and then try to get the JVM logs. I'll
> respond back here if I find something useful.
>
> > Do you get this error in lucene:core:ecjLintMain and not during compile?
> Then this is https://issues.apache.org/jira/browse/LUCENE-10185, solved
> already.
>
> Ahh. I should've been clearer with my comment. The error we see is because
> we have forked the class and have modified it a bit.
> I just assumed that the upstream Lucene package would've also gotten errors
> on the JDK17 build because it was untouched.
>
> -
> Gautam Worah.
>
>
> On Tue, Oct 19, 2021 at 5:07 AM Michael Sokolov 
> wrote:
>
> > > I would a bit careful: On our Jenkins server running with AMD Ryzen CPU
> > it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests
> and
> > stay unkillable (only a hard kill with" kill -9"). Previous Java versions
> > don't hang. It happens not all the time (about 1/4th of all builds) and
> due
> > to the fact that the JVM is unresponsible it is not possible to get a
> stack
> > trace with "jstack". If you know a way to get the stack trace, I'd happy
> to
> > get help.
> >
> > ooh that sounds scary. I suppose one could maybe get core dumps using
> > the right signal and debug that way? Oh wait you said only 9 works,
> > darn! How about attaching using gdb? Do we maintain GC logs for these
> > Jenkins builds? Maybe something suspicious would show up there.
> >
> > By the way the JDK is absolutely "responsible" in this situation! Not
> > responsive maybe ...
> >
> > On Tue, Oct 19, 2021 at 4:46 AM Uwe Schindler  wrote:
> > >
> > > Hi,
> > >
> > > > Hey,
> > > >
> > > > Our team at Amazon Product Search recently ran our internal
> benchmarks
> > with
> > > > JDK 17.
> > > > We saw a ~5% increase in throughput and are in the process of
> > > > experimenting/enabling it in production.
> > > > We also plan to test the new Corretto Generational Shenandoah GC.
> > >
> > > I would a bit careful: On our Jenkins server running with AMD Ryzen CPU
> > it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests
> and
> > stay unkillable (only a hard kill with" kill -9"). Previous Java versions
> > don't hang. It happens not all the time (about 1/4th of all builds) and
> due
> > to the fact that the JVM is unresponsible it is not possible to get a
> stack
> > trace with "jstack". If you know a way to get the stack trace, I'd happy
> to
> > get help.
> > >
> > > Once I figured out what makes it hang, I will open issues in OpenJDK (I
> > am OpenJDK member/editor). I have now many stuck JVMs running to analyze
> on
> > the server, so you're invited to help! At the moment, I have no time to
> > take care, so any help is useful.
> > >
> > > > On a side note, the Lucene codebase still uses the deprecated (as of
> > > > JDK17) AccessController
> > > > in the RamUsageEstimator class.
> > > > We suppressed the warning for now (based on recommendations
> > > >  > > > dev/202106.mbox/%3CJIRA.13369440.1617476525000.615331.16239514800
> > > > 5...@atlassian.jira%3E>
> > > > from the Apache Derby mailing list).
> > >
> > > This should not be an issue, because we compile Lucene with javac
> > parameter "--release 11", so it won't show any warning that you need to
> > suppress. Looks like your build system at Amazon is not the original one
> by
> > Lucene's Gradle, which shows no warnings at all.
> > >
> > > Uwe
> > >
> > > > Gautam Worah.
> > > >
> > > >
> > > > On Mon, Oct 18, 2021 at 3:02 PM Michael McCandless <
> > > > luc...@mikemccandless.com> wrote:
> > > >
> > > > > Also, I try to semi-aggressively upgrade Lucene's nightly
> benchmarks
> > to new
> > > > > JDK releases and leave an annotation on the nightly charts:
> > > > > https://home.apache.org/~mikemccand/lucenebench/
> > > > >
> > > > > I just now upgraded to JDK 17 and kicked off a new benchmark run
> ...
> > in a
> > > > > few hours it should show the new data points and then I'll try to
> > remember
> > > > > to annotate it tomorrow.
> > > > >
> > > > > So let's see whether nightly benchmarks uncover any performance
> > changes
> > > > > from JDK17 :)
> > > > >
> > > > > Mike McCandless
> > > > >
> > > > > http://blog.mikemccandless.com
> > > > >
> > > > >
> > > > > On Mon, Oct 18, 2021 at 5:36 PM Robert Muir 
> > wrote:
> > > > >
> > > > > > We test different releases on different platforms (e.g. Linux,
> > Windows,
> > > > > > Mac).
> > > > > > We also test EA