Re: [JEXL] intended thread safety of JexlFeatures and JexlPermissions
Your are correct, the engine (and the parser) do use its own JexlFeatures copies (expressionFeatures/scriptFeatures members) that are never modified after creation. An equivalent rule applies for JexlOptions btw, copied for isolation for each evaluation. Those classes, by themselves, even if they are not thread-safe, never cause a thread-safety threat since they are always private unmodified copies used during evaluation. But this does not protect against the possibility of 'thread-adverse' reserved-wordset or namespace-predicate implementations used to construct the JexlFeature instances. IMHO, trying to protect against those potential mistakes would only restrict the possibilities of valid usages. Henrib - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [ALL] Java CI badge does not agree with workflow status
On Fri, 20 Oct 2023 at 17:35, Alex Herbert wrote: > > Navigating to the Java CI workflow you can generate a badge using the > ... menu in the top right next to the search filter. This creates a > badge with a different URL to that currently in use: > > Current: > [](https://github.com/apache/commons-build-plugin/actions) > > Generated: > [](https://github.com/apache/commons-build-plugin/actions/workflows/maven.yml) > > Using the badge URL shows a green badge. Thanks, very useful. Did not know that is where they came from. > Another working version from Statistics is the same E.g.: > > https://github.com/apache/commons-statistics/actions/workflows/maven.yml/badge.svg > > So try a badge update with a different URL. It seems the build-plugin template is wrong, and needs to be fixed. > Alex > > > > On Fri, 20 Oct 2023 at 16:03, sebb wrote: > > > > The build-plugin readme shows a failing Java CI build: > > > > https://github.com/apache/commons-build-plugin > > > > However the build has been working for ages: > > > > https://github.com/apache/commons-build-plugin/actions/workflows/maven.yml > > > > Any idea how this can be fixed? > > > > Not much point having the badge if it is stale. > > > > Sebb > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > > For additional commands, e-mail: dev-h...@commons.apache.org > > > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [lang] RandomStringUtilsTest.testRandomStringUtilsHomog fails a lot
I opened a PR after changing the expected failure probability to 1e-5. I had an old version of the GH build file when I estimated it used 4 runs. The latest CI runs 4 JDKs on 3 platforms plus CodeQL and coverage. So this is 14 runs. We should see failures with a probability of: (1 - (1 - 1e-5)**14) = 0.0001399, or approximately 1 in 7143. Compare that to previously: (1 - (1 - 1e-3)**14) = 0.01391, or approximately 1 in 71.89. If this is still a problem moving forward then we can replace with a fixed random number generator calling the underlying method and test coverage of the original method by other means. Alex On Fri, 20 Oct 2023 at 20:16, Gary D. Gregory wrote: > > Hi Alex, > > I'd prefer if you could give a shot at adjusting this test when you can take > the time. > > TY, > Gary > > On 2023/10/20 18:17:35 Alex Herbert wrote: > > On Fri, 20 Oct 2023 at 18:55, Alex Herbert wrote: > > > > > > The chi-square critical value (13.82) is correct: > > > > > > >>> from scipy.stats import chi2 > > > >>> chi2(2).isf(0.001) > > > 13.815510557964274 > > > > > > The test seems to fail with the expected frequency when run locally. I > > > annotated with: > > > > > > @RepeatedTest(value = 10) > > > > > > I observe 93 failures (just under 1 in 1000). So it is strange this > > > fails a lot on the GH CI build. > > > > > > We could just use a fixed Random argument to the call that is > > > ultimately performing the random string generation: > > > > > > random(count, 0, chars.length, false, false, chars, random()); > > > > > > Switch the test to: > > > > > > Random rng = new Random(0xdeadbeef) > > > > > > gen = RandomStringUtils.random(6, 0, 3, false, false, chars, rng); > > > > > > You will see a drop in coverage by not exercising the public API. > > > > > > The alternative is to change the chi-square critical value: > > > > > > 1 in 10,000: 18.420680743952364 > > > 1 in 100,000: 23.025850929940457 > > > 1 in 1,000,000: 27.631021115928547 > > > > > > Alex > > > > Also note that although the test fails 1 in 1000 times, we run 4 > > builds in CI (coverage + 3 JDKS). So we expect to see failure with a > > p-value of: > > > > 1 - (1 - 0.001)^4 = 0.00399 > > > > This is the probability of not seeing a failure subtracted from 1. It > > is approximately 1 in 250. > > > > I did check the computation of the chi-square statistic and AFAIK it is > > correct. > > > > My suggestion for a first change is to bump the critical value to the > > level for 1 in 100,000. We should then see failures every 25,000 GH > > builds. If the frequency is more than that then we have to assume that > > the ThreadLocalRandom instance is not uniformly sampling from the set > > of size 3. I find this unlikely as the underlying algorithm for > > ThreadLocalRandom is good [1]. > > > > Alex > > > > [1] IIRC it passes the Test U01 BigCrush test for random generators > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > > For additional commands, e-mail: dev-h...@commons.apache.org > > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [JEXL] intended thread safety of JexlFeatures and JexlPermissions
On Fri, 20 Oct 2023 at 19:09, Henri Biestro wrote: > > > JexlPermissions concrete classes are but since this is an interface, anyone > could create a non-thread safe instance and use it. The same way a > JexlFeatures could be corrupted by being constructed with a non-thread safe > namespace predicate (making side-effects etc). > And for JexlFeatures, using a concurrent set for reserved names (although a > strange idea) could be a valid usage (counting the number of times a var name > is declared for instance). > > Back to your question, both of them are *expected* to be thread-safe if you > want to use the same JexlEngine and JexlScript across threads. There is no > way to ensure this and I suppose altering documentation would not ease your > worry. JexlFeatures is not thread-safe, but AFAICT JexlEngine does not need JexlFeatures to be thread-safe, because it takes its own copy which is stored in a final field as part of construction. That should ensure the mutable fields are published safely. Of course if changes are made subsequently to the copy, that would change matters. If JexlFeatures were immutable there would be a lot less to be concerned about. Sebb > > On 2023/10/20 16:26:30 sebb wrote: > > Are instances of the classes JexlFeatures and JexlPermissions intended > > to be thread-safe? > > > > I have seen mention that they can be shared between threads. > > > > However JexlFeatures is does not appear to be thread-safe (has mutable > > fields which are not safely published). Not looked at JexlPermissions > > in detail. > > > > Sebb > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > > For additional commands, e-mail: dev-h...@commons.apache.org > > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [lang] RandomStringUtilsTest.testRandomStringUtilsHomog fails a lot
Hi Alex, I'd prefer if you could give a shot at adjusting this test when you can take the time. TY, Gary On 2023/10/20 18:17:35 Alex Herbert wrote: > On Fri, 20 Oct 2023 at 18:55, Alex Herbert wrote: > > > > The chi-square critical value (13.82) is correct: > > > > >>> from scipy.stats import chi2 > > >>> chi2(2).isf(0.001) > > 13.815510557964274 > > > > The test seems to fail with the expected frequency when run locally. I > > annotated with: > > > > @RepeatedTest(value = 10) > > > > I observe 93 failures (just under 1 in 1000). So it is strange this > > fails a lot on the GH CI build. > > > > We could just use a fixed Random argument to the call that is > > ultimately performing the random string generation: > > > > random(count, 0, chars.length, false, false, chars, random()); > > > > Switch the test to: > > > > Random rng = new Random(0xdeadbeef) > > > > gen = RandomStringUtils.random(6, 0, 3, false, false, chars, rng); > > > > You will see a drop in coverage by not exercising the public API. > > > > The alternative is to change the chi-square critical value: > > > > 1 in 10,000: 18.420680743952364 > > 1 in 100,000: 23.025850929940457 > > 1 in 1,000,000: 27.631021115928547 > > > > Alex > > Also note that although the test fails 1 in 1000 times, we run 4 > builds in CI (coverage + 3 JDKS). So we expect to see failure with a > p-value of: > > 1 - (1 - 0.001)^4 = 0.00399 > > This is the probability of not seeing a failure subtracted from 1. It > is approximately 1 in 250. > > I did check the computation of the chi-square statistic and AFAIK it is > correct. > > My suggestion for a first change is to bump the critical value to the > level for 1 in 100,000. We should then see failures every 25,000 GH > builds. If the frequency is more than that then we have to assume that > the ThreadLocalRandom instance is not uniformly sampling from the set > of size 3. I find this unlikely as the underlying algorithm for > ThreadLocalRandom is good [1]. > > Alex > > [1] IIRC it passes the Test U01 BigCrush test for random generators > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > > - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [lang] RandomStringUtilsTest.testRandomStringUtilsHomog fails a lot
On Fri, 20 Oct 2023 at 18:55, Alex Herbert wrote: > > The chi-square critical value (13.82) is correct: > > >>> from scipy.stats import chi2 > >>> chi2(2).isf(0.001) > 13.815510557964274 > > The test seems to fail with the expected frequency when run locally. I > annotated with: > > @RepeatedTest(value = 10) > > I observe 93 failures (just under 1 in 1000). So it is strange this > fails a lot on the GH CI build. > > We could just use a fixed Random argument to the call that is > ultimately performing the random string generation: > > random(count, 0, chars.length, false, false, chars, random()); > > Switch the test to: > > Random rng = new Random(0xdeadbeef) > > gen = RandomStringUtils.random(6, 0, 3, false, false, chars, rng); > > You will see a drop in coverage by not exercising the public API. > > The alternative is to change the chi-square critical value: > > 1 in 10,000: 18.420680743952364 > 1 in 100,000: 23.025850929940457 > 1 in 1,000,000: 27.631021115928547 > > Alex Also note that although the test fails 1 in 1000 times, we run 4 builds in CI (coverage + 3 JDKS). So we expect to see failure with a p-value of: 1 - (1 - 0.001)^4 = 0.00399 This is the probability of not seeing a failure subtracted from 1. It is approximately 1 in 250. I did check the computation of the chi-square statistic and AFAIK it is correct. My suggestion for a first change is to bump the critical value to the level for 1 in 100,000. We should then see failures every 25,000 GH builds. If the frequency is more than that then we have to assume that the ThreadLocalRandom instance is not uniformly sampling from the set of size 3. I find this unlikely as the underlying algorithm for ThreadLocalRandom is good [1]. Alex [1] IIRC it passes the Test U01 BigCrush test for random generators - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [JEXL] intended thread safety of JexlFeatures and JexlPermissions
JexlPermissions concrete classes are but since this is an interface, anyone could create a non-thread safe instance and use it. The same way a JexlFeatures could be corrupted by being constructed with a non-thread safe namespace predicate (making side-effects etc). And for JexlFeatures, using a concurrent set for reserved names (although a strange idea) could be a valid usage (counting the number of times a var name is declared for instance). Back to your question, both of them are *expected* to be thread-safe if you want to use the same JexlEngine and JexlScript across threads. There is no way to ensure this and I suppose altering documentation would not ease your worry. On 2023/10/20 16:26:30 sebb wrote: > Are instances of the classes JexlFeatures and JexlPermissions intended > to be thread-safe? > > I have seen mention that they can be shared between threads. > > However JexlFeatures is does not appear to be thread-safe (has mutable > fields which are not safely published). Not looked at JexlPermissions > in detail. > > Sebb > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > > - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [lang] RandomStringUtilsTest.testRandomStringUtilsHomog fails a lot
The chi-square critical value (13.82) is correct: >>> from scipy.stats import chi2 >>> chi2(2).isf(0.001) 13.815510557964274 The test seems to fail with the expected frequency when run locally. I annotated with: @RepeatedTest(value = 10) I observe 93 failures (just under 1 in 1000). So it is strange this fails a lot on the GH CI build. We could just use a fixed Random argument to the call that is ultimately performing the random string generation: random(count, 0, chars.length, false, false, chars, random()); Switch the test to: Random rng = new Random(0xdeadbeef) gen = RandomStringUtils.random(6, 0, 3, false, false, chars, rng); You will see a drop in coverage by not exercising the public API. The alternative is to change the chi-square critical value: 1 in 10,000: 18.420680743952364 1 in 100,000: 23.025850929940457 1 in 1,000,000: 27.631021115928547 Alex On Fri, 20 Oct 2023 at 18:44, Elliotte Rusty Harold wrote: > > It's possible the chi square test is miscalculated. Perhaps some stats > expert can check that. It's also possible the chi square test isn't > the right one to use here. Again, consult a stats expert. > > It's also very possible that the randomness is not nearly as random as > it's supposed to be. That's incredibly common, and that might be > noticeable given the very short three-letter character set [a, b, c] > being picked from. > > On Fri, Oct 20, 2023 at 1:31 PM Gary D. Gregory wrote: > > > > Despite the failure comment: > > > > RandomStringUtilsTest.testRandomStringUtilsHomog:474 test homogeneity -- > > will fail about 1 in 1000 times ==> expected: but was: > > > > This test fails a LOT more than once every 1000 times, based on how many > > GitHub builds I need to restart every week. > > > > What can be done to make this test more resilient? > > > > TY! > > Gary > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > > For additional commands, e-mail: dev-h...@commons.apache.org > > > > > -- > Elliotte Rusty Harold > elh...@ibiblio.org > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [lang] RandomStringUtilsTest.testRandomStringUtilsHomog fails a lot
It's possible the chi square test is miscalculated. Perhaps some stats expert can check that. It's also possible the chi square test isn't the right one to use here. Again, consult a stats expert. It's also very possible that the randomness is not nearly as random as it's supposed to be. That's incredibly common, and that might be noticeable given the very short three-letter character set [a, b, c] being picked from. On Fri, Oct 20, 2023 at 1:31 PM Gary D. Gregory wrote: > > Despite the failure comment: > > RandomStringUtilsTest.testRandomStringUtilsHomog:474 test homogeneity -- will > fail about 1 in 1000 times ==> expected: but was: > > This test fails a LOT more than once every 1000 times, based on how many > GitHub builds I need to restart every week. > > What can be done to make this test more resilient? > > TY! > Gary > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > -- Elliotte Rusty Harold elh...@ibiblio.org - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
[lang] RandomStringUtilsTest.testRandomStringUtilsHomog fails a lot
Despite the failure comment: RandomStringUtilsTest.testRandomStringUtilsHomog:474 test homogeneity -- will fail about 1 in 1000 times ==> expected: but was: This test fails a LOT more than once every 1000 times, based on how many GitHub builds I need to restart every week. What can be done to make this test more resilient? TY! Gary - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [ALL] Java CI badge does not agree with workflow status
Navigating to the Java CI workflow you can generate a badge using the ... menu in the top right next to the search filter. This creates a badge with a different URL to that currently in use: Current: [](https://github.com/apache/commons-build-plugin/actions) Generated: [](https://github.com/apache/commons-build-plugin/actions/workflows/maven.yml) Using the badge URL shows a green badge. Another working version from Statistics is the same E.g.: https://github.com/apache/commons-statistics/actions/workflows/maven.yml/badge.svg So try a badge update with a different URL. Alex On Fri, 20 Oct 2023 at 16:03, sebb wrote: > > The build-plugin readme shows a failing Java CI build: > > https://github.com/apache/commons-build-plugin > > However the build has been working for ages: > > https://github.com/apache/commons-build-plugin/actions/workflows/maven.yml > > Any idea how this can be fixed? > > Not much point having the badge if it is stale. > > Sebb > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [JEXL] intended thread safety of JexlFeatures and JexlPermissions
In general, I assume nothing is thread-safe unless documented as such. Gary On Fri, Oct 20, 2023, 12:27 PM sebb wrote: > Are instances of the classes JexlFeatures and JexlPermissions intended > to be thread-safe? > > I have seen mention that they can be shared between threads. > > However JexlFeatures is does not appear to be thread-safe (has mutable > fields which are not safely published). Not looked at JexlPermissions > in detail. > > Sebb > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > >
[JEXL] intended thread safety of JexlFeatures and JexlPermissions
Are instances of the classes JexlFeatures and JexlPermissions intended to be thread-safe? I have seen mention that they can be shared between threads. However JexlFeatures is does not appear to be thread-safe (has mutable fields which are not safely published). Not looked at JexlPermissions in detail. Sebb - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
[ALL] Java CI badge does not agree with workflow status
The build-plugin readme shows a failing Java CI build: https://github.com/apache/commons-build-plugin However the build has been working for ages: https://github.com/apache/commons-build-plugin/actions/workflows/maven.yml Any idea how this can be fixed? Not much point having the badge if it is stale. Sebb - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
[BeanUtils] Java 21 failure converting time and timestamps
It seems that on top of our double trouble with [text], we have run into another conversion issue with Java 21, this time in [BeanUtils]. I'd like help figuring this one out: Error: org.apache.commons.beanutils2.sql.converters.SqlTimeConverterTestCase.testLocale -- Time elapsed: 0.009 s <<< FAILURE! junit.framework.AssertionFailedError: Converting 'java.lang.String' value '3:06 pm' threw org.apache.commons.beanutils2.ConversionException: Error converting 'String' to 'java.sql.Time' using pattern 'h:mm a', localized pattern 'h:mm a', errorIndex 4, calendar type GregorianCalendar, this org.apache.commons.beanutils2.sql.converters.SqlTimeConverter[UseDefault=false, UseLocaleFormat=true] at junit.framework.Assert.fail(Assert.java:57) at junit.framework.TestCase.fail(TestCase.java:223) at org.apache.commons.beanutils2.converters.AbstractDateConverterTest.validConversion(AbstractDateConverterTest.java:572) at org.apache.commons.beanutils2.sql.converters.SqlTimeConverterTestCase.testLocale(SqlTimeConverterTestCase.java:118) [INFO] Running org.apache.commons.beanutils2.sql.converters.SqlTimestampConverterTestCase Error: Tests run: 11, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.016 s <<< FAILURE! -- in org.apache.commons.beanutils2.sql.converters.SqlTimestampConverterTestCase Error: org.apache.commons.beanutils2.sql.converters.SqlTimestampConverterTestCase.testLocale -- Time elapsed: 0.010 s <<< FAILURE! junit.framework.AssertionFailedError: Converting 'java.lang.String' value '3/21/06, 3:06 PM' threw org.apache.commons.beanutils2.ConversionException: Error converting 'String' to 'java.sql.Timestamp' using pattern 'M/d/yy, h:mm a', localized pattern 'M/d/yy, h:mm a', errorIndex 13, calendar type GregorianCalendar, this org.apache.commons.beanutils2.sql.converters.SqlTimestampConverter[UseDefault=false, UseLocaleFormat=true] at junit.framework.Assert.fail(Assert.java:57) at junit.framework.TestCase.fail(TestCase.java:223) at org.apache.commons.beanutils2.converters.AbstractDateConverterTest.validConversion(AbstractDateConverterTest.java:572) at org.apache.commons.beanutils2.sql.converters.SqlTimestampConverterTestCase.testLocale(SqlTimestampConverterTestCase.java:138) TY! Gary - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [Text] Java 21 failure with a double
Thank you Alex!
The PR is now merged.
Gary
On Fri, Oct 20, 2023, 4:09 AM Alex Herbert wrote:
> I've implemented a fix for this in PR 467 [1]. Builds now pass on JDK 21.
>
> This does not change the DoubleFormat class. It adds javadoc to state
> that the formatting is dependent on Double.toString.
>
> The tests have been updated to use a different range for the random
> doubles so the tests across formats should be testing roughly the same
> precision. They should not test the full precision output (i.e. 17
> digits).
>
> Note that this change identified another mismatch between the
> reference DecimalFormat and the DoubleFormat when a zero trails the
> decimal point. The DecimalFormat class can omit this trailing zero and
> the decimal point:
>
> new DecimalFormat("##0.0##E0").format(1.129e-4) => 113E-6(not
> 113.0E-6)
>
> The inelegant solution I used was to remove .0 from the DecimalFormat
> output and recheck the string is a match. This allows the above case
> to pass. There are no other cases in the 1000 random numbers that hit
> this mismatch case.
>
> A new test has been added for full precision output. The test requires
> that the output format can be parsed back to the same number, i.e. no
> loss of precision occurs when formatting. This seemed simpler than
> writing custom code to check against the digits output from
> Double.toString to ensure all digits are present.
>
> Alex
>
> [1] https://github.com/apache/commons-text/pull/467
>
> On Mon, 16 Oct 2023 at 18:37, Alex Herbert
> wrote:
> >
> > TLDR; The Double.toString(double) value is different for
> > -9.354004711977437E17 on JDK 21 and earlier JDKs:
> >
> > JDK 21: -9.354004711977437E17
> > JDK 17: -9.3540047119774374E17
> >
> > The DoubleFormat class is built upon Double.toString. So the test
> > fails due to this change.
> >
> > ---
> >
> > More details:
> >
> > On JDK 21 Double.toString is dropping the last digit (a 4) as it is
> > not required to uniquely represent the double from the next double up
> > or down. Note the javadoc for toString states: "This decimal is
> > (almost always) the shortest one that rounds to m according to the
> > round to nearest rounding policy of IEEE 754 floating-point
> > arithmetic." So this is not the closest decimal representation of the
> > double, just the closest required for rounding to the double.
> >
> > I do not think this is a bug. But we should document that the
> > DoubleFormat class is built upon the Double.toString representation of
> > a double. This string may not be the closest decimal representation of
> > the double. Thus final digit errors can be observed when using the
> > entire character output of Double.toString compared to other
> > formatting classes such as java.lang.DecimalFormat or the numerical
> > string representation provided by java.lang.BigDecimal (see examples
> > below).
> >
> > Q. How to fix the test?
> >
> > The DoubleFormatTest is using 1000 random double values with a base-2
> > exponent of -100 to 100. Then testing against a plain text format of
> > 0.00##. The exponent of 9.35e17 is 59. It is so large there are no
> > digits after the decimal point. So the test is checking if
> > DoubleFormat is accurate to 17 decimal digits of precision, which it
> > is not in this case. The other tests using the random numbers test the
> > formats:
> >
> > #,##0.0##
> > 0.0##E0
> > ##0.0##E0
> >
> > So the scientific and engineering tests are only checking 4 and 6
> > decimal digits. But the plain formats are checking up to 17 digits due
> > to the large exponent of the random numbers. This does not seem very
> > fair.
> >
> > Fix 1: A simple fix would be to reduce the exponent range for random
> > numbers so that the plain numbers should always have at least 4 digits
> > after the decimal point.
> >
> > However, changing the test like this would reduce its power. It would
> > not have alerted us to this failure.
> >
> > Fix 2: An alternative would be to change the test assertion to perform
> > the current check, and if it fails perform a test that all digits from
> > Double.toString are present in the formatted string in the same order
> > (i.e. the class has used the entire output from Double.toString and so
> > is at its limit). This is non-trivial as the double value is tested in
> > all the installed locales which may change the digits and other
> > formatting characters. The assertion would have to create a reverse
> > parsing method based on the locale.
> >
> > Note: I did try using the DecimalFormat used to specify formatting the
> > expected string to parse the string with DecimalFormat.parse. It
> > worked on the original root locale but it failed on locale "he"; this
> > may be another bug in DoubleFormat since the locale can parse its own
> > output:
> >
> > jshell> var df = new java.text.DecimalFormat("0.0##",
> > java.text.DecimalFormatSymbols.getInstance(Locale.forLanguageTag("he")))
> > jshell> x
> > x ==> -9.3540047119774374E17
> > jshell>
JDK 21 Is Now GA, a New VS Code Extension, and an Annotation Processing Heads-up
Greetings!
JDK 21 has been released (General Availability) on September 19th as planned.
You can find "The Arrival of Java 21" announcement here [1], and some
additional Java 21 materials in the "Topics of Interest" section below. On
behalf of the entire Java team, let me send our thanks to all of you. Through
your active participation in this program, you are helping shape the Java
platform!
Needless to say, that Java 21 is an important release, so may I ask you to send
me a brief email with the Java 21 support status of your project(s): Already
supported - Plan to support short-term - Don't plan to support short-term ?
And now that JDK 21 is out, let's shift our attention to JDK 22 which will
enter the Rampdown Phase in less than 50 days on December 7 [2].
I want to conclude this update by briefly mentioning three different
initiatives to are relevant to this group as they are, in their own way and at
various levels, contributing to adopt newer Java releases more rapidly: the
Class-File API, Oracle's Java Platform extension for VS Code, and the Java
Playground.
### The Class-File API
The Class-File API is a new standard API for parsing, generating, and
transforming Java class files. One of its unique aspects is that it will
co-evolve with the class-file format, which overtime will greatly reduce the
friction of implementing new class-file features. With the fast-paced evolution
of the Java platform, this was much-needed. This API should soon be previewed
and as it matures, we expect the JDK to switch from using various custom
class-file libraries to this standard API. We also expect that overtime
frameworks relying on bytecode manipulation will also benefit from using this
new JDK class-file library. For more information, please check this recent
Newscast [3] for an overview, Brian Goetz's JVMLS session [4] for more details
and design considerations, and JEP 457: Class-File API (Preview) [5] for the
technical details.
### Oracle's Java Platform extension for Visual Studio Code
Oracle has just announced [6] a new Visual Studio Code extension for Java
developers. Unlike other VS Code extensions, this new extension is using under
the hood the `javac` compiler for code editing and compilation, and OpenJDK's
debugger interface for debugging. This enables us to offer VS Code IDE support
for new JDK features as soon as they are introduced, even during JDK Early
Access phases. To this effect, this VS Code Extension will support the current
JDK releases as well as the next upcoming JDK version. For more information,
please check the announcement [6].
### The Java Playground
The Java Playground [7] is an online sandbox that helps testing and exploring
new Java language features. No setup required, just type your Java snippet in
your browser and run it! Right now, the Playground is using Java 21 with
Preview Features enabled, and it will switch to a new Java version as soon as
there is a new Java language features integrated in OpenJDK Early-Access
builds. The Playground is focusing mostly on Project Amber and is certainly not
mean to be some sort of a lightweight online-IDE, it is instead a learning tool
to play with new Java language feature shortly after they have been integrated
into the platform.
[1] https://inside.java/2023/09/19/the-arrival-of-java-21/
[2] https://mail.openjdk.org/pipermail/jdk-dev/2023-September/008269.html
[3] https://www.youtube.com/watch?v=bQ2Rwpyj_Ks
[4] https://www.youtube.com/watch?v=pcg-E_qyMOI
[5] https://openjdk.org/jeps/457
[6] https://inside.java/2023/10/18/announcing-vscode-extension/
[7] https://dev.java/playground
## Heads-Up - JDK 22: Implicit Annotation Processing Behavior Change
As discussed in the July 2023 Quality Outreach update [8], starting in JDK 21
javac emits a note if _implicit_ annotation processing is being used, that is,
if one or more annotation processors are found and run from the class path when
no explicit annotation processing configuration options are used.
The note is reported since, quoting from the note text: "A future release of
javac may disable annotation processing unless at least one processor is
specified by name (-processor), or a search path is specified
(--processor-path, --processor-module-path), or annotation processing is
enabled explicitly (-proc:only, -proc:full)."
That future version of javac has arrived in JDK 22 b19+ with JDK-8306819
("Consider disabling the compiler's default active annotation processing"). In
the situation where a note was emitted in JDK 21, in JDK 22 no note is emitted,
and annotation processors are *not* run. To restore the previous behavior with
respect to running annotation processors, add the '-proc:full' javac option.
Feedback on the annotation processing policy change can be sent to compiler-dev
[9].
[8] https://mail.openjdk.org/pipermail/quality-discuss/2023-July/001122.html
[9] https://mail.openjdk.org/mailman/listinfo/compiler-dev
## JDK
Re: [Text] Java 21 failure with a double
I've implemented a fix for this in PR 467 [1]. Builds now pass on JDK 21.
This does not change the DoubleFormat class. It adds javadoc to state
that the formatting is dependent on Double.toString.
The tests have been updated to use a different range for the random
doubles so the tests across formats should be testing roughly the same
precision. They should not test the full precision output (i.e. 17
digits).
Note that this change identified another mismatch between the
reference DecimalFormat and the DoubleFormat when a zero trails the
decimal point. The DecimalFormat class can omit this trailing zero and
the decimal point:
new DecimalFormat("##0.0##E0").format(1.129e-4) => 113E-6(not 113.0E-6)
The inelegant solution I used was to remove .0 from the DecimalFormat
output and recheck the string is a match. This allows the above case
to pass. There are no other cases in the 1000 random numbers that hit
this mismatch case.
A new test has been added for full precision output. The test requires
that the output format can be parsed back to the same number, i.e. no
loss of precision occurs when formatting. This seemed simpler than
writing custom code to check against the digits output from
Double.toString to ensure all digits are present.
Alex
[1] https://github.com/apache/commons-text/pull/467
On Mon, 16 Oct 2023 at 18:37, Alex Herbert wrote:
>
> TLDR; The Double.toString(double) value is different for
> -9.354004711977437E17 on JDK 21 and earlier JDKs:
>
> JDK 21: -9.354004711977437E17
> JDK 17: -9.3540047119774374E17
>
> The DoubleFormat class is built upon Double.toString. So the test
> fails due to this change.
>
> ---
>
> More details:
>
> On JDK 21 Double.toString is dropping the last digit (a 4) as it is
> not required to uniquely represent the double from the next double up
> or down. Note the javadoc for toString states: "This decimal is
> (almost always) the shortest one that rounds to m according to the
> round to nearest rounding policy of IEEE 754 floating-point
> arithmetic." So this is not the closest decimal representation of the
> double, just the closest required for rounding to the double.
>
> I do not think this is a bug. But we should document that the
> DoubleFormat class is built upon the Double.toString representation of
> a double. This string may not be the closest decimal representation of
> the double. Thus final digit errors can be observed when using the
> entire character output of Double.toString compared to other
> formatting classes such as java.lang.DecimalFormat or the numerical
> string representation provided by java.lang.BigDecimal (see examples
> below).
>
> Q. How to fix the test?
>
> The DoubleFormatTest is using 1000 random double values with a base-2
> exponent of -100 to 100. Then testing against a plain text format of
> 0.00##. The exponent of 9.35e17 is 59. It is so large there are no
> digits after the decimal point. So the test is checking if
> DoubleFormat is accurate to 17 decimal digits of precision, which it
> is not in this case. The other tests using the random numbers test the
> formats:
>
> #,##0.0##
> 0.0##E0
> ##0.0##E0
>
> So the scientific and engineering tests are only checking 4 and 6
> decimal digits. But the plain formats are checking up to 17 digits due
> to the large exponent of the random numbers. This does not seem very
> fair.
>
> Fix 1: A simple fix would be to reduce the exponent range for random
> numbers so that the plain numbers should always have at least 4 digits
> after the decimal point.
>
> However, changing the test like this would reduce its power. It would
> not have alerted us to this failure.
>
> Fix 2: An alternative would be to change the test assertion to perform
> the current check, and if it fails perform a test that all digits from
> Double.toString are present in the formatted string in the same order
> (i.e. the class has used the entire output from Double.toString and so
> is at its limit). This is non-trivial as the double value is tested in
> all the installed locales which may change the digits and other
> formatting characters. The assertion would have to create a reverse
> parsing method based on the locale.
>
> Note: I did try using the DecimalFormat used to specify formatting the
> expected string to parse the string with DecimalFormat.parse. It
> worked on the original root locale but it failed on locale "he"; this
> may be another bug in DoubleFormat since the locale can parse its own
> output:
>
> jshell> var df = new java.text.DecimalFormat("0.0##",
> java.text.DecimalFormatSymbols.getInstance(Locale.forLanguageTag("he")))
> jshell> x
> x ==> -9.3540047119774374E17
> jshell> df.parse(df.format(x)).doubleValue()
> $32 ==> -9.3540047119774374E17
>
> Fix 3: Implement fix 1, plus add a test for full length precision
> using only the root locale. This can quickly test the output is at the
> limit by checking the string creates the original input double using
> Double.parseDouble, i.e. the
