I took a closer look at the build file and reasoning in the linked issue.
The second scenario I mentioned above is, as I suspected, a result of
incorrect wiring of Gradle tasks and dependencies. This has been a known
issue for years, so I'll put it aside for now.
The more important discussion is an abstract one - the responsibility of
the build system vs that of the test harness. The issue reasons that running
gradle :base:test --tests BooleanPropertyTest // runs the
"BooleanPropertyTest" in javafx.base
twice in a row with no changes should still rerun the test. I disagree.
Most of the written tests are deterministic and are not flaky (e.g.,
concurrency issues). Running a test for assertEquals(2+2, 4) more than once
is meaningless.
The issue then goes on to explain why tests should be rerun:
The most common reason is to ensure that a test you are running is stable
> on the platform you are testing on. Other reasons might include a test that
> has randomness (which is generally undesirable, but there are some limited
> cases where it might be used).
(Does 'BooleanPropertyTest' belong to these?) Testing the stability of a
test is not the responsibility of the build system, it's the responsibility
of the test harness. This is what JUnit annotations like @RepeatedTest and
@ParametrizedTest are for. The build system should be able to tell whether
a test met your expectations or not (I'm not using failed or passed
deliberately). If a test meets your expectations, it can be marked as
up-to-date as long as the code on which it depends hasn't changed. It is up
to the test harness to define what "met your expectations" means. @RepeatedTest
allows a test to pass even if it fails a given number of times
('failureThreshold'). If you just force reruns of the test in Gradle, you
displace the success condition because it's not part of your test design
anymore.
On the same token, I can ask why we need @ParametrizedTest? We can do it
with Gradle by re-running the test with an args array of the parameters and
seeing which of the runs fail. But I think it's obvious that this sort of
testing shouldn't be controlled by Gradle, it should be controlled by
JUnit. Gradle is only interested in the end result.
I wonder, if I put @RepeatedTest(1_000_000) on all the tests in 'base', can
any of them really sometimes fail and sometimes not? Can `HBoxTest` in
'graphics'? 'CheckBoxTest' in 'controls'? If I can know which problematic
tests prompted this solution then I can find out why the tests didn't rerun
when they should have. More often than not, it's a wiring issue.
In any case, I prepared a PR that wires FORCE_TESTS to the test task
configuration, with a default that preserves the current behavior. Will
submit it after I make sure it works as intended.
On Wed, Aug 5, 2026 at 10:24 PM Nir Lisker <[email protected]> wrote:
> My issue is that it runs tests that can't be affected by the changes you
> made. For example, a change in the media module will run the tests of base,
> graphics, and controls, even though these are not aware of media sources
> (save 'exports to' in module-info).
>
> I think that there are two scenarios that ended up being combined here.
> The one mentioned in the issue is rerunning tests that pass; this is valid
> in non-deterministic tests. The other one is what I wrote above where tests
> that can't be affected are rerun; this is never valid.
>
> I'll see if there's a way to separate these scenarios. Defaulting
> FORCE_TESTS to true and wiring it to the test task configuration can also
> work, maybe as a last resort.
>
> On Tue, Aug 4, 2026 at 5:48 PM Kevin Rushforth <[email protected]>
> wrote:
>
>> This was intentionally done via the following:
>>
>> JDK-8307076: gradle test should always run tests [1][2]
>>
>> I see that I forgot to remove the unused FORCE_TESTS property.
>>
>> I am very much in favor of always running tests, at least by default, as
>> I said in the bug and PR. I would oppose changing the default for those
>> same reasons. As I mentioned there, gradle's test harness is the only UI
>> test framework I've used where the "run tests" command doesn't
>> unconditionally run tests.
>>
>> If you want to retain the ability to run only if gradle thinks you need
>> to, then I wouldn't oppose the idea of qualifying it with "FORCE_TESTS"
>> as long as the default was changed to true, matching today's behavior.
>>
>> -- Kevin
>>
>> [1] https://bugs.openjdk.org/browse/JDK-8307076
>> [2] https://github.com/openjdk/jfx/pull/1120
>>
>> On 8/4/2026 4:24 AM, Nir Lisker wrote:
>> > Hi all,
>> >
>> > The Gradle build defines:
>> >
>> > defineProperty("FORCE_TESTS", "false")
>> > ext.IS_FORCE_TESTS = Boolean.parseBoolean(FORCE_TESTS);
>> >
>> > However, it's never used. Meanwhile, the allprojects configuration for
>> the
>> > 'test' task declares:
>> >
>> > // Always run tests
>> > outputs.upToDateWhen { false }
>> >
>> > It's not clear why all tests need to be run every time. I suspect that
>> the
>> > line should be
>> >
>> > outputs.upToDateWhen { IS_FORCE_TESTS }
>> >
>> > This can save a lot of time during development iterations as there's no
>> > reason to run unaffected tests.
>> >
>> > Are there any insights into this?
>> >
>> > - Nir
>>
>>
