[
https://issues.apache.org/jira/browse/KAFKA-14682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton updated KAFKA-14682:
----------------------------------
Description:
We've started using [strict
stubbing|https://javadoc.io/static/org.mockito/mockito-core/4.6.1/org/mockito/junit/MockitoJUnitRunner.StrictStubs.html]
for unit tests written with Mockito, which is supposed to automatically fail
tests when they set up mock expectations that go unused.
However, these failures are not reported during Jenkins builds, even if they
are reported when building/testing locally.
In at least one case, this difference appears to be because our [Jenkins
build|https://github.com/apache/kafka/blob/6d11261d5deaca300e273bebe309f9e4f814f815/Jenkinsfile#L32-L35]
uses the custom {{unitTest}} and {{integrationTest}} tasks defined in the
project's [Gradle build
file|https://github.com/apache/kafka/blob/6d11261d5deaca300e273bebe309f9e4f814f815/build.gradle#L452-L543],
instead of the {{test}} task. Some IDEs (such as IntelliJ) may use the latter
instead of the former, which can cause tests to fail due to unnecessary
stubbings when being run in that IDE but not when being built on Jenkins.
It's possible that, because the custom test tasks filter out some tests from
running, Mockito does not check for unnecessary stubbings in order to avoid
incorrectly failing tests that set up mocks in, e.g., a {{@BeforeEach}} method.
This exact behavior has been reported elsewhere as a [Gradle
issue|https://github.com/gradle/gradle/issues/10694]; based on [discussion on
that
thread|https://github.com/gradle/gradle/issues/10694#issuecomment-1374911274],
it appears this is a known and somewhat-intentional limitation of Mockito:
{quote}I spent some time trying to solve this and eventually I stumbled upon
this piece in Mockito's JUnit runner:
[https://github.com/mockito/mockito/blob/main/src/main/java/org/mockito/internal/runners/StrictRunner.java#L47-L53]
// only report when:
// 1. if all tests from given test have ran (filter requested is false)
// Otherwise we would report unnecessary stubs even if the user runs just
single test
// from the class
// 2. tests are successful (we don't want to add an extra failure on top of any
existing
// failure, to avoid confusion)
(1) suggests that skipping unused stub validation is the intended behavior when
the user filters a single test from the class. However, this behavior applies
to any type of filter.
And Gradle indeed applies a {{CategoryFilter}} if categories are configured:
[https://github.com/rieske/gradle/blob/e82029abb559d620fdfecb4708a95c6313af625c/subprojects/testing-jvm-infrastructure/src/main/java/org/gradle/api/internal/tasks/testing/junit/JUnitTestClassExecutor.java#L70-L96]
Which then causes Mockito to not validate unused stubs.
{quote}
was:
We've started using [strict
stubbing|https://javadoc.io/static/org.mockito/mockito-core/4.6.1/org/mockito/junit/MockitoJUnitRunner.StrictStubs.html]
for unit tests written with Mockito, which is supposed to automatically fail
tests when they set up mock expectations that go unused.
However, these failures are not reported during Jenkins builds, even if they
are reported when building/testing locally.
In at least one case, this difference appears to be because our [Jenkins
build|https://github.com/apache/kafka/blob/6d11261d5deaca300e273bebe309f9e4f814f815/Jenkinsfile#L32-L35]
uses the custom {{unitTest}} and {{integrationTest}} tasks defined in the
project's [Gradle build
file|https://github.com/apache/kafka/blob/6d11261d5deaca300e273bebe309f9e4f814f815/build.gradle#L452-L543],
instead of the {{test}} task. Some IDEs (such as IntelliJ) may use the latter
instead of the former, which can cause tests to fail due to unnecessary
stubbings when being run in that IDE but not when being built on Jenkins.
It's possible that, because the custom test tasks filter out some tests from
running, Mockito does not check for unnecessary stubbings in order to avoid
incorrectly failing tests that set up mocks in, e.g., a {{@BeforeEach}} method.
This exact behavior has been reported elsewhere as a [Gradle
issue|https://github.com/gradle/gradle/issues/10694]; based on [discussion on
that
thread|https://github.com/gradle/gradle/issues/10694#issuecomment-1374911274],
it appears this is a known and somewhat-intentional limitation of Mockito:
{quote}I spent some time trying to solve this and eventually I stumbled upon
this piece in Mockito's JUnit runner:
[https://github.com/mockito/mockito/blob/main/src/main/java/org/mockito/internal/runners/StrictRunner.java#L47-L53]
{{ }}
{{}}
{{ // only report when:
// 1. if all tests from given test have ran (filter requested is
false)
// Otherwise we would report unnecessary stubs even if the user
runs just single test
// from the class
// 2. tests are successful (we don't want to add an extra failure
on top of any existing
// failure, to avoid confusion)}}
(1) suggests that skipping unused stub validation is the intended behavior when
the user filters a single test from the class. However, this behavior applies
to any type of filter.
And Gradle indeed applies a {{CategoryFilter}} if categories are configured:
[https://github.com/rieske/gradle/blob/e82029abb559d620fdfecb4708a95c6313af625c/subprojects/testing-jvm-infrastructure/src/main/java/org/gradle/api/internal/tasks/testing/junit/JUnitTestClassExecutor.java#L70-L96]
Which then causes Mockito to not validate unused stubs.
{quote}
> Unused stubbings are not reported by Mockito during CI builds
> -------------------------------------------------------------
>
> Key: KAFKA-14682
> URL: https://issues.apache.org/jira/browse/KAFKA-14682
> Project: Kafka
> Issue Type: Test
> Components: unit tests
> Reporter: Chris Egerton
> Assignee: Chris Egerton
> Priority: Major
>
> We've started using [strict
> stubbing|https://javadoc.io/static/org.mockito/mockito-core/4.6.1/org/mockito/junit/MockitoJUnitRunner.StrictStubs.html]
> for unit tests written with Mockito, which is supposed to automatically fail
> tests when they set up mock expectations that go unused.
> However, these failures are not reported during Jenkins builds, even if they
> are reported when building/testing locally.
> In at least one case, this difference appears to be because our [Jenkins
> build|https://github.com/apache/kafka/blob/6d11261d5deaca300e273bebe309f9e4f814f815/Jenkinsfile#L32-L35]
> uses the custom {{unitTest}} and {{integrationTest}} tasks defined in the
> project's [Gradle build
> file|https://github.com/apache/kafka/blob/6d11261d5deaca300e273bebe309f9e4f814f815/build.gradle#L452-L543],
> instead of the {{test}} task. Some IDEs (such as IntelliJ) may use the
> latter instead of the former, which can cause tests to fail due to
> unnecessary stubbings when being run in that IDE but not when being built on
> Jenkins.
> It's possible that, because the custom test tasks filter out some tests from
> running, Mockito does not check for unnecessary stubbings in order to avoid
> incorrectly failing tests that set up mocks in, e.g., a {{@BeforeEach}}
> method.
>
> This exact behavior has been reported elsewhere as a [Gradle
> issue|https://github.com/gradle/gradle/issues/10694]; based on [discussion on
> that
> thread|https://github.com/gradle/gradle/issues/10694#issuecomment-1374911274],
> it appears this is a known and somewhat-intentional limitation of Mockito:
> {quote}I spent some time trying to solve this and eventually I stumbled upon
> this piece in Mockito's JUnit runner:
> [https://github.com/mockito/mockito/blob/main/src/main/java/org/mockito/internal/runners/StrictRunner.java#L47-L53]
> // only report when:
> // 1. if all tests from given test have ran (filter requested is false)
> // Otherwise we would report unnecessary stubs even if the user runs just
> single test
> // from the class
> // 2. tests are successful (we don't want to add an extra failure on top of
> any existing
> // failure, to avoid confusion)
>
> (1) suggests that skipping unused stub validation is the intended behavior
> when the user filters a single test from the class. However, this behavior
> applies to any type of filter.
> And Gradle indeed applies a {{CategoryFilter}} if categories are configured:
> [https://github.com/rieske/gradle/blob/e82029abb559d620fdfecb4708a95c6313af625c/subprojects/testing-jvm-infrastructure/src/main/java/org/gradle/api/internal/tasks/testing/junit/JUnitTestClassExecutor.java#L70-L96]
> Which then causes Mockito to not validate unused stubs.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
