Mohsen Dehghankar created SCB-2718:
--------------------------------------
Summary: Fix flaky test failure in
TestOperationGenerator#apiOperationThenResponse
Key: SCB-2718
URL: https://issues.apache.org/jira/browse/SCB-2718
Project: Apache ServiceComb
Issue Type: Bug
Reporter: Mohsen Dehghankar
h3. What is the purpose of this PR
- This PR fixes the flaky test
`org.apache.servicecomb.swagger.generator.core.TestOperationGenerator.apiOperationThenResponse`
- The mentioned test may fail or pass without changes made to the source code
when it is run in different JVMs due to the non-deterministic order of the
returned array from `getDeclaredAnnotations`. Because there is no assumption on
the order of the returned array from this method in [Java
documentation|https://docs.oracle.com/javase/8/docs/api/java/lang/reflect/AccessibleObject.html#getDeclaredAnnotations--].
h3. Why the test fails
- When processing the annotations of a method in
`AbstractOperationGenerator#scanMethodAnnotations()`, the result of
`method.getAnnotations()` is used which uses
`AccessibleObject.getDeclaredAnnotations()` which does not guarantee a
deterministic result order based on Java specifications.
- This results in processing annotations in arbitrary order, so the expected
assertions in both `TestOperationGenerator#apiOperationThenResponse` and
`TestOperationGenerator#responseThenApiOperation` might fail due to this change
in orders of annotations.
h3. Reproduce the test failure
- Run the test with 'NonDex' maven plugin. The command to recreate the flaky
test failure is `mvn edu.illinois:nondex-maven-plugin:2.1-SNAPSHOT:nondex -pl
swagger/swagger-generator/generator-core
-Dtest=TestOperationGenerator#apiOperationThenResponse`
- Fixing the flaky test now may prevent flaky test failures in future Java
versions.
h3. Expected Result
- The tests should run successfully when run with 'NonDex' maven with the same
command for recreating the flaky test.
h3. Actual Result
- We get the following error:
```
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.258 s
<<< FAILURE! - in
org.apache.servicecomb.swagger.generator.core.TestOperationGenerator
[ERROR]
org.apache.servicecomb.swagger.generator.core.TestOperationGenerator.apiOperationThenResponse
Time elapsed: 0.25 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <null> but was:
<io.swagger.models.properties.StringProperty@8f5d1e11>
at
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at
org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at org.junit.jupiter.api.AssertNull.failNotNull(AssertNull.java:50)
at org.junit.jupiter.api.AssertNull.assertNull(AssertNull.java:35)
at org.junit.jupiter.api.AssertNull.assertNull(AssertNull.java:30)
at org.junit.jupiter.api.Assertions.assertNull(Assertions.java:276)
at
org.apache.servicecomb.swagger.generator.core.TestOperationGenerator.apiOperationThenResponse(TestOperationGenerator.java:114)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
at
org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
at
org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
at
org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
at
org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
at
org.apache.maven.surefire.junitplatform.LazyLauncher.execute(LazyLauncher.java:55)
at
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:223)
at
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:175)
at
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:139)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:456)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:169)
at
org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:595)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:581)
```
h3. Fix
- First, we should sort the `method.getAnnotations()` result deterministically.
- Secondly, there should be no different in the `assertEqual` expressions
between these two tests: `apiOperationThenResponse` and
`responseThenApiOperation`
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
