LucasEby opened a new issue, #24968: URL: https://github.com/apache/pulsar/issues/24968
### Search before reporting - [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Read release policy - [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker. ### User environment Broker version: 4.2.0-SNAPSHOT Broker Operating system and hardware type: Linux x86_64 GNU/Linux Broker Java version: 17.0.16 ### Issue Description The below test would compares the generated documentation of CmdGenerateDocsTest#testGenerateDocs. The test assumes a deterministic order of the generated documentation but the order of the listed help and name options was not guaranteed. Due to two sources of nondeterminism hidden in the test and production code, the options could be listed in a different order. Picocli's CommandLine code uses [Class#getDeclaredFields](https://docs.oracle.com/javase/8/docs/api/java/lang/Class.html#getDeclaredFields) internally (via reflection), which has a non-deterministic specification to retrieve the @Option annotations and orders these options accordingly. Even though picocli supports an order attribute on @Option annotations, [CommandLine](https://github.com/remkop/picocli/blob/main/src/main/java/picocli/CommandLine.java) never sorts by this order - it just maintains whatever order the reflection API gave it. CmdGenerateDocs#generateDocument then iterates over the order that the `args` meth od returns, with the assumption that the order is deterministic. Unfortunately, the ordering can change due to different environments producing the contents in different orders despite the logical contents being the same. Harmless re-ordering could flip the tests from pass to fail despite the data being semantically the same. - `org.apache.pulsar.docs.tools.CmdGenerateDocsTest.testGenerateDocs` I discovered the problem with the [NonDex](https://github.com/TestingResearchIllinois/NonDex) tool. NonDex systematically detects incorrect tests that rely on non-deterministic behaviors in Java APIs—like assuming order of name/value pairs in json files — by exploring all specification-allowed outcomes. It does this by instrumenting undetermined APIs and randomizing the returned order/behavior within what the spec allows. Failures it exposes reliably indicate flawed assumptions in order that were never promised and should be fixed. ### Error messages ```text [INFO] Running org.apache.pulsar.docs.tools.CmdGenerateDocsTest [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 3.273 s <<< FAILURE! -- in org.apache.pulsar.docs.tools.CmdGenerateDocsTest [ERROR] org.apache.pulsar.docs.tools.CmdGenerateDocsTest.testGenerateDocs -- Time elapsed: 0.007 s <<< FAILURE! java.lang.AssertionError: expected [# test $ pulsar test options |Flag|Description|Default| |---|---|---| | `-h, --help` | Show this help message|false| | `-n, --name` | Name|null| ] but found [# test $ pulsar test options |Flag|Description|Default| |---|---|---| | `-n, --name` | Name|null| | `-h, --help` | Show this help message|false| ] at org.testng.Assert.fail(Assert.java:110) at org.testng.Assert.failNotEquals(Assert.java:1577) at org.testng.Assert.assertEqualsImpl(Assert.java:149) at org.testng.Assert.assertEquals(Assert.java:131) at org.testng.Assert.assertEquals(Assert.java:655) at org.testng.Assert.assertEquals(Assert.java:665) at org.apache.pulsar.docs.tools.CmdGenerateDocsTest.testGenerateDocs(CmdGenerateDocsTest.java:79) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:569) at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139) at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47) at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76) at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) [INFO] [INFO] Results: [INFO] [ERROR] Failures: [ERROR] org.apache.pulsar.docs.tools.CmdGenerateDocsTest.testGenerateDocs [INFO] Run 1: PASS [ERROR] Run 2: CmdGenerateDocsTest.testGenerateDocs:79 expected [# test $ pulsar test options |Flag|Description|Default| |---|---|---| | `-h, --help` | Show this help message|false| | `-n, --name` | Name|null| ] but found [# test $ pulsar test options |Flag|Description|Default| |---|---|---| | `-n, --name` | Name|null| | `-h, --help` | Show this help message|false| ] [INFO] [INFO] [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0 ``` ### Reproducing the issue You can replicate the problem with the [NonDex](https://github.com/TestingResearchIllinois/NonDex) tool command: ``` mvn -pl pulsar-docs-tools -Dtest=org.apache.pulsar.docs.tools.CmdGenerateDocsTest#testGenerateDocs -DnondexRuns=10 -DreuseForks=false edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dcheckstyle.skip=true -DnondexRunsWithoutShuffling=0 ``` ### Additional information _No response_ ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
