rRajivramachandran opened a new pull request, #8183: URL: https://github.com/apache/cloudstack/pull/8183
### Description The PR fixes the following test which could produce non-deterministic/flaky pass/fail result `com.cloud.server.StatsCollectorTest.java#persistVirtualMachineStatsTestPersistsSuccessfully` ### Setup: Java version: 11.0.20.1 Maven version: 3.8.8 ### Test failure reproduction: The test `com.cloud.server.StatsCollectorTest.java#persistVirtualMachineStatsTestPersistsSuccessfully` compares two JSON Strings and asserts if they are equal. However, the order of key value pairs is not guaranteed in a JSON string. Hence the test can fail. This issue was verified using the [NonDex plugin](https://github.com/TestingResearchIllinois/NonDex). #### Steps: ``` git clone https://github.com/apache/cloudstack.git cd cloudstack mvn install -pl server -am -DskipTests mvn -pl server edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=com.cloud.server.StatsCollectorTest.java#persistVirtualMachineStatsTestPersistsSuccessfully ``` ### Root cause and Fix Test Failure in Nondex Mode ``` Results: [INFO] [ERROR] Failures: [ERROR] StatsCollectorTest.persistVirtualMachineStatsTestPersistsSuccessfully:335 expected:<{"vmId":2,"[cpuUtilization":6.0,"networkReadKBs":7.0,"networkWriteKBs":8.0,"diskReadIOs":12.0,"diskWriteIOs":13.0,"diskReadKBs":10.0,"diskWriteKBs":11.0,"memoryKBs":3.0,"intFreeMemoryKBs":4.0,"targetMemoryKBs":5.0,"numCPUs":9,"entityType":"vm"]}> but was:<{"vmId":2,"[networkWriteKBs":8.0,"memoryKBs":3.0,"entityType":"vm","cpuUtilization":6.0,"diskReadKBs":10.0,"networkReadKBs":7.0,"diskReadIOs":12.0,"targetMemoryKBs":5.0,"numCPUs":9,"intFreeMemoryKBs":4.0,"diskWriteIOs":13.0,"diskWriteKBs":11.0]}> [INFO] [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0 ``` The following error occurs when comparing two json strings in this [line](https://github.com/apache/cloudstack/blob/99ded8169beef89a664df7580b2f039569491172/server/src/test/java/com/cloud/server/StatsCollectorTest.java#L333). The expected string is a hardcoded json string. The actual string is obtained by converting the class attributes to JSON(using the Gson library) [here](https://github.com/apache/cloudstack/blob/99ded8169beef89a664df7580b2f039569491172/server/src/main/java/com/cloud/server/StatsCollector.java#L1900). The comparision fails in some cases since the key orderings between the JSON data could be different. ### Fix The Fix converts both the JSON strings to TreeMaps(which maintain key order) and compares these for equality. This makes the test more stable without losing out on effectiveness ### Types of changes - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] New feature (non-breaking change which adds functionality) - [x] Bug fix (non-breaking change which fixes an issue) - [ ] Enhancement (improves an existing feature and functionality) - [ ] Cleanup (Code refactoring and cleanup, that may add test cases) - [ ] build/CI ### Feature/Enhancement Scale or Bug Severity This code modifies only test files and should not affect anything in production #### Feature/Enhancement Scale - [ ] Major - [x] Minor #### Bug Severity - [ ] BLOCKER - [ ] Critical - [ ] Major - [x] Minor - [ ] Trivial ### Screenshots (if appropriate): ### How Has This Been Tested? - The test passes when run with the NonDex tool. (`mvn -pl server edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=com.cloud.server.StatsCollectorTest.java#persistVirtualMachineStatsTestPersistsSuccessfully`) - The entire test suite of the module passes. (`mvn -pl server test`) #### How did you try to break this feature and the system with this change? The change only creates a new test utility and calls it within a single test. Hence it should not affect any other code <!-- see how your change affects other areas of the code, etc. --> <!-- Please read the [CONTRIBUTING](https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md) document --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
