santhoshct opened a new pull request, #17938:
URL: https://github.com/apache/kafka/pull/17938
### Summary
This pull request introduces a new script, develocity_reports.py, designed
to enhance our detailed reports on flaky tests. It leverages the Develocity API
to fetch and analyze test results, focusing on identifying and reporting
quarantined tests with high failure rates. The script is intended to help
developers/CI quickly identify problematic tests that require attention,
thereby improving the overall quality of our codebase.
### Changes
- New Script Addition: Introduced develocity_reports.py to the
.github/scripts directory.
- Functionality:
- Fetches test results from the Develocity API for a specified project and
test type.
- Analyzes test outcomes to identify flaky and failed tests.
- Generates reports highlighting high-priority quarantined tests based on
failure rates and quarantine duration.
- Provides detailed timelines and statistics for each test and test case.
- Logging: Integrated logging to track the script's execution and handle
exceptions gracefully.
- Configuration: Allows configuration of API base URL, authentication token,
project name, and thresholds for quarantine and failure rates.
- Output: Produces a console report summarizing the most problematic tests,
including detailed statistics and recent execution timelines.
### Testing
- Manual tested. Example output would be like this.
org.apache.kafka.tiered.storage.integration.OffloadAndTxnConsumeFromLeaderTest
==============================================================================
Quarantined for 14 days
Container Failure Rate: 14.51%
Recent Failure Rate: 14.51%
Container Statistics:
Total Runs: 1013
Failed: 4
Flaky: 143
Passed: 866
Container Recent Executions:
Date/Time (UTC) Outcome Build ID
------------------------------------------------
2024-11-25 02:51 passed 5kqy57pu3uwxs
2024-11-25 03:18 passed rgjbtcmlfk7so
2024-11-25 03:18 passed hhko6esalsqco
2024-11-25 03:30 passed i35nqmpusibpw
2024-11-25 03:31 flaky jddx23jdksg5m
Test Cases (Last 7 Days):
------------------------------------------------
→ executeTieredStorageTest(String, String)[1]
Failure Rate: 10.92%
Runs: 476 | Failed: 0 | Flaky: 52 | Passed: 424
Recent Executions:
Date/Time (UTC) Outcome Build ID
--------------------------------------------
2024-11-25 03:18 passed hhko6esalsqco
2024-11-25 03:30 passed i35nqmpusibpw
2024-11-25 03:31 flaky jddx23jdksg5m
→ executeTieredStorageTest(String, String)[2]
Failure Rate: 5.25%
Runs: 476 | Failed: 0 | Flaky: 25 | Passed: 451
Recent Executions:
Date/Time (UTC) Outcome Build ID
--------------------------------------------
2024-11-25 03:18 passed hhko6esalsqco
2024-11-25 03:30 passed i35nqmpusibpw
2024-11-25 03:31 passed jddx23jdksg5m
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
