potiuk opened a new pull request, #64446:
URL: https://github.com/apache/airflow/pull/64446
Add infrastructure to track and report on flaky UI E2E tests across CI runs.
**What this does:**
- Modifies `ui-e2e-tests.yml` to extract structured failure and fixme test
data from Playwright JSON results after each run, uploading them as artifacts
- Adds a new `e2e-flaky-tests-report.yml` scheduled workflow (daily at
midnight UTC) that:
- Downloads test report artifacts from the last 10 CI runs
- Aggregates failures across runs and browsers to identify flaky tests
- Identifies flaky candidates (tests failing ≥30% of runs)
- Posts a formatted report to `#airflow-3-ui` Slack channel with:
- Summary numbers (runs analyzed, unique failures, flaky candidates,
fixme count)
- Flaky candidate details with failure rates and short error excerpts
- All failures sorted by frequency
- Tests currently marked `test.fixme` waiting for fixes
- Also writes a GitHub Actions step summary with markdown tables
**New files:**
- `scripts/ci/extract_e2e_test_results.py` — parses Playwright JSON, outputs
`failures.json` + `fixme_tests.json`
- `scripts/ci/analyze_e2e_flaky_tests.py` — aggregates across runs,
identifies flaky candidates, formats Slack Block Kit report
- `scripts/tests/ci/test_extract_e2e_test_results.py` — 13 unit tests
- `scripts/tests/ci/test_analyze_e2e_flaky_tests.py` — 12 unit tests
**Note:** Requires `SLACK_BOT_TOKEN` secret to have permission to post to
`#airflow-3-ui` channel.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Claude Opus 4.6)
Generated-by: Claude Code (Claude Opus 4.6) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
