zozo123 opened a new pull request, #74172:
URL: https://github.com/apache/airflow/pull/74172
Provider DB test jobs have been reporting failing tests as green.
`scripts/ci/testing/run_unit_tests.sh` treats exit code 1 from
`breeze testing providers-tests --run-in-parallel --run-db-tests-only` as
"No DB tests were
collected ...; treating as success". Exit code 1 does not mean that. In a
parallel run, breeze
exits 1 when any test type returns a non-zero code (`parallel.py`
`finalize_async_tasks` /
`check_async_run_results`), so the rule cannot tell a failing test from an
empty selection. Only
a timeout (exit 2) still failed the job. Core DB jobs never had this
exception, so the same
failure turns them red.
The rule was added in #64222 to handle a real case. After #63791,
`--run-db-tests-only`
deselects non-DB tests at collection time. A provider test type with no DB
tests therefore
collects nothing, and pytest exits 5. That is the only point where "no DB
tests" is known
precisely: per test type, before breeze folds every return code into 1. So
this PR:
- in breeze `_run_test`, treats pytest exit code 5 as success for that test
type, only when
`--run-db-tests-only` is set, and prints `No DB tests collected for
<type>`. Any other code is
passed through unchanged.
- removes the `RESULT == 1` exception from `providers_tests()`, so it
handles exit codes the same
way `core_tests()` does.
The breeze change is not limited to providers. A core test type with no DB
tests would now pass
the DB job too. No core test type is in that situation today.
Evidence from CI (all read-only):
- I scanned all 79 scheduled canary runs on main since 2026-09-14 (40
ci-amd, 39 ci-arm), which
contain 1,279 provider DB jobs. 29 jobs printed "treating as success".
Every one of them had a
real test failure, and every one concluded `success`. They are the 17
DB-prov jobs of ci-amd run
37088278777 and the 12 of ci-arm run 37055158665 (head 66d751bfd2). All
failed
`test_edge_executor.py::TestEdgeExecutor::test_scheduler_restart_adopts_queued_edge_task`,
which was later fixed by #74122.
- Example: job 111105083352 (MySQL 8.0, Python 3.10). Log line 3425
`NOK for Test: Providers[-amazon,celery,google,standard]: Return code:
1.`, 4323
`1 failed, 2649 passed, 70 skipped, 14671 deselected`, 4574 `+
RESULT=1`, 4576
`No DB tests were collected for providers; treating as success.`, 4578
`Providers tests completed successfully`. Conclusion: success.
- Control in the same run: job 111105084513 (Pendulum2, providers, scope
All) hit the same
failing test and got the same `+ RESULT=1` (line 5460), then failed with
"The providers test
All failed! Giving up" (line 5463).
- The only provider DB job in the window that went red was a timeout: job
110190202345,
`+ RESULT=2`.
- PR run 36643870541: four provider DB jobs passed with 2 failing celery
tests
(`test_celery_executor.py::TestCeleryExecutor::test_cleanup_stuck_queued_tasks`
and
`::test_revoke_task`). Jobs: 109666320921 (MySQL), 109666320818 (Postgres
14), 109666321750
(LatestSQLAlchemy), 109666321542 (MinSQLAlchemy).
- The case the rule was written for still happens. In PR run 36973419258,
job 110734506798 log
line 3090 shows `collected 511 items / 511 deselected / 0 selected`, line
2494 shows
`NOK for Test: Providers[apache.kafka,common.compat,common.messaging]:
Return code: 5.` and line
3135 shows the rule firing. With this change, that test type reports `OK`
and the job still
passes.
Testing:
- `uv run --project dev/breeze pytest
dev/breeze/tests/test_run_test_args.py`: 17 passed, both
with and without `CI=true`.
- `uv run --project scripts pytest
scripts/tests/ci/testing/test_run_unit_tests.py`: 2 passed.
- The new tests fail without the source changes. The breeze test fails in the
`db-only-no-tests-collected` case (`assert 5 == 0`), and the script test
fails because the job
exits 0 instead of 1.
- `uv run --project dev/breeze pytest dev/breeze/tests`: 1211 passed, 3
failed. The 3 failures
are in `test_publish_docs_to_s3.py` (a missing `.build/.suppress_colour`
in a fresh worktree).
They fail the same way without this change.
- `prek run --stage pre-commit` on the changed files passed. The
Docker-based `shellcheck` hook was
skipped; local `shellcheck -x -a` (0.11.0) on the script is clean.
- I also ran the real `breeze testing providers-tests|core-tests
--run-in-parallel --run-db-tests-only`
command, called from the real `run_unit_tests.sh`, with only `docker
compose run` replaced by
real pytest runs on small fixtures. Results:
| Scenario (per test type) | Before | After
|
|---------------------------------------|-------------------|-------------------|
| providers: all pass | green | green
|
| providers: a test fails | green (masked) | red (exit 1)
|
| providers: one type has no DB tests | green | green
|
| providers: failure + no-DB type | green (masked) | red (exit 1)
|
| providers: compose exits 1 / 137 | green (masked) | red (exit 1)
|
| providers: exception in a worker | green (masked) | red (exit 1)
|
| providers: total timeout | red (exit 2) | red (exit 2)
|
| core: a test fails | red (exit 1) | red (exit 1)
|
| core: one type has no DB tests | red (exit 1) | green
|
Note: `v3-3-test` has the same rule. `v3-2-test` and `v3-1-test` do not,
because backport #64233
was closed unmerged.
related: #64222, #64154, #63791
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 5.5)
Generated-by: Claude Code (Opus 5.5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
