srchilukoori opened a new pull request, #65982:
URL: https://github.com/apache/airflow/pull/65982
Follow-up to #65742 as requested by @shahar1. Adds a system test DAG that
exercises the `cancel_on_kill` behavior introduced when `DataprocSubmitTrigger`
and `DataprocSubmitJobDirectTrigger` migrated to `BaseTrigger.on_kill()`.
**Changes:**
New file:
`providers/google/tests/system/google/cloud/dataproc/example_dataproc_cancel_on_kill.py`
The DAG contains two tests sharing a single Dataproc cluster:
| Test | What it does | What it proves |
|---|---|---|
| A (happy path) | Deferrable submit with `cancel_on_kill=True` runs SparkPi
to completion | Trigger plumbing with `cancel_on_kill=True` does not break
normal deferrable execution |
| B (cancel path) | Submits a long-running SparkPi (1M slices)
asynchronously, cancels it via `DataprocHook.cancel_job()`, asserts `CANCELLED`
state | The `cancel_job()` call that `on_kill()` delegates to actually cancels
running Dataproc jobs |
**Testing:**
System tests require GCP credentials and a project with Dataproc API
enabled. I do not have access to a GCP environment to run this test end-to-end.
The test was verified via:
- `python3 -m py_compile` — syntax OK
- `ruff check` / `ruff format` — all checks passed
- All pre-commit hooks passed
- Pattern consistency verified against existing Dataproc system tests
(`example_dataproc_spark_deferrable.py`, `example_dataproc_spark_async.py`,
`example_dataproc_start_from_trigger.py`)
To run manually:
```bash
SYSTEM_TESTS_ENV_ID=<id> SYSTEM_TESTS_GCP_PROJECT=<project> \
pytest
providers/google/tests/system/google/cloud/dataproc/example_dataproc_cancel_on_kill.py
-s
```
related: #65742
---
- [X] Yes (please specify the tool below)
Generated-by: Claude following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
