Yanislava opened a new pull request, #77242:
URL: https://github.com/apache/airflow/pull/77242
closes: #39463
related: #61541
### Why
`ExternalTaskSensor` only sees DAGs of the Airflow it runs in. Teams that
run separate Airflow instances (or are in the middle of migrating from Airflow
2 to Airflow 3) have no built-in way to wait for each other's DAGs. The
workarounds are either a hand-written `HttpSensor` with a `response_check`, or
asking the other team to push asset events / trigger runs through the API,
which needs changes on their side.
### What
`RemoteExternalTaskSensor` in the http provider. It subclasses
`ExternalTaskSensor` and takes the same arguments (`external_task_id(s)`,
`allowed_states`, `failed_states`, `skipped_states`, `execution_delta`,
`execution_date_fn`, `check_existence`, `soft_fail`), plus `http_conn_id` and
`api_version`:
- `api_version="v1"`: remote Airflow 2, `/api/v1`, basic auth or no auth.
- `api_version="v2"` (default): remote Airflow 3, `/api/v2`, login/password
exchanged for a token at `/auth/token` on every poke (so a long wait does not
outlive the token).
Dates, failing and skipping reuse the `ExternalTaskSensor` helpers
(`_get_dttm_filter`, `_handle_failed_states`, `_handle_skipped_states`, all
present since standard 1.0.0). The states come from the remote list endpoints:
- whole DAG: `dagRuns/list` counted with `page_limit=1` / `total_entries`
(at most one run per `dag_id` + logical date);
- tasks: `taskInstances/list`, all pages, one request per date. A date is
done only when **every** external task has task instances and **all** of them
(every map index of a mapped task) are in an allowed state; any task instance
in a failed / skipped state fails / skips the sensor. Counting task instances
and dividing by the number of tasks, as `ExternalTaskSensor` does, would let
map indexes of one task stand in for another task, and a mapped task alone
would never reach the expected count.
`check_existence=True` checks the remote DAG and tasks once. Empty, blank or
duplicate ids (e.g. an empty `{{ params.x }}`) fail before any request, since
`ExternalTaskSensor` validates them before templates are rendered.
Errors: 429 and 5xx raise `AirflowException` (task retries apply, the remote
Airflow may be restarting); other 4xx raise `AirflowFailException` (the same
request would fail again).
Not in this PR, to keep it reviewable: task groups
(`external_task_group_id`), deferrable mode, a link to the remote UI, and a
remote counterpart of `TriggerDagRunOperator` (#61541). Both unsupported
options raise a clear `ValueError` at DAG parse time.
### How the comments on the issues are addressed
- **Where it lives** (#39463, #61541): in the http provider, built on
`HttpHook`, so connection extras (headers, `verify`, proxies) work as for any
HTTP connection. The standard provider becomes an optional `standard` extra of
the http provider instead of a try/except stub class.
- **A separate provider on `apache-airflow-client`** (#39463): the 2.x
client only speaks `/api/v1` and the 3.x client only `/api/v2`, and they cannot
be installed side by side, while a common real case is Airflow 3 waiting for
Airflow 2. Plain HTTP covers both and adds no dependency.
- **Is `HttpSensor` enough?** (#39463): it does not compute the logical date
of the external run (`execution_delta` / `execution_date_fn`), does not fail
fast when the external run fails (it waits until the timeout), cannot obtain an
Airflow 3 token, and does not check existence.
- **Subclassing instead of duplicating** (#61541): done; `poke` and
`execute` are overridden. `execute` is overridden so the parent's deferrable
branch (which reads the local metadata DB) can never be reached.
`poke` is overridden as a whole because the counting in
`ExternalTaskSensor._poke_af3` is a nested function. If preferred, it can be
extracted into a method in a follow-up so that subclasses override only the
counting.
### Testing
- Unit tests with mocked HTTP: both API versions, basic auth, token exchange
(new token per poke, 401), DAG runs, task instances incl. mapped tasks and
pagination, `execution_delta`, time zones, failed / soft_fail / skipped states,
4xx vs 429/5xx, empty and duplicate ids, `check_existence` (missing DAG,
missing task, checked once).
- Against real instances, with the sensor class copied verbatim from this
branch into a DAG file:
- Airflow 3.4 → Airflow 2.10 (`v1`, no auth) and Airflow 3.4 → Airflow 3.4
(`v2`, login exchanged for a token).
- Cases: DAG success; task in a task group with `check_existence`; mapped
task (3 map indexes); task + mapped task together; no run (waits, then skipped
with `soft_fail`); failed run (fails on the first poke, not at the timeout);
missing DAG; empty task id (fails before any request).
- Found on the way and documented: an Airflow 3 manual run triggered
without a logical date cannot be matched (same for `ExternalTaskSensor`).
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code
Generated-by: Claude Code following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01LAG776D1f3b6nF6tdW1zD5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]