Yanislava opened a new pull request, #77242:
URL: https://github.com/apache/airflow/pull/77242

   closes: #39463
   related: #61541
   
   ### Why
   
   `ExternalTaskSensor` only sees DAGs of the Airflow it runs in. Teams that 
run separate Airflow instances (or are in the middle of migrating from Airflow 
2 to Airflow 3) have no built-in way to wait for each other's DAGs. The 
workarounds are either a hand-written `HttpSensor` with a `response_check`, or 
asking the other team to push asset events / trigger runs through the API, 
which needs changes on their side.
   
   ### What
   
   `RemoteExternalTaskSensor` in the http provider. It subclasses 
`ExternalTaskSensor` and takes the same arguments (`external_task_id(s)`, 
`allowed_states`, `failed_states`, `skipped_states`, `execution_delta`, 
`execution_date_fn`, `check_existence`, `soft_fail`), plus `http_conn_id` and 
`api_version`:
   
   - `api_version="v1"`: remote Airflow 2, `/api/v1`, basic auth or no auth.
   - `api_version="v2"` (default): remote Airflow 3, `/api/v2`, login/password 
exchanged for a token at `/auth/token` on every poke (so a long wait does not 
outlive the token).
   
   Dates, failing and skipping reuse the `ExternalTaskSensor` helpers 
(`_get_dttm_filter`, `_handle_failed_states`, `_handle_skipped_states`, all 
present since standard 1.0.0). The states come from the remote list endpoints:
   
   - whole DAG: `dagRuns/list` counted with `page_limit=1` / `total_entries` 
(at most one run per `dag_id` + logical date);
   - tasks: `taskInstances/list`, all pages, one request per date. A date is 
done only when **every** external task has task instances and **all** of them 
(every map index of a mapped task) are in an allowed state; any task instance 
in a failed / skipped state fails / skips the sensor. Counting task instances 
and dividing by the number of tasks, as `ExternalTaskSensor` does, would let 
map indexes of one task stand in for another task, and a mapped task alone 
would never reach the expected count.
   
   `check_existence=True` checks the remote DAG and tasks once. Empty, blank or 
duplicate ids (e.g. an empty `{{ params.x }}`) fail before any request, since 
`ExternalTaskSensor` validates them before templates are rendered.
   
   Errors: 429 and 5xx raise `AirflowException` (task retries apply, the remote 
Airflow may be restarting); other 4xx raise `AirflowFailException` (the same 
request would fail again).
   
   Not in this PR, to keep it reviewable: task groups 
(`external_task_group_id`), deferrable mode, a link to the remote UI, and a 
remote counterpart of `TriggerDagRunOperator` (#61541). Both unsupported 
options raise a clear `ValueError` at DAG parse time.
   
   ### How the comments on the issues are addressed
   
   - **Where it lives** (#39463, #61541): in the http provider, built on 
`HttpHook`, so connection extras (headers, `verify`, proxies) work as for any 
HTTP connection. The standard provider becomes an optional `standard` extra of 
the http provider instead of a try/except stub class.
   - **A separate provider on `apache-airflow-client`** (#39463): the 2.x 
client only speaks `/api/v1` and the 3.x client only `/api/v2`, and they cannot 
be installed side by side, while a common real case is Airflow 3 waiting for 
Airflow 2. Plain HTTP covers both and adds no dependency.
   - **Is `HttpSensor` enough?** (#39463): it does not compute the logical date 
of the external run (`execution_delta` / `execution_date_fn`), does not fail 
fast when the external run fails (it waits until the timeout), cannot obtain an 
Airflow 3 token, and does not check existence.
   - **Subclassing instead of duplicating** (#61541): done; `poke` and 
`execute` are overridden. `execute` is overridden so the parent's deferrable 
branch (which reads the local metadata DB) can never be reached.
   
   `poke` is overridden as a whole because the counting in 
`ExternalTaskSensor._poke_af3` is a nested function. If preferred, it can be 
extracted into a method in a follow-up so that subclasses override only the 
counting.
   
   ### Testing
   
   - Unit tests with mocked HTTP: both API versions, basic auth, token exchange 
(new token per poke, 401), DAG runs, task instances incl. mapped tasks and 
pagination, `execution_delta`, time zones, failed / soft_fail / skipped states, 
4xx vs 429/5xx, empty and duplicate ids, `check_existence` (missing DAG, 
missing task, checked once).
   - Against real instances, with the sensor class copied verbatim from this 
branch into a DAG file:
     - Airflow 3.4 → Airflow 2.10 (`v1`, no auth) and Airflow 3.4 → Airflow 3.4 
(`v2`, login exchanged for a token).
     - Cases: DAG success; task in a task group with `check_existence`; mapped 
task (3 map indexes); task + mapped task together; no run (waits, then skipped 
with `soft_fail`); failed run (fails on the first poke, not at the timeout); 
missing DAG; empty task id (fails before any request).
     - Found on the way and documented: an Airflow 3 manual run triggered 
without a logical date cannot be matched (same for `ExternalTaskSensor`).
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes — Claude Code
   
   Generated-by: Claude Code following [the 
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   
   https://claude.ai/code/session_01LAG776D1f3b6nF6tdW1zD5
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] Http remote external tas... Yanislava (via GitHub) via commits

Reply via email to