dzpan0 opened a new pull request, #67832:
URL: https://github.com/apache/airflow/pull/67832
## Description
Follow-up to #64394, which fixed `ExternalTaskSensor`'s
`check_existence=True` for the AF2. On AF3, the deferrable path still defers
silently when the external Dag, task, or task group does not exist, because the
existence check has no access to the metadata database.
This PR adds two Execution API endpoints that let the sensor verify external
references in a single round trip, and wires the sensor to use them before
deferring:
- `GET /dags/{dag_id}/tasks/existence?task_ids=...`
- `GET /dags/{dag_id}/task-groups/existence?task_group_ids=...`
Each returns its input partitioned into `existing` and `missing`. Passing an
empty `task_ids` list works as a Dag-existence probe (a missing Dag surfaces as
`DAG_NOT_FOUND`, which the sensor translates to `ExternalDagNotFoundError`).
Missing tasks or task groups surface as `ExternalTaskNotFoundError` and
`ExternalTaskGroupNotFoundError`, respectively, matching the AF2 path's
behavior.
The Task SDK exposes them via two new `RuntimeTaskInstance` accessors:
`get_dag_tasks_existence` and `get_dag_task_groups_existence`.
## Why
Existing dag-level Execution API routes can't distinguish a Dag that doesn't
exist from a Dag that exists, but has no DagRuns: `GET /dag_runs/count` returns
0 and `GET /dag_runs/previous` returns None in both cases. `GET /dags/{dag_id}`
settles the existence question on its own, but says nothing about specific
tasks or task groups. These new endpoints fill that gap by combining the
existence check with a per-id partition.
These endpoints can answer that cheaply and explicitly, which is useful for
any future operator or sensor that needs to validate references against an
external dag before deferring or scheduling work. The partitioned response
shape (`existing` / `missing`) is also more informative than per-id checks when
validating a list. Callers get one round trip instead of N, and a single
response covers all four outcomes (all present, all missing, partial overlap,
dag itself missing).
## Testing
Unit tests added across the API route, datamodels, SDK client, supervisor
handler, runtime accessor and sensor layers.
related: #40745
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
http://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [X ] Yes (please specify the tool below)
Claude (Opus 4.7) following the project guidelines.
<!--
Generated-by: [Tool Name] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
