jason810496 opened a new pull request, #75795:
URL: https://github.com/apache/airflow/pull/75795
# Load the serialized Dag in ti_run only for stub tasks
- related: #74869, #75387
- needs `backport-to-v3-4-test` before 3.4.0 RC.
## Why
Starting any task deserialized its whole Dag, so that `ti_run` could find
out whether the task was a stub and read its `arg_bindings`. On a Dag cache
miss that is a full `serialized_dag` read plus deserialization per task start,
paid while the route holds `SELECT ... FOR UPDATE` on the `task_instance` row,
and for a Python task the result is thrown away.
The version gate in front of it does not help:
`client_supports_arg_bindings()` tests the client's API version, not the kind
of task, and every current Task SDK is past that version.
## How
- The serialized `is_stub` flag is denormalized onto `task_instance`, next
to `operator` and `custom_operator_name`, which are written from the task the
same way. `ti_run` already fetches and locks that row, so reading one more
column costs nothing.
- The route loads the task's definition only when the flag is set:
```python
serialized_task = get_task(dag_bag, ti, session=session) if ti.is_stub
else None
```
- Null means "not a stub", with no fallback. Every null row predates the
migration, so it was written by 3.3, which has no `arg_bindings` at all, and
such a task behaves exactly as it did there. `refresh_from_task` fills the
column in on a clear.
- `execution_timeout` comes from the same definition, so it is now resolved
for a task a language SDK runs and left unset for a Python task. That is the
split #75387 makes permanent by having the Python task runner send its own
timeout to the supervisor. Until it merges, a stuck Python task has no
supervisor backstop on `main`; the runner's own `AirflowTaskTimeout` still
fires. 3.4.0 is unaffected, having no `TIRunContext.execution_timeout` at all
(#75469).
### For the release manager
`main` has moved past the 3.4.0 chain head, and a 3.4.0 revision cannot sit
after 3.5.0 ones or `db migrate --to-version 3.4.0` would stop short of it. The
new revision is therefore **inserted**, and the two 3.5.0 revisions after it
are renumbered:
```
ee86eed19e24 0148_3_4_0_add_pending_partition_key_to_apdr
b4f1a7d2c903 0149_3_4_0_add_is_stub_to_task_instance <- new
c71065a1b14f 0150_3_5_0_add_session_id_to_job <- was 0149,
down_revision rewritten
38c8df3fcb9c 0151_3_5_0_add_dag_run_dag_id_run_after_index <- was 0150,
unchanged
```
A dev database already at `c71065a1b14f` or later has a history that no
longer matches and needs a reset. The head is unchanged, and the column is
nullable with no default and no backfill, so the upgrade itself stays
metadata-only on PostgreSQL 11+ and instant on MySQL 8.
## What
- Add the nullable `task_instance.is_stub` column and write it from
`task.is_stub` in `TaskInstance.refresh_from_task` and
`TaskInstance.insert_mapping`.
- Select it in `ti_run` and derive the task's definition, `arg_bindings` and
`execution_timeout` from it only when it is set.
- Tests: a regular task loads no Dag at all, a null flag is treated as a
regular task, both write paths carry the flag, and every recorded TypeScript,
Java and Go payload marks its own tasks as stubs while a Go or Java
`TriggerDagRun` task, which is the Python operator itself, does not.
- `java_native.json` gains `is_stub`. That fixture was recorded in #74043,
before #74440 made the Java SDK emit the flag.
No SDK changes: TypeScript, Java and Go already write `is_stub` for Dags
defined in them.
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes, with help of Claude Code Opus 5 following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
