hkc-8010 opened a new issue, #67238:
URL: https://github.com/apache/airflow/issues/67238
### Apache Airflow version
2.11.2
### If "Other Airflow 2 version" selected, which one?
N/A
### What happened?
We have multiple anonymized production incidents on Airflow 2.11.2 with
`CeleryExecutor` where retry attempts that definitely ran are missing from
`task_instance_history`.
The visible user symptom is that the Task logs / tries UI does not show all
real attempts. But the more important observation is that this is not just a UI
problem: one or more retry attempts are actually absent from
`task_instance_history` even though scheduler and worker logs prove those
attempts executed.
Observed pattern in affected runs:
- the surviving `task_instance` row reflects the current or final attempt
- one or more earlier retry attempts are missing from `task_instance_history`
- `/tries` therefore omits those attempts because it is built from
`task_instance_history` plus the current `task_instance`
- in at least one case, logs for a missing attempt were still retrievable
directly, which makes the UI inconsistency more confusing
### What you think should happen instead?
Every retry attempt that actually executes should be preserved in
`task_instance_history`.
If a task reaches `deferred`, `up_for_retry`, `failed`, or another terminal
transition for a given try number, that try should still exist in
`task_instance_history` afterward, and `/tries` should list it.
### How to reproduce
I do not yet have a minimal standalone reproducer, but the repeated field
pattern is:
1. Run Airflow 2.11.2 with `CeleryExecutor`
2. Use a task that can retry, including cases that may defer and then retry
again
3. Let the task execute multiple tries
4. Inspect scheduler logs for the task/run and confirm a given try number
was sent to the executor and finished
5. Query `task_instance_history` for that same task/run
Observed result in affected runs:
- `task_instance.try_number` advances normally
- some earlier try numbers below the current try are missing from
`task_instance_history`
- `/tries` omits those missing attempts
### Operating System
Linux / Kubernetes
### Versions of Apache Airflow Providers
Not yet isolated to a provider-specific issue.
### Deployment
Other Kubernetes deployment
### Deployment details
These incidents were observed on Astro Hosted deployments running Runtime
13.6.0 (`Airflow 2.11.2+astro.2`) with:
- `CeleryExecutor`
- two scheduler replicas present
- PostgreSQL metadata DB
I am filing this upstream because the symptom is directly in core retry
history persistence (`task_instance_history`), not in a provider package.
### Anything else?
This does **not** look like a UI-only bug.
## Concrete example 1
Dag: `cs-forecast-cl-data-preprocessing-bk-eks-intg`
Task: `publish_data.pyspark`
Run: `manual__2026-04-20T13:05:04+00:00`
Direct DB query from the scheduler container after the incident:
```text
{'current_try_number': 3, 'current_state': 'failed', 'current_start_date':
'2026-04-22 13:00:30.668381+00:00', 'current_end_date': '2026-04-22
13:06:51.673156+00:00', 'current_hostname': '10.94.10.200',
'current_external_executor_id': 'a29904ce-e180-4b40-80d6-366e3a3b8cd2'}
history_rows=
{'try_number': 1, 'state': 'success', 'start_date': '2026-04-20
13:19:53.454647+00:00', 'end_date': '2026-04-20 13:25:16.544094+00:00',
'hostname': '10.94.23.102', 'external_executor_id':
'b7ba76f7-d337-4634-a5b6-989dd041eef1'}
{'history_try_numbers': [1], 'missing_try_numbers': [2]}
```
So the current row says `try_number=3`, but `task_instance_history` contains
only try `1`. Try `2` is missing.
Scheduler and worker logs prove try 2 actually ran:
```text
[2026-04-22T12:51:27.388+] {scheduler_job_runner.py:692} INFO - Sending
TaskInstanceKey(dag_id='cs-forecast-cl-data-preprocessing-bk-eks-intg',
task_id='publish_data.pyspark', run_id='manual__2026-04-20T13:05:04+00:00',
try_number=2, map_index=-1) to CeleryExecutor with priority 2 and queue default
[2026-04-22 12:51:27,841: INFO/ForkPoolWorker-3] Running on host 10.94.10.200
[2026-04-22T12:51:46.816+] {scheduler_job_runner.py:813} INFO -
TaskInstance Finished: dag_id=cs-forecast-cl-data-preprocessing-bk-eks-intg,
task_id=publish_data.pyspark, run_id=manual__2026-04-20T13:05:04+00:00,
map_index=-1, run_start_date=2026-04-22 12:51:28.238431+00:00,
run_end_date=None, run_duration=323.089447, state=deferred,
executor=CeleryExecutor(parallelism=25), executor_state=success, try_number=2,
max_tries=2, job_id=584134, pool=default_pool, queue=default,
priority_weight=2, operator=PySparkOperator, queued_dttm=2026-04-22
12:51:27.386468+00:00, queued_b
