seanmuth commented on issue #65708:
URL: https://github.com/apache/airflow/issues/65708#issuecomment-5609167396
Adding a data point from an Astro Hosted deployment (Astronomer's managed
Airflow), Runtime 3.1-15 (Airflow 3.1.8) — confirms this is not limited to
3.2.x, and confirms the concurrency-correlation observation with actual numbers.
**Same exact signature**, both variants seen in this thread:
- Fatal: `Starting call to 'airflow.sdk.api.client.Client.request'...` →
`API server error, status_code=409, previous_state=success`
(`supervisor.py:617`, uncaught) → task exits with `exit_code=1`,
`final_state=up_for_retry`, despite the task's own work having already
completed successfully.
- Non-fatal sibling, same root cause: heartbeat hits the same
409/`previous_state=success` conflict, caught by the existing
`ServerResponseError` handler (`supervisor.py:1115`, `"Server indicated the
task shouldn't be running anymore"`) and just SIGTERMs the process — no crash,
no misleading `up_for_retry`.
**Confirms PR #63355 is the right shape of fix** ("if API says TI is already
in requested state... task runner will not fail the process") — verified it's
not present in 3.1.8/3.2.x (`git compare` against the merge commit: diverged
from 3.1.8, 3.2.0, 3.2.1, 3.2.2; first contained in 3.3.0).
**New evidence on the concurrency correlation:** cross-environment
comparison over the same 24h window, filtering on this exact error signature —
one high-task-density environment showed ~487 occurrences across its cluster,
while a much lower-density environment (older Runtime, same platform, no shared
code) showed only 5. On the DB side: RDS Performance Insights (Database Load,
Top SQL) looked unremarkable around one occurrence, but that view is built from
active/CPU-consuming sessions and wouldn't surface an idle-in-transaction
pileup even if present. Broader CloudWatch RDS metrics for that same window
show a real, if modest, correlation: write IOPS/throughput/transaction-log
generation all spike right at the time of the failure, with DB load, connection
count, and CPU climbing over the following several minutes (DB load ~14→20,
connections ~252→316, CPU ~46%→53%). Not dramatic, but a genuine load increase
lands right on the incident — consistent with the heartbeat-que
uing theory raised earlier in this thread, though we can't confirm the
specific mechanism (e.g., can't rule in/out idle-in-transaction sessions
specifically). We're now watching `pg_stat_activity` live (`state != 'idle'`,
runtime > 5s) to try to catch it during a genuine load spike.
One detail from an earlier comment worth a second look: the report showing
ingress logs with only *one* state-update request sent, yet two different
apiserver pods logging update activity for the same TI. We saw something
adjacent to this — two separate `"...this is the 1st time calling it"`
tenacity-`before` log lines within ~300ms of each other in the same worker
pod's log, for what should be non-competing calls. Neither of us has a full
explanation for that piece yet; flagging in case it points at something below
the request-retry layer (e.g. a stale/pooled connection landing on a different
backend pod) rather than a pure application-level duplicate call.
---
Filed by Claude (Sonnet 5) on behalf of Sean Muth, from a support
investigation on an Astro Hosted deployment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
