zoomzoomTnT commented on PR #76381:
URL: https://github.com/apache/airflow/pull/76381#issuecomment-6754862676
Thanks @fat-catTW and @kaxil — I was looking at duplicate durable-agent
calls because a retry that re-runs finished model/tool steps can spend real
tokens.
The intended mechanism, as I read it on main:
- On Airflow < 3.3 the journal is a single ObjectStorage JSON file
(`DurableStorage`), keyed by a hash of dag/task/run/map_index.
- On 3.3+ it lives in the AIP-103 task state store
(`TaskStateStoreDurableStorage`).
- `DurableJournal` / `DurableRun` claim each step by position, name and
fingerprint and replay the recorded payload on retry; `JournalStep.record`
writes *after* the live step succeeds.
- A missing journal is a normal first attempt. Today's `_load_cache` also
treats `OSError`, `JSONDecodeError` and `ValueError` as empty, so a transient
read failure or a corrupt file looks like "never ran" and the retry calls the
model and tools again.
This PR's fail-closed change on read (and on corrupt journals) looks right
for that case, and the review notes about `s3:ListBucket`, the warning wording,
`_cache = {}` after `delete_steps`, the changelog, and `path=` in the logs all
look addressed in the latest push.
I don't want to step on the existing review. Two questions:
1. Would you welcome another pair of eyes on the latest revision, or any
leftover work on this PR?
2. Separately, retries are still at-least-once when a step succeeds but is
not recorded (`JournalStep.record` / `save_step` returning False, or a crash
between the live call and the write). That's warned in the journal, but it's
another way the same cost shows up. Happy to look at that, and at whether the
3.3+ store can fail-open on a bad `get`, in a follow-up if that's useful.
Happy to help either as a reviewer or with a follow-up PR — just let me know
what would be most useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
