YAshhh29 opened a new pull request, #75916:
URL: https://github.com/apache/airflow/pull/75916
On Airflow before 3.3, durable agent tasks keep their journal in a JSON file
through ObjectStorage. When that storage failed, or the file was damaged, the
journal could fail a task whose work had already succeeded, or quietly throw
away the previous attempt's steps. This makes that backend behave like the task
state store one already does: storage problems are logged and skipped, and they
never fail the task.
**What was wrong**
`DurableStorageProtocol` says a backend may skip a write it can't make
without failing the step, and `TaskStateStoreDurableStorage` (Airflow >= 3.3)
already does. `DurableStorage` didn't: an error from `_write` went straight out
of `save_step`. Most saves happen after the work they record, so a failed write
could:
- fail a task whose work had already succeeded, when the journal lists more
than one agent run, because `DurableJournal.cleanup` rewrites the run list
through `save_step`
- raise a storage error in place of the agent's own error, because
`DurableRun.fail` saves the run meta before re-raising
- fail a tool step after the tool's side effect had already run
Reads had problems too. A read that failed with an `OSError` was treated as
an empty journal without a warning, so the next save overwrote the previous
attempt's steps. A file holding JSON that isn't an object failed the task with
an `AttributeError` on the first lookup.
**What changes**
- A failed write is skipped with a warning and `save_step` returns `False`,
so a retry runs that step again. In memory the key keeps its previous value, so
a later save doesn't write the skipped entry either.
- If the journal file can't be read, for a reason other than being missing
or invalid, it's left in place for the next attempt. This attempt runs live
without recording, and `delete_steps` leaves the file alone.
- A file that isn't a valid journal is logged and replaced: one cut off
mid-write, JSON that isn't an object, or bytes that aren't UTF-8 text. The last
case is the second commit. Without it, such a file counted as unreadable, so
that task instance would never journal again.
The durable execution docs now describe this for the ObjectStorage backend
too.
**Testing**
On Linux (WSL, Ubuntu 24.04) the durable tests give 302 passed and 3
skipped, because the optional `agents` and `strands` packages aren't installed.
There's also 1 error, in
`TestSummary.test_summary_names_the_tools_a_retry_runs_again`. That test runs
on the in-memory test storage and doesn't touch this code; the error is a local
fixture import (`cap_structlog` needs `airflow_shared`, which isn't installed
in a plain uv env).
I wrote the non-UTF-8 case test-first: it failed before the fix and passes
after. While developing the first commit, each new guard in `storage.py` was
removed one at a time, and a test failed every time. The prek pre-commit hooks
pass.
<details>
<summary>Test runs (WSL, Ubuntu 24.04, Python 3.12.16)</summary>
New case before the fix (`f8165fb566` with only the test change applied):
```
$ uv run --project providers/common/ai pytest
providers/common/ai/tests/unit/common/ai/durable/test_storage.py -k corrupt -v
...test_a_corrupt_journal_is_reported_and_started_over[cut off mid-write]
PASSED
...test_a_corrupt_journal_is_reported_and_started_over[not a JSON object]
PASSED
...test_a_corrupt_journal_is_reported_and_started_over[not UTF-8 text] FAILED
E AssertionError: assert False
E + where False = save_step('k', {...})
1 failed, 2 passed, 20 deselected
```
After the fix:
```
...test_a_corrupt_journal_is_reported_and_started_over[not UTF-8 text] PASSED
3 passed, 20 deselected
```
Full durable suite:
```
$ uv run --project providers/common/ai pytest
providers/common/ai/tests/unit/common/ai/durable -q
302 passed, 3 skipped, 1 warning, 1 error
ERROR
...test_journal.py::TestSummary::test_summary_names_the_tools_a_retry_runs_again
- ModuleNotFoundError: No module named 'airflow_shared'
```
</details>
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 5.5)
I used Claude Code to help trace the journal code paths and to draft this
description. I ran the tests on Linux myself, wrote the non-UTF-8 test and fix
test-first, and went through every change before pushing.
Generated-by: Claude Code (Opus 5.5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
