kaxil commented on code in PR #76381:
URL: https://github.com/apache/airflow/pull/76381#discussion_r4607175123


##########
providers/common/ai/tests/unit/common/ai/durable/test_storage.py:
##########
@@ -84,6 +84,44 @@ def test_a_retry_reads_what_the_previous_attempt_wrote(self, 
storage):
     def test_missing_key_is_none(self, storage):
         assert storage.load_step("missing") is None
 
+    @patch.object(storage_module, "log", spec=FilteringBoundLogger)
+    @patch.object(ObjectStoragePath, "read_text", autospec=True, 
side_effect=OSError("S3 is unavailable"))
+    def 
test_a_storage_read_failure_stops_the_task_before_it_can_replay_live(self, _, 
mock_log, storage):
+        with pytest.raises(OSError, match="S3 is unavailable"):
+            storage.load_step("k")
+
+        mock_log.warning.assert_called_once_with(
+            "Durable: could not read the journal; failing this attempt so 
finished steps don't run again",
+            path=str(storage._get_path()),
+        )
+
+    @pytest.mark.parametrize(
+        ("contents", "expected_exception"),
+        [(b'{"k":', json.JSONDecodeError), (b"\xff", UnicodeDecodeError)],
+        ids=["truncated-json", "invalid-encoding"],
+    )
+    @patch.object(storage_module, "log", spec=FilteringBoundLogger)
+    def test_a_corrupt_journal_blocks_reads_and_writes(

Review Comment:
   The `truncated-json` case makes the same checks as 
`TestJournalThatIsNotAnObject[cut off mid-write]` below, by a longer route 
through the private `_cache`. Could the `b"\xff"` case and the warning 
assertion move into that parametrize, and this test go?



##########
providers/common/ai/src/airflow/providers/common/ai/durable/storage.py:
##########
@@ -96,8 +96,21 @@ def _load_cache(self) -> dict[str, Any]:
             if not isinstance(cache, dict):
                 # JSON that is not an object holds no steps, just like a 
cut-off file.
                 raise ValueError("the journal file does not hold a JSON 
object")
-        except (FileNotFoundError, OSError, json.JSONDecodeError, ValueError):
+        except FileNotFoundError:
             cache = {}
+        except OSError:
+            log.warning(
+                "Durable: could not read the journal; failing this attempt so 
finished steps don't run again",
+                path=str(path),
+            )
+            raise
+        except (json.JSONDecodeError, ValueError):

Review Comment:
   Retrying can't fix a corrupt journal (the docs added here say so too), but 
this re-raises it as `JSONDecodeError` / `ValueError`, so a task with 
`retries=3` spends every retry and its delay failing the same way before anyone 
hears about it. `DurableJournalError` is the provider's error for a step that 
fails identically on every attempt, and it skips retries. Could this branch 
`raise DurableJournalError(...) from err` instead? The read-failure branch 
above should stay retryable, since an outage can clear.



##########
providers/common/ai/src/airflow/providers/common/ai/durable/storage.py:
##########
@@ -96,8 +96,21 @@ def _load_cache(self) -> dict[str, Any]:
             if not isinstance(cache, dict):
                 # JSON that is not an object holds no steps, just like a 
cut-off file.

Review Comment:
   This comment still describes the old fall-through to an empty journal. Both 
cases raise now, so maybe "JSON that is not an object is as corrupt as a 
cut-off file"?



##########
providers/common/ai/docs/durable_execution.rst:
##########
@@ -171,6 +171,17 @@ Without the option, the task fails with:
 
     ValueError: Durable execution on Airflow before 3.3 needs [common.ai] 
durable_cache_path to be set, to say where its journal is kept. Example: 
durable_cache_path = file:///tmp/airflow_durable_cache
 
+For S3, the IAM role must also have ``s3:ListBucket`` permission on the 
bucket, in
+addition to permissions to read, write, and delete journal objects. Without
+``s3:ListBucket``, S3 returns ``403 Access Denied`` instead of ``404 Not 
Found``
+when the journal does not exist. This is treated as a storage read failure, so 
even
+the first task attempt fails. Read failures stop the attempt to avoid repeating
+completed LLM calls or tool operations when an existing journal cannot be read.
+
+A corrupt journal also fails the task attempt and is left unchanged. Retrying 
alone

Review Comment:
   How does a user find the journal to inspect? The file name is a sha256 of 
the task instance, so could this say it's the `path` field on the `Durable: 
journal is corrupt` warning?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to