Lee-W commented on code in PR #76861:
URL: https://github.com/apache/airflow/pull/76861#discussion_r4533338044
##########
providers/common/ai/src/airflow/providers/common/ai/durable/journal.py:
##########
@@ -160,6 +177,28 @@ def record(self, payload: Any) -> bool:
self.durable_run.journal.stats.recorded[self.kind] += 1
log.debug("Durable: recorded step", position=self.position,
step=self.name)
return True
+ self._not_recorded()
+ return False
+
+ def cannot_record(self, error: Exception) -> None:
+ """
+ Note that the step's result has no form the journal can hold, such as
a value with no JSON form.
+
+ :raises DurableJournalError: When the journal fails on such results,
as it does by default.
+ Otherwise the step is left unrecorded, as when storage skips a
write, and runs again on retry.
+ """
+ if not self.replayable:
+ # Never recorded anyway, so there is nothing to fail or report.
+ return
+ if self.durable_run.journal.fail_on_unrecordable:
+ # The same value fails the same way on every retry, so retrying
the task cannot help.
+ raise DurableJournalError(
+ f"durable execution could not record a step's result, because
it is not JSON-serializable: {error}"
+ ) from error
+ log.warning("Durable: step result is not JSON-serializable",
step=self.name, error=str(error))
+ self._not_recorded()
+
+ def _not_recorded(self) -> None:
Review Comment:
```suggestion
def _mark_not_recorded(self) -> None:
```
##########
providers/common/ai/docs/durable_execution.rst:
##########
@@ -366,11 +366,15 @@ A toolset can opt out of replay. ``ManagedAgentToolset``
does by default
(``replayable=False``), because a managed agent acts on a system Airflow
cannot observe:
its calls run again on every attempt, while the steps around them still replay.
-A result has to survive a JSON round trip to be recorded. One that cannot be
encoded
-fails the task with ``DurableJournalError`` and no retry, since it would fail
the same
-way every time; the model could not have read it either. A write the task
state store
-refuses does not fail the task: the task log names
-the step, and a retry runs it again.
+A result has to survive a JSON round trip to be recorded. Bytes a tool
returns, on their
Review Comment:
Nit: we cover bytes in dicts and lists and in dataclass or Pydantic fields,
but not bytearray, memoryview or sets of bytes. Those fail or come back changed
(see the _tag_bytes comment). We can mention it here (e.g., Only bytes is
converted; bytearray, memoryview and sets or frozensets of bytes are not)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
