uranusjr commented on code in PR #74794:
URL: https://github.com/apache/airflow/pull/74794#discussion_r4548646282
##########
airflow-core/src/airflow/dag_processing/processor.py:
##########
@@ -257,8 +261,11 @@ def _parse_file(msg: DagFileParseRequest, log:
FilteringBoundLogger) -> DagFileP
)
if msg.callback_requests:
+ targets_loaded = any(_is_callback_target_loaded(bag, request) for
request in msg.callback_requests)
Review Comment:
But doesn’t the schema change anyway with DagCallbacksStarted being added?
##########
airflow-core/src/airflow/dag_processing/processor.py:
##########
@@ -183,6 +200,32 @@ class DagFileParsingResult(BaseModel):
]
[email protected]()
+class _DagParseCommsDecoder(CommsDecoder[ToDagProcessor, ToManager]):
+ _received_api_error: bool = attrs.field(default=False, init=False)
+ _pending_request_ids: set[int] = attrs.field(factory=set, init=False)
+
+ @property
+ def api_request_failed(self) -> bool:
+ """Include unanswered requests interrupted by the Dag import
timeout."""
+ return self._received_api_error or bool(self._pending_request_ids)
+
+ def _make_frame(self, msg: ToManagerMessage):
+ frame = super()._make_frame(msg)
+ self._pending_request_ids.add(frame.id)
+ return frame
+
+ def _from_frame(self, frame) -> ToDagProcessor | None:
+ # Match IDs so another response cannot hide an interrupted sync or
async request.
+ self._pending_request_ids.discard(frame.id)
+ try:
+ return super()._from_frame(frame)
+ except AirflowRuntimeError as error:
+ if error.error.error == ErrorType.API_SERVER_ERROR:
+ self._received_api_error = True
+ raise
Review Comment:
Would it be better for the Task SDK to expose this instead, e.g. a public
`CommsDecoder.has_failed_requests`? Overriding here would mean the dag
processor will need to be in the same package as the rest of the SDK. It’s a
choice, but I’m not sure we want to commit yet.
##########
airflow-core/src/airflow/dag_processing/processor.py:
##########
@@ -143,8 +153,15 @@ class DagFileParsingResult(BaseModel):
type: Literal["DagFileParsingResult"] = "DagFileParsingResult"
+class DagCallbacksStarted(BaseModel):
+ """Request permission to begin a callback batch after importing its Dag
file."""
Review Comment:
Is this docstring stale? The server does not seem to do anything about
permission.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]