ephraimbuddy commented on code in PR #74794: URL: https://github.com/apache/airflow/pull/74794#discussion_r4536135627
########## airflow-core/src/airflow/dag_processing/processor.py: ########## @@ -183,6 +193,21 @@ class DagFileParsingResult(BaseModel): ] [email protected]() +class _DagParseCommsDecoder(CommsDecoder[ToDagProcessor, ToManager]): + api_request_failed: bool = attrs.field(default=False, init=False) + """Whether the Dag processor could not answer a request of this process through the Execution API.""" + + def _from_frame(self, frame) -> ToDagProcessor | None: Review Comment: Unanswered requests now count as API failures for both `send` and `asend`, and only their matching responses clear them. An import timeout therefore retries the callback batch when none of its targets loaded, within the existing retry deadline. Regression tests cover interrupted imports and response-ID matching. ########## airflow-core/src/airflow/dag_processing/manager.py: ########## @@ -2716,10 +2853,15 @@ def terminate(self): processor.kill(signal.SIGTERM, escalation_delay=5.0) def end(self): - """Kill all child processes on exit since we don't want to leave them as orphaned.""" + """Kill child processes so none are orphaned, then acknowledge work that already finished.""" pids_to_kill = [p.pid for p in self._processors.values()] if pids_to_kill: kill_child_processes_by_pids(pids_to_kill) + # Claims the server releases after this Job ends are delivered again, repeating their callbacks. + try: + self.work_source.flush(deadline=math.inf) + except Exception: + self.log.exception("Unable to acknowledge requested work before exiting") Review Comment: Retired registrations now produce a warning without a traceback during shutdown. Unexpected errors still retain their traceback. The test checks both paths; mutation testing confirmed that replacing `log.exception` with `log.error` fails the traceback assertion. ########## airflow-core/src/airflow/config_templates/config.yml: ########## @@ -3314,6 +3314,32 @@ dag_processor: - how often to refresh DAGs from those sources - how many files to parse concurrently options: + execution_api_token_file: Review Comment: Added `[dag_processor] use_execution_api`, defaulting to `False`, independently of `execution_api_token_file`. A token file alone keeps database mode. Future credential sources can use the same switch, and #74796 now uses it for probes, rollout strategy and replica validation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
