JoelEmmanuelCloud commented on code in PR #76410:
URL: https://github.com/apache/airflow/pull/76410#discussion_r4639209367


##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -435,6 +454,41 @@ def __init__(
         self.executors: list[BaseExecutor] = executors if executors else 
ExecutorLoader.init_executors()
         self.executor: BaseExecutor = self.executors[0]
 
+        # Each executor's get_health() runs on its own daemon thread, never on 
the scheduler loop's own
+        # thread: a check that talks to a broker or a cluster control plane 
can be slow or hang, and
+        # must not be able to delay task scheduling. A pool isn't needed since 
each executor is only
+        # ever checked by one in-flight call at a time (see 
_check_executor_health); a daemon thread
+        # also means a hung check can't block process exit the way a 
ThreadPoolExecutor worker would.
+        self._executor_health_check_futures: dict[str, tuple[Future, float]] = 
{}
+        # A probe that ignores the timeout passed to it would otherwise occupy 
its slot forever: once
+        # a pending future has been sitting this many intervals, it's reported 
as stalled instead of
+        # silently skipped on every later tick (see _check_executor_health).
+        self._executor_health_check_stall_after_intervals = 3
+        self._stalled_health_checks: set[str] = set()
+
+        self._executor_health_check_interval = conf.getfloat(
+            "scheduler", "executor_health_check_interval", fallback=30.0
+        )
+        # Must be comfortably below the check interval so a probe that honours 
it always finishes
+        # before the next tick fires; half the interval is the starting point.
+        self._executor_health_check_timeout = 
self._executor_health_check_interval / 2
+
+        # Only executors that actually override get_health() get a Job row at 
all: one that was never
+        # going to report anything isn't a gap in coverage (see 
_configured_executor_keys in
+        # airflow_health.py, which applies this same rule from the 
/monitor/health side). An executor
+        # with no name is also skipped: _executor_health_key has nothing 
stable to key its row under.
+        # getattr rather than a plain attribute access: a test double standing 
in for an executor
+        # (e.g. a bare, non-speced MagicMock) has no real get_health of its 
own to compare, and should
+        # be treated the same as one that doesn't override it, not raise 
AttributeError.
+        from airflow.executors.base_executor import BaseExecutor as 
_BaseExecutor
+
+        self._executor_health_keys: dict[str, str | None] = {
+            key: executor.team_name
+            for executor in self.executors
+            if getattr(type(executor), "get_health", _BaseExecutor.get_health) 
is not _BaseExecutor.get_health

Review Comment:
   swapped it for a BaseExecutor.overrides_get_health() classmethod, shared by 
both places that needed this check (also covers your nit below).
   
   checked it through issubclass(type(executor), BaseExecutor) instead of 
getattr. a MagicMock(spec=BaseExecutor) passes isinstance, since mock fakes 
that check, but calling a method on a mock never runs the real code, it just 
hands back another mock, so checking type() instead is what actually keeps 
mocks out of this.
   
   while testing this change i also found a real bug nearby: a timer got 
registered at delay=self.job.heartrate, and heartrate can be 0 (several 
existing tests set it that way on purpose). a zero delay timer re-queues itself 
at "now" every single pass through the event scheduler, which never lets a 
non-blocking run() return, so the scheduler loop just hangs forever. floored it 
to 1 second.
   



##########
airflow-core/src/airflow/api/common/airflow_health.py:
##########
@@ -250,6 +377,28 @@ def get_airflow_health() -> dict[str, Any]:
         dag_processor_status = HealthStatus.UNHEALTHY
         dag_processor_detailed_status = DetailedHealthStatus.DOWN
 
+    # Omitted entirely, not just reported unhealthy, when no configured 
executor implements a health
+    # check: a reporting mechanism nothing has opted into isn't a gap in 
coverage, and showing it as
+    # "unhealthy" on every deployment that hasn't adopted it yet would be a 
false alarm, not a signal.
+    report_executor_health = bool(_configured_executor_keys())
+    if report_executor_health:
+        try:
+            executor_jobs = get_jobs_health(EXECUTOR_JOB_TYPE)
+            executor_detailed_status = _executor_detailed_status(executor_jobs)
+            executor_status = (
+                HealthStatus.UNHEALTHY
+                if executor_detailed_status == DetailedHealthStatus.DOWN
+                else HealthStatus.HEALTHY
+            )
+            if executor_jobs and executor_jobs[0].latest_heartbeat:
+                latest_executor_heartbeat = 
executor_jobs[0].latest_heartbeat.isoformat()

Review Comment:
   dennis said you two already talked about this and decided against it, so 
leaving this one as is.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to