vincbeck commented on code in PR #76410:
URL: https://github.com/apache/airflow/pull/76410#discussion_r4647559444
##########
airflow-core/src/airflow/jobs/job.py:
##########
@@ -57,6 +57,32 @@ def __str__(self) -> str:
return self.value
+# Executors don't run as their own process the way
SchedulerJob/TriggererJob/DagProcessorJob do, so
+# there is no ExecutorJobRunner to own this constant. The scheduler writes
rows with this job_type
+# directly when it records an executor's health, reusing the Job table and
JobState exactly as the
+# other components do instead of introducing a parallel status model.
+EXECUTOR_JOB_TYPE = "ExecutorJob"
Review Comment:
`airflow jobs check --local` (or `--hostname`) without `--job-type` now also
counts the `ExecutorJob` rows. On a scheduler host with a reporting executor,
it fails with "Found 2 alive jobs. Expected only one." The docs and the chart
pass `--job-type`, but whoever runs it without one gets a failing probe after
upgrading. Should `jobs check` skip `ExecutorJob` rows when no job type is
given?
##########
airflow-core/src/airflow/api/common/airflow_health.py:
##########
@@ -120,6 +123,53 @@ def _dag_processor_instance_health(job: Job) -> dict[str,
Any]:
}
+@lru_cache
+def _configured_executor_names() -> dict[str, ExecutorName]:
+ """
+ Map every configured executor's health key to its ``ExecutorName``, for
display purposes.
+
+ Cached for the same reason :func:`_configured_executor_keys` is: the
configured executors can't
+ change while the process runs.
+ """
+ try:
+ return {repr(name): name for name in
ExecutorLoader.get_executor_names(validate_teams=False)}
+ except Exception:
+ # A health probe must not fail on malformed executor config; the row's
own key is still shown.
+ log.warning("Could not read the executor configuration", exc_info=True)
+ return {}
+
+
+def _executor_instance_health(job: Job) -> dict[str, Any]:
+ # job.executor_class is the internal identity key (e.g.
":CeleryExecutor:"), not the name the user
+ # configured. Reporting the alias or module path they wrote in [core]
executor instead is a better
+ # user experience; the key itself is only shown as a fallback, e.g. for a
since-reconfigured executor.
+ name = _configured_executor_names().get(job.executor_class) if
job.executor_class else None
+ return {
+ **_job_instance_health(job, "latest_executor_heartbeat"),
+ "executor": (name.alias or name.module_path) if name else
job.executor_class,
+ "team_names": job.team_names,
+ "health_check_result": job.health_check_result,
+ }
+
+
+def _live_scheduler_job_ids() -> set[int]:
+ """Ids of the currently-alive SchedulerJob rows, used to age out a dead
scheduler's own rows."""
+ return {job.id for job in get_jobs_health(SchedulerJobRunner.job_type) if
job.is_alive()}
+
+
+def _live_executor_jobs(jobs: list[Job]) -> list[Job]:
+ """
+ Narrow executor rows down to live instances whose owning scheduler is also
still alive.
+
+ A row can stay technically alive for a little while after its scheduler
process has already
+ died (its heartbeat hasn't crossed the staleness threshold yet), so
liveness on the row alone
+ isn't enough; cross-checking the owner avoids briefly reporting a dead
scheduler's executor as
+ healthy.
+ """
+ live_scheduler_ids = _live_scheduler_job_ids()
Review Comment:
nit: `_live_executor_jobs` is called twice per request (in
`_executor_detailed_status` and for `instances`), and each call runs
`get_jobs_health(SchedulerJob)` again, although `get_airflow_health` already
fetched the scheduler jobs. Can you compute the live scheduler ids once and
pass them in?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]