nosami opened a new issue, #69554: URL: https://github.com/apache/airflow/issues/69554
### Apache Airflow version 3.2.0 (also present on `main`) ### What happened When running the **CeleryExecutor**, task subprocess stdout/stderr is **not** forwarded to the Celery worker's own stdout, so it never reaches a container-level log collector (e.g. Kubernetes/AKS `ContainerLogV2`, Loki, etc.). The **LocalExecutor** and the **KubernetesExecutor** per-task pod both forward task logs to stdout, so migrating a deployment from KubernetesExecutor to CeleryExecutor silently drops task-level logs from any stdout-based collector. Task logs still reach the remote/base task-log handler (files/blob) and the UI — only the container-stdout stream is affected. This is a companion asymmetry to the JSON-rendering gap fixed in #68912 / #68916, but it is a **separate** flag and is **not** addressed by that fix. ### What you think should happen instead The CeleryExecutor worker should have a supported way (config option, ideally `[celery]`/`[logging]`) to also emit task logs to the worker stdout — i.e. parity with LocalExecutor / the Kubernetes pod path — instead of the value being hard-coded per call site. ### Root cause Whether task logs are forwarded to the worker's stdout logger is controlled by the keyword-only `subprocess_logs_to_stdout` argument of `supervise()`, which **defaults to `False`**: - Default `False`: https://github.com/apache/airflow/blob/3.2.0/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L479 and https://github.com/apache/airflow/blob/3.2.0/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L2012 - The stdout logger is only added to the log forwarder's `target_loggers` when the flag is `True`: https://github.com/apache/airflow/blob/3.2.0/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L569-L572 and https://github.com/apache/airflow/blob/3.2.0/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L1505-L1507 The call sites are inconsistent: - **LocalExecutor** passes `subprocess_logs_to_stdout=True`: https://github.com/apache/airflow/blob/3.2.0/airflow-core/src/airflow/executors/local_executor.py#L144-L152 - **Kubernetes pod path** (`execute_workload`) passes `subprocess_logs_to_stdout=True`: https://github.com/apache/airflow/blob/3.2.0/task-sdk/src/airflow/sdk/execution_time/execute_workload.py#L66-L77 - **CeleryExecutor** calls `supervise(...)` **without** the argument, so it defaults to `False`: https://github.com/apache/airflow/blob/providers-celery/3.17.1/providers/celery/src/airflow/providers/celery/executors/celery_executor_utils.py#L202-L210 There is currently no `[celery]`/`[logging]` config option to toggle this for the Celery worker, so operators must monkeypatch `supervise`'s default to restore the KubernetesExecutor behaviour. ### How to reproduce 1. Run a deployment with `[core] executor = CeleryExecutor` (Airflow 3.2.0, `apache-airflow-providers-celery==3.17.1`). 2. Run any task that writes to stdout/stderr (e.g. a `BashOperator` echo, or an rclone/ssh subprocess). 3. Observe the **Celery worker container stdout** (e.g. `kubectl logs <worker-pod>` / `docker logs`): the task's stdout/stderr lines are absent (they appear in the task log file / UI only). 4. Repeat with LocalExecutor or the KubernetesExecutor pod path: the same lines **do** appear on the process stdout. ### Proposed solution Add a config option honored by the Celery worker's task execution path — consistent with the `[celery] json_logs` / `[logging] json_logs` fallback pattern added in #68916 — and pass it through to `supervise(...)`: ```python # providers/celery/.../executors/celery_executor_utils.py, in execute_workload() subprocess_logs_to_stdout = conf.getboolean("celery", "logs_to_stdout", fallback=None) if subprocess_logs_to_stdout is None: subprocess_logs_to_stdout = conf.getboolean("logging", "worker_logs_to_stdout", fallback=False) supervise( ..., subprocess_logs_to_stdout=subprocess_logs_to_stdout, ) ``` (Exact config names open to maintainer preference; the key point is Celery parity + a supported toggle rather than a hard-coded default.) ### Are you willing to submit a PR? Happy to test against 3.2.0 and open a PR if maintainers agree on the config surface. ### Anything else Related: #68912 / #68916 fixed the *rendering* (`json_output`) half of this Kubernetes→Celery asymmetry; this issue tracks the *forwarding* (`subprocess_logs_to_stdout`) half, which remains hard-coded. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
