This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 27609598e1b Document OpenSearch logging override needed on Airflow
3.0.0 – 3.2.0 (#65209)
27609598e1b is described below
commit 27609598e1b875a395a678b80dfa787ee7d93e80
Author: Jarek Potiuk <[email protected]>
AuthorDate: Tue Apr 14 22:54:45 2026 +0200
Document OpenSearch logging override needed on Airflow 3.0.0 – 3.2.0
(#65209)
* Document OpenSearch logging override needed on Airflow 3.0.0 – 3.2.0
The wiring of OpensearchTaskHandler in the stock airflow_local_settings.py
landed in Airflow 3.2.1. On 3.0.0 – 3.2.0 users must ship a custom
logging_config_class to swap the task handler. Document the override and
note that only provider 1.9.0+ is compatible with Airflow 3.
* Clarify scope: override is only for UI log rendering
Make explicit in the doc that the airflow_local_settings.py override
is only needed so the Airflow UI can read task logs back from OpenSearch
on 3.0.0 – 3.2.0. Workers write logs as usual either way.
* Soften provider 1.9.0 compatibility note
Earlier provider versions can still be installed on Airflow 3.x — they
just cannot render task logs from OpenSearch in the UI.
---
providers/opensearch/docs/logging/index.rst | 76 ++++++++++++++++++++++++++++-
1 file changed, 74 insertions(+), 2 deletions(-)
diff --git a/providers/opensearch/docs/logging/index.rst
b/providers/opensearch/docs/logging/index.rst
index 22c0314223b..34f2dc359b9 100644
--- a/providers/opensearch/docs/logging/index.rst
+++ b/providers/opensearch/docs/logging/index.rst
@@ -20,8 +20,9 @@
Writing logs to Opensearch
-----------------------------
-*Added in provider version 1.5.0*
-Available only with Airflow>=3.0
+Only ``apache-airflow-providers-opensearch`` **1.9.0+** is compatible with
Airflow 3 for
+viewing task logs in the UI. Earlier provider versions can still be installed
on Airflow
+3.x, but the UI will not be able to render task logs from OpenSearch.
Airflow can be configured to read task logs from Opensearch and optionally
write logs to stdout in standard or json format. These logs can later be
collected and forwarded to the cluster using tools like fluentd, logstash or
others.
@@ -72,6 +73,77 @@ To output task logs to OpenSearch directly, the following
config could be used:
write_to_os = True
target_index = [name of the index to store logs]
+.. _opensearch-airflow-3-0-to-3-2-local-settings:
+
+Enabling the OpenSearch task handler on Airflow 3.0.0 – 3.2.0
+'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
+
+This section is **only about reading task logs back into the Airflow UI**.
Tasks running
+on workers will write logs as usual (to local files, stdout, or — with
appropriate log
+shipping — to OpenSearch) regardless of the override below. Without the
override on
+Airflow 3.0.0 – 3.2.0, logs reach OpenSearch fine but the **UI cannot render
them**
+because no handler is registered to fetch them back.
+
+The wiring that registers ``OpensearchTaskHandler`` inside the stock
+``airflow_local_settings.py`` (the file that builds
``DEFAULT_LOGGING_CONFIG``) only landed
+in Airflow **3.2.1**. On Airflow **3.0.0 – 3.2.0** installing the provider is
not enough:
+to make the UI's log viewer fetch logs from OpenSearch you must ship a custom
logging
+config that swaps the ``task`` handler. The handler self-registers as the
remote-log
+reader on construction (via ``REMOTE_TASK_LOG`` on 3.0/3.1 and
``_ActiveLoggingConfig``
+on 3.2), so swapping the handler class is the only change required.
+
+Create a module on the Python path — for example
``config/airflow_local_settings.py`` —
+and point Airflow at it via ``[logging] logging_config_class``:
+
+.. code-block:: python
+
+ from airflow.config_templates.airflow_local_settings import (
+ BASE_LOG_FOLDER,
+ DEFAULT_LOGGING_CONFIG,
+ )
+ from airflow.providers.common.compat.sdk import conf
+
+ OPENSEARCH_HOST = conf.get("opensearch", "host", fallback=None)
+
+ if OPENSEARCH_HOST:
+ DEFAULT_LOGGING_CONFIG["handlers"]["task"] = {
+ "class":
"airflow.providers.opensearch.log.os_task_handler.OpensearchTaskHandler",
+ "formatter": "airflow",
+ "base_log_folder": str(BASE_LOG_FOLDER),
+ "end_of_log_mark": "end_of_log",
+ "host": OPENSEARCH_HOST,
+ "port": conf.getint("opensearch", "port", fallback=9200),
+ "username": conf.get("opensearch", "username"),
+ "password": conf.get("opensearch", "password"),
+ "write_stdout": conf.getboolean("opensearch", "write_stdout"),
+ "json_format": conf.getboolean("opensearch", "json_format"),
+ "json_fields": conf.get("opensearch", "json_fields"),
+ "host_field": conf.get("opensearch", "host_field",
fallback="host"),
+ "offset_field": conf.get("opensearch", "offset_field",
fallback="offset"),
+ "write_to_opensearch": conf.getboolean("opensearch",
"write_to_os", fallback=False),
+ "target_index": conf.get("opensearch", "target_index",
fallback="airflow-logs"),
+ }
+
+Then, in ``airflow.cfg``:
+
+.. code-block:: ini
+
+ [logging]
+ remote_logging = True
+ logging_config_class = config.airflow_local_settings.DEFAULT_LOGGING_CONFIG
+
+On Airflow **3.2.1+** this override is unnecessary — the stock
``airflow_local_settings.py``
+already contains an ``elif OPENSEARCH_HOST:`` branch, so configuring the
``[opensearch]``
+section in ``airflow.cfg`` is sufficient.
+
+.. note::
+
+ Reading task logs from OpenSearch works on Airflow 3.0.0 – 3.2.0 with the
override
+ above. The ``write_to_os = True`` direct-write path depends on the
remote-log-IO
+ plumbing that is only fully wired through the supervisor in Airflow
**3.2.1+**. On
+ 3.0.0 – 3.2.0, prefer shipping logs with fluentd / logstash rather than
enabling
+ ``write_to_os``.
+
.. _write-logs-elasticsearch-tls:
Writing logs to Opensearch over TLS
