I would like it if we drop the log template model entirely. It’s also possible to write to TI field. And better to not allow any context var in log template.
On Sun, Sep 14, 2025 at 12:01 AM Zhe-You Liu <[email protected]> wrote: > Hi all, > > While reviewing “Fix the write-to-es feature #53821” [1], I found that the > ElasticsearchTaskHandler (and OpensearchTaskHandler as well) has been > broken for a long time since Airflow 3, as we block the DB session, but > ElasticsearchTaskHandler still depends on the LogTemplate [2] db model for > the log writing path [3]. To fix the ElasticsearchTaskHandler in the > Airflow 3 security model, we can either: > > 1. > > Drop LogTemplate db model support for ElasticsearchTaskHandler, > > or > > 2. > > Still support the LogTemplate feature with ElasticsearchTaskHandler by > fetching the LogTemplate db model before we block the DB session and > attaching the LogTemplate attribute to the ExecuteTask workload. > > For 1. > > It will be much simpler and will also decouple the ES provider from Airflow > Core, in my opinion, but with no backward compatibility (we need to > introduce a new major version and restrict Airflow 3 to only use the new ES > major version). > > Since only ElasticsearchTaskHandler leverages the LogTemplate db model, we > add the LogTemplate db model in Airflow Core just to support ES and OS > providers, but all the other remote log handlers don’t require the > LogTemplate db model for either the log read or write path. It seems we > couple the ES provider with Airflow Core. > > For 2. > > Full compatibility for the ES provider, but much more work needs to be done > to support ES under the Airflow 3 security model. > > Here is the full discussion [4] and more context on how we can solve it > properly without a workaround by bypassing the DB session restriction. > > [1] https://github.com/apache/airflow/pull/53821 > [2] > > https://github.com/apache/airflow/blob/0b968e997c9320bf54e53cdec6fd6c617bbdede4/airflow-core/src/airflow/models/tasklog.py#L27-L28 > [3] > > https://github.com/Owen-CH-Leung/airflow/blob/fix-write-to-es-feature/providers/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py#L147 > [4] https://github.com/apache/airflow/pull/53821#discussion_r2314359557 > > Looking forward to your feedback! Thanks! > > Best, > Jason >
