pankajkoti opened a new issue, #53405: URL: https://github.com/apache/airflow/issues/53405
### Apache Airflow version main (development) ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? While further looking at https://github.com/apache/airflow/issues/53401, I am trying to understand more the spans and traces config. It came to observance that span queries are still happening although traces is disabled. The Airflow scheduler continues to: update span_status on dag_run and task_instance, and execute, every scheduling loop, a query that scans those tables for ``` WHERE span_status = 'should_end' ``` This adds an unnecessary SELECT … every loop; on large task_instance tables, it showed up in `pg_stat_activity` with noticeably slow queries until we added an index on `span_status` Impact: - Unnecessary DB queries during Scheduler operations - Performance degradation in environments where tracing is not used - Missed heartbeats due to blocking/slow queries ### What you think should happen instead? I was checking the latest main branch and noticed, for example, that the [call](https://github.com/apache/airflow/blame/912ccace3532e3c05a46f6a1cf9457c0de41db6b/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L1280) to the _end_spans_of_externally_ended_ops() method doesn’t appear to check whether tracing is enabled in the config. From what I can see, the Scheduler calls this method unconditionally. There may be other similar cases, but in general, it would be good to ensure that no span-related logic or queries are executed when tracing is disabled (which is the default setting). ### How to reproduce 1. Start up an Airflow deployment with the default configuration (i.e. tracing disabled). 2. Observe the database activity — span-related queries are executed during the Scheduler operation, even though tracing is disabled 3. If the task_instance table contains a large number of records, these span-related queries can become slow and degrade performance (until [this issue](https://github.com/apache/airflow/issues/53401) is addressed). 4. These queries can be easily identified in the query logs or by profiling the database. 5. Adding info logs to the `_end_spans_of_externally_ended_ops()` method would further confirm that it’s being invoked unconditionally (I haven’t tried this yet, but it’s straightforward to trace through the code and verify the call path). ### Operating System Linux ### Versions of Apache Airflow Providers _No response_ ### Deployment Astronomer ### Deployment details _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
