arkadiuszbach commented on issue #50708:
URL: https://github.com/apache/airflow/issues/50708#issuecomment-4055740119
1. Memory leak in croniter is causing leak in Dag Processor when it gets
next_dag_run info:
- The leak: https://github.com/pallets-eco/croniter/pull/185 - it will be
fixed in 6.0.1 which is not released yet
- And its called here over and over again by the Dag Processor
https://github.com/apache/airflow/blob/74b0fd98f9876374c52a2d4acd891497c3b93774/airflow-core/src/airflow/timetables/_cron.py#L162
via:
https://github.com/apache/airflow/blob/01ce2deda4e23d6bd6310f7f4b1b8b9f5e9b6eac/airflow-core/src/airflow/serialization/definitions/dag.py#L205
2. structlog.BytesLogger is leaking file_handles via WRITE_LOCKS
- For each opened file It keeps the locks within dictionary with
file_handles used as a keys:
https://github.com/hynek/structlog/blob/67886c2390e3513a83a2512b607c5afeecf1aae2/src/structlog/_output.py#L21
- This list is never cleaned, even if the file is closed, once the file
is opened the lock stays there forever
- Each time Dag processor is parsing the DAG file it is reopening the
log_file and it creates another lock + file handle:
https://github.com/apache/airflow/blob/da77f37468483e0616446f80698455aa6017f84d/airflow-core/src/airflow/dag_processing/manager.py#L976-L984
The file is closed later, but the lock will not be released along file
handle).
To fix it I imported the WRITE_LOCKS from structlog and after each
close just removed the lock using the file handle:
- import: ```from structlog._output import WRITE_LOCKS```
- ```del WRITE_LOCKS[processor.logger_filehandle]``` after:
https://github.com/apache/airflow/blob/da77f37468483e0616446f80698455aa6017f84d/airflow-core/src/airflow/dag_processing/manager.py#L902
https://github.com/apache/airflow/blob/da77f37468483e0616446f80698455aa6017f84d/airflow-core/src/airflow/dag_processing/manager.py#L936
https://github.com/apache/airflow/blob/da77f37468483e0616446f80698455aa6017f84d/airflow-core/src/airflow/dag_processing/manager.py#L1188
- Other places where strucklog.BytesLogger is used over and over again to
open some files would also leak. I just had a very small leak on celery workers
running a lot of tasks daily, so I patched supervisor as seems like it
initializes the log file each time task is started on celery:
https://github.com/apache/airflow/blob/3e7876fdc7756f4df6b5b29de935cca0b8f8df1b/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L1982-L1984
Added `del WRITE_LOCKS[processor.logger_filehandle]`
After:
https://github.com/apache/airflow/blob/3e7876fdc7756f4df6b5b29de935cca0b8f8df1b/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L2112
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
