[GitHub] [airflow] Taragolis commented on issue #27065: Log files are still being cached causing ever-growing memory usage when scheduler is running
Taragolis commented on issue #27065: URL: https://github.com/apache/airflow/issues/27065#issuecomment-1293984562 > BTW. I've heard VERY bad things about EFS when EFS is used to share DAGs. It has profound impact on stability and performance of Airlfow if you have big number of DAGs unless you pay big bucks for IOPS. I've heard that from many people. > This is the moment when I usually STRONGLY recommend GitSync instead: https://medium.com/apache-airflow/shared-volumes-in-airflow-the-good-the-bad-and-the-ugly-22e9f681afca It's always it depends on configuration and monitoring. I personally have this issue might be in Airflow 2.1.x and I do not know is it actually related to Airflow itself or some other stuff. Work with EFS definitely take more effort rather than GitSync. Just for someone who might found this thread in the future with EFS performance degradation might help: **Disable save python bytecodes inside of NFS (AWS EFS) mount** + Mount as Read-Only + Disable Python bytecode by set `PYTHONDONTWRITEBYTECODE=x` + Or set location for bytecodes by set `PYTHONPYCACHEPREFIX` for example to `/tmp/pycaches` Throughput in mode Bursting in first looks like miracle but when all Bursting Capacity go to zero it could turn into your life into the hell. Each newly created EFS share has about 2.1 TB Bursting capacity. What could be done here: - Switch to Provisional Throughput mode permanently which might cost a lot, something like 6 USD per 1 MiB/sec without VAT - Switch to Provisional Throughput mode only when Bursting Capacity less than some amount, like 0.5 TB, and switch back when Bursting Capacity exceed limit 2.1 TB. Unfortunately there is no autoscaling so it would be manual or combination of CloudWatch Alerting + AWS Lambda.  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #27065: Log files are still being cached causing ever-growing memory usage when scheduler is running
Taragolis commented on issue #27065: URL: https://github.com/apache/airflow/issues/27065#issuecomment-1293884630 I also mount default `logs` directory to NFS (AWS EFS) so I could only suggest my personal configuration which use for a long time 1. Change default dag processor manager log location outside of NFS, e.g. `AIRFLOW__LOGGING__DAG_PROCESSOR_MANAGER_LOG_LOCATION = "/tmp/airflow/logs/dag_processor_manager/dag_processor_manager.log"` 2. Increase print stats interval `AIRFLOW__SCHEDULER__PRINT_STATS_INTERVAL = 300` which could reduce final size of file -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #27065: Log files are still being cached causing ever-growing memory usage when scheduler is running
Taragolis commented on issue #27065: URL: https://github.com/apache/airflow/issues/27065#issuecomment-1293838786 > wait i'm confused, so it is NFS design choice not to remove the cache file after it's written to an actual file? > >  https://nfs.sourceforge.net/#faq_d2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #27065: Log files are still being cached causing ever-growing memory usage when scheduler is running
Taragolis commented on issue #27065: URL: https://github.com/apache/airflow/issues/27065#issuecomment-1293745606 `.nfs*` files should be related to NFS not to Airflow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
