Re: [I] Scheduler slow down when too many completed dag runs [airflow]
github-actions[bot] commented on issue #54283: URL: https://github.com/apache/airflow/issues/54283#issuecomment-4455854460 This issue has been closed because it has not received response from the issue author. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Scheduler slow down when too many completed dag runs [airflow]
github-actions[bot] closed issue #54283: Scheduler slow down when too many completed dag runs URL: https://github.com/apache/airflow/issues/54283 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Scheduler slow down when too many completed dag runs [airflow]
github-actions[bot] commented on issue #54283: URL: https://github.com/apache/airflow/issues/54283#issuecomment-440366 This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Scheduler slow down when too many completed dag runs [airflow]
eladkal commented on issue #54283: URL: https://github.com/apache/airflow/issues/54283#issuecomment-4185060908 If anyone can share exact reproduce steps that would be helpful -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Scheduler slow down when too many completed dag runs [airflow]
w0ut0 commented on issue #54283: URL: https://github.com/apache/airflow/issues/54283#issuecomment-3759091459 We have something similar (the same?) where we have 1 DAG that runs every 5 seconds and spawns +/- ti's. Over time, this grows a lot, and the dag_processor goes OOM when trying to process this DAG. Changing the dag_id in the python file fixes this OOM behaviour, until after a couple of weeks we bump into the issue again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Scheduler slow down when too many completed dag runs [airflow]
Arunodoy18 commented on issue #54283: URL: https://github.com/apache/airflow/issues/54283#issuecomment-3711372812 The degradation seems related to unbounded growth of dag_run and task_instance metadata and how the scheduler scans these tables on each loop. I would like to think and work on this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Scheduler slow down when too many completed dag runs [airflow]
ephraimbuddy commented on issue #54283: URL: https://github.com/apache/airflow/issues/54283#issuecomment-3471640227 Just to note that I haven't made progress in reproducing this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Scheduler slow down when too many completed dag runs [airflow]
phanikumv commented on issue #54283: URL: https://github.com/apache/airflow/issues/54283#issuecomment-3431902659 @ephraimbuddy could you please check this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Scheduler slow down when too many completed dag runs [airflow]
widewing commented on issue #54283: URL: https://github.com/apache/airflow/issues/54283#issuecomment-3192958932 @vikramkoka we are using postgresql 14.14, and pgbouncer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Scheduler slow down when too many completed dag runs [airflow]
vikramkoka commented on issue #54283: URL: https://github.com/apache/airflow/issues/54283#issuecomment-3192934245 @widewing What database (and version) are you using? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
