vchiapaikeo commented on issue #27296: URL: https://github.com/apache/airflow/issues/27296#issuecomment-1297480661
Hey @potiuk - thanks for the response. I forgot to mention that we've already disabled this configuration: ``` AIRFLOW__SCHEDULER__SCHEDULE_AFTER_TASK_EXECUTION: "False" ``` That said, we have been starting to see more stability with our scheduler lately. Another config that I failed to provide on our end was this: ``` AIRFLOW__CORE__PARALLELISM: "0" ``` This seems to have caused the schedulers to take on more load than they could handle and also likely contributed to a greater likelihood of locks on the db. We've adjusted parallelism to 500. The default of 32 seems too low for us and we've also noticed that more schedulers can also contribute to a higher likelihood of locking. Our plan is to launch with 3 schedulers (each with a parallelism of 500). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
