GitHub user stephsi created a discussion: Task Queue time for Tasks executed using KubernetesExecutor
Hello everyone! I have not found much information regarding my question, thus reaching out here: We are running Airflow 2.9.2 on Openshift using the KubernetesExecutor. Many of our tasks are quite short around 5-15 seconds, but the queue time still is around 25-30 seconds for every task execution, even after tweaking various settings. The airflow pods have limits: of 500m CPU and 512MB of memory and of course the airflow image is cached on the nodes and we use the ifnotpresent pull policy. **Thus my question: is 25-30 seconds normal for the queueing time of airflow tasks executed by the KubernetesExecutor?** Due to the lack of good answers, I questioned chatgpt as well and got a wide estimate of 10-50 seconds. I know that the queing time can depend on many aspects, but with this poll I want to get a feeling if lower queueing time for this kind of setup are even possible. Maybe the results also help others when trying to optimize their airflow deployment. Thanks GitHub link: https://github.com/apache/airflow/discussions/55175 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
