Miretpl commented on issue #69048: URL: https://github.com/apache/airflow/issues/69048#issuecomment-4928028403
> I also tried increasing parsing_processes and min_file_process_interval, but these had no effect on age_seconds. It seems that the staleness is intrinsic to the heartbeat mechanism, not to parsing load. I even confirmed this with the processor fully idle (min_file_process_interval=300, no files to parse) and the age still stayed in the 30–60s range. This indicates insufficient resources on the database, such as CPU or IOPS. > Would it be possible/correct to update this default in the Helm chart then? AIRFLOW__DAG_PROCESSOR__HEALTH_CHECK_THRESHOLD=120 To be honest, I don't see the reason to do that. At least for now, there is a lack of information regarding particular components' configurations and resources, as well as the dags' characteristics themselves, which are impacting the load on both the dag processor and the database. From my experience with quite big Airflow environments (comparing, e.g. to the data from the Airflow Survey), when the database is correctly configured with quality gates for dags code and some tweaks to dag processor configuration (depending on the dags characteristics), the default settings are sufficient. Also, adjusting this configuration is easy, and I would say not problematic. If adjusting it wasn't possible, it would be a totally different story. I will convert this issue to a discussion since we don't have enough data to determine whether it is environment-specific or an Apache Airflow issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
