GitHub user raphaelauv added a comment to the discussion: Proposal: deterministic schedule jitter so daily DAGs don't all fire at the same time
there are already many concurrent setting to limit spike AIRFLOW__CORE__DEFAULT_POOL_TASK_SLOT_COUNT=128 AIRFLOW__CORE__PARALLELISM=32 AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG=16 AIRFLOW__CORE__MAX_MAX_ACTIVE_RUNS_PER_DAG=2 if things must start at midnight , then you want your airflow to start the maximum number of things at midnight, if it put pressure on your hardware or external resource then update your scaling or max PARALLELISM GitHub link: https://github.com/apache/airflow/discussions/69027#discussioncomment-17470594 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
