chandakanant commented on issue #57961:
URL: https://github.com/apache/airflow/issues/57961#issuecomment-5678688785
Hi @arkadiuszbach & @pierrejeambrun, is this issue solved by upgrading
Airflow to 3.2.0? I am also facing the above issue.
I have 2 Airflow environments, i.e., staging and production. Staging Airflow
is working fine, but in production I am getting the above error when running
multiple DAGs with multiple tasks. Both have the same configurations.
config:
api:
base_url: "${var.environment == "production" ?
"https://airflow.${var.region}a.com"; :
"https://airflow.${var.region}a.${var.environment}.com"}";
worker_timeout: 600
api_auth:
jwt_secret: "airflow-jwt-secret"
webserver:
page_size: 25
base_url: "${var.environment == "production" ?
"https://airflow.${var.region}a.com"; :
"https://airflow.${var.region}a.${var.environment}.com"}";
core:
dags_folder: /opt/airflow/dags/repo/dags_us
max_active_tasks_per_dag: 50
max_active_runs_per_dag: 16
dagbag_import_timeout: 600
non_pooled_task_slot_count: 1000
dag_concurrency: 10000
parallelism: 500
default_pool_task_slot_count: 1000
default_timezone: "utc"
dag_processor:
min_file_process_interval: 120
logging:
fab_logging_level: "ERROR"
logging_level: "INFO"
database:
sql_alchemy_max_overflow: 50
sql_alchemy_pool_enabled: true
sql_alchemy_pool_pre_ping: true
sql_alchemy_pool_recycle: 600
sql_alchemy_pool_size: 300
scheduler:
dag_file_processor_timeout : 1000
enable_health_check: true
job_heartbeat_sec: 300
max_dagruns_per_loop_to_schedule: 175
max_dagruns_to_create_per_loop: 175
max_tis_per_query: 300
scheduler_health_check_threshold: 120
scheduler_zombie_task_threshold: 300
kubernetes_executor:
worker_pods_creation_batch_size : 1000
delete_worker_pods: true
delete_worker_pods_on_failure: false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
